summaryrefslogtreecommitdiff
path: root/ceilometer/publisher
Commit message (Collapse)AuthorAgeFilesLines
* Make TCP publisher log warning instead of failing.Jaromír Wysoglad2023-04-241-3/+7
| | | | Change-Id: I56ca759c2d43830f4e342130f27b75bdba65c469
* Merge "Post individual JSONs instead of the whole list"Zuul2023-02-211-1/+1
|\
| * Post individual JSONs instead of the whole listMariusz Karpiarz2022-10-281-1/+1
| | | | | | | | Change-Id: I5114222c1ead7858c0ed9ddafe7c23496608a6c6
* | Merge "Add TCP publisher"Zuul2023-01-202-1/+97
|\ \
| * | Add TCP publisherjwysogla2023-01-132-1/+97
| | | | | | | | | | | | | | | | | | | | | This commit adds a TCP publisher. The publisher works similarly to the UDP publisher. Change-Id: Iac662018039e74ad59ac9c7fa4db994da540ef2f
* | | Merge "NoUniqueMatch: ClientException on Gnocchi publisher"Zuul2022-12-211-8/+35
|\ \ \
| * | | NoUniqueMatch: ClientException on Gnocchi publisherRafael Weingärtner2022-12-081-8/+35
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Ceilometer can ignore/discard measurements that come from possible VMs that can be used to host Gnocchi. The assumption is that one can use OpenStack itself to host the VMs that are used to run Gnocchi. The configuration is called `filter_project`. This configuration can be used in `event_pipeline.yaml` and `pipeline.yaml` configuration files. However, when this config is not used, it has a default project name for Gnocchi as `service`, as we can see in the following code snippet: ``` def __init__(self, conf, parsed_url): super(GnocchiPublisher, self).__init__(conf, parsed_url) # TODO(jd) allow to override Gnocchi endpoint via the host in the URL options = urlparse.parse_qs(parsed_url.query) self.filter_project = options.get('filter_project', ['service'])[-1] ``` Which means that if somebody creates a project called `service`, this project would not push measurements to Gnocchi. This configuration is then used by the following code: ``` def gnocchi_project_id(self): if self._gnocchi_project_id is not None: return self._gnocchi_project_id with self._gnocchi_project_id_lock: if self._gnocchi_project_id is None: try: project = self._ks_client.projects.find( name=self.filter_project, domain=self.filter_domain) except ka_exceptions.NotFound: LOG.warning('filtered project not found in keystone,' ' ignoring the filter_project ' 'option') self.filter_project = None return None except Exception: LOG.exception('fail to retrieve filtered project ') raise self._gnocchi_project_id = project.id LOG.debug("filtered project found: %s", self._gnocchi_project_id) return self._gnocchi_project_id ``` Basically, this method will look for the project ID of the project name that is configured in `filter_project` option. If it does not find any project, it returns None, and it changes the value of `filter_project` to None as well. Before this `gnocchi_project_id` method/property is called, there is a verification if `filter_project` is None. Therefore, it is assumed that when we set the value of `filter_project` to None, this method (`gnocchi_project_id`) would not be called anymore. However, that is not taking into account concurrency parallel executions. In the code, we can see `with self._gnocchi_project_id_lock:` statement, which seems to execute locking in the execution flow. However, that will not always be the case because multiple concurrent calls can be queued in that part of the code, and when the first one finishes setting the `filter_project` to None, the others will execute with this variable as None, which will cause Keystone command `self._ks_client.projects.find` to find/list all projects. That command was designed to list/find only one project; therefore, when it finds more than one project, it throws an error. That is the cause for the exception we were seeing from time to time in the log files. Change-Id: I3b4ac918015b2fd3fbe24047c3eb13419f580b27
* | | | Merge "Change oslo_cache implementation"Zuul2022-12-211-25/+6
|\ \ \ \ | |/ / / |/| | |
| * | | Change oslo_cache implementationYadnesh Kulkarni2022-11-211-25/+6
| | |/ | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | As of now to leverage caching, oslo_cache library has to be imported and configured everytime it's needed. This change migrates such implementatio to use `cache_utils.py` which returns a cache client to perform caching operations. This eliminates the purpose of importing oslo_cache everytime when needed. To get a cache client: ``` from ceilometer import cache_utils . cache_client = cache_utils.get_client(conf) ``` Signed-off-by: Yadnesh Kulkarni <ykulkarn@redhat.com> Change-Id: I14f9e1cbe84a953b092c3a88345d5faa9bcc9fb2
* | | Merge "Exclude metrics from "service" project"Zuul2022-12-081-2/+2
|\ \ \ | |_|/ |/| |
| * | Exclude metrics from "service" projectYadnesh Kulkarni2022-09-071-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Upon installing ceilometer and gnocchi using devstack, "filter_project" in "pipeline.yaml" is set to "gnocchi_swift" project. However, no such project exists in keystone. ``` publishers: - gnocchi://?archive_policy=ceilometer-low&filter_project=gnocchi_swift ``` This change, will set "filter_project" to "service" project which will exclude all metrics from it. Also add the name of the project to the logs which couldn't be found in keystone. Closes-Bug: #1988923 Signed-off-by: Yadnesh Kulkarni <ykulkarn@redhat.com> Change-Id: Ifc215f86134729665c142bb9bf64f66ca8de2a29
* | | Improve logging for Gnocchi publisherRafael Weingärtner2022-11-061-4/+17
| |/ |/| | | | | Change-Id: I2f26746225a76df66e999490c0055101d325b114
* | Merge "Properly handle 'resource_id' as None for Gnocchi publisher"Zuul2022-10-181-1/+12
|\ \
| * | Properly handle 'resource_id' as None for Gnocchi publisherRafael Weingärtner2022-09-211-1/+12
| |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This issue is related to [1]. In Python 3 (I did not check exactly in which version), the following code was throwing an error `data.sort(key=operator.attrgetter('resource_id'))`. Therefore, this patch is proposing a solution for such situations to properly handle samples where the `resource_id` is None. Of course, `resource_id` cannot be None in Gnocchi. However, that can be properly handled by removing the metric to be pushed from the `gnocchi_resource.yml` file. Which is what we were doing when we proposed [1]. The problem is that the sort is happening before that part of the code is executed, to decide if the sample should be ignored or not. [1] https://review.opendev.org/c/openstack/ceilometer/+/746717 Change-Id: Ie8eb42df3d5b9505160c9e9d6b86bdaa9a02d16a
* | Merge "Add user/project names to polled samples"Zuul2022-09-301-0/+2
|\ \
| * | Add user/project names to polled samplesYadnesh Kulkarni2022-09-131-0/+2
| |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Project and user names would be first fetched from cache, if not found, they will be requested from keystone and then cached. Using cache will significanlty reduce the number of calls made to keystone. If ceilometer is configured with no caching backend, the results will always be fetched by querying request to keystone. A new config option, `tenant_name_discovery` is introduced to operate this feature. This feature is optional and is disabled by default. No attempts to identify names will be made if uuids are found to be `None`. Signed-off-by: Yadnesh Kulkarni <ykulkarn@redhat.com> Change-Id: Iee5dbf09a1fd3ac571746fc66d2683eb8e6a1b27
* | Merge "Remove unnecessary logic for missing oslo.cache"Zuul2022-09-291-3/+1
|\ \ | |/ |/|
| * Remove unnecessary logic for missing oslo.cacheTakashi Kajinami2022-09-051-3/+1
| | | | | | | | | | | | | | | | The oslo.cache library is now included in the requirements thus should be always installed. This change removes the logic to handle missing oslo.cache. Change-Id: I1cc09aa0db5f19e913030b528c51052fd4ef331d
* | Merge "Remove GenericHardwareDeclarativePollster"Zuul2022-08-121-55/+0
|\ \ | |/ |/|
| * Remove GenericHardwareDeclarativePollsterTakashi Kajinami2022-07-011-55/+0
| | | | | | | | | | | | | | | | | | ... and discovery/inspector plugins depending on the pollster. These were implemented to gather metrics via SNMP daemon in TripleO-managed deployment but these are no longer valid since Telemetry services and Nova were removed from undercloud. Change-Id: If9a6b695ba799c766314a88328ea8a779407acc0
* | Remove support for neutron-lbaasTakashi Kajinami2022-06-081-14/+0
|/ | | | | | | | | | ... because it is of no use since neutron-lbaas was retired and was replaced by Octavia. This feature was officially deprecated during Yoga cycle[1] so we are ready to remove it. [1] 8917c73964eb764c3c4beb65d3713b36938181dd Change-Id: Ic145c23cc0b0372ef78f4a45ffb084bec24936c3
* Merge "Notify and update snapshot metrics"Zuul2021-10-141-1/+4
|\
| * Notify and update snapshot metricsMatthias Runge2021-06-301-1/+4
| | | | | | | | | | | | | | when a volume with an attached snapshot is transferred to a different project. Change-Id: I907b9b1c912d9794ada796494b172d290901993a
* | Replace oslo_utils.fnmatch with fnmatchTakashi Kajinami2021-07-061-1/+1
|/ | | | | | | | | | | The oslo_utils.fnmatch module was added to solve an issue in py2.7 but it is no longer required because py2.7 is no longer supported. The module was deprecated since oslo.utils 4.9.1[1] and the stdlib's fnmatch module should be used instead. [1] 4c893c92f551c9dd2a7cfbe7ae8171ad8139df0b Change-Id: Ibae52796df829f151c3088026a326deaf859bd44
* Replace six with python3 code stylekuangcx2021-01-135-29/+20
| | | | | | Co-authored by: Matthias Runge <mrunge@redhat.com> Change-Id: I85a4d79396874670f1b36cb91cfba5da812c2839
* Merge "Remove six.moves"Zuul2020-10-096-7/+6
|\
| * Remove six.moveswangzihao2020-09-276-7/+6
| | | | | | | | | | | | | | | | | | | | | | Remove six.moves Replace the following items with Python 3 style code. - six.moves.urllib - six.moves.xrange - six.moves.range urlparse instead of url_parse Change-Id: I2a66e69d7c1401d0bbdb9d8e8b0a7b5400aee6d2
* | Merge "Only process 'resource_id' if metric is configured for Gnocchi push"Zuul2020-09-271-2/+2
|\ \ | |/ |/|
| * Only process 'resource_id' if metric is configured for Gnocchi pushRafael Weingärtner2020-09-111-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Somne errors are being caused by the new neutron implementation of network metering (https://review.opendev.org/#/c/735605/). The samples there (in the new implementation) use a different format than the expected by the default "bandwith" and "packages" meter notification configuration (in ceilometer notification). When enabling the new Neutron implementation, we need to remove the following meters (out of the network resource): - bandwith - packages Those metrics are defined by default in Ceilometer-notification. The problem is that those definitions expect the old format of Neutron metering messages. Therefore, whenever we enable the granular extension for Neutron metering system, we need to disable them. However, even if we disable them, we have a problem. For all metrics (even the ignored ones) Ceilometer-notification was executing the following: ``` resource_id = resource_id.replace('/', '_') ``` This causes an error because the `packages` and `bandwith` metrics generated with the default ceilometer-notification will not have a resource_id. Change-Id: I04e098d3e1fa211ac7cff37681389de50f38197a
* | Add logic for event_update handling and set transfer event as event_updatepleimer2020-08-282-0/+32
|/ | | | | | | | | This patch implements logic for updating resources in gnocchi when an event_update happens. Additionally, it adds volume.transfer events to the event_definitions.yaml and attributes for said event in gnocchi_resources.yaml Change-Id: I15bc4e1589f92fc856876a563122c1a1e07d4c4a
* Fix failing unit tests for new msgpack versionRabi Mishra2020-08-141-1/+1
| | | | | | | | | | | | | This fixes the failing unit tests with the newly msgpack version. Also depends on a requirements patch that bans pecan 1.4.0 that has broken gnocchi and hence ceilometer which uses gnocchi as the default backend. Task: 40652 Depends-on: https://review.opendev.org/746261 Change-Id: Id75b7d4004a50c588a22c7eeb46ad1d79ee7a189
* Remove [ and ] around parameter values in http urlMatthias Runge2020-07-211-1/+1
| | | | | | | | | | Without this patch, using a publisher like - http://host.com/?a=b&c=d leads to calling a http post to http://host.com/?a=['b']&c=['d'] URLs without additional parameters are not affected. Change-Id: I7c4f03b5d4412ec2844cb24d32ce575442c9fd3c
* Trigger volume deletes only on volume deletedMatthias Runge2020-06-231-2/+2
| | | | | | | | | | | | | | | | | It was possible to create a volume and to do the following operations with the volume; at the end, the volume itself is marked as ended, but the volume is still available. openstack volume create --size x foo openstack volume snapshot create --volume <id> foo_snap openstack volume create --snapshot <id_of_snap> clone openstack volume delete --purge <id_of_foo> openstack metric resource show <id_of_foo> This commit fixes that issue. Change-Id: I55f0667c71ba4f3a33959a44bef43c3b05bc097c
* Add volume.manage to metrics.Matthias Runge2020-05-281-1/+6
| | | | | | | | This allows to monitor cinder managed ceph volumes to be monitored. The second addition here allows the same for snapshots. Change-Id: I7f045fa618e78351e05ad69bc9580e98487f0c29
* Update hacking for Python3Andreas Jaeger2020-03-312-2/+3
| | | | | | | | | | | The repo is Python 3 now, so update hacking to version 3.0 which supports Python 3. Fix problems found. Update local hacking checks for new flake8. Change-Id: I129bc38e6663836e12610dd50a20c74dbc79891c
* Include project_id in Prometheus publisherLance Albertson2019-11-261-2/+2
| | | | | | | | It's quite useful to have a project ID included in Prometheus data so that one can filter data based on projects. Currently it only includes the resource_id which can be difficult to track for a project. Change-Id: Ied85839d089a019c0511cb5600d6bdaa811d5d1a
* Merge "Dynamic pollster system to support non-OpenStack APIs"Zuul2019-11-211-1/+2
|\
| * Dynamic pollster system to support non-OpenStack APIsRafael Weingärtner2019-11-181-1/+2
| | | | | | | | | | | | | | | | The goal of this PR is to add the support for non-OpenStack APIs into Ceilometer. An example of such API is the RadosGW usage API. Change-Id: If5e1c9bce9e2709746338e043b20d328d8fb4504
* | Add loadbalancer resource type.Jorge Niedbalski2019-11-071-0/+12
|/ | | | | | | | | Adds the loadbalancer resource type for usage with the network.services.lb.* metrics. Change-Id: Ib6e78438c3da0e22d93f720f00cdeadf0ed7a91f Closes-Bug: #1848286 Signed-off-by: Jorge Niedbalski <jnr@metaklass.org>
* Merge "publisher: Contribute the Monasca publisher"Zuul2019-11-012-0/+479
|\
| * publisher: Contribute the Monasca publisherJoseph Davis2019-10-072-0/+479
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The Ceilosca (monasca-ceilometer) publisher has been around since before the Mitaka release and has been used in production for years. The MonascaPublisher acts as another Ceilometer publisher and sends selected metrics on to the Monasca API for storage, aggregation, alarming, etc. Once metrics are in Monasca, they may be retrieved through the Monasca API or with the python-monascaclient. This Ceilosca functionality is a key component for metering in several distributions and is used in many customer installations. With the removal of the Ceilometer v2 API (which allowed the removal of the Ceilosca storage driver, shrinking the Ceilosca code base) and continuing changes to Ceilometer, a tighter integration with the ceilometer repo may be beneficial to keep both Monasca and Telemetry in sync. Change-Id: I2cbce160503e23dfbde375722a3bd100ec86494e Story: 2001239 Task: 5769
* | Fix samples with dots in sample nameYanos Angelopoulos2019-10-031-5/+7
| | | | | | | | | | | | | | | | | | This patch fixes Prometheus publisher behavior when Ceilometer metric names contain dots by replacing them with underscores. Prometheus does not accept metrics with dots. Change-Id: If4799a1b17001c0535413a26ec5d2f427e52f168 Signed-off-by: Yanos Angelopoulos <yanos@admin.grnet.gr>
* | Merge "Added metric backup.size to remove warning in ↵Zuul2019-09-231-0/+1
|\ \ | |/ |/| | | ceilometer-agent-notification log"
| * Added metric backup.size to remove warning in ceilometer-agent-notification logprankul2019-01-171-0/+1
| | | | | | | | | | Change-Id: I07a129088b1a0063c8885ffc16251be2994f8e77 Closes-bug: #1810148
* | metering data lost randomlyzhang-shaoman2019-05-051-3/+8
| | | | | | | | | | | | | | | | | | Thread lock has been added for flush(), while queue may be modified in other places, which has not been protected, so need to add lock in such places too. Change-Id: I6b46f07690f2fc165fe27bf509ee8926656fd6b9 Closes-Bug: #1827768
* | Fix error print formatzhufl2019-04-291-2/+2
| | | | | | | | | | | | | | This is to fix the error print format of "%{query}s" and "%{resource_type}s". Change-Id: Ibf5dfcf4465246db4d49fb517b630a4c16dea05f
* | Merge "Added snapshot delete event"Zuul2019-04-181-1/+3
|\ \
| * | Added snapshot delete eventDmitriy Rabotjagov2018-11-211-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | Previously snapshot delete process wasn't sended to gnocchi, as snapshot.delete event wasn't descriped in gnocchi_resources.yml This has been fixed, and snapshot delete now fills resource ended_at field in gnocchi. Change-Id: I47a25c50287fbac16f0e5915f166cfb5709d6bad
* | | Add availability_zone attribute to gnocchi instance resources.Sam Morrison2019-03-212-0/+23
| |/ |/| | | | | | | | | | | | | | | | | | | | | | | Since moving to libvirt nova-compute poller we no longer have the availability_zone attribute of an instance available. We can capture instance.create.end events and create gnocchi resources based on these which do have the availability_zone availabile. Currently we are only handling delete events for resources, this patch adds the handling of create events for this purpose. Change-Id: Ieddbddd4ddb6af11d5158f5c90c87ae4f847bb96
* | Replace deprecated snmp metricsMehdi Abaakouk2018-10-172-3/+17
|/ | | | Change-Id: I9306afe72601baf87ca4767c15e719ec6d23dfe7
View Lorry Controller Status