Dynamic pollsters: enable operation on attributes

This PR enables the use of python operation to transform the attributes that
 are extracted in the JSON response that the Dynamic pollster handle. By
 enabling operators to define transformations on the fly, we can provide even
 more flexibility for the dynamic pollsters' system.

One example of use case is the RadosGW that uses <project_id$project_id> as the
 username. With this implementation, one can create configurations in the
 dynamic pollster to clean that variable. It is as simple as defining
 `resource_id_attribute: "user | value.split('$')[0].strip()"`

The operations are separated by `|` symbol. The first element of the expression
 is the key to be retrieved from the JSON object. The other elements are
 operations that can be applied to the `value` variable. The value variable is
 the variable we use to hold the data.

Depends-On: https://review.opendev.org/#/c/693088/
Change-Id: I9ee209410491b3f04259e1a5c62ac20461070ae1
Signed-off-by: Rafael Weingärtner <rafael@apache.org>

3 files changed, 123 insertions, 4 deletions

diff --git a/ceilometer/polling/dynamic_pollster.py b/ceilometer/polling/dynamic_pollster.py
index f501c1b5..6d4662a6 100644
--- a/ceilometer/polling/dynamic_pollster.py
+++ b/ceilometer/polling/dynamic_pollster.py
@@ -264,5 +264,28 @@ class DynamicPollster(plugin_base.PollsterBase):
     def retrieve_attribute_nested_value(self, json_object, attribute_key):
         LOG.debug("Retrieving the nested keys [%s] from [%s].",
                   attribute_key, json_object)
+
+        keys_and_operations = attribute_key.split("|")
+        attribute_key = keys_and_operations[0].strip()
+
         nested_keys = attribute_key.split(".")
-        return reduce(operator.getitem, nested_keys, json_object)
+        value = reduce(operator.getitem, nested_keys, json_object)
+
+        # We have operations to be executed against the value extracted
+        if len(keys_and_operations) > 1:
+            for operation in keys_and_operations[1::]:
+                # The operation must be performed onto the 'value' variable
+                if 'value' not in operation:
+                    raise declarative.DynamicPollsterDefinitionException(
+                        "The attribute field operation [%s] must use the ["
+                        "value] variable." % operation,
+                        self.pollster_definitions)
+
+                LOG.debug("Executing operation [%s] against value[%s].",
+                          operation, value)
+
+                value = eval(operation.strip())
+
+                LOG.debug("Result [%s] of operation [%s] against value [%s].",
+                          value, operation)
+        return value
diff --git a/ceilometer/tests/unit/polling/test_dynamic_pollster.py b/ceilometer/tests/unit/polling/test_dynamic_pollster.py
index 3cbccf6e..52a2b916 100644
--- a/ceilometer/tests/unit/polling/test_dynamic_pollster.py
+++ b/ceilometer/tests/unit/polling/test_dynamic_pollster.py
@@ -451,3 +451,52 @@ class TestDynamicPollster(base.BaseTestCase):
             json_object, key)
 
         self.assertEqual(sub_value, returned_value)
+
+    def test_retrieve_attribute_nested_value_with_operation_on_attribute(self):
+        # spaces here are added on purpose at the end to make sure we
+        # execute the strip in the code before the eval
+        key = "key.subKey | value + 1|value / 2  |  value * 3"
+
+        value1 = [{"d": 2}, {"g": {"h": "val"}}]
+        sub_value = 1
+        expected_value_after_operations = 3
+        json_object = {"key": {"subKey": sub_value, "subkey2": value1}}
+
+        pollster = dynamic_pollster.DynamicPollster(
+            self.pollster_definition_only_required_fields)
+
+        returned_value = pollster.retrieve_attribute_nested_value(
+            json_object, key)
+
+        self.assertEqual(expected_value_after_operations, returned_value)
+
+    def test_retrieve_attribute_nested_value_simulate_radosgw_processing(self):
+        key = "user | value.split('$') | value[0] | value.strip()"
+
+        json_object = {"categories": [
+            {
+                "bytes_received": 0,
+                "bytes_sent": 357088,
+                "category": "complete_multipart",
+                "ops": 472,
+                "successful_ops": 472
+            }],
+            "total": {
+                "bytes_received": 206739531986,
+                "bytes_sent": 273793180,
+                "ops": 119690,
+                "successful_ops": 119682
+            },
+            "user":
+                " 00ab8d7e76fc4$00ab8d7e76fc45a37776732"
+        }
+
+        expected_value_after_operations = "00ab8d7e76fc4"
+
+        pollster = dynamic_pollster.DynamicPollster(
+            self.pollster_definition_only_required_fields)
+
+        returned_value = pollster.retrieve_attribute_nested_value(json_object,
+                                                                  key)
+
+        self.assertEqual(expected_value_after_operations, returned_value)
diff --git a/doc/source/admin/telemetry-dynamic-pollster.rst b/doc/source/admin/telemetry-dynamic-pollster.rst
index 72421f78..2c829836 100644
--- a/doc/source/admin/telemetry-dynamic-pollster.rst
+++ b/doc/source/admin/telemetry-dynamic-pollster.rst
@@ -24,9 +24,6 @@ dynamic pollster system:
    fashion. This feature is "a nice" to have, but is currently not
    implemented.
 
-*  APIs that return a list of entries directly, without a first key for the
-   list. An example is Aodh alarm list.
-
 
 The dynamic pollsters system configuration (for OpenStack APIs)
 ---------------------------------------------------------------
@@ -329,3 +326,53 @@ ones), we can use the `successful_ops`.
   project_id_attribute: "user"
   resource_id_attribute: "user"
   response_entries_key: "summary"
+
+Operations on extracted attributes
+----------------------------------
+
+The dynamic pollster system can execute Python operations to transform the
+attributes that are extracted from the JSON response that the system handles.
+
+One example of use case is the RadosGW that uses <project_id$project_id> as the
+username (which is normally mapped to the Gnocchi resource_id). With this
+feature (operations on extracted attributes), one can create configurations in
+the dynamic pollster to clean/normalize that variable. It is as simple as
+defining `resource_id_attribute: "user | value.split('$')[0].strip()"`
+
+The operations are separated by `|` symbol. The first element of the expression
+is the key to be retrieved from the JSON object. The other elements are
+operations that can be applied to the `value` variable. The value variable
+is the variable we use to hold the data being extracted. The previous
+example can be rewritten as:
+`resource_id_attribute: "user | value.split ('$') | value[0] | value.strip()"`
+
+As follows we present a complete configuration for a RadosGW dynamic
+pollster that is removing the `$` symbol, and getting the first part of the
+String.
+
+.. code-block:: yaml
+
+  ---
+
+  - name: "dynamic.radosgw.api.request.successful_ops"
+  sample_type: "gauge"
+  unit: "request"
+  value_attribute: "total.successful_ops"
+  url_path: "http://rgw.service.stage.i.ewcs.ch/admin/usage"
+  module: "awsauth"
+  authentication_object: "S3Auth"
+  authentication_parameters: "<access_key>, <secret_key>,
+  <rados_gateway_server>"
+  user_id_attribute: "user | value.split ('$') | value[0]"
+  project_id_attribute: "user | value.split ('$') | value[0]"
+  resource_id_attribute: "user | value.split ('$') | value[0]"
+  response_entries_key: "summary"
+
+The Dynamic pollster configuration options that support this feature are the
+following:
+
+* value_attribute
+* response_entries_key
+* user_id_attribute
+* project_id_attribute
+* resource_id_attribute