remove configuration from contributor guide

- move polling configuration to admin data-collection docs
- move pipeline partitioning to admin data-pipeline docs
  - add a note that it's only required for transformations
- publisher section is already in admin docs
- pipeline configuration is already in admin docs

Change-Id: Ib29ec71d835b8625b518ede98e00078e7de3d282
(cherry picked from commit 96b4d1f7e5864e0ef3a40f59b812f3cd3823f7d6)

4 files changed, 82 insertions, 222 deletions

diff --git a/doc/source/admin/telemetry-data-collection.rst b/doc/source/admin/telemetry-data-collection.rst
index 03d69bf5..192f2038 100644
--- a/doc/source/admin/telemetry-data-collection.rst
+++ b/doc/source/admin/telemetry-data-collection.rst
@@ -37,7 +37,7 @@ RESTful API (deprecated in Ocata)
 
 
 Notifications
-~~~~~~~~~~~~~
+=============
 
 All OpenStack services send notifications about the executed operations
 or system state. Several notifications carry information that can be
@@ -307,7 +307,7 @@ between two ``datetime`` fields from one notification.
 .. _Polling-Configuration:
 
 Polling
-~~~~~~~
+=======
 
 The Telemetry service is intended to store a complex picture of the
 infrastructure. This goal requires additional information than what is
@@ -322,6 +322,54 @@ closer interaction with the compute hosts. To solve this issue,
 Telemetry uses an agent based architecture to fulfill the requirements
 against the data collection.
 
+Configuration
+-------------
+
+Polling rules are defined by the `polling.yaml` file. It defines the pollsters
+to enable and the interval they should be polled.
+
+Each source configuration encapsulates meter name matching which matches
+against the entry point of pollster. It also includes: polling
+interval determination, optional resource enumeration or discovery.
+
+All samples generated by polling are placed on the queue to be handled by
+the pipeline configuration loaded in the notification agent.
+
+The polling definition may look like the following::
+
+    ---
+    sources:
+      - name: 'source name'
+        interval: 'how often the samples should be generated'
+        meters:
+          - 'meter filter'
+        resources:
+          - 'list of resource URLs'
+        discovery:
+          - 'list of discoverers'
+
+The *interval* parameter in the sources section defines the cadence of sample
+generation in seconds.
+
+Polling plugins are invoked according to each source's section whose *meters*
+parameter matches the plugin's meter name. Its matching logic functions the
+same as pipeline filtering.
+
+The optional *resources* section of a polling source allows a list of
+static resource URLs to be configured. An amalgamated list of all
+statically defined resources are passed to individual pollsters for polling.
+
+The optional *discovery* section of a polling source contains the list of
+discoverers. These discoverers can be used to dynamically discover the
+resources to be polled by the pollsters.
+
+If both *resources* and *discovery* are set, the final resources passed to the
+pollsters will be the combination of the dynamic resources returned by the
+discoverers and the static resources defined in the *resources* section.
+
+Agents
+------
+
 There are three types of agents supporting the polling mechanism, the
 ``compute agent``, the ``central agent``, and the ``IPMI agent``. Under
 the hood, all the types of polling agents are the same
@@ -363,7 +411,7 @@ polling plug-ins to be loaded by using the ``pollster-list`` option:
    used.
 
 Compute agent
--------------
+~~~~~~~~~~~~~
 
 This agent is responsible for collecting resource usage data of VM
 instances on individual compute nodes within an OpenStack deployment.
@@ -397,7 +445,7 @@ each hypervisor supported by the Telemetry service.
     Telemetry supports Libvirt, which hides the hypervisor under it.
 
 Central agent
--------------
+~~~~~~~~~~~~~
 
 This agent is responsible for polling public REST APIs to retrieve additional
 information on OpenStack resources not already surfaced via notifications,
@@ -426,7 +474,7 @@ database connection. The samples are sent via AMQP to the notification agent.
 .. _telemetry-ipmi-agent:
 
 IPMI agent
-----------
+~~~~~~~~~~
 
 This agent is responsible for collecting IPMI sensor data and Intel Node
 Manager data on individual compute nodes within an OpenStack deployment.
diff --git a/doc/source/admin/telemetry-data-pipelines.rst b/doc/source/admin/telemetry-data-pipelines.rst
index 87d9fdad..aac436b4 100644
--- a/doc/source/admin/telemetry-data-pipelines.rst
+++ b/doc/source/admin/telemetry-data-pipelines.rst
@@ -523,7 +523,6 @@ specified. A sample ``publishers`` section in the
        - udp://10.0.0.2:1234
        - notifier://?policy=drop&max_queue_length=512&topic=custom_target
 
-
 Deprecated publishers
 ---------------------
 
@@ -615,3 +614,32 @@ The level of support differs in case of the configured back end:
      - DB2 NoSQL does not have native TTL
        nor ``ceilometer-expirer`` support.
 
+Pipeline Partitioning
+~~~~~~~~~~~~~~~~~~~~~
+
+.. note::
+
+   Partitioning is only required if pipelines contain transformations. It has
+   secondary benefit of supporting batching in certain publishers.
+
+On large workloads, multiple notification agents can be deployed to handle the
+flood of incoming messages from monitored services. If transformations are
+enabled in the pipeline, the notification agents must be coordinated to ensure
+related messages are routed to the same agent. To enable coordination, set the
+``workload_partitioning`` value in ``notification`` section.
+
+To distribute messages across agents, ``pipeline_processing_queues`` option
+should be set. This value defines how many pipeline queues to create which will
+then be distributed to the active notification agents. It is recommended that
+the number of processing queues, at the very least, match the number of agents.
+
+Increasing the number of processing queues will improve the distribution of
+messages across the agents. It will also help batching which minimises the
+requests to Gnocchi storage backend. It will also increase the load the on
+message queue as it uses the queue to shard data.
+
+.. warning::
+
+   Decreasing the number of processing queues may result in lost data as any
+   previously created queues may no longer be assigned to active agents. It
+   is only recommended that you **increase** processing queues.
diff --git a/doc/source/contributor/configuration.rst b/doc/source/contributor/configuration.rst
deleted file mode 100644
index ded6b66d..00000000
--- a/doc/source/contributor/configuration.rst
+++ /dev/null
@@ -1,215 +0,0 @@
-..
-      Copyright 2012 New Dream Network, LLC (DreamHost)
-
-      Licensed under the Apache License, Version 2.0 (the "License"); you may
-      not use this file except in compliance with the License. You may obtain
-      a copy of the License at
-
-          http://www.apache.org/licenses/LICENSE-2.0
-
-      Unless required by applicable law or agreed to in writing, software
-      distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
-      WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
-      License for the specific language governing permissions and limitations
-      under the License.
-
-===============
- Configuration
-===============
-
-Polling
-~~~~~~~
-
-Polling rules are defined by the `polling.yaml` file. It defines the pollsters
-to enable and the interval they should be polled. See :doc:`plugins` for
-details on how to write and plug in your plugins.
-
-Each source configuration encapsulates meter name matching which matches
-against the entry point of pollster. It also includes: polling
-interval determination, optional resource enumeration or discovery.
-
-All samples generated by polling are placed on the queue to be handled by
-the pipeline configuration loaded in the notification agent.
-
-The polling definition may look like the following::
-
-    ---
-    sources:
-      - name: 'source name'
-        interval: 'how often the samples should be generated'
-        meters:
-          - 'meter filter'
-        resources:
-          - 'list of resource URLs'
-        discovery:
-          - 'list of discoverers'
-
-The *interval* parameter in the sources section should be defined in seconds.
-It determines the cadence of sample generation.
-
-A given polling plugin is invoked according to each source section
-whose *meters* parameter matches the plugin's meter name.  That is,
-the matching source sections are combined by union, not intersection,
-of the prescribed time series. It functions the same as Pipelines_
-filtering
-
-The optional *resources* section of a pipeline source allows a list of
-static resource URLs to be configured. An amalgamated list of all
-statically configured resources for a set of pipeline sources with a
-common interval is passed to individual pollsters matching those pipelines.
-
-The optional *discovery* section of a pipeline source contains the list of
-discoverers. These discoverers can be used to dynamically discover the
-resources to be polled by the pollsters defined in this pipeline. The name
-of the discoverers should be the same as the related names of plugins in
-:file:`setup.cfg`.
-
-If *resources* or *discovery* section is not set, the default value would
-be an empty list. If both *resources* and *discovery* are set, the final
-resources passed to the pollsters will be the combination of the dynamic
-resources returned by the discoverers and the static resources defined
-in the *resources* section. If there are some duplications between the
-resources returned by the discoverers and those defined in the *resources*
-section, the duplication will be removed before passing those resources
-to the pollsters.
-
-There are three ways a pollster can get a list of resources to poll, as the
-following in descending order of precedence:
-
-    1. From the per-pipeline configured discovery and/or static resources.
-    2. From the per-pollster default discovery.
-    3. From the per-agent default discovery.
-
-
-.. _Pipeline-Configuration:
-
-Pipelines
-~~~~~~~~~
-
-Pipelines describe a coupling between sources of samples and the
-corresponding sinks for transformation and publication of the samples.
-
-A source is a set of samples that should be passed to specified sinks. These
-samples may come from polling agents or service notifications.
-
-A sink on the other hand is a consumer of samples, providing logic for
-the transformation and publication of samples emitted from related sources.
-Each sink configuration is concerned `only` with the transformation rules
-and publication conduits for samples.
-
-In effect, a sink describes a chain of handlers. The chain starts with
-zero or more transformers and ends with one or more publishers. The first
-transformer in the chain is passed samples from the corresponding source,
-takes some action such as deriving rate of change, performing unit conversion,
-or aggregating, before passing the modified sample to the next step.
-
-The chains end with one or more publishers. This component makes it possible
-to persist the data into storage through the message bus or to send it to one
-or more external consumers. One chain can contain multiple publishers, see the
-:ref:`multi-publisher` section.
-
-
-Pipeline configuration
-----------------------
-
-Pipeline configuration by default, is stored in a separate configuration file,
-called :file:`pipeline.yaml`, next to the :file:`ceilometer.conf` file. The
-pipeline configuration file can be set in the *pipeline_cfg_file* parameter in
-:file:`ceilometer.conf`. Multiple chains can be defined in one configuration
-file.
-
-The chain definition looks like the following::
-
-    ---
-    sources:
-      - name: 'source name'
-        sinks
-          - 'sink name'
-    sinks:
-      - name: 'sink name'
-        transformers: 'definition of transformers'
-        publishers:
-          - 'list of publishers'
-
-The *name* parameter of a source is unrelated to anything else;
-nothing references a source by name, and a source's name does not have
-to match anything.
-
-There are several ways to define the list of meters for a pipeline source. The
-list of valid meters can be found in the :ref:`measurements` section. There is
-a possibility to define all the meters, or just included or excluded meters,
-with which a source should operate:
-
-* To include all meters, use the ``*`` wildcard symbol.
-* To define the list of meters, use either of the following:
-
-  * To define the list of included meters, use the ``meter_name`` syntax
-  * To define the list of excluded meters, use the ``!meter_name`` syntax
-  * For meters, which identify a complex Sample field, use the wildcard
-    symbol to select all, e.g. for ``disk.read.bytes``, use ``disk.*``
-
-The above definition methods can be used in the following combinations:
-
-* Only the wildcard symbol
-* The list of included meters
-* The list of excluded meters
-* Wildcard symbol with the list of excluded meters
-
-.. note::
-    At least one of the above variations should be included in the meters
-    section. Included and excluded meters cannot co-exist in the same
-    pipeline. Wildcard and included meters cannot co-exist in the same
-    pipeline definition section.
-
-The *transformers* section of a pipeline sink provides the possibility to add a
-list of transformer definitions. The names of the transformers should be the
-same as the names of the related extensions in :file:`setup.cfg`. For a more
-detailed description, please see the `transformers`_ section of the
-Administrator Guide of Ceilometer.
-
-.. _transformers: https://docs.openstack.org/admin-guide/telemetry-data-collection.html#transformers
-
-The *publishers* section contains the list of publishers, where the samples
-data should be sent after the possible transformations. The names of the
-publishers should be the same as the related names of the plugins in
-:file:`setup.cfg`.
-
-The default configuration can be found in `pipeline.yaml`_. For more details about
-how to configure publishers, see :ref:`publisher-configuration`.
-
-.. _pipeline.yaml: https://git.openstack.org/cgit/openstack/ceilometer/tree/ceilometer/pipeline/data/pipeline.yaml
-
-Pipeline Processing
--------------------
-
-On large workloads, multiple notification agents can be deployed to handle the
-flood of incoming messages from monitored services. If transformations are
-enabled in the pipeline, the notification agents must be coordinated to ensure
-related messages are routed to the same agent. To enable coordination, set the
-``workload_partitioning`` value in ``notification`` section.
-
-To distribute messages across agents, ``pipeline_processing_queues`` option
-should be set. This value defines how many pipeline queues to create which will
-then be distributed to the active notification agents. It is recommended that
-the number of processing queues, at the very least, match the number of agents.
-
-Increasing the number of processing queues will improve the distribution of
-messages across the agents. It will also help batching which minimises the
-requests to Gnocchi storage backend.
-
-.. warning::
-
-   Decreasing the number of processing queues may result in lost data as any
-   previously created queues may no longer be assigned to active agents. It
-   is only recommended that you **increase** processing queues.
-
-
-.. _pipeline-publishers:
-
-Publishers
-~~~~~~~~~~
-
-For more information about publishers see the `publishers`_ section of the
-Administrator Guide of Ceilometer.
-
-.. _publishers: https://docs.openstack.org/admin-guide/telemetry-data-pipelines.html#publishers
diff --git a/doc/source/contributor/index.rst b/doc/source/contributor/index.rst
index ccb34140..1d57ec15 100644
--- a/doc/source/contributor/index.rst
+++ b/doc/source/contributor/index.rst
@@ -28,7 +28,6 @@ Developer reference
    architecture
    measurements
    events
-   configuration
    plugins
    new_resource_types
    devstack