Make collector configurable in DevStack plugin

In DevStack OSProfiler is by default configured to use messaging
driver with Ceilometer backend. User can change the driver, but
still needs to install collector/storage manually. This patch
introduces configuration option `OSPROFILER_COLLECTOR` which is
used to specify which collector/storage to install into DevStack.

Currently 2 values are supported:
 * `redis` to install Redis server and use Redis driver
 * <empty> to keep the default behavior and use messaging driver

To test the patch on DevStack, the following lines are needed in local.conf:
  enable_plugin osprofiler https://git.openstack.org/openstack/osprofiler refs/changes/06/527406/5
  OSPROFILER_BRANCH=refs/changes/06/527406/5
  OSPROFILER_COLLECTOR=redis

Change-Id: Ia9763605db95b3f35c8b0e51211f96ee0dd3a82d

4 files changed, 92 insertions, 35 deletions

diff --git a/devstack/README.rst b/devstack/README.rst
index 9e5b752..2cf7d20 100644
--- a/devstack/README.rst
+++ b/devstack/README.rst
@@ -5,29 +5,58 @@ Enabling OSProfiler using DevStack
 This directory contains the files necessary to run OpenStack with enabled
 OSProfiler in DevStack.
 
-OSProfiler has different drivers for trace processing. The default driver uses
-Ceilometer to process and store trace events. Other drivers may connect
-to databases directly and do not require Ceilometer.
+OSProfiler can send trace data into different collectors. There are 2 parameters
+that control this:
+
+* ``OSPROFILER_COLLECTOR`` specifies which collector to install in DevStack.
+  By default OSProfiler plugin does not install anything, thus default
+  messaging driver with Ceilometer storage will be used.
+
+  Possible values:
+
+  * ``<empty>`` - default messaging driver with Ceilometer is used
+  * ``redis`` - Redis is installed
+
+  The default value of ``OSPROFILER_CONNECTION_STRING`` is set automatically
+  depending on ``OSPROFILER_COLLECTOR`` value.
+
+* ``OSPROFILER_CONNECTION_STRING`` specifies which driver is used by OSProfiler.
+
+  Possible values:
+
+  * ``messaging://`` - use messaging as trace collector (with the transport configured by oslo.messaging)
+  * ``redis://host:port`` - use Redis as trace storage
+  * ``elasticsearch://host:port`` - use Elasticsearch as trace storage
+  * ``mongodb://host:port`` - use MongoDB as trace storage
+  * ``loginsight://username:password@host`` - use LogInsight as trace collector/storage
+
 
 To configure DevStack and enable OSProfiler edit ``${DEVSTACK_DIR}/local.conf``
 file and add the following to ``[[local|localrc]]`` section:
 
-  * to use specified driver::
+* to use Redis collector::
+
+      enable_plugin osprofiler https://git.openstack.org/openstack/osprofiler master
+      OSPROFILER_COLLECTOR=redis
 
-    enable_plugin osprofiler https://git.openstack.org/openstack/osprofiler master
-    OSPROFILER_CONNECTION_STRING=<connection string value>
+  OSProfiler plugin will install Redis and configure OSProfiler to use Redis driver
 
-    the driver is chosen depending on the value of
-    ``OSPROFILER_CONNECTION_STRING`` variable (refer to the next section for
-    details)
+* to use specified driver::
 
-  * to use default Ceilometer driver::
+      enable_plugin osprofiler https://git.openstack.org/openstack/osprofiler master
+      OSPROFILER_CONNECTION_STRING=<connection string value>
 
-    enable_plugin panko https://git.openstack.org/openstack/panko master
-    enable_plugin ceilometer https://git.openstack.org/openstack/ceilometer master
-    enable_plugin osprofiler https://git.openstack.org/openstack/osprofiler master
+  the driver is chosen depending on the value of
+  ``OSPROFILER_CONNECTION_STRING`` variable (refer to the next section for
+  details)
 
-  .. note:: The order of enabling plugins matters.
+* to use default Ceilometer driver::
+
+      enable_plugin panko https://git.openstack.org/openstack/panko master
+      enable_plugin ceilometer https://git.openstack.org/openstack/ceilometer master
+      enable_plugin osprofiler https://git.openstack.org/openstack/osprofiler master
+
+  Note: the order of enabling plugins matters.
 
 Run DevStack as normal::
 
@@ -52,3 +81,9 @@ Example: enable ElasticSearch driver with the server running on localhost::
 
     OSPROFILER_CONNECTION_STRING=elasticsearch://127.0.0.1:9200
 
+**OSPROFILER_COLLECTOR** - controls which collector to install into DevStack.
+The driver is then chosen automatically based on the collector. Empty value assumes
+that the default messaging driver with Ceilometer is used.
+Example: enable Redis collector::
+
+    OSPROFILER_COLLECTOR=redis
diff --git a/devstack/lib/osprofiler b/devstack/lib/osprofiler
index dbbebd4..5c49543 100644
--- a/devstack/lib/osprofiler
+++ b/devstack/lib/osprofiler
@@ -1,16 +1,7 @@
-# lib/osprofiler
-# Functions to control the configuration and operation of the **osprofiler**
-
-# Dependencies:
-#
-# - ``functions`` file
-# - ``DEST``, ``DATA_DIR``, ``STACK_USER`` must be defined
+#!/bin/bash
 
-# ``stack.sh`` calls the entry points in this order:
-#
-# - install_osprofiler
-# - configure_osprofiler
-# - init_osprofiler
+# lib/osprofiler
+# Functions to control the configuration and operation of the **OSProfiler**
 
 # Save trace setting
 XTRACE=$(set +o | grep xtrace)
@@ -44,7 +35,31 @@ export CEILOMETER_NOTIFICATION_TOPICS=notifications,profiler
 # Functions
 # ---------
 
-# configure_osprofiler() - Nothing for now
+function install_redis() {
+    if is_fedora; then
+        install_package redis python-redis
+    elif is_ubuntu; then
+        install_package redis-server python-redis
+    elif is_suse; then
+        install_package redis python-redis
+    else
+        exit_distro_not_supported "redis installation"
+    fi
+
+    start_service redis
+}
+
+function install_osprofiler_collector() {
+    if [ -z "$OSPROFILER_COLLECTOR" ]; then
+        OSPROFILER_CONNECTION_STRING=${OSPROFILER_CONNECTION_STRING:-"messaging://"}
+    elif [ "$OSPROFILER_COLLECTOR" == "redis" ]; then
+        install_redis
+        OSPROFILER_CONNECTION_STRING=${OSPROFILER_CONNECTION_STRING:-"redis://localhost:6379"}
+    else
+        die $LINENO "OSProfiler collector $OSPROFILER_COLLECTOR is not supported"
+    fi
+}
+
 function configure_osprofiler() {
 
     for conf in ${CONF_FILES[@]}; do
@@ -56,6 +71,14 @@ function configure_osprofiler() {
             iniset $conf profiler connection_string $OSPROFILER_CONNECTION_STRING
         fi
     done
+
+    # Insert osprofiler filter into Neutron paste configuration
+    if [ -f $Q_API_PASTE_FILE ]; then
+        VAL=$(iniget $Q_API_PASTE_FILE composite:neutronapi_v2_0 keystone)
+        VAL=${VAL/catch_errors/catch_errors osprofiler}
+        iniset $Q_API_PASTE_FILE composite:neutronapi_v2_0 keystone "$VAL"
+    fi
+
     if [ -f $CEILOMETER_CONF ]
     then
         iniset $CEILOMETER_CONF event store_raw info
@@ -65,8 +88,3 @@ function configure_osprofiler() {
 
 # Restore xtrace
 $XTRACE
-
-# Tell emacs to use shell-script-mode
-## Local variables:
-## mode: shell-script
-## End:
diff --git a/devstack/plugin.sh b/devstack/plugin.sh
index 348da47..b2b6b6d 100644
--- a/devstack/plugin.sh
+++ b/devstack/plugin.sh
@@ -1,3 +1,4 @@
+#!/bin/bash
 # DevStack extras script to install osprofiler
 
 # Save trace setting
@@ -6,8 +7,12 @@ set -o xtrace
 
 source $DEST/osprofiler/devstack/lib/osprofiler
 
-if [[ "$1" == "stack" && "$2" == "post-config" ]]; then
-    echo_summary "Configuring OSprofiler"
+if [[ "$1" == "stack" && "$2" == "pre-install" ]]; then
+    echo_summary "Configuring system services for OSProfiler"
+    install_osprofiler_collector
+
+elif [[ "$1" == "stack" && "$2" == "post-config" ]]; then
+    echo_summary "Configuring OSProfiler"
     configure_osprofiler
 fi
 
diff --git a/devstack/settings b/devstack/settings
index fea08c2..ac689c9 100644
--- a/devstack/settings
+++ b/devstack/settings
@@ -4,6 +4,5 @@
 # of profiling in OpenStack services: profiling is only performed for
 # requests that specify one of these keys in HTTP headers.
 OSPROFILER_HMAC_KEYS=${OSPROFILER_HMAC_KEYS:-"SECRET_KEY"}
-OSPROFILER_CONNECTION_STRING=${OSPROFILER_CONNECTION_STRING:-"messaging://"}
 
 enable_service osprofiler