summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorBoris Pavlovic <boris@pavlovic.me>2014-07-03 21:53:17 +0400
committerBoris Pavlovic <boris@pavlovic.me>2014-07-03 22:08:56 +0400
commit5939394d2e218bb013d5f4eba517bc5c35a8df1a (patch)
tree7be2750461750c42b828f1736c2290333b9afa9b
parent0835a2a3b17757c76ad18c0e197bc3b6c670e29f (diff)
downloadosprofiler-5939394d2e218bb013d5f4eba517bc5c35a8df1a.tar.gz
Update README.rst0.2.1
Change-Id: If29b5a1fba689d4a078e10317e29055cb3b86951
-rw-r--r--README.rst305
1 files changed, 150 insertions, 155 deletions
diff --git a/README.rst b/README.rst
index db3f927..19bcc46 100644
--- a/README.rst
+++ b/README.rst
@@ -7,248 +7,243 @@ OSProfiler is an OpenStack cross-project profiling library.
Background
----------
-OpenStack consists of multiple projects. Each project, in turn, has multiple services. To process some request, e.g. to boot a virtual machine, OpenStack uses multiple services from different projects. In case something works too slowly, it's extremely complicated to understand what exactly goes wrong and to locate the bottleneck.
+OpenStack consists of multiple projects. Each project, in turn, has multiple
+services. To process some request, e.g. to boot a virtual machine, OpenStack
+uses multiple services from different projects. In case something works too
+slowly, it's extremely complicated to understand what exactly goes wrong and to
+locate the bottleneck.
-To resolve this issue, we introduce a tiny but powerful library, **osprofiler**, that is going to be used by all OpenStack projects and their python clients. This library generates 1 trace per request, that goes through all services invoved, and builds a tree of calls (see an `example <http://pavlovic.me/rally/profiler/>`_).
+To resolve this issue, we introduce a tiny but powerful library,
+**osprofiler**, that is going to be used by all OpenStack projects and their
+python clients. To be able to generate 1 trace per request, that goes through
+all services invoved, and builds a tree of calls (see an
+`example <http://pavlovic.me/rally/profiler/>`_).
-*osprofiler* maintains a trace stack, which is shared between different services calling each other, thus being able to track nested method calls even across different OpenStack projects. By using a thread-safe storage, *osprofiler* takes care of the fact that each request to a certain service gets processed in a separated thread (so that each request results in a separate trace stack being maintained by this library).
-*osproifiler* calls a special driver (*notify()*) on start & stop of every event. These notifications contain some special info that allows the user to track the stack of calls + meta info. In the case of OpenStack, notifications are expected to be sent to *Ceilometer* using the *oslo.messaging notifier API*. This enables to use *Ceilometer* as a centralized collecter, as well as to have an OpenStack API to retrieve all notifications related to one trace. To be more specific, *osprofiler* supports `retrieving all notifications in a single request <https://gist.github.com/boris-42/9a8f905d3c5bc7496984>`_ and, optionally, their analysis by producing a `tree of calls <https://gist.github.com/boris-42/c3c3ee1c2c7db40de236>`_.
+Why not cProfile and etc?
+-------------------------
-To restrict its usage by non-allowed persons, *osprofiler* supports `HMAC authentication <http://en.wikipedia.org/wiki/Hash-based_message_authentication_code>`_. With this authentication mechanism enabled (and the same HMAC key set in the *osprofiler* section of the *api-paste.ini* file across all services involved), only users that know the HMAC key are able to use the profiling library and create traces.
+**The scope of this lib is a quite different:**
+* We are interested in getting one trace of points from different service,
+ not tracing all python calls inside one process
-Usage
------
+* This library should be easy integrable in OpenStack. It means that:
-This simple example below demonstates how to use *osprofiler*, as well as what are the contents of the trace stack maintained by this library at each step:
+ * It shouldn't require too much changes in code bases of projects
-.. parsed-literal::
+ * We should be able to turn it off fully
- from osprofiler import profile
+ * We should be able to keep it turned on in lazy mode (trace on request)
- profiler.init(base_id=1, parent_id=1) *# Initializes a thread-safe stack for storing traces.*
- *# In case there is no init() call, all future start()/stop()*
- *# calls will be ignored by osprofiler.*
- *# stack = [1, 1]*
+OSprofiler API version 0.2.0
+----------------------------
- profiler.start(name="code block 1") *# stack = [1, 1, uuid1], sends notification with (base_id=1, parent_id=1, trace_id=uuid1)*
+There are couple of things that you should know about API before learning it.
- profiler.start(name="code block 2") *# stack = [1, 1, uuid1, uuid2], sends notification with (base_id=1, parent_id=uuid1, trace_id=uuid2)*
- profiler.stop() *# stack = [1, 1, uuid1], sends notification with (base_id=1, parent_id=uuid1, trace_id=uuid2)*
+* **3 ways to add new trace point**
- profiler.start(name="code block 3") *# stack = [1, 1, uuid1, uuid3], sends notification with (base_id=1, parent_id=uuid1, trace_id=uuid3)*
- profiler.stop() *# stack = [1, 1, rand1], sends notification with (base_id=1, parent_id=uuid1, trace_id=uuid3)*
+ .. parsed-literal::
- profiler.stop() *# stack = [1, 1], sends notification with (base_id=1, parent_id=1, trace_id=1)*
+ from osprofiler import profiler
+ def some_func():
+ profiler.start("point_name", {"any_info_about_point": "in_this_dict"})
+ # your code
+ profiler.stop({"any_info_about_point": "in_this_dict"})
-After running this example, there will be 6 notifications for 3 events, and *osprofiler* will be able to restore the tree of calls, if needed.
+ @profiler.Trace("point_name", {"any_info_about_point": "in_this_dict"})
+ def some_func2():
+ pass
-The alternative syntax uses the `Trace object <https://github.com/stackforge/osprofiler/blob/master/osprofiler/profiler.py#L64>`_ from the profiler module, which should be used in a *with*-statement:
+ def some_func3():
+ with profiler.trace("point_name", {"any_info_about_point": "in_this_dict"}):
+ # some code here
+* **How works profiler actually?**
-.. parsed-literal::
+ * **@profiler.Trace()** and **profiler.trace()** are just syntax sugar,
+ that just call profiler.start() & profiler.stop() methods.
- from osprofiler import profiler
+ * It sends to **collector** 1 message per every call of profiler.start()
+ & profiler.stop(). So every trace points crates 2 records in collector.
+ *(more about collector later)*
- ...
+ * Trace points support nesting, so next sample will work properly:
- with profiler.Trace(name="code block name", info={...}):
- *# Code to be profiled*
+ .. parsed-literal::
+ profiler.start("parent_point")
+ profiler.start("child_point")
+ profiler.stop()
+ profiler.stop()
-The following example shows a real custom *notification function* from the *oslo.messaging library*:
+ In sample above, we will create 2 points:
+ This is implemented in quite simple manner. We have one stack that
+ contains ids of all trace points. E.g.:
-*oslo.messaging.profiler*
+ .. parsed-literal::
-.. parsed-literal::
+ profiler.start("parent_point") # trace_stack.push(<new_uuid>)
+ # send to collector -> trace_stack[-2:]
- **import osprofiler.notifier**
+ profiler.start("parent_point") # trace_stack.push(<new_uuid>)
+ # send to collector -> trace_stack[-2:]
+ profiler.stop() # send to collector -> trace_stack[-2:]
+ # trace_stack.pop()
- **from oslo.messaging.notify import notifier**
+ profiler.stop() # send to collector -> trace_stack[-2:]
+ # trace_stack.pop()
+ So with this information from collector, we can restore order and nesting
+ of all points.
- def set_notifier(context, transport, project, service, host):
- """Sets OSprofiler's notifer based on oslo.messaging notifier API.
+* **What is actually send to to collector?**
- OSProfiler will call this method on every call of profiler.start() and
- profiler.stop(). These messages will be collected by Ceilometer, which
- which allows end users to retrieve traces via Ceilometer API.
+ Trace points are presented in collector as 2 messages (start and stop)
- :context: the request context
- :transport: oslo.messaging transport
- :project: project name (e.g. nova, cinder, glance...)
- :service: service name, that sends notification (nova-conductor)
- :host: service's host name
- """
+ .. parsed-literal::
+ {
+ "name": <point_name>-(start|stop)
+ "base_id": <uuid>,
+ "parent_id": <uuid>,
+ "trace_id": <uuid>,
+ "info": <dict>
+ }
- *# Notifier based on oslo.messaging notifier API*
- **_notifier = notifier.Notifier(transport, publisher_id=host,
- driver="messaging", topic="profiler")**
+ * base_id - is <uuid> that is equal for all trace points that belongs
+ to one trace, it is done to simplify process of retrieving
+ all trace points related to one trace from collector
+ * parent_id - is <uuid> that has parent trace point
+ * trace_id - is <uuid> of current trace point
+ * info - it's dictionary that contains user information passed via calls of
+ profiler start & stop methods.
- **def notifier_func(payload):**
- """This method will be called on profiler.start() and profiler.stop().
- :payload: Contains information about trace element.
- In payload dict there are always 3 ids:
- "base_id" - uuid that is common for all notifications related
- to one trace. Used to simplify retrieving of all
- trace elements from Ceilometer.
- "parent_id" - uuid of parent element in trace
- "trace_id" - uuid of current element in trace
- Using parent_id and trace_id it's quite simple to build tree
- of trace elements, which simplify analyze of trace.
- """
- payload["project"] = project
- payload["service"] = service
- _notifier.info(context, "profiler.%s" % service, payload)
+* **Setting up Collector.**
- *# Setting the notifier function in osprofiler.*
- **osprofiler.notifier.set_notifier(notifier_func)**
+ Profiler doesn't include any collector for trace points, end user should
+ provide method that will send message to collector. Let's take a look at
+ trivial sample, where collector is just a file:
+ .. parsed-literal::
-This notifier is perfectly suited for integrating *osprofiler* with other OpenStack projects, e.g. Nova (the corresponding code sample can be found in the **Integration example** section below).
+ import json
+ from osprofiler import notifier
+ def send_info_to_file_collector(info, context=None):
+ with open("traces", "a") as f:
+ f.write(json.dumps())
-Library contents
-----------------
+ notifier.set(send_info_to_file_collector)
-Along with the basic profiling algorithm implementation, the *osprofiler* library includes a range of **profiling applications**, available out-of-box.
+ So now on every **profiler.start()** and **profiler.stop()** call we will
+ write info about trace point to the end of **traces** file.
-First, there is **SQLAlchemy profiling**. It's well known that OpenStack haevily uses *SQLAlchemy*. SQLAlchemy is a cool stuff that allows to add handlers for different events before and after SQL execution. With *osprofiler*, we can run *profiler.start()* before execution with info that contains an SQL request and call *profiler.stop()* after execution. This allows to collect the information about all DB calls across all services. To enable this DB profiling, just *osprofiler* to an **engine** instance (this will allow it to add event listeners to points before and after SQL execution):
-.. parsed-literal::
+* **Initialization of profiler.**
- from osprofiler import sqlalchemy
+ If profiler is not initialized, all calls of profiler.start() and
+ profiler.stop() are ignored.
- sqlalchemy.add_tracing(*<sqlalchemy_module>*, *<engine_instance>*, *<name>*)
+ Initialization is quite simple.
+ .. parsed-literal::
-Second, there is **Web profiling**. The main interaction mechanism between different OpenStack projects is issuing requests to each other. Thus, to be able to be easliy integrated with OpenStack projects, *osprofiler* supports processing such requests via a special **WSGI Middleware** for tracing Web applications:
+ from osprofiler import profiler
-.. parsed-literal::
+ profiler.init("SECRET_HMAC_KEY", base_id=<uuid>, parent_id=<uuid>)
- import webob.dec
+ "SECRET_HMAC_KEY" - will be discussed later, cause it's related to the
+ integration of OSprofiler & OpenStack.
- from osprofiler import profiler
+ **base_id** and **trace_id** will actually initialize trace_stack in
+ profiler, e.g. stack_trace = [base_id, trace_id].
- ...
- class WsgiMiddleware(object):
- """WSGI Middleware that enables tracing for an application."""
+Integration with OpenStack
+--------------------------
- ...
+There are 4 topics related to integration OSprofiler & OpenStack:
- @webob.dec.wsgify
- def __call__(self, request):
- if not self.enabled:
- return request.get_response(self.application)
+* **What to use as centralized collector**
- *# Trace info is passed through headers. In case the trace info header*
- *# is not present, no profiling will be done.*
- **trace_info_enc = request.headers.get("X-Trace-Info")**
- *# HMAC key is passed through headers as well.*
- **trace_hmac = request.headers.get("X-Trace-HMAC")**
+ We decided to use Ceilometer, because:
- ...
+ * It's already integrated in OpenStack, so it's quite simple to send
+ notifications to it from every project.
- if trace_info_enc:
- trace_raw = utils.binary_decode(trace_info_enc)
- *# Validating the HMAC key.*
- try:
- **validate_hmac(trace_raw, trace_hmac, self.hmac_key)**
- except IOError:
- pass
- else:
- trace_info = json.loads(trace_raw)
+ * There is a OpenStack API in Ceilometer that allows us to retrieve all
+ messages related to one trace. Take a look at
+ *osprofiler.parsers.ceilometer:get_notifications*
- *# Initializing the profiler with the info retrieved from headers.*
- **profiler.init(trace_info.get("base_id"),
- trace_info.get("parent_id"),
- self.hmac_key)**
- *# The trace info that will be sent to the notifier.*
- info = {
- "request": {
- "host_url": request.host_url,
- "path": request.path,
- "query": request.query_string,
- "method": request.method,
- "scheme": request.scheme
- }
- }
+* **How to setup profiler notifier, to send messages to this collector**
- *# Profiling the request.*
- **with profiler.Trace(self.name, info=info):
- return request.get_response(self.application)**
+ We decided to use olso.messaging Notifier API, because:
- *# If there is no trace info header, just process the request without profiling.*
- return request.get_response(self.application)
+ * oslo.messaging is integrated in all projects
+ * It's the simplest way to send notification to Ceilometer, take a look at:
+ *osprofiler.notifiers.messaging.Messaging:notify* method
-Integration example
--------------------
+ * We don't need to add any new CONF options in projects
-**OSProfiler** can be easily integrated with any of the core OpenStack projects, e.g. `Nova <https://github.com/boris-42/nova/commit/9ebe86bf5b4cc7150251396cfb302dd05e89085d>`_. Basically, it requires setting up the corresponding notifier function for the *WSGI service*, and then adding the *osprofiler WSGI middleware* in a special *api-paste.ini* file. The example below shows how it looks for Nova:
+* **How to initialize profiler, to have one trace cross all services**
-*nova.service*
+ To enable cross service profiling we actually need to do send from caller
+ to callee (base_id & trace_id). So callee will be able to init his profiler
+ with these values.
-.. parsed-literal::
+ In case of OpenStack there are 2 kinds interaction between to services:
- ...
+ * REST API
- from oslo.messaging import profiler
+ It's well know that there are python clients for every projects,
+ that generates proper HTTP request, and parses response to objects.
+ These python clients are used in 2 cases:
- class Service(service.Service):
- """Service object for binaries running on hosts."""
+ * User access OpenStack
- def __init__(self, host, binary, topic, manager, ...):
+ * Service from Project 1 would like to access Service from Project 2
- ...
- *# Set the notifier function with the admin request context*
- *# and corresponding project & service parameters*
- **profiler.set_notifier(context.get_admin_context().to_dict(),
- rpc.TRANSPORT, "nova", binary, host)**
- ...
+ So what we need is to:
+ * Put in python clients headers with trace info (if profiler is inited)
- class WSGIService(object):
- """Provides ability to launch API from a 'paste' configuration."""
+ * Add OSprofiler WSGI middleware to service, that will init profiler, if
+ there are special trace headers.
- def __init__(self, name, loader=None, use_ssl=False, max_url_len=None):
+ Actually the algorithm is a bit more complex. Python client are signed
+ trace info, and WSGI middleware checks that it's signed with HMAC that
+ is specified in api-paste.ini. So only user that knows HMAC key in
+ api-paste.ini can init properly profiler and send trace info that will
+ be actually processed.
- ...
- *# Set the notifier function with the admin request context*
- *# and corresponding project & service parameters*
- **profiler.set_notifier(context.get_admin_context().to_dict(),
- rpc.TRANSPORT, "nova", name, self.host)**
- ...
+ * RPC API
+ RPC calls are used for interaction between services of one project. As
+ we all known for RPC projects are using oslo.messaging. So the best way
+ to enable cross service tracing (inside on project). Is to add trace
+ info to all messages (if profiler is inited). And initialize profiler
+ on callee side.
-*etc/nova/api-paste.ini*
+* **What points should be by default tracked**
-.. parsed-literal::
+ I think that for all projects we should include by default 3 kinds o points:
- ...
- [composite:openstack_compute_api_v2]
- use = call:nova.api.auth:pipeline_factory
- noauth = compute_req_id faultwrap sizelimit **osprofiler** noauth ...
- keystone = compute_req_id faultwrap sizelimit **osprofiler** authtoken ...
- keystone_nolimit = compute_req_id faultwrap sizelimit **osprofiler** authtoken ...
+ * All HTTP calls
- ...
- [filter:osprofiler]
- paste.filter_factory = osprofiler.web:WsgiMiddleware.factory
- hmac_key = SECRET_KEY
- enabled = yes
+ * All RPC calls
+
+ * All DB calls