Add support for configuring and testing tracing

This adds support for configuring tracing in Zuul along with
basic documentation of the configuration.

It also adds test infrastructure that runs a gRPC-based collector
so that we can test tracing end-to-end, and exercises a simple
test span.

Change-Id: I4744dc2416460a2981f2c90eb3e48ac93ec94964

3 files changed, 99 insertions, 0 deletions

diff --git a/doc/source/admin.rst b/doc/source/admin.rst
index e9c210d3b..c7e2431f5 100644
--- a/doc/source/admin.rst
+++ b/doc/source/admin.rst
@@ -12,5 +12,6 @@ Service Administration
    operation
    authentication
    monitoring
+   tracing
    client
    troubleshooting
diff --git a/doc/source/configuration.rst b/doc/source/configuration.rst
index 7df062438..ced3ad781 100644
--- a/doc/source/configuration.rst
+++ b/doc/source/configuration.rst
@@ -61,6 +61,81 @@ Statsd
       If present, this will be prefixed to all of the keys before
       transmitting to the statsd server.
 
+Tracing
+~~~~~~~
+
+.. attr:: tracing
+
+   Information about the optional OpenTelemetry tracing configuration.
+   See :ref:`tracing` for more information.
+
+   .. attr:: enabled
+      :required:
+
+      To enable tracing, set this value to ``true``.  This is the only
+      required parameter in order to export to a collector running
+      locally.
+
+   .. attr:: protocol
+      :default: grpc
+
+      The OTLP wire protocol to use.
+
+      .. value:: grpc
+
+         Use gRPC (the default).
+
+      .. value:: http/protobuf
+
+         Use HTTP with protobuf encoding.
+
+   .. attr:: endpoint
+
+      The endpoint to use.  The default is protocol specific, but
+      defaults to localhost in all cases.
+
+   .. attr:: service_name
+      :default: zuul
+
+      The service name may be specified here.  Multiple Zuul
+      installations should use different values.
+
+   .. attr:: tls_cert
+
+      The path to the PEM encoded certificate file.  Used only by
+      :value:`tracing.protocol.grpc`.
+
+   .. attr:: tls_key
+
+      The path to the PEM encoded key file.  Used only by
+      :value:`tracing.protocol.grpc`.
+
+   .. attr:: tls_ca
+
+      The path to the PEM encoded CA certificate file.  Used only by
+      :value:`tracing.protocol.grpc`.
+
+   .. attr:: certificate_file
+
+      The path to the PEM encoded certificate file used to verify the
+      endpoint.  Used only by :value:`tracing.protocol.http/protobuf`.
+
+   .. attr:: insecure
+
+      Whether to allow an insecure connection.  Used only by
+      :value:`tracing.protocol.grpc`.
+
+   .. attr:: timeout
+      :default: 10000
+
+      The timeout for outgoing data in milliseconds.
+
+   .. attr:: compression
+
+      The compression algorithm to use.  Available values depend on
+      the protocol and endpoint.  The only universally supported value
+      is ``gzip``.
+
 ZooKeeper
 ~~~~~~~~~
 
diff --git a/doc/source/tracing.rst b/doc/source/tracing.rst
new file mode 100644
index 000000000..e973a77dd
--- /dev/null
+++ b/doc/source/tracing.rst
@@ -0,0 +1,23 @@
+:title: Tracing
+
+.. _tracing:
+
+Tracing
+=======
+
+Zuul includes support for distributed `tracing`_ as described by the
+OpenTelemetry project.  This allows operators (and potentially users)
+to visualize the progress of events and queue items through the
+various Zuul components as an aid to debugging.
+
+OpenTelemetry defines several observability signals such as traces,
+metrics, and logs.  Zuul uses other systems for metrics and logs; only
+traces are exported via OpenTelemetry.
+
+Zuul supports the OpenTelemetry Protocol (OTLP) for exporting traces.
+Many observability systems support receiving traces via OTLP
+(including Jaeger tracing).
+
+Related configuration is in the :attr:`tracing` section of ``zuul.conf``.
+
+_`distributed tracing`: https://opentelemetry.io/docs/concepts/observability-primer/#distributed-traces