Tracing: implement span save/restore

This adds methods to allow us to save and restore spans using
ZooKeeper data.

Additionally, we subclass the tracing.Span class so that we can
transparently handle timestamps which are stored as floating point
numbers rather than integer nanoseconds.

To exercise the new features, emit spans for QueueItems and BuildSets.

Because most of our higher-level (parent) spans may start on
one host and end on another, we save the full information about
the span in ZK and restore it whenever we do anything with it,
including starting child spans.  This works well for starting
a Build span given a BuildSet, since both objects are used by
the executor client and so the span information for both is
available.

However, there are cases where we would like to have child spans
and we do not have the full information of the parent, such as
any children of the Build span on the executor.  We could
duplicate all the information of the Build span in ZK and send
it along with the build request, but we really only need a few
bits of info to start a remote child span.  In OpenTelemetry,
this is called trace propogation, and there are some tools for
this which assume that the implicit trace context is being used
and formats information for an HTTP header.  We could use those
methods, but this change adds a simpler API that is well suited
to our typical json-serialization method of propogation.

To use it, we will add a small extra dictionary to build and merge
requests.  This should serialize to about 104 bytes.

So that we can transparantly handle upgrades from having no
saved state for spans and span context in our ZK data, have our
tracing API return a NonRecordingSpan when we try to restore
from a None value.  This code uses tracing.INVALID_SPAN or
tracing.INVALID_SPAN_CONTEXT which are suitable constants.  They
are sufficiently real for the purpose of context managers, etc.

The only down side is that any child spans of these will be
real, actual reported spans, so in these cases, we will emit
what we intend to be child spans as actual parent traces.
Since this should only happen when a user first enables tracing
on an already existing system, that seems like a reasonable
trade-off.  As new objects are populated, the spans will be emitted
as expected.

The trade off here is that our operational code can be much
simpler as we can avoid null value checks and any confusion regarding
context managers.

In particular, we can just assume that tracing spans and contexts
are always valid.

Change-Id: If55b06572b5e95f8c21611b2a3c23f7fd224a547

1 files changed, 4 insertions, 1 deletions

diff --git a/tests/otlp_fixture.py b/tests/otlp_fixture.py
index cd2329483..633296fac 100644
--- a/tests/otlp_fixture.py
+++ b/tests/otlp_fixture.py
@@ -11,11 +11,11 @@
 # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
 # License for the specific language governing permissions and limitations
 # under the License.
-
 from concurrent import futures
 
 import fixtures
 import grpc
+from opentelemetry import trace
 from opentelemetry.proto.collector.trace.v1.trace_service_pb2_grpc import (
     TraceServiceServicer,
     add_TraceServiceServicer_to_server
@@ -45,6 +45,9 @@ class OTLPFixture(fixtures.Fixture):
         self.server = grpc.server(self.executor)
         add_TraceServiceServicer_to_server(TraceServer(self), self.server)
         self.port = self.server.add_insecure_port('[::]:0')
+        # Reset global tracer provider
+        trace._TRACER_PROVIDER_SET_ONCE = trace.Once()
+        trace._TRACER_PROVIDER = None
 
     def _setUp(self):
         self.server.start()