Use more accurate timestamps for InfluxDB.metrics-timestamp-precision

This changes the timestamp of metrics to be more accurate/unique by
using Time#to_f combined with a small random jitter value. This
combination hopefully reduces the amount of collisions, though there's
no way to fully prevent any from occurring.

Fixes gitlab-com/operations#175

1 files changed, 21 insertions, 1 deletions

diff --git a/lib/gitlab/metrics/metric.rb b/lib/gitlab/metrics/metric.rb
index 7ea9555cc8c..1cd1ca30f70 100644
--- a/lib/gitlab/metrics/metric.rb
+++ b/lib/gitlab/metrics/metric.rb
@@ -2,6 +2,8 @@ module Gitlab
   module Metrics
     # Class for storing details of a single metric (label, value, etc).
     class Metric
+      JITTER_RANGE = 0.000001..0.001
+
       attr_reader :series, :values, :tags, :created_at
 
       # series - The name of the series (as a String) to store the metric in.
@@ -16,11 +18,29 @@ module Gitlab
 
       # Returns a Hash in a format that can be directly written to InfluxDB.
       def to_hash
+        # InfluxDB overwrites an existing point if a new point has the same
+        # series, tag set, and timestamp. In a highly concurrent environment
+        # this means that using the number of seconds since the Unix epoch is
+        # inevitably going to collide with another timestamp. For example, two
+        # Rails requests processed by different processes may end up generating
+        # metrics using the _exact_ same timestamp (in seconds).
+        #
+        # Due to the way InfluxDB is set up there's no solution to this problem,
+        # all we can do is lower the amount of collisions. We do this by using
+        # Time#to_f which returns the seconds as a Float providing greater
+        # accuracy. We then add a small random value that is large enough to
+        # distinguish most timestamps but small enough to not alter the amount
+        # of seconds.
+        #
+        # See https://gitlab.com/gitlab-com/operations/issues/175 for more
+        # information.
+        time = @created_at.to_f + rand(JITTER_RANGE)
+
         {
           series:    @series,
           tags:      @tags,
           values:    @values,
-          timestamp: @created_at.to_i * 1_000_000_000
+          timestamp: (time * 1_000_000_000).to_i
         }
       end
     end