summaryrefslogtreecommitdiff
path: root/config.ru
diff options
context:
space:
mode:
authorAleksei Lipniagov <alipniagov@gitlab.com>2019-08-09 14:56:22 +0300
committerAleksei Lipniagov <alipniagov@gitlab.com>2019-08-12 13:52:15 +0300
commit3f9815865c38aeec0007b4ef3c636be51967b661 (patch)
tree0790f5bb23fedf11d706bf0abfd203045ad97e2f /config.ru
parent7a48b4d6b0f3e92887407768d639d13489d0d495 (diff)
downloadgitlab-ce-3f9815865c38aeec0007b4ef3c636be51967b661.tar.gz
Fix metric files being wiped after the app starts
When we hit our app with the initial request, in `warmup`, some metrics already being created as well as corresponding files. If we do `multiproc_file_dir` cleanup after that, we delete the files from the dir while keeping them in memory which leads to the incorrect behavior: the metric is being updated in in-memory, while is not present in the db, not sent to Prometheus as the result.
Diffstat (limited to 'config.ru')
-rw-r--r--config.ru15
1 files changed, 15 insertions, 0 deletions
diff --git a/config.ru b/config.ru
index 6f6fb85d8fa..d3c6e304998 100644
--- a/config.ru
+++ b/config.ru
@@ -17,7 +17,22 @@ end
require ::File.expand_path('../config/environment', __FILE__)
+
+# The following is necessary to ensure stale Prometheus metrics don't accumulate over time.
+# It needs to be done as early as here to ensure metrics files aren't deleted.
+# After we hit our app in `warmup`, first metrics and corresponding files already being created,
+# for example in `lib/gitlab/metrics/requests_rack_middleware.rb`.
+def cleanup_prometheus_multiproc_dir
+ if dir = ::Prometheus::Client.configuration.multiprocess_files_dir
+ old_metrics = Dir[File.join(dir, '*.db')]
+
+ FileUtils.rm_rf(old_metrics)
+ end
+end
+
warmup do |app|
+ cleanup_prometheus_multiproc_dir
+
client = Rack::MockRequest.new(app)
client.get('/')
end