Memoize GitLab logger to reduce open file descriptors

We see that in gitlab-org/gitlab-ee#3664 that if we log a lot of
data in Sidekiq workers, the number of open file descriptors
reaches over 1000. To avoid this, we can memoize the logger per
thread via RequestStore.

Closes gitlab-org/gitlab-ee#3664

3 files changed, 27 insertions, 2 deletions

diff --git a/changelogs/unreleased/sh-memoize-logger.yml b/changelogs/unreleased/sh-memoize-logger.yml
new file mode 100644
index 00000000000..1b6567ce72f
--- /dev/null
+++ b/changelogs/unreleased/sh-memoize-logger.yml
@@ -0,0 +1,5 @@
+---
+title: Memoize GitLab logger to reduce open file descriptors
+merge_request:
+author:
+type: fixed
diff --git a/lib/gitlab/logger.rb b/lib/gitlab/logger.rb
index 6bffd410ed0..a42e312b5d3 100644
--- a/lib/gitlab/logger.rb
+++ b/lib/gitlab/logger.rb
@@ -13,7 +13,7 @@ module Gitlab
     end
 
     def self.read_latest
-      path = Rails.root.join("log", file_name)
+      path = self.full_log_path
 
       return [] unless File.readable?(path)
 
@@ -22,7 +22,15 @@ module Gitlab
     end
 
     def self.build
-      new(Rails.root.join("log", file_name))
+      RequestStore[self.cache_key] ||= new(self.full_log_path)
+    end
+
+    def self.full_log_path
+      Rails.root.join("log", file_name)
+    end
+
+    def self.cache_key
+      'logger:'.freeze + self.full_log_path.to_s
     end
   end
 end
diff --git a/spec/lib/gitlab/app_logger_spec.rb b/spec/lib/gitlab/app_logger_spec.rb
new file mode 100644
index 00000000000..c86d30ce6df
--- /dev/null
+++ b/spec/lib/gitlab/app_logger_spec.rb
@@ -0,0 +1,12 @@
+require 'spec_helper'
+
+describe Gitlab::AppLogger, :request_store do
+  subject { described_class }
+
+  it 'builds a logger once' do
+    expect(::Logger).to receive(:new).and_call_original
+
+    subject.info('hello world')
+    subject.error('hello again')
+  end
+end