Merge branch 'benchmark-suite' into 'master'

Basic RSpec/benchmark-ips powered benchmark suite

Corresponding issue: #2909, see the commit messages for more details.

A few things to note:

1. The current use of `subject` isn't exactly easy on the eyes due to them having to return a Proc, I'm not sure yet how (and if) we can work around this.
2. The maximum amount of iterations in the current `User.by_login` benchmark is arbitrary, we might have to adjust it once said method's performance has been improved.
3. Benchmarks currently take 2 seconds to warm up and 5 seconds to run (benchmark-ips defaults). 
4. The custom RSpec matcher file (`benchmark_matchers.rb`) is a bit messy, any feedback on this would be appreciated

Any comments/feedback on this would be greatly appreciated.

See merge request !1503

2 files changed, 70 insertions, 0 deletions

diff --git a/doc/development/README.md b/doc/development/README.md
index 6bc8e1888db..d5bf166ad32 100644
--- a/doc/development/README.md
+++ b/doc/development/README.md
@@ -8,3 +8,4 @@
 - [UI guide](ui_guide.md) for building GitLab with existing css styles and elements
 - [Migration Style Guide](migration_style_guide.md) for creating safe migrations
 - [How to dump production data to staging](dump_db.md)
+- [Benchmarking](benchmarking.md)
diff --git a/doc/development/benchmarking.md b/doc/development/benchmarking.md
new file mode 100644
index 00000000000..88e18ee95f9
--- /dev/null
+++ b/doc/development/benchmarking.md
@@ -0,0 +1,69 @@
+# Benchmarking
+
+GitLab CE comes with a set of benchmarks that are executed for every build. This
+makes it easier to measure performance of certain components over time.
+
+Benchmarks are written as RSpec tests using a few extra helpers. To write a
+benchmark, first tag the top-level `describe`:
+
+```ruby
+describe MaruTheCat, benchmark: true do
+
+end
+```
+
+This ensures the benchmark is executed separately from other test collections.
+It also exposes the various RSpec matchers used for writing benchmarks to the
+test group.
+
+Next, lets write the actual benchmark:
+
+```ruby
+describe MaruTheCat, benchmark: true do
+  let(:maru) { MaruTheChat.new }
+
+  describe '#jump_in_box' do
+    benchmark_subject { maru.jump_in_box }
+
+    it { is_expected.to iterate_per_second(9000) }
+  end
+end
+```
+
+Here `benchmark_subject` is a small wrapper around RSpec's `subject` method that
+makes it easier to specify the subject of a benchmark. Using RSpec's regular
+`subject` would require us to write the following instead:
+
+```ruby
+subject { -> { maru.jump_in_box } }
+```
+
+The `iterate_per_second` matcher defines the amount of times per second a
+subject should be executed. The higher the amount of iterations the better.
+
+By default the allowed standard deviation is a maximum of 30%. This can be
+adjusted by chaining the `with_maximum_stddev` on the `iterate_per_second`
+matcher:
+
+```ruby
+it { is_expected.to iterate_per_second(9000).with_maximum_stddev(50) }
+```
+
+This can be useful if the code in question depends on external resources of
+which the performance can vary a lot (e.g. physical HDDs, network calls, etc).
+However, in most cases 30% should be enough so only change this when really
+needed.
+
+## Benchmarks Location
+
+Benchmarks should be stored in `spec/benchmarks` and should follow the regular
+Rails specs structure. That is, model benchmarks go in `spec/benchmark/models`,
+benchmarks for code in the `lib` directory go in `spec/benchmarks/lib`, etc.
+
+## Underlying Technology
+
+The benchmark setup uses [benchmark-ips][benchmark-ips] which takes care of the
+heavy lifting such as warming up code, calculating iterations, standard
+deviation, etc.
+
+[benchmark-ips]: https://github.com/evanphx/benchmark-ips