Auto merge of #5986 - bundler:seg-jobs-count, r=indirect

[2.0] Auto-configure job count

Closes https://github.com/bundler/bundler/pull/5808.

The description of that issue, copied verbatim:

This argument comes in two parts, but luckily, the first one is both easier to understand and hopefully to agree with.

Background: Bundler 1.4.0 added support for parallel installation via the `--jobs` param.  Soon after, [this blog post](http://archlever.blogspot.com/2013/09/lies-damned-lies-and-truths-backed-by.html) (probably greatly amplified by [this Thoughtbot blog post](https://robots.thoughtbot.com/parallel-gem-installing-using-bundler)) popularized the recommendation "set `--jobs` to `nproc - 1`".

Not long after, probably also inspired by the popularity of this tip, this "n - 1 jobs" advice got codified into Bundler itself: https://github.com/bundler/bundler/commit/66acd02de593a6c7ee271bcbce3917eb3a01825a

However, my assertion here is _Bundler should not do that_.

The first argument (the easy one) is just that it's not doing what the user asks for.  For all the people following the (seemingly popular) tip to set their jobs to `nproc - 1`, they're actually ending up with the probably-worse `- 2`.  Even worse than that, if a user does a conservative `--jobs 2`, they're getting _no benefit_ — Bundler is quietly taking their parallelization back down to "no parallelization".

Hopefully that's a sufficient argument on its own, but the part II is that this blanket advice is probably out-of-date anyway.

Using [this script](https://gist.github.com/tjschuck/ca1d37a8869d1cc01313171b4b318094), I repeatedly installed 29 gems (installing them to a `vendor/` dir and deleting it in between runs).  I averaged the time over 10 runs per --jobs value, but the trend holds regardless of how many runs you do.

Note that these numbers are for a machine with 2 physical cores and 4 virtual ones (a Mac, reporting 2 and 4 respectively from `sysctl -n hw.physicalcpu` and `sysctl -n hw.ncpu`, the latter corresponding to Linux's `nproc`).

```
~/Code/tmp/bundler_jobs_bench ☠  ./bundler_jobs_bench.sh

Installing 29 gems repeatedly...

===============================================
Using Bundler version 1.15.1 (current release version)
===============================================

--jobs 1  5.58435780 seconds on average (across 10 runs)
--jobs 2  5.35010690 seconds on average (across 10 runs)
--jobs 3  3.93493610 seconds on average (across 10 runs)
--jobs 4  3.86082760 seconds on average (across 10 runs)
--jobs 5  3.24673650 seconds on average (across 10 runs)
--jobs 6  3.49340190 seconds on average (across 10 runs)
--jobs 7  3.26473430 seconds on average (across 10 runs)
--jobs 8  3.34560500 seconds on average (across 10 runs)

===============================================
Using development version (no more n - 1 jobs)
===============================================

--jobs 1  4.32629540 seconds on average (across 10 runs)
--jobs 2  3.48100690 seconds on average (across 10 runs)
--jobs 3  3.30937880 seconds on average (across 10 runs)
--jobs 4  3.30868200 seconds on average (across 10 runs)
--jobs 5  3.54932920 seconds on average (across 10 runs)
--jobs 6  3.36123440 seconds on average (across 10 runs)
--jobs 7  3.96490630 seconds on average (across 10 runs)
--jobs 8  3.39955640 seconds on average (across 10 runs)
```

From the above, you can see:

1. In the first block, no notable change between `--jobs 1` and `--jobs 2` — that's because they're currently the same.

2. In both, a best time corresponding to the value that (effectively) matches nproc, _not_ nproc - 1.

3. Regardless of nproc coming out best in this run, there is close enough performance among the range of `nproc - 1` through to `nproc * 2` that it doesn't seem like anything in particular (like the `- 1` removed in this commit) should be codified — people seeking to particularly optimize their bundler runtimes should do their own tweaking of the value, and it should be respected as given.

5 files changed, 43 insertions, 10 deletions

diff --git a/lib/bundler/feature_flag.rb b/lib/bundler/feature_flag.rb
index 574fad9900..d6618379da 100644
--- a/lib/bundler/feature_flag.rb
+++ b/lib/bundler/feature_flag.rb
@@ -30,6 +30,7 @@ module Bundler
     settings_flag(:allow_bundler_dependency_conflicts) { bundler_2_mode? }
     settings_flag(:allow_offline_install) { bundler_2_mode? }
     settings_flag(:auto_clean_without_path) { bundler_2_mode? }
+    settings_flag(:auto_config_jobs) { bundler_2_mode? }
     settings_flag(:cache_all) { bundler_2_mode? }
     settings_flag(:cache_command_is_package) { bundler_2_mode? }
     settings_flag(:console_command) { !bundler_2_mode? }
diff --git a/lib/bundler/installer.rb b/lib/bundler/installer.rb
index 2bb405bc7a..91cb0ec55e 100644
--- a/lib/bundler/installer.rb
+++ b/lib/bundler/installer.rb
@@ -178,14 +178,36 @@ module Bundler
     # installation is SO MUCH FASTER. so we let people opt in.
     def install(options)
       force = options["force"]
-      jobs = options.delete(:jobs) do
-        if can_install_in_parallel?
-          [Bundler.settings[:jobs].to_i - 1, 1].max
+      jobs = installation_parallelization(options)
+      install_in_parallel jobs, options[:standalone], force
+    end
+
+    def installation_parallelization(options)
+      if jobs = options.delete(:jobs)
+        return jobs
+      end
+
+      return 1 unless can_install_in_parallel?
+
+      auto_config_jobs = Bundler.feature_flag.auto_config_jobs?
+      if jobs = Bundler.settings[:jobs]
+        if auto_config_jobs
+          jobs
         else
-          1
+          [jobs.pred, 1].max
         end
+      elsif auto_config_jobs
+        processor_count
+      else
+        1
       end
-      install_in_parallel jobs, options[:standalone], force
+    end
+
+    def processor_count
+      require "etc"
+      Etc.nprocessors
+    rescue
+      1
     end
 
     def load_plugins
diff --git a/lib/bundler/installer/parallel_installer.rb b/lib/bundler/installer/parallel_installer.rb
index 95d9575c44..7a8003140b 100644
--- a/lib/bundler/installer/parallel_installer.rb
+++ b/lib/bundler/installer/parallel_installer.rb
@@ -87,6 +87,7 @@ module Bundler
       @force = force
       @specs = all_specs.map {|s| SpecInstallation.new(s) }
       @spec_set = all_specs
+      @rake = @specs.find {|s| s.name == "rake" }
     end
 
     def call
@@ -218,6 +219,8 @@ module Bundler
     # are installed.
     def enqueue_specs
       @specs.select(&:ready_to_enqueue?).each do |spec|
+        next if @rake && !@rake.installed? && spec.name != @rake.name
+
         if spec.dependencies_installed? @specs
           spec.state = :enqueued
           worker_pool.enq spec
diff --git a/lib/bundler/settings.rb b/lib/bundler/settings.rb
index 30bd6b39db..5f01e7a376 100644
--- a/lib/bundler/settings.rb
+++ b/lib/bundler/settings.rb
@@ -14,6 +14,7 @@ module Bundler
       allow_offline_install
       auto_clean_without_path
       auto_install
+      auto_config_jobs
       cache_all
       cache_all_platforms
       cache_command_is_package
@@ -58,6 +59,7 @@ module Bundler
     ].freeze
 
     NUMBER_KEYS = %w[
+      jobs
       redirect
       retry
       ssl_verify_mode
diff --git a/spec/quality_spec.rb b/spec/quality_spec.rb
index 81aa9f88cd..7a34269831 100644
--- a/spec/quality_spec.rb
+++ b/spec/quality_spec.rb
@@ -169,9 +169,9 @@ RSpec.describe "The library itself" do
 
   it "documents all used settings" do
     exemptions = %w[
+      auto_config_jobs
       cache_command_is_package
       console_command
-      default_cli_command
       deployment_means_frozen
       forget_cli_options
       gem.coc
@@ -179,12 +179,11 @@ RSpec.describe "The library itself" do
       inline
       lockfile_uses_separate_rubygems_sources
       use_gem_version_promoter_for_major_updates
-      warned_version
       viz_command
     ]
 
     all_settings = Hash.new {|h, k| h[k] = [] }
-    documented_settings = exemptions
+    documented_settings = []
 
     Bundler::Settings::BOOL_KEYS.each {|k| all_settings[k] << "in Bundler::Settings::BOOL_KEYS" }
     Bundler::Settings::NUMBER_KEYS.each {|k| all_settings[k] << "in Bundler::Settings::NUMBER_KEYS" }
@@ -200,8 +199,14 @@ RSpec.describe "The library itself" do
       documented_settings = File.read("man/bundle-config.ronn")[/LIST OF AVAILABLE KEYS.*/m].scan(/^\* `#{key_pattern}`/).flatten
     end
 
-    documented_settings.each {|s| all_settings.delete(s) }
-    exemptions.each {|s| all_settings.delete(s) }
+    documented_settings.each do |s|
+      all_settings.delete(s)
+      expect(exemptions.delete(s)).to be_nil, "setting #{s} was exempted but was actually documented"
+    end
+
+    exemptions.each do |s|
+      expect(all_settings.delete(s)).to be_truthy, "setting #{s} was exempted but unused"
+    end
     error_messages = all_settings.map do |setting, refs|
       "The `#{setting}` setting is undocumented\n\t- #{refs.join("\n\t- ")}\n"
     end