Auto merge of #4103 - jingweno:master, r=indirect

Retry `Fetcher#specs` with `Bundler::Retry`

As mentioned in https://github.com/heroku/heroku-buildpack-ruby/pull/435, bundler failed at the step of fetching version metadata:

```
Fetching version metadata from https://rubygems.org/Net::HTTPInternalServerError: <?xml version=\"1.0\" encoding=\"UTF-8\"?>
       <Error><Code>InternalError</Code><Message>We encountered an internal error.
       Please try
       again.</Message><RequestId>E7402EA19C5D6803</RequestId><HostId>hzuvaA1JAZX6ST+OL4ARYeAqZ/tgkM2yOjZgBq6Panu10YzWtfNNozOg8N5qR3gxFE/sUfYGP48=</HostId></Error>
```

It turns out the retry logic doesn’t cover fetching source index, gem metadata, version metadata and dependency metadata. A new method `Fetcher#specs_with_retry` is added to wrap `Fetcher#specs` with `Bundler::Retry` which reties when bundler fails in the mentioned cases.

/cc @schneems

2 files changed, 12 insertions, 5 deletions

diff --git a/lib/bundler/fetcher.rb b/lib/bundler/fetcher.rb
index 465a361762..4ebbe7acf8 100644
--- a/lib/bundler/fetcher.rb
+++ b/lib/bundler/fetcher.rb
@@ -92,6 +92,13 @@ module Bundler
         "Your network or your gem server is probably having issues right now."
     end
 
+    # return the specs in the bundler format as an index with retries
+    def specs_with_retry(gem_names, source)
+      Bundler::Retry.new("fetcher").attempts do
+        specs(gem_names, source)
+      end
+    end
+
     # return the specs in the bundler format as an index
     def specs(gem_names, source)
       old = Bundler.rubygems.sources
diff --git a/lib/bundler/source/rubygems.rb b/lib/bundler/source/rubygems.rb
index 8e3173c666..b42cdc3eef 100644
--- a/lib/bundler/source/rubygems.rb
+++ b/lib/bundler/source/rubygems.rb
@@ -337,7 +337,7 @@ module Bundler
           # gather lists from non-api sites
           index_fetchers.each do |f|
             Bundler.ui.info "Fetching source index from #{f.uri}"
-            idx.use f.specs(nil, self)
+            idx.use f.specs_with_retry(nil, self)
           end
 
           # because ensuring we have all the gems we need involves downloading
@@ -350,7 +350,7 @@ module Bundler
           if allow_api
             api_fetchers.each do |f|
               Bundler.ui.info "Fetching gem metadata from #{f.uri}", Bundler.ui.debug?
-              idx.use f.specs(dependency_names, self)
+              idx.use f.specs_with_retry(dependency_names, self)
               Bundler.ui.info "" unless Bundler.ui.debug? # new line now that the dots are over
             end
 
@@ -363,7 +363,7 @@ module Bundler
               idxcount = idx.size
               api_fetchers.each do |f|
                 Bundler.ui.info "Fetching version metadata from #{f.uri}", Bundler.ui.debug?
-                idx.use f.specs(idx.dependency_names, self), true
+                idx.use f.specs_with_retry(idx.dependency_names, self), true
                 Bundler.ui.info "" unless Bundler.ui.debug? # new line now that the dots are over
               end
               break if idxcount == idx.size
@@ -378,7 +378,7 @@ module Bundler
               # if there are any cross-site gems we missed, get them now
               api_fetchers.each do |f|
                 Bundler.ui.info "Fetching dependency metadata from #{f.uri}", Bundler.ui.debug?
-                idx.use f.specs(unmet, self)
+                idx.use f.specs_with_retry(unmet, self)
                 Bundler.ui.info "" unless Bundler.ui.debug? # new line now that the dots are over
               end if unmet.any?
             else
@@ -389,7 +389,7 @@ module Bundler
           unless allow_api
             api_fetchers.each do |f|
               Bundler.ui.info "Fetching source index from #{f.uri}"
-              idx.use f.specs(nil, self)
+              idx.use f.specs_with_retry(nil, self)
             end
           end
         end