Auto merge of #6661 - eregon:consistent-encoding-for-reading-files, r=deivid-rodriguez

Use UTF-8 for reading files including Gemfile

### What was the end-user problem that led to this PR?

See #6660 and https://github.com/oracle/truffleruby/issues/1410.

### What was your diagnosis of the problem?

The above issue details the problem.

### What is your fix for the problem, implemented in this PR?

To read the Gemfile and other files in Bundler with the default source encoding of Ruby, UTF-8, instead of the binary encoding which cannot interpret non-US-ASCII characters.

### Why did you choose this fix out of the possible options?

Because it's what Ruby does for other source files.

Fixes #6660.

(cherry picked from commit e71418eeb1c16aa6bad8712b0a74696a6d8f1e36)

3 files changed, 32 insertions, 3 deletions

diff --git a/lib/bundler.rb b/lib/bundler.rb
index bd6fe4d0bf..52bbc57fb5 100644
--- a/lib/bundler.rb
+++ b/lib/bundler.rb
@@ -410,7 +410,7 @@ EOF
 
     def read_file(file)
       SharedHelpers.filesystem_access(file, :read) do
-        File.open(file, "rb", &:read)
+        File.open(file, "r:UTF-8", &:read)
       end
     end
 
@@ -431,7 +431,7 @@ EOF
 
     def load_gemspec_uncached(file, validate = false)
       path = Pathname.new(file)
-      contents = File.open(file, "r:UTF-8", &:read)
+      contents = read_file(file)
       spec = if contents.start_with?("---") # YAML header
         eval_yaml_gemspec(path, contents)
       else
diff --git a/lib/bundler/env.rb b/lib/bundler/env.rb
index 58fe20dbe7..78880adfc2 100644
--- a/lib/bundler/env.rb
+++ b/lib/bundler/env.rb
@@ -61,7 +61,7 @@ module Bundler
     end
 
     def self.read_file(filename)
-      File.read(filename.to_s).strip
+      Bundler.read_file(filename.to_s).strip
     rescue Errno::ENOENT
       "<No #{filename} found>"
     rescue => e
diff --git a/spec/install/gemfile_spec.rb b/spec/install/gemfile_spec.rb
index 945d9f485d..0a17fb3f82 100644
--- a/spec/install/gemfile_spec.rb
+++ b/spec/install/gemfile_spec.rb
@@ -110,4 +110,33 @@ RSpec.describe "bundle install" do
       end
     end
   end
+
+  context "with a Gemfile containing non-US-ASCII characters" do
+    it "reads the Gemfile with the UTF-8 encoding by default" do
+      skip "Ruby 1.8 has no encodings" if RUBY_VERSION < "1.9"
+
+      install_gemfile <<-G
+        str = "Il était une fois ..."
+        puts "The source encoding is: " + str.encoding.name
+      G
+
+      expect(out).to include("The source encoding is: UTF-8")
+      expect(out).not_to include("The source encoding is: ASCII-8BIT")
+      expect(out).to include("Bundle complete!")
+    end
+
+    it "respects the magic encoding comment" do
+      skip "Ruby 1.8 has no encodings" if RUBY_VERSION < "1.9"
+
+      # NOTE: This works thanks to #eval interpreting the magic encoding comment
+      install_gemfile <<-G
+        # encoding: iso-8859-1
+        str = "Il #{"\xE9".b}tait une fois ..."
+        puts "The source encoding is: " + str.encoding.name
+      G
+
+      expect(out).to include("The source encoding is: ISO-8859-1")
+      expect(out).to include("Bundle complete!")
+    end
+  end
 end