Merge branch 'fix-git-blame-syntax-highlighting' into 'master'

Fix git blame syntax highlighting when different commits break up lines

### What does this MR do?

This MR fixes a bug where syntax highlighting would not work properly in certain lines when doing a `git blame` view of a file. Also, this MR reuses the Rugments lexer and formatter for each `git blame` commit block, which should make things faster and reduce overhead.

### Why was this MR needed?

When doing a `git blame`, GitLab feeds the Rugments lexer/formatter blocks of code based on the latest commit for each line.  However, this can cause lexer to fail because the state is cleared between each block of code, which causes `highlight` to fallback to the plaintext lexer. For example, notice this unhighlighted line in [this file](https://gitlab.common-lisp.net/cmucl/cmucl/blame/master/src/assembly/assemfile.lisp):

![image](https://gitlab.com/gitlab-org/gitlab-ce/uploads/b403fe5c2b883882a9eeea7e0409c583/image.png)

The fixed version looks like this:

![image](https://gitlab.com/stanhu/gitlab-ce/uploads/32c4d7c8ff15a7d59b364dd988f7c657/image.png)

This MR requires a patch to rugments to allow the `continue` option to be passed to the lexer.

* https://github.com/rumpelsepp/rugments/pull/24
* https://github.com/rumpelsepp/rugments/pull/23
* Also submitted to rouge: https://github.com/jneen/rouge/pull/267

### What are the relevant issue numbers?

Closes #1521

See merge request !697

6 files changed, 43 insertions, 9 deletions

diff --git a/CHANGELOG b/CHANGELOG
index 46a052e1a19..35724ae6027 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -12,6 +12,7 @@ v 7.12.0 (unreleased)
   - Disable "New Issue" and "New Merge Request" buttons when features are disabled in project settings (Stan Hu)
   - Remove Rack Attack monkey patches and bump to version 4.3.0 (Stan Hu)
   - Fix clone URL losing selection after a single click in Safari and Chrome (Stan Hu)
+  - Fix git blame syntax highlighting when different commits break up lines (Stan Hu)
   - Allow to configure location of the `.gitlab_shell_secret` file. (Jakub Jirutka)
   - Disabled expansion of top/bottom blobs for new file diffs
   - Update Asciidoctor gem to version 1.5.2. (Jakub Jirutka)
diff --git a/Gemfile b/Gemfile
index 8eb1f04000c..26981f3e0a3 100644
--- a/Gemfile
+++ b/Gemfile
@@ -277,4 +277,4 @@ end
 gem "newrelic_rpm"
 
 gem 'octokit', '3.7.0'
-gem "rugments"
+gem "rugments", "~> 1.0.0.beta7"
diff --git a/Gemfile.lock b/Gemfile.lock
index 80e4a44c1da..7dbc3b4ffa9 100644
--- a/Gemfile.lock
+++ b/Gemfile.lock
@@ -531,7 +531,7 @@ GEM
     rubyntlm (0.5.0)
     rubypants (0.2.0)
     rugged (0.22.2)
-    rugments (1.0.0.beta6)
+    rugments (1.0.0.beta7)
     safe_yaml (0.9.7)
     sanitize (2.1.0)
       nokogiri (>= 1.4.4)
@@ -781,7 +781,7 @@ DEPENDENCIES
   rqrcode-rails3
   rspec-rails (= 2.99)
   rubocop (= 0.28.0)
-  rugments
+  rugments (~> 1.0.0.beta7)
   sanitize (~> 2.0)
   sass-rails (~> 4.0.2)
   sdoc
diff --git a/app/helpers/blob_helper.rb b/app/helpers/blob_helper.rb
index 9fe5f82f02f..50df3801703 100644
--- a/app/helpers/blob_helper.rb
+++ b/app/helpers/blob_helper.rb
@@ -1,6 +1,6 @@
 module BlobHelper
-  def highlight(blob_name, blob_content, nowrap = false)
-    formatter = Rugments::Formatters::HTML.new(
+  def highlight(blob_name, blob_content, nowrap: false, continue: false)
+    @formatter ||= Rugments::Formatters::HTML.new(
       nowrap: nowrap,
       cssclass: 'code highlight',
       lineanchors: true,
@@ -8,11 +8,11 @@ module BlobHelper
     )
 
     begin
-      lexer = Rugments::Lexer.guess(filename: blob_name, source: blob_content)
-      result = formatter.format(lexer.lex(blob_content)).html_safe
+      @lexer ||= Rugments::Lexer.guess(filename: blob_name, source: blob_content).new
+      result = @formatter.format(@lexer.lex(blob_content, continue: continue)).html_safe
     rescue
       lexer = Rugments::Lexers::PlainText
-      result = formatter.format(lexer.lex(blob_content)).html_safe
+      result = @formatter.format(lexer.lex(blob_content)).html_safe
     end
 
     result
diff --git a/app/views/projects/blame/show.html.haml b/app/views/projects/blame/show.html.haml
index 462f5b7afb0..8019c7f4569 100644
--- a/app/views/projects/blame/show.html.haml
+++ b/app/views/projects/blame/show.html.haml
@@ -32,5 +32,5 @@
                 %code
                   :erb
                     <% lines.each do |line| %>
-                      <%= highlight(@blob.name, line, true).html_safe %>
+                      <%= highlight(@blob.name, line, nowrap: true, continue: true).html_safe %>
                     <% end %>
diff --git a/spec/helpers/blob_helper_spec.rb b/spec/helpers/blob_helper_spec.rb
new file mode 100644
index 00000000000..e49e4e6d5d8
--- /dev/null
+++ b/spec/helpers/blob_helper_spec.rb
@@ -0,0 +1,33 @@
+require 'spec_helper'
+
+describe BlobHelper do
+  describe 'highlight' do
+    let(:blob_name) { 'test.lisp' }
+    let(:no_context_content) { ":type \"assem\"))" }
+    let(:blob_content) { "(make-pathname :defaults name\n#{no_context_content}" }
+    let(:split_content) { blob_content.split("\n") }
+
+    it 'should return plaintext for unknown lexer context' do
+      result = highlight(blob_name, no_context_content, nowrap: true, continue: false)
+      expect(result).to eq('<span id="LC1" class="line">:type &quot;assem&quot;))</span>')
+    end
+
+    it 'should highlight single block' do
+      expected = %Q[<span id="LC1" class="line"><span class="p">(</span><span class="nb">make-pathname</span> <span class="ss">:defaults</span> <span class="nv">name</span></span>
+<span id="LC2" class="line"><span class="ss">:type</span> <span class="s">&quot;assem&quot;</span><span class="p">))</span></span>]
+
+      expect(highlight(blob_name, blob_content, nowrap: true, continue: false)).to eq(expected)
+    end
+
+    it 'should highlight continued blocks' do
+      # Both lines have LC1 as ID since formatter doesn't support continue at the moment
+      expected = [
+        '<span id="LC1" class="line"><span class="p">(</span><span class="nb">make-pathname</span> <span class="ss">:defaults</span> <span class="nv">name</span></span>',
+        '<span id="LC1" class="line"><span class="ss">:type</span> <span class="s">&quot;assem&quot;</span><span class="p">))</span></span>'
+      ]
+
+      result = split_content.map{ |content| highlight(blob_name, content, nowrap: true, continue: true) }
+      expect(result).to eq(expected)
+    end
+  end
+end