Merge branch 'master' into lua-scanner

18 files changed, 272 insertions, 227 deletions

diff --git a/.gitignore b/.gitignore
index c03ec75..caab594 100644
--- a/.gitignore
+++ b/.gitignore
@@ -22,4 +22,4 @@ bench/test.div.html
 diff.html
 etc/CodeRay.tmproj
 *.swp
-etc
\ No newline at end of file
+old-stuff
diff --git a/.travis.yml b/.travis.yml
index 63a9b0b..7c29c56 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -6,7 +6,7 @@ rvm:
   - jruby-19mode
   - rbx-18mode
   - rbx-19mode
-  # - ruby-head  # test again later: RedCloth not compiling
+  - ruby-head  # test again later: RedCloth not compiling
   - jruby-head
   - ree
 branches:
diff --git a/README.textile b/CREDITS.textile
index 543dc47..4c58c54 100644
--- a/README.textile
+++ b/CREDITS.textile
@@ -1,43 +1,4 @@
-h1. CodeRay !https://secure.travis-ci.org/rubychan/coderay.png!:https://secure.travis-ci.org/rubychan/coderay
-
-h2. About
-
-CodeRay is a Ruby library for syntax highlighting.
-
-You put your code in, and you get it back colored; Keywords, strings, floats, comments - all in different colors. And with line numbers.
-
-*Syntax Highlighting*…
-
-* makes code easier to read and maintain
-* lets you detect syntax errors faster
-* helps you to understand the syntax of a language
-* looks nice
-* is what everybody wants to have on their website
-* solves all your problems and makes the girls run after you
-
-
-h2. Installation
-
-bc. gem install coderay
-
-h3. Dependencies
-
-CodeRay needs Ruby 1.8.7+ or 1.9.2+. It also runs on Rubinius and JRuby.
-
-h2. Example Usage
-
-bc.. require 'coderay'
-
-html = CodeRay.scan("puts 'Hello, world!'", :ruby).div(:line_numbers => :table)
-
-p. 
-
-h2. Documentation
-
-See "http://coderay.rubychan.de/doc/":http://coderay.rubychan.de/doc/.
-
-
-h2. Credits
+h1. Credits
 
 h3. Special Thanks to
 
diff --git a/Changes.textile b/Changes.textile
index 42af2c8..3984b83 100644
--- a/Changes.textile
+++ b/Changes.textile
@@ -1,83 +1,93 @@
 h1=. CodeRay Version History
- 
+
 p=. _This files lists all changes in the CodeRay library since the 0.9.8 release._
- 
-{{toc}}
- 
-h2. Next Version
- 
+
+h2. Changes in 1.1
+
+* Diff scanner: Highlight inline changes in multi-line changes [#99]
+* Remove double-click toggle handler from HTML table output
+* Display line numbers in HTML @:table@ mode even for single-line code (remove special case) [#41, thanks to Ariejan de Vroom]
+
+h2. Changes in 1.0.9
+
+* Fix Ruby scanner: Ruby 1.9 hash syntax @{ key: value }@ is highlighted correctly. [GH #106, thanks to Seth Vargo]
+* Fix HTML scanner: Accept DTDs. [GH #83]
+* Fix PHP scanner: Accept Unicode. [GH #40, thanks to Lance Li]
+
+h2. Changes in 1.0.8
+
 * add @:string/:char@, remove @:regexp/:function@ color from Terminal encoder [GH #29, thanks to Kyrylo Silin]
 * allow @-@ in line number anchor prefix for HTML encoder [GH #32, thanks to shurizzle]
 * Fix HTML scanner: Don't crash if HTML in a diff contains a JavaScript tag.
 
 h2. Changes in 1.0.7
- 
+
 * Changed license from LGPL to MIT. [GH-25, thanks to jessehu]
 * Fix issue with plugin files not being loaded. [GH-20, thanks to Will Read]
 * Fix HTML scanner bug: Don't choke on boolean attributes. [GH-26, thanks to jugglinmike]
 
 h2. Changes in 1.0.6
- 
+
 * New option @:break_lines@ for the HTML encoder (splits tokens at line breaks). [GH-15, thanks to Etienne Massip]
 * Improved speed of @:line_numbers => :inline@ option for the HTML encoder.
 * Fixed wrong HTML file type. (was @:page@) [GH-16, thanks to Doug Hammond]
 * The CSS Scanner now highlights tokens like @url(...)@ as @:function@ instead of @:string@. [GH-13, thanks to Joel Holdbrooks]
 
 h2. Changes in 1.0.5
- 
+
 Fixes:
- 
+
 * @autoload@ calls do not depend on @coderay/lib@ being in the load path (GitHub issue #6; thanks to tvon, banister, envygeeks, and ConradIrwin)
 * avoid dark blue as terminal color (GitHub issue #9; thanks to shevegen)
 
 h2. Changes in 1.0.4
- 
+
 Fixes in the CSS scanner:
- 
+
 * understands the unit "s" (seconds)
 * ignores unexpected curly braces
 * code inside of diffs is highlighted correctly
 
 h2. Changes in 1.0.3
- 
+
 New:
- 
+
 * .tmproj files are recognized as XML.
- 
+
 Fixes:
- 
+
 * Removed files are highlighted inside diffs generated by git.
 
 h2. Changes in 1.0.2
- 
+
 Fixes:
- 
+
 * .erb files are recognized as ERB.
 
 h2. Changes in 1.0.1
- 
+
 New:
- 
+
 * YAML scanner allows "-" and "/" in key names
- 
+
 Changes:
- 
+
 * HTML page output has no white border anymore (alpha style)
- 
+
 Fixes:
- 
+
 * fixed warning in the output of "coderay stylesheet"
 * fixed additional scrollbar in code when last line contains an eyecatcher
 * minor fixes in the tests (issue github-#4)
 
 h2. Changes in 1.0
- 
+
 CodeRay 1.0 is a major rewrite of the library, and incompatible to earlier versions.
- 
+
 The command line and programmer interfaces are similar to 0.9, but the internals have completely changed.
 
 h3. General changes
- 
+
 * *NEW*: The new Diff scanner colorizes code inside of the diff, and highlights inline changes.
 * *NEW*: Extended support and usage of HTML5 and CSS 3 features.
 * *NEW*: Direct Streaming
@@ -89,18 +99,18 @@ h3. General changes
 * *IMPROVED* Tests: There are more of them now!
 
 h3. Direct Streaming
- 
+
 CodeRay 1.0 introduces _Direct Streaming_ as a faster and simpler alternative to Tokens. It means that all Scanners,
 Encoders and Filters had to be rewritten, and that older scanners using the Tokens API are no longer compatible with
 this version.
- 
+
 The main benefits of this change are:
- 
+
 * more speed (benchmarks show 10% to 50% more tokens per second compared to CodeRay 0.9)
 * the ability to stream output into a pipe on the command line
 * a simpler API
 * less code
- 
+
 Changes related to the new tokens handling include:
 * *CHANGED*: The Scanners now call Encoders directly; tokens are not added to a Tokens array, but are send to the
   Encoder as a method call. The Tokens representation (which can be seen as a cache now) is still present, but as a
@@ -117,16 +127,16 @@ Changes related to the new tokens handling include:
   and have been removed.
 
 h3. Command Line
- 
+
 The @coderay@ executable was rewritten and has a few new features:
- 
+
 * *NEW* Ability to stream into a pipe; try @coderay file | more -r@
 * *NEW* help
 * *IMPROVED*: more consistent parameter handling
 * *REMOVED* @coderay_stylesheet@ executable; use @coderay stylesheet [name]@.
 
 h3. @Tokens@
- 
+
 * *NEW* methods @count@, @begin_group@, @end_group@, @begin_line@, and @end_line@.
 * *REMOVED* methods @#stream?@, @#each_text_token@.
 * *REMOVED* methods @#optimize@, @#fix@, @#split_into_lines@ along with their bang! variants.
@@ -134,11 +144,11 @@ h3. @Tokens@
 * *REMOVED* special implementation of @#each@ taking a filter parameter. Use @TokenKindFilter@ instead.
 
 h3. *RENAMED*: @TokenKinds@
- 
+
 Renamed from @Tokens::ClassOfKind@ (was also @Tokens::AbbreviationForKind@ for a while).
 The term "token class" is no longer used in CodeRay. Instead, tokens have _kinds_.
 See "#122":http://odd-eyed-code.org/issues/122.
- 
+
 * *CHANGED* all token CSS classes to readable names.
 * *ADDED* token kinds @:filename@, @:namespace@, and @:eyecatcher@.
 * *RENAMED* @:pre_constant@ and @:pre_type@ to @:predefined_constant@ and @predefined_type@.
@@ -150,23 +160,23 @@ See "#122":http://odd-eyed-code.org/issues/122.
   @:NO_HIGHLIGHT@ to @false@.
 
 h3. @Duo@
- 
+
 * *NEW* method @call@ for allowing code like @CodeRay::Duo[:python => :yaml].(code)@ in Ruby 1.9.
 
 h3. @Encoders::CommentFilter@
- 
+
 * *NEW* alias @:remove_comments@
 
 h3. @Encoders::Filter@
- 
+
 * *NEW* option @tokens@.
 * *CHANGED*: Now it simply delegates to the output.
 * *REMOVED* @include_text_token?@ and @include_block_token?@ methods.
 
 h3. @Encoders::HTML@
- 
+
 The HTML encoder was cleaned up and simplified.
- 
+
 * *NEW*: HTML5 and CSS 3 compatible.
   See "#215":http://odd-eyed-code.org/issues/215.
 * *ADDED* support for @:line_number_anchors@.
@@ -180,11 +190,11 @@ The HTML encoder was cleaned up and simplified.
 * *RENAMED* @Output#numerize@ to @#number@, which is an actual English word.
 
 h3. @Encoders::LinesOfCode@
- 
+
 * *CHANGED*: @compile@ and @finish@ methods are now protected.
 
 h3. *Renamed*: @Encoders::Terminal@ (was @Encoders::Term@)
- 
+
 * *RENAMED* from @Encoders::Term@, added @:term@ alias.
 * *CLEANUP*: Use @#setup@'s @super@, don't use @:procedure@ token class.
 * *CHANGED*: @#token@'s second parameter is no longer optional.
@@ -192,21 +202,21 @@ h3. *Renamed*: @Encoders::Terminal@ (was @Encoders::Term@)
 * *FIXED* handling of line tokens.
 
 h3. @Encoders::Text@
- 
+
 * *FIXED* default behavior of stripping the trailing newline.
 
 h3. *RENAMED*: @Encoders::TokenKindFilter@ (was @Encoders::TokenClassFilter@)
- 
+
 * *NEW*: Handles token groups.
   See "#223":http://odd-eyed-code.org/issues/223.
 * *RENAMED* @include_block_token?@ to @include_group?@.
 
 h3. @Encoders::Statistic@
- 
+
 * *CHANGED*: Tokens actions are counted separately.
 
 h3. @Scanners::Scanner@
- 
+
 * *NEW* methods @#file_extension@ and @#encoding@.
 * *NEW*: The @#tokenize@ method also takes an Array of Strings as source. The
   code is highlighted as one and split into parts of the input lengths
@@ -219,11 +229,11 @@ h3. @Scanners::Scanner@
 * *CHANGED*: @#column@ starts counting with 1 instead of 0
 
 h3. *NEW*: @Scanners::Clojure@
- 
+
 Thanks to Licenser, CodeRay now supports the Clojure language.
 
 h3. @Scanners::CSS@
- 
+
 * *NEW*: Rudimentary support for the @attr@, @counter@, and @counters@ functions.
   See "#224":http://odd-eyed-code.org/issues/224.
 * *NEW*: Rudimentary support for CSS 3 colors.
@@ -231,7 +241,7 @@ h3. @Scanners::CSS@
 * *CHANGED*: Comments are scanned as one token instead of three.
 
 h3. @Scanners::Debug@
- 
+
 * *NEW*: Support for line tokens (@begin_line@ and @end_line@ represented by @[@ and @]@.)
 * *FIXED*: Don't send @:error@ and @nil@ tokens for buggy input any more.
 * *FIXED*: Closes unclosed tokens at the end of @scan_tokens@.
@@ -239,32 +249,32 @@ h3. @Scanners::Debug@
 * *CHANGED*: Raises an error when trying to end an invalid token group.
 
 h3. @Scanners::Delphi@
- 
+
 * *FIXED*: Closes open string groups.
 
 h3. @Scanners::Diff@
- 
+
 * *NEW*: Highlighting of code based on file names.
   See ticket "#52":http://odd-eyed-code.org/issues/52.
-  
+
   Use the @:highlight_code@ option to turn this feature off. It's enabled
   by default.
-  
+
   This is a very original feature. It enables multi-language highlighting for
   diff files, which is especially helpful for CodeRay development itself. The
   updated version of the scanner test suite generated .debug.diff.html files
   using this.
-  
+
   Note: This is still experimental. Tokens spanning more than one line
   may get highlighted incorrectly. CodeRay tries to keep scanner states
   between the lines and changes, but the quality of the results depend on
   the scanner.
 * *NEW*: Inline change highlighting, as suggested by Eric Thomas.
   See ticket "#227":http://odd-eyed-code.org/issues/227 for details.
-  
+
   Use the @:inline_diff@ option to turn this feature off. It's enabled by
   default.
-  
+
   For single-line changes (that is, a single deleted line followed by a single
   inserted line), this feature surrounds the changed parts with an
   @:eyecatcher@ group which appears in a more saturated background color.
@@ -281,38 +291,38 @@ h3. @Scanners::Diff@
 h3. *RENAMED*: @Scanners::ERB@ (was @Scanners::RHTML@)
 
 h3. *NEW*: @Scanners::HAML@
- 
+
 It uses the new :state options of the HTML and Ruby scanners.
- 
+
 Some rare cases are not considered (like @#{...}@ snippets inside of :javascript blocks),
 but it highlights pretty well.
 
 h3. @Scanners::HTML@
- 
+
 * *FIXED*: Closes open string groups.
 
 h3. @Scanners::JavaScript@
- 
+
 * *IMPROVED*: Added @NaN@ and @Infinity@ to list of predefined constants.
 * *IMPROVED* recognition of RegExp literals with leading spaces.
 
 h3. @Scanners::Java@
- 
+
 * *NEW*: Package names are highlighted as @:namespace@.
   See "#210":http://odd-eyed-code.org/issues/210.
 
 h3. *REMOVED*: @Scanners::NitroXHTML@
- 
+
 Nitro is "dead":http://www.nitrohq.com/.
 
 h3. *RENAMED*: @Scanners::Text@ (was @Scanners::Plaintext@)
- 
+
 * *IMPROVED*: Just returns the string without scanning (faster).
- 
+
  This is much faster than scanning until @/\z/@ in Ruby 1.8.
 
 h3. @Scanners::Python@
- 
+
 * *CHANGED*: Docstrings are highlighted as @:comment@.
   See "#190":http://odd-eyed-code.org/issues/190.
 
@@ -322,7 +332,7 @@ Copied from @Scanners::Debug@, highlights the token dump instead of importing it
 name suffix now.
 
 h3. @Scanners::Ruby@
- 
+
 * *ADDED* more predefined keywords (see http://murfy.de/ruby-constants).
 * *IMPROVED* support for singleton method definitions.
   See "#147":http://odd-eyed-code.org/issues/147.
@@ -330,44 +340,44 @@ h3. @Scanners::Ruby@
   (eg. @GL.PushMatrix@).
 * *NEW*: Highlight buggy floats (like .5) as @:error@.
 * *CLEANUP* of documentation, names of constants and variables, state handling.
-  
+
   Moved @StringState@ class from @patterns.rb@ into a separate file.
 * *NEW*: Complicated rule for recognition of @foo=@ style method names.
 * *NEW*: Handles @:keep_state@ option (a bit; experimental).
-  
+
   Actually, Ruby checks if there is @[~>=]@, but not @=>@ following the name.
-  
+
 * *REMOVED* @EncodingError@
 
 h3. *REMOVED* @Scanners::Scheme@
- 
+
 * It is too buggy, and nobody was using it. To be added again when it's fixed.
   See "#59":http://odd-eyed-code.org/issues/59.
 
 h3. @Scanners::SQL@
- 
+
 * *IMPROVED*: Extended list of keywords and functions (thanks to
   Joshua Galvez, Etienne Massip, and others).
- 
+
   See "#221":http://odd-eyed-code.org/issues/221.
 * *FIXED*: Closes open string groups.
 * *FIXED*: Words after @.@ are always recognized as @:ident@.
 
 h3. @Scanners::YAML@
- 
+
 * *FIXED*: Allow spaces before colon in mappings.
- 
+
   See "#231":http://odd-eyed-code.org/issues/231.
 
 h3. *NEW*: @Styles::Alpha@
 
 A style that uses transparent HSLA colors as defined in CSS 3. See "#199":http://odd-eyed-code.org/issues/199.
- 
+
 It also uses the CSS 3 property @user-select: none@ to keep the user from selecting the line numbers. This is especially
 nice for @:inline@ line numbers. See "#226":http://odd-eyed-code.org/issues/226.
 
 h3. @WordList@
- 
+
 Stripped down to 19 LOC.
 
 * *RENAMED* @CaseIgnoringWordList@ to @WordList::CaseIgnoring@.
@@ -375,14 +385,14 @@ Stripped down to 19 LOC.
 * *REMOVED* block option.
 
 h3. @FileType@
- 
+
 * *NEW*: Recognizes @.gemspec@, @.rjs@, @.rpdf@ extensions, @Gemfile@, and @Capfile@ as Ruby.
-  
+
   Thanks to the authors of the TextMate Ruby bundle!
 * *REMOVED* @FileType#shebang@ is a protected method now.
 
 h3. @Plugin@
- 
+
 * *IMPROVED*: @register_for@ sets the @plugin_id@; it can now be a @Symbol@.
 * *ADDED* @PluginHost#const_missing@ method: Plugins are loaded automatically.
   Using @Scanners::JavaScript@ in your code loads @scanners/java_script.rb@.
@@ -391,19 +401,19 @@ h3. @Plugin@
 * *CHANGED* the default plugin key from @nil@ to @:default@.
 
 h3. @GZip@
- 
+
 * *MOVED* into @CodeRay@ namespace.
 * *MOVED* file from @gzip_simple.rb@ to @gzip.rb@.
 * *REMOVED* @String@ extensions.
 
 h3. More API changes
- 
+
 * *FIXED* @Encoders::HTML#token@'s second parameter is no longer optional.
 * *CHANGED* @Encoders::HTML::Output@'s API.
 * *REMOVED* lots of unused methods.
- 
+
 The helper classes were cleaned up; see above for details.
- 
+
 * *CHANGED* @Plugin@ API was simplified and stripped of all unnecessary features.
 * *CHANGED* Moved @GZip@ and @FileType@ libraries into @CodeRay@; cleaned them up.
 
diff --git a/README.markdown b/README.markdown
new file mode 100644
index 0000000..42bdab6
--- /dev/null
+++ b/README.markdown
@@ -0,0 +1,27 @@
+# CodeRay [![Build Status](https://travis-ci.org/rubychan/coderay.png)](https://travis-ci.org/rubychan/coderay)
+
+## About
+
+CodeRay is a Ruby library for syntax highlighting.
+
+You put your code in, and you get it back colored; Keywords, strings, floats, comments - all in different colors. And with line numbers.
+
+## Installation
+
+`gem install coderay`
+
+### Dependencies
+
+CodeRay needs Ruby 1.8.7+ or 1.9.2+. It also runs on Rubinius and JRuby.
+
+## Example Usage
+
+```ruby
+require 'coderay'
+
+html = CodeRay.scan("puts 'Hello, world!'", :ruby).div(:line_numbers => :table)
+````
+
+## Documentation
+
+See [http://coderay.rubychan.de/doc/](http://coderay.rubychan.de/doc/).
diff --git a/bench/bench.rb b/bench/bench.rb
index 45dc5b0..1889eed 100644
--- a/bench/bench.rb
+++ b/bench/bench.rb
@@ -108,7 +108,7 @@ N.times do
   $file_created = here('test.' +
     ($dump_output ? 'dump' : $hl.file_extension))
   File.open($file_created, 'wb') do |f|
-    f.write $o
+    # f.write $o
   end
   Dir.chdir(here) do
     FileUtils.copy 'test.dump', 'example.dump' if $dump_output
diff --git a/coderay.gemspec b/coderay.gemspec
index e686035..328b94c 100644
--- a/coderay.gemspec
+++ b/coderay.gemspec
@@ -17,12 +17,14 @@ Gem::Specification.new do |s|
   s.summary     = 'Fast syntax highlighting for selected languages.'
   s.description = 'Fast and easy syntax highlighting for selected languages, written in Ruby. Comes with RedCloth integration and LOC counter.'
   
+  s.license = 'MIT'
+  
   s.platform              = Gem::Platform::RUBY
   s.required_ruby_version = '>= 1.8.6'
   
   readme_file = 'README_INDEX.rdoc'
   
-  s.files         = `git ls-files -- lib/* test/functional/* Rakefile #{readme_file} LICENSE`.split("\n")
+  s.files         = `git ls-files -- lib/* test/functional/* Rakefile #{readme_file} MIT-LICENSE`.split("\n")
   s.test_files    = `git ls-files --       test/functional/*`.split("\n")
   s.executables   = `git ls-files -- bin/*`.split("\n").map { |f| File.basename(f) }
   s.require_paths = ['lib']
diff --git a/lib/coderay/encoders/html/numbering.rb b/lib/coderay/encoders/html/numbering.rb
index e717429..332145b 100644
--- a/lib/coderay/encoders/html/numbering.rb
+++ b/lib/coderay/encoders/html/numbering.rb
@@ -1,15 +1,15 @@
 module CodeRay
 module Encoders
-
+  
   class HTML
-
+    
     module Numbering  # :nodoc:
-
+      
       def self.number! output, mode = :table, options = {}
         return self unless mode
-
+        
         options = DEFAULT_OPTIONS.merge options
-
+        
         start = options[:line_number_start]
         unless start.is_a? Integer
           raise ArgumentError, "Invalid value %p for :line_number_start; Integer expected." % start
@@ -56,12 +56,17 @@ module Encoders
             raise ArgumentError, 'Invalid value %p for :bolding; false or Integer expected.' % bold_every
           end
         
-        line_count = output.count("\n")
-        position_of_last_newline = output.rindex(RUBY_VERSION >= '1.9' ? /\n/ : ?\n)
-        if position_of_last_newline
+        if position_of_last_newline = output.rindex(RUBY_VERSION >= '1.9' ? /\n/ : ?\n)
           after_last_newline = output[position_of_last_newline + 1 .. -1]
           ends_with_newline = after_last_newline[/\A(?:<\/span>)*\z/]
-          line_count += 1 if not ends_with_newline
+          
+          if ends_with_newline
+            line_count = output.count("\n")
+          else
+            line_count = output.count("\n") + 1
+          end
+        else
+          line_count = 1
         end
         
         case mode
@@ -74,30 +79,30 @@ module Encoders
             line_number += 1
             "<span class=\"line-numbers\">#{indent}#{line_number_text}</span>#{line}"
           end
-
+        
         when :table
           line_numbers = (start ... start + line_count).map(&bolding).join("\n")
           line_numbers << "\n"
           line_numbers_table_template = Output::TABLE.apply('LINE_NUMBERS', line_numbers)
-
+          
           output.gsub!(/<\/div>\n/, '</div>')
           output.wrap_in! line_numbers_table_template
           output.wrapped_in = :div
-
+        
         when :list
           raise NotImplementedError, 'The :list option is no longer available. Use :table.'
-
+        
         else
           raise ArgumentError, 'Unknown value %p for mode: expected one of %p' %
             [mode, [:table, :inline]]
         end
-
+        
         output
       end
-
+      
     end
-
+    
   end
-
+  
 end
 end
diff --git a/lib/coderay/encoders/html/output.rb b/lib/coderay/encoders/html/output.rb
index 9132d94..de6f6ea 100644
--- a/lib/coderay/encoders/html/output.rb
+++ b/lib/coderay/encoders/html/output.rb
@@ -124,7 +124,7 @@ module Encoders
 
       TABLE = Template.new <<-TABLE
 <table class="CodeRay"><tr>
-  <td class="line-numbers" title="double click to toggle" ondblclick="with (this.firstChild.style) { display = (display == '') ? 'none' : '' }"><pre><%LINE_NUMBERS%></pre></td>
+  <td class="line-numbers"><pre><%LINE_NUMBERS%></pre></td>
   <td class="code"><pre><%CONTENT%></pre></td>
 </tr></table>
       TABLE
diff --git a/lib/coderay/scanners/diff.rb b/lib/coderay/scanners/diff.rb
index 9e899c3..38efaf4 100644
--- a/lib/coderay/scanners/diff.rb
+++ b/lib/coderay/scanners/diff.rb
@@ -20,7 +20,7 @@ module Scanners
       
       line_kind = nil
       state = :initial
-      deleted_lines = 0
+      deleted_lines_count = 0
       scanners = Hash.new do |h, lang|
         h[lang] = Scanners[lang].new '', :keep_tokens => true, :keep_state => true
       end
@@ -30,7 +30,7 @@ module Scanners
       until eos?
         
         if match = scan(/\n/)
-          deleted_lines = 0 unless line_kind == :delete
+          deleted_lines_count = 0 unless line_kind == :delete
           if line_kind
             encoder.end_line line_kind
             line_kind = nil
@@ -99,37 +99,59 @@ module Scanners
             end
             next
           elsif match = scan(/-/)
-            deleted_lines += 1
-            encoder.begin_line line_kind = :delete
-            encoder.text_token match, :delete
-            if options[:inline_diff] && deleted_lines == 1 && check(/(?>.*)\n\+(?>.*)$(?!\n\+)/)
-              content_scanner_entry_state = content_scanner.state
-              skip(/(.*)\n\+(.*)$/)
-              head, deletion, insertion, tail = diff self[1], self[2]
-              pre, deleted, post = content_scanner.tokenize [head, deletion, tail], :tokens => Tokens.new
-              encoder.tokens pre
-              unless deleted.empty?
-                encoder.begin_group :eyecatcher
-                encoder.tokens deleted
-                encoder.end_group :eyecatcher
+            deleted_lines_count += 1
+            if options[:inline_diff] && deleted_lines_count == 1 && (changed_lines_count = 1 + check(/.*(?:\n\-.*)*/).count("\n")) && match?(/(?>.*(?:\n\-.*){#{changed_lines_count - 1}}(?:\n\+.*){#{changed_lines_count}})$(?!\n\+)/)
+              deleted_lines  = Array.new(changed_lines_count) { |i| skip(/\n\-/) if i > 0; scan(/.*/) }
+              inserted_lines = Array.new(changed_lines_count) { |i| skip(/\n\+/)         ; scan(/.*/) }
+              
+              deleted_lines_tokenized  = []
+              inserted_lines_tokenized = []
+              for deleted_line, inserted_line in deleted_lines.zip(inserted_lines)
+                pre, deleted_part, inserted_part, post = diff deleted_line, inserted_line
+                content_scanner_entry_state = content_scanner.state
+                deleted_lines_tokenized  << content_scanner.tokenize([pre, deleted_part, post], :tokens => Tokens.new)
+                content_scanner.state = content_scanner_entry_state || :initial
+                inserted_lines_tokenized << content_scanner.tokenize([pre, inserted_part, post], :tokens => Tokens.new)
               end
-              encoder.tokens post
-              encoder.end_line line_kind
-              encoder.text_token "\n", :space
-              encoder.begin_line line_kind = :insert
-              encoder.text_token '+', :insert
-              content_scanner.state = content_scanner_entry_state || :initial
-              pre, inserted, post = content_scanner.tokenize [head, insertion, tail], :tokens => Tokens.new
-              encoder.tokens pre
-              unless inserted.empty?
-                encoder.begin_group :eyecatcher
-                encoder.tokens inserted
-                encoder.end_group :eyecatcher
+              
+              for pre, deleted_part, post in deleted_lines_tokenized
+                encoder.begin_line :delete
+                encoder.text_token '-', :delete
+                encoder.tokens pre
+                unless deleted_part.empty?
+                  encoder.begin_group :eyecatcher
+                  encoder.tokens deleted_part
+                  encoder.end_group :eyecatcher
+                end
+                encoder.tokens post
+                encoder.end_line :delete
+                encoder.text_token "\n", :space
+              end
+              
+              for pre, inserted_part, post in inserted_lines_tokenized
+                encoder.begin_line :insert
+                encoder.text_token '+', :insert
+                encoder.tokens pre
+                unless inserted_part.empty?
+                  encoder.begin_group :eyecatcher
+                  encoder.tokens inserted_part
+                  encoder.end_group :eyecatcher
+                end
+                encoder.tokens post
+                changed_lines_count -= 1
+                if changed_lines_count > 0
+                  encoder.end_line :insert
+                  encoder.text_token "\n", :space
+                end
               end
-              encoder.tokens post
+              
+              line_kind = :insert
+              
             elsif match = scan(/.*/)
+              encoder.begin_line line_kind = :delete
+              encoder.text_token '-', :delete
               if options[:highlight_code]
-                if deleted_lines == 1
+                if deleted_lines_count == 1
                   content_scanner_entry_state = content_scanner.state
                 end
                 content_scanner.tokenize match, :tokens => encoder unless match.empty?
diff --git a/lib/coderay/scanners/html.rb b/lib/coderay/scanners/html.rb
index 49c346d..3ba3b79 100644
--- a/lib/coderay/scanners/html.rb
+++ b/lib/coderay/scanners/html.rb
@@ -101,7 +101,7 @@ module Scanners
           when :initial
             if match = scan(/<!--(?:.*?-->|.*)/m)
               encoder.text_token match, :comment
-            elsif match = scan(/<!DOCTYPE(?:.*?>|.*)/m)
+            elsif match = scan(/<!(\w+)(?:.*?>|.*)|\]>/m)
               encoder.text_token match, :doctype
             elsif match = scan(/<\?xml(?:.*?\?>|.*)/m)
               encoder.text_token match, :preprocessor
diff --git a/lib/coderay/scanners/php.rb b/lib/coderay/scanners/php.rb
index 8acfff5..6c68834 100644
--- a/lib/coderay/scanners/php.rb
+++ b/lib/coderay/scanners/php.rb
@@ -1,4 +1,4 @@
-# encoding: ASCII-8BIT
+# encoding: utf-8
 module CodeRay
 module Scanners
   
@@ -11,7 +11,6 @@ module Scanners
     
     register_for :php
     file_extension 'php'
-    encoding 'BINARY'
     
     KINDS_NOT_LOC = HTML::KINDS_NOT_LOC
     
@@ -211,7 +210,7 @@ module Scanners
       
       HTML_INDICATOR = /<!DOCTYPE html|<(?:html|body|div|p)[> ]/i
       
-      IDENTIFIER = /[a-z_\x7f-\xFF][a-z0-9_\x7f-\xFF]*/i
+      IDENTIFIER = 'ä'[/[[:alpha:]]/] == 'ä' ? Regexp.new('[[:alpha:]_[^\0-\177]][[:alnum:]_[^\0-\177]]*') : Regexp.new('[a-z_\x7f-\xFF][a-z0-9_\x7f-\xFF]*', true)
       VARIABLE = /\$#{IDENTIFIER}/
       
       OPERATOR = /
diff --git a/lib/coderay/scanners/ruby.rb b/lib/coderay/scanners/ruby.rb
index 2be98a6..c5cf1e2 100644
--- a/lib/coderay/scanners/ruby.rb
+++ b/lib/coderay/scanners/ruby.rb
@@ -94,18 +94,27 @@ module Scanners
             if !method_call_expected &&
                match = scan(unicode ? /#{patterns::METHOD_NAME}/uo :
                                       /#{patterns::METHOD_NAME}/o)
-              value_expected = false
+              
               kind = patterns::IDENT_KIND[match]
-              if kind == :ident
-                if match[/\A[A-Z]/] && !(match[/[!?]$/] || match?(/\(/))
-                  kind = :constant
+              if kind == :ident && value_expected != :colon_expected && scan(/:(?!:)/)
+                value_expected = true
+                encoder.text_token match, :key
+                encoder.text_token ':',   :operator
+              else
+                value_expected = false
+                if kind == :ident
+                  if match[/\A[A-Z]/] && !(match[/[!?]$/] || match?(/\(/))
+                    kind = :constant
+                  end
+                elsif kind == :keyword
+                  state = patterns::KEYWORD_NEW_STATE[match]
+                  if patterns::KEYWORDS_EXPECTING_VALUE[match]
+                    value_expected = match == 'when' ? :colon_expected : true
+                  end
                 end
-              elsif kind == :keyword
-                state = patterns::KEYWORD_NEW_STATE[match]
-                value_expected = true if patterns::KEYWORDS_EXPECTING_VALUE[match]
+                value_expected = true if !value_expected && check(/#{patterns::VALUE_FOLLOWS}/o)
+                encoder.text_token match, kind
               end
-              value_expected = true if !value_expected && check(/#{patterns::VALUE_FOLLOWS}/o)
-              encoder.text_token match, kind
               
             elsif method_call_expected &&
                match = scan(unicode ? /#{patterns::METHOD_AFTER_DOT}/uo :
@@ -119,9 +128,9 @@ module Scanners
               value_expected = check(/#{patterns::VALUE_FOLLOWS}/o)
               
             # OPERATORS #
-            elsif !method_call_expected && match = scan(/ (\.(?!\.)|::) | (?: \.\.\.? | ==?=? | [,\(\[\{] )() | [\)\]\}] /x)
+            elsif !method_call_expected && match = scan(/ (\.(?!\.)|::) | ( \.\.\.? | ==?=? | [,\(\[\{] ) | [\)\]\}] /x)
               method_call_expected = self[1]
-              value_expected = !method_call_expected && self[2]
+              value_expected = !method_call_expected && !!self[2]
               if inline_block_stack
                 case match
                 when '{'
@@ -213,7 +222,7 @@ module Scanners
               encoder.text_token match, :integer
               
             elsif match = scan(/ %=? | <(?:<|=>?)? | \? /x)
-              value_expected = true
+              value_expected = match == '?' ? :colon_expected : true
               encoder.text_token match, :operator
               
             elsif match = scan(/`/)
diff --git a/lib/coderay/scanners/ruby/patterns.rb b/lib/coderay/scanners/ruby/patterns.rb
index a52198e..ed071d2 100644
--- a/lib/coderay/scanners/ruby/patterns.rb
+++ b/lib/coderay/scanners/ruby/patterns.rb
@@ -1,9 +1,9 @@
 # encoding: utf-8
 module CodeRay
 module Scanners
-
+  
   module Ruby::Patterns  # :nodoc: all
-
+    
     KEYWORDS = %w[
       and def end in or unless begin
       defined? ensure module redo super until
@@ -12,7 +12,7 @@ module Scanners
       while alias class elsif if not return
       undef yield
     ]
-
+    
     # See http://murfy.de/ruby-constants.
     PREDEFINED_CONSTANTS = %w[
       nil true false self
@@ -24,19 +24,19 @@ module Scanners
       RUBY_PLATFORM RUBY_RELEASE_DATE RUBY_REVISION RUBY_VERSION
       __FILE__ __LINE__ __ENCODING__
     ]
-
+    
     IDENT_KIND = WordList.new(:ident).
       add(KEYWORDS, :keyword).
       add(PREDEFINED_CONSTANTS, :predefined_constant)
-
+    
     KEYWORD_NEW_STATE = WordList.new(:initial).
       add(%w[ def ], :def_expected).
       add(%w[ undef ], :undef_expected).
       add(%w[ alias ], :alias_expected).
       add(%w[ class module ], :module_expected)
-
-    IDENT = 'ä'[/[[:alpha:]]/] == 'ä' ? /[[:alpha:]_][[:alnum:]_]*/ : /[^\W\d]\w*/
-
+    
+    IDENT = 'ä'[/[[:alpha:]]/] == 'ä' ? Regexp.new('[[:alpha:]_[^\0-\177]][[:alnum:]_[^\0-\177]]*') : /[^\W\d]\w*/
+    
     METHOD_NAME = / #{IDENT} [?!]? /ox
     METHOD_NAME_OPERATOR = /
       \*\*?           # multiplication and power
@@ -57,25 +57,25 @@ module Scanners
     GLOBAL_VARIABLE = / \$ (?: #{IDENT} | [1-9]\d* | 0\w* | [~&+`'=\/,;_.<>!@$?*":\\] | -[a-zA-Z_0-9] ) /ox
     PREFIX_VARIABLE = / #{GLOBAL_VARIABLE} | #{OBJECT_VARIABLE} /ox
     VARIABLE = / @?@? #{IDENT} | #{GLOBAL_VARIABLE} /ox
-
+    
     QUOTE_TO_TYPE = {
       '`' => :shell,
       '/'=> :regexp,
     }
     QUOTE_TO_TYPE.default = :string
-
+    
     REGEXP_MODIFIERS = /[mousenix]*/
-
+    
     DECIMAL = /\d+(?:_\d+)*/
     OCTAL = /0_?[0-7]+(?:_[0-7]+)*/
     HEXADECIMAL = /0x[0-9A-Fa-f]+(?:_[0-9A-Fa-f]+)*/
     BINARY = /0b[01]+(?:_[01]+)*/
-
+    
     EXPONENT = / [eE] [+-]? #{DECIMAL} /ox
     FLOAT_SUFFIX = / #{EXPONENT} | \. #{DECIMAL} #{EXPONENT}? /ox
     FLOAT_OR_INT = / #{DECIMAL} (?: #{FLOAT_SUFFIX} () )? /ox
     NUMERIC = / (?: (?=0) (?: #{OCTAL} | #{HEXADECIMAL} | #{BINARY} ) | #{FLOAT_OR_INT} ) /ox
-
+    
     SYMBOL = /
       :
       (?:
@@ -85,7 +85,7 @@ module Scanners
       )
     /ox
     METHOD_NAME_OR_SYMBOL = / #{METHOD_NAME_EX} | #{SYMBOL} /ox
-
+    
     SIMPLE_ESCAPE = /
         [abefnrstv]
       |  [0-7]{1,3}
@@ -110,7 +110,7 @@ module Scanners
       | \\ #{ESCAPE}
       )
     /mox
-
+    
     # NOTE: This is not completely correct, but
     # nobody needs heredoc delimiters ending with \n.
     HEREDOC_OPEN = /
@@ -122,13 +122,13 @@ module Scanners
         ( [^\n]*? ) \3     # $4 = delim
       )
     /mx
-
+    
     RUBYDOC = /
       =begin (?!\S)
       .*?
       (?: \Z | ^=end (?!\S) [^\n]* )
     /mx
-
+    
     DATA = /
       __END__$
       .*?
@@ -136,7 +136,7 @@ module Scanners
     /mx
     
     RUBYDOC_OR_DATA = / #{RUBYDOC} | #{DATA} /xo
-
+    
     # Checks for a valid value to follow. This enables
     # value_expected in method calls without parentheses.
     VALUE_FOLLOWS = /
diff --git a/lib/coderay/version.rb b/lib/coderay/version.rb
index 87d1cff..bfb5f24 100644
--- a/lib/coderay/version.rb
+++ b/lib/coderay/version.rb
@@ -1,3 +1,3 @@
 module CodeRay
-  VERSION = '1.0.8'
+  VERSION = '1.0.9'
 end
diff --git a/rake_tasks/test.rake b/rake_tasks/test.rake
index f070ccc..a60699d 100644
--- a/rake_tasks/test.rake
+++ b/rake_tasks/test.rake
@@ -1,5 +1,4 @@
 namespace :test do
-  
   desc 'run all sample tests'
   task :samples do
     ruby './sample/suite.rb'
@@ -17,22 +16,33 @@ namespace :test do
   end
   
   scanner_suite = 'test/scanners/suite.rb'
-  task scanner_suite do
-    unless File.exist? scanner_suite
-      puts 'Scanner tests not found; downloading from Subversion...'
-      sh 'svn co http://svn.rubychan.de/coderay-scanner-tests/trunk/ test/scanners/'
-      puts 'Finished.'
-    end
-  end
-  
   desc 'run all scanner tests'
   task :scanners => :update_scanner_suite do
     ruby scanner_suite
   end
   
-  desc 'update scanner test suite from SVN'
-  task :update_scanner_suite => scanner_suite do
-    sh "svn up #{File.dirname(scanner_suite)}"
+  desc 'update scanner test suite from GitHub'
+  task :update_scanner_suite do
+    if File.exist? scanner_suite
+      Dir.chdir File.dirname(scanner_suite) do
+        if File.directory? '.git'
+          puts 'Updating scanner test suite...'
+          sh 'git pull'
+        elsif File.directory? '.svn'
+          raise <<-ERROR
+Found the deprecated Subversion scanner test suite in ./#{File.dirname(scanner_suite)}.
+Please rename or remove it and run again to use the GitHub repository:
+
+  mv test/scanners test/scanners-old
+          ERROR
+        else
+          raise 'No scanner test suite found.'
+        end
+      end
+    else
+      puts 'Downloading scanner test suite...'
+      sh 'git clone https://github.com/rubychan/coderay-scanner-tests.git test/scanners/'
+    end
   end
   
   namespace :scanner do
@@ -72,7 +82,6 @@ namespace :test do
       puts "Skipping."
     end
   end
-  
 end
 
 task :test => %w(test:functional test:units test:exe)
diff --git a/test/executable/suite.rb b/test/executable/suite.rb
index f3495d6..d386f4b 100644
--- a/test/executable/suite.rb
+++ b/test/executable/suite.rb
@@ -14,12 +14,13 @@ class TestCodeRayExecutable < Test::Unit::TestCase
   
   ROOT_DIR = Pathname.new(File.dirname(__FILE__)) + '..' + '..'
   EXECUTABLE = ROOT_DIR + 'bin' + 'coderay'
+  RUBY_COMMAND = RUBY_VERSION < '2.0.0' ? 'ruby -w' : 'ruby'  # Ruby 2 currently throws warnings for bundler
   EXE_COMMAND =
     if RUBY_PLATFORM === 'java' && `ruby --ng -e '' 2> /dev/null` && $?.success?
       # use Nailgun
-      'ruby --ng -wI%s %s'
+      "#{RUBY_COMMAND}--ng -I%s %s"
     else
-      'ruby -wI%s %s'
+      "#{RUBY_COMMAND} -I%s %s"
     end % [ROOT_DIR + 'lib', EXECUTABLE]
   
   def coderay args, options = {}
diff --git a/test/functional/examples.rb b/test/functional/examples.rb
index ff64af3..15f9ca3 100755
--- a/test/functional/examples.rb
+++ b/test/functional/examples.rb
@@ -22,7 +22,7 @@ end
     CODE
     assert_equal <<-DIV, div
 <table class="CodeRay"><tr>
-  <td class="line-numbers" title="double click to toggle" ondblclick="with (this.firstChild.style) { display = (display == '') ? 'none' : '' }"><pre><a href="#n1" name="n1">1</a>
+  <td class="line-numbers"><pre><a href="#n1" name="n1">1</a>
 <a href="#n2" name="n2">2</a>
 <a href="#n3" name="n3">3</a>
 </pre></td>
@@ -38,7 +38,7 @@ end
 <body>
 
 <table class="CodeRay"><tr>
-  <td class="line-numbers" title="double click to toggle" ondblclick="with (this.firstChild.style) { display = (display == '') ? 'none' : '' }"><pre>
+  <td class="line-numbers"><pre><a href="#n1" name="n1">1</a>
 </pre></td>
   <td class="code"><pre>puts <span class="string"><span class="delimiter">&quot;</span><span class="content">Hello, world!</span><span class="delimiter">&quot;</span></span></pre></td>
 </tr></table>