Merge branch 'master' into paint-integrationpaint-integration

99 files changed, 1322 insertions, 5875 deletions

diff --git a/.gitignore b/.gitignore
index dd001c8..deed1a2 100644
--- a/.gitignore
+++ b/.gitignore
@@ -11,5 +11,4 @@ Gemfile.lock
 test/executable/source.rb.html
 test/executable/source.rb.json
 test/scanners
-bench/test.div.html
 old-stuff
diff --git a/.travis.yml b/.travis.yml
index 59bb791..f815698 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -2,7 +2,10 @@ rvm:
   - 1.8.7
   - ree
   - 1.9.3
-  - 2.0.0
+  - 2.0
+  - 2.1
+  - 2.2
+  - 2.3.0
   - ruby-head
   - jruby-18mode
   - jruby-19mode
@@ -14,6 +17,9 @@ branches:
     - master
 matrix:
   allow_failures:
+    - rvm: ruby-head
+    - rvm: jruby-head
     - rvm: rbx-18mode
     - rvm: rbx-19mode
 script: "rake test" # test:scanners"
+sudo: false
diff --git a/Changes.textile b/Changes.textile
index 9570a03..1276a33 100644
--- a/Changes.textile
+++ b/Changes.textile
@@ -2,21 +2,80 @@ h1=. CodeRay Version History
 
 p=. _This files lists all changes in the CodeRay library since the 0.9.8 release._
 
+h2. Changes in 1.1.1
+
+* SQL scanner: Allow @$@ signs in SQL identifiers [#164, thanks to jasir and Ben Basson]
+* SQL scanner: Fix open strings [#163, thanks to Adam]
+* Ruby scanner: Accept number literal suffixes @r@ and @i@ (Ruby 2.1)
+* Ruby scanner: Accept quoted hash keys like @{ "a": boss }@ (Ruby 2.2)
+* Ruby scanner: Accept save navigation operator @&.@ (Ruby 2.3)
+* Ruby scanner: Accept squiggly heredoc @<<~@ (Ruby 2.3)
+* Diff scanner: Prevent running out of regexp stack.
+* HTML encoder: You can keep tabs intact now by setting @tab_width: false@.
+
 h2. Changes in 1.1
 
-* New scanner: Sass [#93]
+New scanners:
+
+* Go [#28, thanks to Eric Guo and Nathan Youngman]
+* Lua [#21, #22, thanks to Quintus]
+* Sass [#93]
+* Taskpaper [#39, thanks to shimomura]
+
+More new stuff:
+
+* @.xaml@ file type [#121, thanks to Kozman Bálint]
+* recognize @Guardfile@, @Vagrantfile@, and @Appraisals@ as Ruby files [#121, thanks to Kozman Bálint]
+* new token kind @:id@ for CSS/Sass [#27]
+* new token kind @:done@ for Taskpaper [#39]
+* new token kind @:map@ for Lua, introducing a nice nested-shades trick [#22, thanks to Quintus and Nathan Youngman]
+* new token kind @:unknown@ for Debug scanner
+* new DebugLint encoder that checks for empty tokens and correct nesting
+
+Improvements:
+
+* CSS scanner uses @:id@ and @:tag@ now [#27]
 * Diff scanner: Highlight inline changes in multi-line changes [#99]
 * JavaScript scanner: Highlight multi-line comments in diff correctly
-* Remove double-click toggle handler from HTML table output
-* Fixes to CSS scanner (floats, pseudoclasses)
-* Plugin does not warn about fallback when default is defined
-* Display line numbers in HTML @:table@ mode even for single-line code (remove special case) [#41, thanks to Ariejan de Vroom]
-* Add .xaml file type [#121, thanks to Kozman Bálint]
-* @CodeRay::TokenKinds@ should not be frozen [#130, thanks to Gavin Kistner]
-* Override Bootstrap's pre word-break setting for line numbers [#102, thanks to lightswitch05]
-* Accept keywords as Ruby 1.9 hash keys [#126]
-* New token type @:id@ for CSS/Sass [#27]
-* CSS scanner uses @:id@ and @:tag@ now [#27]
+* JSON scanner: simplify key/value heuristic, using look-ahead instead of a stack
+* HTML scanner displays style tags and attributes now [#145]
+* Ruby scanner: Accept @%i(…)@ and @%I(…)@ symbol lists (Ruby 2.0) [thanks to Nathan Youngman]
+* Ruby scanner: Accept keywords as Ruby hash keys [#126]
+* performance improvements to several scanners and encoders, especially Terminal and HTML
+* added @:keep_state@ functionality to more scanners so they work nicely with diff now [#116]
+* refactoring and cleanup to achieve better "Code Climate" ratings (but I don't really care)
+* updated and cleaned up the documentation,
+* documented list of TokenKinds
+* Alpha style: tweaked colors for @.binary@, @.local-variable@, and @.predefined-type@
+* @rake generate@ supports Git now instead of Subversion
+
+Removed:
+
+* @Tokens#dump@, @Tokens.load@, @Tokens::Undumping@, and @zlib@ dependency
+* double-click toggle handler from HTML table output
+* @rake_helpers@, @sample@ directories and several other ancient garbage
+
+Fixes:
+
+* fixes to CSS scanner (floats, pseudoclasses, nth-child) [#143]
+* fixed empty tokens and unclosed token groups in HTML, CSS, Diff, Goovy, PHP, Raydebug, Ruby, SQL, and YAML scanners [#144]
+* fixed @:docstring@ token type style
+* fixed several infinite Hash caches and dynamic Symbol creation that might have been exploited by an attacker [#148]
+* fixed HTML encoder when output is a StringIO (eg. when using @-HTML@ as a command line parameter)
+* TokenKinds should not be frozen [#130, thanks to Gavin Kistner]
+* display line numbers in HTML @:table@ mode even for single-line code (remove special case) [#41, thanks to Ariejan de Vroom]
+* override Bootstrap's @pre { word-break: break-all }@ styling for line numbers [#102, thanks to lightswitch05]
+* HTML encoder will not warn about unclosed token groups at the end of the stream
+* fixed problem with coderay/version.rb being loaded twice
+
+Internals:
+
+* The Debug scanner maps unknown token kinds to @:unknown@ (to avoid creating Symbols based on possibly unsafe input).
+* The Raydebug scanner highlights unknown token kinds as @:plain@.
+* The Debug encoder refactored; use DebugLint if you want strict checking now..
+* The Debug encoder will not warn about errors in the token stream.
+* Plugin does not warn about fallback when default is defined.
+* PluginHost now works with Strings instead of Symbols internally (to avoid using @#to_sym@).
 
 h2. Changes in 1.0.9
 
@@ -426,6 +485,3 @@ The helper classes were cleaned up; see above for details.
 
 * *CHANGED* @Plugin@ API was simplified and stripped of all unnecessary features.
 * *CHANGED* Moved @GZip@ and @FileType@ libraries into @CodeRay@; cleaned them up.
-
-
-
diff --git a/Gemfile b/Gemfile
index c3239ec..252454f 100644
--- a/Gemfile
+++ b/Gemfile
@@ -8,11 +8,13 @@ gem 'paint', '~> 0.8.4'
 # Add dependencies to develop your gem here.
 # Include everything needed to run rake, tests, features, etc.
 group :development do
-  gem "bundler", ">= 1.0.0"
+  gem "bundler"
   gem "rake"
   gem "RedCloth", RUBY_PLATFORM == 'java' ? ">= 4.2.7" : ">= 4.0.3"
-  gem "term-ansicolor", '~> 1.2.2'
-  gem "shoulda-context", "~> 1.1.2"
+  gem "term-ansicolor"
+  gem 'tins', '~> 1.6.0'
+  gem "shoulda-context"
+  gem "test-unit"
   gem "json" if RUBY_VERSION < '1.9'
   gem "rdoc"
 end
diff --git a/README.markdown b/README.markdown
index f333655..15b3447 100644
--- a/README.markdown
+++ b/README.markdown
@@ -16,7 +16,7 @@ You put your code in, and you get it back colored; Keywords, strings, floats, co
 
 ### Dependencies
 
-CodeRay needs Ruby 1.8.7, 1.9.3 or 2.0. It also runs on JRuby.
+CodeRay needs Ruby 1.8.7, 1.9.3 or 2.0+. It also runs on JRuby.
 
 ## Example Usage
 
@@ -28,4 +28,4 @@ html = CodeRay.scan("puts 'Hello, world!'", :ruby).div(:line_numbers => :table)
 
 ## Documentation
 
-See [http://coderay.rubychan.de/doc/](http://coderay.rubychan.de/doc/).
+See [rubydoc](http://rubydoc.info/gems/coderay).
diff --git a/Rakefile b/Rakefile
index ba6c34e..c9b1e8a 100644
--- a/Rakefile
+++ b/Rakefile
@@ -1,3 +1,5 @@
+require 'bundler/gem_tasks'
+
 $:.unshift File.dirname(__FILE__) unless $:.include? '.'
 
 ROOT = '.'
@@ -32,4 +34,4 @@ else
     rd.rdoc_dir = 'doc'
   end
   
-end
\ No newline at end of file
+end
diff --git a/bench/bench.rb b/bench/bench.rb
index 1889eed..92f9d07 100644
--- a/bench/bench.rb
+++ b/bench/bench.rb
@@ -1,168 +1,46 @@
-# The most ugly test script I've ever written!
-# Shame on me!
-
-require 'pathname'
-require 'profile' if ARGV.include? '-p'
-
-MYDIR = File.dirname(__FILE__)
-LIBDIR = Pathname.new(MYDIR).join('..', 'lib').cleanpath.to_s
-$:.unshift MYDIR, LIBDIR
+require 'benchmark'
+$: << File.expand_path('../../lib', __FILE__)
 require 'coderay'
 
-@size = ARGV.fetch(2, 100).to_i * 1000
-
-lang = ARGV.fetch(0) do
-  puts <<-HELP
-Usage:
-  ruby bench.rb (c|ruby|dump) (null|text|tokens|count|statistic|yaml|html) [size in kB] [stream]
-
-  SIZE defaults to 100 kB (= 100,000 bytes).
-  SIZE = 0 means the whole input.
-  SIZE is ignored when dump is input.
-
--p generates a profile (slow! use with SIZE = 1)
--o shows the output
-stream enabled streaming mode
-
-Sorry for the strange interface. I will improve it in the next release.
-  HELP
+if ARGV.include? '-h'
+  puts DATA.read
   exit
 end
 
-format = ARGV.fetch(1, 'html').downcase
-
-$stream = ARGV.include? 'stream'
-$optimize = ARGV.include? 'opt'
-$style = ARGV.include? 'style'
-
-require 'benchmark'
-require 'fileutils'
+lang = ARGV.fetch(0, 'ruby')
+data = nil
+File.open(File.expand_path("../example.#{lang}", __FILE__), 'rb') { |f| data = f.read }
+raise 'Example file is empty.' if data.empty?
 
-if format == 'comp'
-  format = 'page'
-  begin
-    require 'syntax'
-    require 'syntax/convertors/html.rb'
-  rescue LoadError
-    puts 'Syntax no found!! (Try % gem install syntax)'
-  end
-end
-
-$dump_input = lang == 'dump'
-$dump_output = format == 'dump'
-require 'coderay/helpers/gzip_simple.rb' if $dump_input
+format = ARGV.fetch(1, 'html').downcase
+encoder = CodeRay.encoder(format)
 
-def here fn = nil
-  return MYDIR unless fn
-  File.join here, fn
+size = ARGV.fetch(2, 1000).to_i * 1000
+unless size.zero?
+  data += data until data.size >= size
+  data = data[0, size]
 end
+size = data.size
+puts "encoding %d kB of #{lang} code to #{format}..." % [(size / 1000.0).round]
 
-n = ARGV.find { |a| a[/^N/] }
-N = if n then n[/\d+/].to_i else 1 end
-$filename = ARGV.include?('strange') ? 'strange' : 'example'
-
-Benchmark.bm(20) do |bm|
-N.times do
-
-  data = nil
-  File.open(here("#$filename." + lang), 'rb') { |f| data = f.read }
-  if $dump_input
-    @size = CodeRay::Tokens.load(data).text.size
-  else
-    raise 'Example file is empty.' if data.empty?
-    unless @size.zero?
-      data += data until data.size >= @size
-      data = data[0, @size]
-    end
-    @size = data.size
-  end
-
-  options = {
-    :tab_width => 2,
-    # :line_numbers => :inline,
-    :css => $style ? :style : :class,
-  }
-  $hl = CodeRay.encoder(format, options) unless $dump_output
-  time = bm.report('CodeRay') do
-    if $stream || true
-      if $dump_input
-        raise 'Can\'t stream dump.'
-      elsif $dump_output
-        raise 'Can\'t dump stream.'
-      end
-      $o = $hl.encode(data, lang, options)
-    else
-      if $dump_input
-        tokens = CodeRay::Tokens.load data
-      else
-        tokens = CodeRay.scan(data, lang)
-      end
-      tokens.optimize! if $optimize
-      if $dump_output
-        $o = tokens.optimize.dump
-      else
-        $o = tokens.encode($hl)
-      end
-    end
-  end
-  $file_created = here('test.' +
-    ($dump_output ? 'dump' : $hl.file_extension))
-  File.open($file_created, 'wb') do |f|
-    # f.write $o
-  end
-  Dir.chdir(here) do
-    FileUtils.copy 'test.dump', 'example.dump' if $dump_output
-  end
-
-  time_real = time.real
-
-  puts "\t%7.2f KB/s (%d.%d KB)" % [((@size / 1000.0) / time_real), @size / 1000, @size % 1000]
-  puts $o if ARGV.include? '-o'
-
-end
+n = ARGV.fetch(3, 5).to_s[/\d+/].to_i
+require 'profile' if ARGV.include? '-p'
+n.times do |i|
+  time = Benchmark.realtime { encoder.encode(data, lang) }
+  puts "run %d: %5.2f s, %4.0f kB/s" % [i + 1, time, size / time / 1000.0]
 end
-puts "Files created: #$file_created"
 
-STDIN.gets if ARGV.include? 'wait'
+STDIN.gets if ARGV.include? '-w'
 
 __END__
-.ruby .normal {}
-.ruby .comment { color: #005; font-style: italic; }
-.ruby .keyword { color: #A00; font-weight: bold; }
-.ruby .method { color: #077; }
-.ruby .class { color: #074; }
-.ruby .module { color: #050; }
-.ruby .punct { color: #447; font-weight: bold; }
-.ruby .symbol { color: #099; }
-.ruby .string { color: #944; background: #FFE; }
-.ruby .char { color: #F07; }
-.ruby .ident { color: #004; }
-.ruby .constant { color: #07F; }
-.ruby .regex { color: #B66; background: #FEF; }
-.ruby .number { color: #F99; }
-.ruby .attribute { color: #7BB; }
-.ruby .global { color: #7FB; }
-.ruby .expr { color: #227; }
-.ruby .escape { color: #277; }
+Usage:
+  ruby bench.rb [lang] [format] [size in kB] [number of runs]
 
-.xml .normal {}
-.xml .namespace { color: #B66; font-weight: bold; }
-.xml .tag { color: #F88; }
-.xml .comment { color: #005; font-style: italic; }
-.xml .punct { color: #447; font-weight: bold; }
-.xml .string { color: #944; }
-.xml .number { color: #F99; }
-.xml .attribute { color: #BB7; }
+  - lang defaults to ruby.
+  - format defaults to html.
+  - size defaults to 1000 kB (= 1,000,000 bytes). 0 uses the whole example input.
+  - number of runs defaults to 5.
 
-.yaml .normal {}
-.yaml .document { font-weight: bold; color: #07F; }
-.yaml .type { font-weight: bold; color: #05C; }
-.yaml .key { color: #F88; }
-.yaml .comment { color: #005; font-style: italic; }
-.yaml .punct { color: #447; font-weight: bold; }
-.yaml .string { color: #944; }
-.yaml .number { color: #F99; }
-.yaml .time { color: #F99; }
-.yaml .date { color: #F99; }
-.yaml .ref { color: #944; }
-.yaml .anchor { color: #944; }
+-h prints this help
+-p generates a profile (slow, use with SIZE = 1)
+-w waits after the benchmark (for debugging memory usw)
diff --git a/bin/coderay b/bin/coderay
index 3dea528..2aebfea 100755
--- a/bin/coderay
+++ b/bin/coderay
@@ -35,7 +35,7 @@ defaults:
 
 common:
   coderay file.rb                      # highlight file to terminal
-  coderay file.rb > file.html          # highlight file to HTML page
+  coderay file.rb -page > file.html    # highlight file to HTML page
   coderay file.rb -div > file.html     # highlight file to HTML snippet
 
 configure output:
@@ -155,8 +155,9 @@ when 'highlight', nil
         puts boom.message
       end
       # puts "I don't know this plugin: #{boom.message[/Could not load plugin (.*?): /, 1]}."
-    rescue CodeRay::Scanners::Scanner::ScanError  # FIXME: rescue Errno::EPIPE
-      # this is sometimes raised by pagers; ignore [TODO: wtf?]
+    rescue CodeRay::Scanners::Scanner::ScanError
+      # this is sometimes raised by pagers; ignore
+    # FIXME: rescue Errno::EPIPE
     ensure
       file.close if output_file
     end
diff --git a/lib/coderay.rb b/lib/coderay.rb
index 88c7cc2..f759ed6 100644
--- a/lib/coderay.rb
+++ b/lib/coderay.rb
@@ -127,14 +127,14 @@ module CodeRay
   
   $CODERAY_DEBUG ||= false
   
-  CODERAY_PATH = File.join File.dirname(__FILE__), 'coderay'
+  CODERAY_PATH = File.expand_path('../coderay', __FILE__)
   
   # Assuming the path is a subpath of lib/coderay/
   def self.coderay_path *path
     File.join CODERAY_PATH, *path
   end
   
-  require coderay_path('version')
+  require 'coderay/version'
   
   # helpers
   autoload :FileType,    coderay_path('helpers', 'file_type')
@@ -166,7 +166,6 @@ module CodeRay
     #
     # See also demo/demo_simple.
     def scan code, lang, options = {}, &block
-      # FIXME: return a proxy for direct-stream encoding
       TokensProxy.new code, lang, options, block
     end
     
diff --git a/lib/coderay/encoders/debug.rb b/lib/coderay/encoders/debug.rb
index c03d3fb..f4db330 100644
--- a/lib/coderay/encoders/debug.rb
+++ b/lib/coderay/encoders/debug.rb
@@ -9,7 +9,6 @@ module Encoders
   #
   # You cannot fully restore the tokens information from the
   # output, because consecutive :space tokens are merged.
-  # Use Tokens#dump for caching purposes.
   # 
   # See also: Scanners::Debug
   class Debug < Encoder
@@ -18,38 +17,26 @@ module Encoders
     
     FILE_EXTENSION = 'raydebug'
     
-    def initialize options = {}
-      super
-      @opened = []
-    end
-    
     def text_token text, kind
-      raise 'empty token' if $CODERAY_DEBUG && text.empty?
       if kind == :space
         @out << text
       else
-        # TODO: Escape (
-        text = text.gsub(/[)\\]/, '\\\\\0') if text.index(/[)\\]/)
-        @out << kind.to_s << '(' << text << ')'
+        text = text.gsub('\\', '\\\\\\\\') if text.index('\\')
+        text = text.gsub(')',  '\\\\)')    if text.index(')')
+        @out << "#{kind}(#{text})"
       end
     end
     
     def begin_group kind
-      @opened << kind
-      @out << kind.to_s << '<'
+      @out << "#{kind}<"
     end
     
     def end_group kind
-      if @opened.last != kind
-        puts @out
-        raise "we are inside #{@opened.inspect}, not #{kind}"
-      end
-      @opened.pop
       @out << '>'
     end
     
     def begin_line kind
-      @out << kind.to_s << '['
+      @out << "#{kind}["
     end
     
     def end_line kind
diff --git a/lib/coderay/encoders/debug_lint.rb b/lib/coderay/encoders/debug_lint.rb
new file mode 100644
index 0000000..a4eba2c
--- /dev/null
+++ b/lib/coderay/encoders/debug_lint.rb
@@ -0,0 +1,63 @@
+module CodeRay
+module Encoders
+  
+  load :lint
+  
+  # = Debug Lint Encoder
+  #
+  # Debug encoder with additional checks for:
+  # 
+  # - empty tokens
+  # - incorrect nesting
+  # 
+  # It will raise an InvalidTokenStream exception when any of the above occurs.
+  # 
+  # See also: Encoders::Debug
+  class DebugLint < Debug
+    
+    register_for :debug_lint
+    
+    def text_token text, kind
+      raise Lint::EmptyToken,       'empty token for %p' % [kind] if text.empty?
+      raise Lint::UnknownTokenKind, 'unknown token kind %p (text was %p)' % [kind, text] unless TokenKinds.has_key? kind
+      super
+    end
+    
+    def begin_group kind
+      @opened << kind
+      super
+    end
+    
+    def end_group kind
+      raise Lint::IncorrectTokenGroupNesting, 'We are inside %s, not %p (end_group)' % [@opened.reverse.map(&:inspect).join(' < '), kind] if @opened.last != kind
+      @opened.pop
+      super
+    end
+    
+    def begin_line kind
+      @opened << kind
+      super
+    end
+    
+    def end_line kind
+      raise Lint::IncorrectTokenGroupNesting, 'We are inside %s, not %p (end_line)' % [@opened.reverse.map(&:inspect).join(' < '), kind] if @opened.last != kind
+      @opened.pop
+      super
+    end
+    
+    protected
+    
+    def setup options
+      super
+      @opened = []
+    end
+    
+    def finish options
+      raise 'Some tokens still open at end of token stream: %p' % [@opened] unless @opened.empty?
+      super
+    end
+    
+  end
+  
+end
+end
diff --git a/lib/coderay/encoders/html.rb b/lib/coderay/encoders/html.rb
index b897f5e..942b9c8 100644
--- a/lib/coderay/encoders/html.rb
+++ b/lib/coderay/encoders/html.rb
@@ -25,7 +25,8 @@ module Encoders
   # == Options
   #
   # === :tab_width
-  # Convert \t characters to +n+ spaces (a number.)
+  # Convert \t characters to +n+ spaces (a number or false.)
+  # false will keep tab characters untouched.
   # 
   # Default: 8
   #
@@ -180,7 +181,7 @@ module Encoders
       
       @break_lines = (options[:break_lines] == true)
       
-      @HTML_ESCAPE = HTML_ESCAPE.merge("\t" => ' ' * options[:tab_width])
+      @HTML_ESCAPE = HTML_ESCAPE.merge("\t" => options[:tab_width] ? ' ' * options[:tab_width] : "\t")
       
       @opened = []
       @last_opened = nil
@@ -193,18 +194,19 @@ module Encoders
     
     def finish options
       unless @opened.empty?
-        warn '%d tokens still open: %p' % [@opened.size, @opened] if $CODERAY_DEBUG
         @out << '</span>' while @opened.pop
         @last_opened = nil
       end
       
-      @out.extend Output
-      @out.css = @css
-      if options[:line_numbers]
-        Numbering.number! @out, options[:line_numbers], options
+      if @out.respond_to? :to_str
+        @out.extend Output
+        @out.css = @css
+        if options[:line_numbers]
+          Numbering.number! @out, options[:line_numbers], options
+        end
+        @out.wrap! options[:wrap]
+        @out.apply_title! options[:title]
       end
-      @out.wrap! options[:wrap]
-      @out.apply_title! options[:title]
       
       if defined?(@real_out) && @real_out
         @real_out << @out
@@ -286,7 +288,7 @@ module Encoders
     
     def make_span_for_kinds method, hint
       Hash.new do |h, kinds|
-        h[kinds.is_a?(Symbol) ? kinds : kinds.dup] = begin
+        begin
           css_class = css_class_for_kinds(kinds)
           title     = HTML.token_path_to_hint hint, kinds if hint
           
@@ -298,6 +300,9 @@ module Encoders
               "<span#{title}#{" class=\"#{css_class}\"" if css_class}>"
             end
           end
+        end.tap do |span|
+          h.clear if h.size >= 100
+          h[kinds] = span
         end
       end
     end
@@ -310,8 +315,8 @@ module Encoders
     
     def break_lines text, style
       reopen = ''
-      @opened.each_with_index do |k, index|
-        reopen << (@span_for_kinds[index > 0 ? [k, *@opened[0...index]] : k] || '<span>')
+      @opened.each_with_index do |kind, index|
+        reopen << (@span_for_kinds[index > 0 ? [kind, *@opened[0...index]] : kind] || '<span>')
       end
       text.gsub("\n", "#{'</span>' * @opened.size}#{'</span>' if style}\n#{reopen}#{style}")
     end
diff --git a/lib/coderay/encoders/html/numbering.rb b/lib/coderay/encoders/html/numbering.rb
index 332145b..a1b9c04 100644
--- a/lib/coderay/encoders/html/numbering.rb
+++ b/lib/coderay/encoders/html/numbering.rb
@@ -26,7 +26,7 @@ module Encoders
               "<a href=\"##{anchor}\" name=\"#{anchor}\">#{line}</a>"
             end
           else
-            proc { |line| line.to_s }  # :to_s.to_proc in Ruby 1.8.7+
+            :to_s.to_proc
           end
         
         bold_every = options[:bold_every]
@@ -75,7 +75,7 @@ module Encoders
           line_number = start
           output.gsub!(/^.*$\n?/) do |line|
             line_number_text = bolding.call line_number
-            indent = ' ' * (max_width - line_number.to_s.size)  # TODO: Optimize (10^x)
+            indent = ' ' * (max_width - line_number.to_s.size)
             line_number += 1
             "<span class=\"line-numbers\">#{indent}#{line_number_text}</span>#{line}"
           end
diff --git a/lib/coderay/encoders/lint.rb b/lib/coderay/encoders/lint.rb
new file mode 100644
index 0000000..88c8bd1
--- /dev/null
+++ b/lib/coderay/encoders/lint.rb
@@ -0,0 +1,59 @@
+module CodeRay
+module Encoders
+  
+  # = Lint Encoder
+  #
+  # Checks for:
+  # 
+  # - empty tokens
+  # - incorrect nesting
+  # 
+  # It will raise an InvalidTokenStream exception when any of the above occurs.
+  # 
+  # See also: Encoders::DebugLint
+  class Lint < Debug
+    
+    register_for :lint
+    
+    InvalidTokenStream         = Class.new StandardError
+    EmptyToken                 = Class.new InvalidTokenStream
+    UnknownTokenKind           = Class.new InvalidTokenStream
+    IncorrectTokenGroupNesting = Class.new InvalidTokenStream
+    
+    def text_token text, kind
+      raise EmptyToken,       'empty token for %p' % [kind] if text.empty?
+      raise UnknownTokenKind, 'unknown token kind %p (text was %p)' % [kind, text] unless TokenKinds.has_key? kind
+    end
+    
+    def begin_group kind
+      @opened << kind
+    end
+    
+    def end_group kind
+      raise IncorrectTokenGroupNesting, 'We are inside %s, not %p (end_group)' % [@opened.reverse.map(&:inspect).join(' < '), kind] if @opened.last != kind
+      @opened.pop
+    end
+    
+    def begin_line kind
+      @opened << kind
+    end
+    
+    def end_line kind
+      raise IncorrectTokenGroupNesting, 'We are inside %s, not %p (end_line)' % [@opened.reverse.map(&:inspect).join(' < '), kind] if @opened.last != kind
+      @opened.pop
+    end
+    
+    protected
+    
+    def setup options
+      @opened = []
+    end
+    
+    def finish options
+      raise 'Some tokens still open at end of token stream: %p' % [@opened] unless @opened.empty?
+    end
+    
+  end
+  
+end
+end
diff --git a/lib/coderay/encoders/statistic.rb b/lib/coderay/encoders/statistic.rb
index 2315d9e..b2f8b83 100644
--- a/lib/coderay/encoders/statistic.rb
+++ b/lib/coderay/encoders/statistic.rb
@@ -67,7 +67,6 @@ Token Types (%d):
       @type_stats['TOTAL'].count += 1
     end
     
-    # TODO Hierarchy handling
     def begin_group kind
       block_token ':begin_group', kind
     end
diff --git a/lib/coderay/encoders/terminal.rb b/lib/coderay/encoders/terminal.rb
index 500e5d8..c7ae014 100644
--- a/lib/coderay/encoders/terminal.rb
+++ b/lib/coderay/encoders/terminal.rb
@@ -19,105 +19,135 @@ module CodeRay
       register_for :terminal
       
       TOKEN_COLORS = {
-        :annotation => "\e[35m",
-        :attribute_name => "\e[33m",
+        :debug => "\e[1;37;44m",
+        
+        :annotation => "\e[34m",
+        :attribute_name => "\e[35m",
         :attribute_value => "\e[31m",
-        :binary => "\e[1;35m",
+        :binary => {
+          :self => "\e[31m",
+          :char => "\e[1;31m",
+          :delimiter => "\e[1;31m",
+        },
         :char => {
-          :self => "\e[36m", :delimiter => "\e[1;34m"
+          :self => "\e[35m",
+          :delimiter => "\e[1;35m"
         },
-        :class => "\e[1;35m",
+        :class => "\e[1;35;4m",
         :class_variable => "\e[36m",
         :color => "\e[32m",
-        :comment => "\e[37m",
-        :complex => "\e[1;34m",
-        :constant => "\e[1;34m\e[4m",
-        :decoration => "\e[35m",
-        :definition => "\e[1;32m",
-        :directive => "\e[32m\e[4m",
-        :doc => "\e[46m",
-        :doctype => "\e[1;30m",
-        :doc_string => "\e[31m\e[4m",
-        :entity => "\e[33m",
-        :error => "\e[1;33m\e[41m",
+        :comment => {
+          :self => "\e[1;30m",
+          :char => "\e[37m",
+          :delimiter => "\e[37m",
+        },
+        :constant => "\e[1;34;4m",
+        :decorator => "\e[35m",
+        :definition => "\e[1;33m",
+        :directive => "\e[33m",
+        :docstring => "\e[31m",
+        :doctype => "\e[1;34m",
+        :done => "\e[1;30;2m",
+        :entity => "\e[31m",
+        :error => "\e[1;37;41m",
         :exception => "\e[1;31m",
         :float => "\e[1;35m",
         :function => "\e[1;34m",
-        :global_variable => "\e[42m",
+        :global_variable => "\e[1;32m",
         :hex => "\e[1;36m",
-        :include => "\e[33m",
+        :id => "\e[1;34m",
+        :include => "\e[31m",
         :integer => "\e[1;34m",
-        :key => "\e[35m",
-        :label => "\e[1;15m",
+        :imaginary => "\e[1;34m",
+        :important => "\e[1;31m",
+        :key => {
+          :self => "\e[35m",
+          :char => "\e[1;35m",
+          :delimiter => "\e[1;35m",
+        },
+        :keyword => "\e[32m",
+        :label => "\e[1;33m",
         :local_variable => "\e[33m",
-        :octal => "\e[1;35m",
-        :operator_name => "\e[1;29m",
+        :namespace => "\e[1;35m",
+        :octal => "\e[1;34m",
+        :predefined => "\e[36m",
         :predefined_constant => "\e[1;36m",
-        :predefined_type => "\e[1;30m",
-        :predefined => "\e[4m\e[1;34m",
-        :preprocessor => "\e[36m",
+        :predefined_type => "\e[1;32m",
+        :preprocessor => "\e[1;36m",
         :pseudo_class => "\e[1;34m",
         :regexp => {
-          :self => "\e[31m",
-          :content => "\e[31m",
-          :delimiter => "\e[1;29m",
+          :self => "\e[35m",
+          :delimiter => "\e[1;35m",
           :modifier => "\e[35m",
+          :char => "\e[1;35m",
         },
-        :reserved => "\e[1;31m",
+        :reserved => "\e[32m",
         :shell => {
-          :self => "\e[42m",
-          :content => "\e[1;29m",
-          :delimiter => "\e[37m",
+          :self => "\e[33m",
+          :char => "\e[1;33m",
+          :delimiter => "\e[1;33m",
+          :escape => "\e[1;33m",
         },
         :string => {
-          :self => "\e[32m",
-          :modifier => "\e[1;32m",
-          :escape => "\e[1;36m",
-          :delimiter => "\e[1;32m",
-          :char => "\e[1;36m",
+          :self => "\e[31m",
+          :modifier => "\e[1;31m",
+          :char => "\e[1;35m",
+          :delimiter => "\e[1;31m",
+          :escape => "\e[1;31m",
+        },
+        :symbol => {
+          :self => "\e[33m",
+          :delimiter => "\e[1;33m",
         },
-        :symbol => "\e[1;32m",
-        :tag => "\e[1;34m",
+        :tag => "\e[32m",
         :type => "\e[1;34m",
         :value => "\e[36m",
-        :variable => "\e[1;34m",
+        :variable => "\e[34m",
         
-        :insert => "\e[42m",
-        :delete => "\e[41m",
-        :change => "\e[44m",
-        :head => "\e[45m"
+        :insert => {
+          :self => "\e[42m",
+          :insert => "\e[1;32;42m",
+          :eyecatcher => "\e[102m",
+        },
+        :delete => {
+          :self => "\e[41m",
+          :delete => "\e[1;31;41m",
+          :eyecatcher => "\e[101m",
+        },
+        :change => {
+          :self => "\e[44m",
+          :change => "\e[37;44m",
+        },
+        :head => {
+          :self => "\e[45m",
+          :filename => "\e[37;45m"
+        },
       }
+      
       TOKEN_COLORS[:keyword] = TOKEN_COLORS[:reserved]
       TOKEN_COLORS[:method] = TOKEN_COLORS[:function]
-      TOKEN_COLORS[:imaginary] = TOKEN_COLORS[:complex]
-      TOKEN_COLORS[:begin_group] = TOKEN_COLORS[:end_group] =
-        TOKEN_COLORS[:escape] = TOKEN_COLORS[:delimiter]
+      TOKEN_COLORS[:escape] = TOKEN_COLORS[:delimiter]
       
     protected
       
       def setup(options)
         super
         @opened = []
-        @subcolors = nil
+        @color_scopes = [TOKEN_COLORS]
       end
       
     public
       
       def text_token text, kind
-        if color = (@subcolors || TOKEN_COLORS)[kind]
-          if Hash === color
-            if color[:self]
-              color = color[:self]
-            else
-              @out << text
-              return
-            end
-          end
+        if color = @color_scopes.last[kind]
+          color = color[:self] if color.is_a? Hash
           
           @out << color
-          @out << text.gsub("\n", "\e[0m\n" + color)
+          @out << (text.index("\n") ? text.gsub("\n", "\e[0m\n" + color) : text)
           @out << "\e[0m"
-          @out << @subcolors[:self] if @subcolors
+          if outer_color = @color_scopes.last[:self]
+            @out << outer_color
+          end
         else
           @out << text
         end
@@ -130,40 +160,33 @@ module CodeRay
       alias begin_line begin_group
       
       def end_group kind
-        if @opened.empty?
-          # nothing to close
-        else
-          @opened.pop
+        if @opened.pop
+          @color_scopes.pop
           @out << "\e[0m"
-          @out << open_token(@opened.last)
+          if outer_color = @color_scopes.last[:self]
+            @out << outer_color
+          end
         end
       end
       
       def end_line kind
-        if @opened.empty?
-          # nothing to close
-        else
-          @opened.pop
-          # whole lines to be highlighted,
-          # eg. added/modified/deleted lines in a diff
-          @out << (@line_filler ||= "\t" * 100 + "\e[0m")
-          @out << open_token(@opened.last)
-        end
+        @out << (@line_filler ||= "\t" * 100)
+        end_group kind
       end
       
     private
       
       def open_token kind
-        if color = TOKEN_COLORS[kind]
-          if Hash === color
-            @subcolors = color
+        if color = @color_scopes.last[kind]
+          if color.is_a? Hash
+            @color_scopes << color
             color[:self]
           else
-            @subcolors = {}
+            @color_scopes << @color_scopes.last
             color
           end
         else
-          @subcolors = nil
+          @color_scopes << @color_scopes.last
           ''
         end
       end
diff --git a/lib/coderay/helpers/file_type.rb b/lib/coderay/helpers/file_type.rb
index a5d83ff..7de34d5 100644
--- a/lib/coderay/helpers/file_type.rb
+++ b/lib/coderay/helpers/file_type.rb
@@ -38,7 +38,7 @@ module CodeRay
           (TypeFromExt[ext2.downcase] if ext2) ||
           TypeFromName[name] ||
           TypeFromName[name.downcase]
-        type ||= shebang(filename) if read_shebang
+        type ||= type_from_shebang(filename) if read_shebang
         
         type
       end
@@ -63,7 +63,7 @@ module CodeRay
       
     protected
       
-      def shebang filename
+      def type_from_shebang filename
         return unless File.exist? filename
         File.open filename, 'r' do |f|
           if first_line = f.gets
@@ -77,54 +77,58 @@ module CodeRay
     end
     
     TypeFromExt = {
-      'c'        => :c,
-      'cfc'      => :xml,
-      'cfm'      => :xml,
-      'clj'      => :clojure,
-      'css'      => :css,
-      'diff'     => :diff,
-      'dpr'      => :delphi,
-      'erb'      => :erb,
-      'gemspec'  => :ruby,
-      'groovy'   => :groovy,
-      'gvy'      => :groovy,
-      'h'        => :c,
-      'haml'     => :haml,
-      'htm'      => :html,
-      'html'     => :html,
-      'html.erb' => :erb,
-      'java'     => :java,
-      'js'       => :java_script,
-      'json'     => :json,
-      'mab'      => :ruby,
-      'pas'      => :delphi,
-      'patch'    => :diff,
-      'phtml'    => :php,
-      'php'      => :php,
-      'php3'     => :php,
-      'php4'     => :php,
-      'php5'     => :php,
-      'prawn'    => :ruby,
-      'py'       => :python,
-      'py3'      => :python,
-      'pyw'      => :python,
-      'rake'     => :ruby,
-      'raydebug' => :raydebug,
-      'rb'       => :ruby,
-      'rbw'      => :ruby,
-      'rhtml'    => :erb,
-      'rjs'      => :ruby,
-      'rpdf'     => :ruby,
-      'ru'       => :ruby,
-      'rxml'     => :ruby,
-      'sass'     => :sass,
-      'sql'      => :sql,
-      'tmproj'   => :xml,
-      'xaml'     => :xml,
-      'xhtml'    => :html,
-      'xml'      => :xml,
-      'yaml'     => :yaml,
-      'yml'      => :yaml,
+      'c'         => :c,
+      'cfc'       => :xml,
+      'cfm'       => :xml,
+      'clj'       => :clojure,
+      'css'       => :css,
+      'diff'      => :diff,
+      'dpr'       => :delphi,
+      'erb'       => :erb,
+      'gemspec'   => :ruby,
+      'go'        => :go, 
+      'groovy'    => :groovy,
+      'gvy'       => :groovy,
+      'h'         => :c,
+      'haml'      => :haml,
+      'htm'       => :html,
+      'html'      => :html,
+      'html.erb'  => :erb,
+      'java'      => :java,
+      'js'        => :java_script,
+      'json'      => :json,
+      'lua'       => :lua,
+      'mab'       => :ruby,
+      'pas'       => :delphi,
+      'patch'     => :diff,
+      'phtml'     => :php,
+      'php'       => :php,
+      'php3'      => :php,
+      'php4'      => :php,
+      'php5'      => :php,
+      'prawn'     => :ruby,
+      'py'        => :python,
+      'py3'       => :python,
+      'pyw'       => :python,
+      'rake'      => :ruby,
+      'raydebug'  => :raydebug,
+      'rb'        => :ruby,
+      'rbw'       => :ruby,
+      'rhtml'     => :erb,
+      'rjs'       => :ruby,
+      'rpdf'      => :ruby,
+      'ru'        => :ruby,  # config.ru
+      'rxml'      => :ruby,
+      'sass'      => :sass,
+      'sql'       => :sql,
+      'taskpaper' => :taskpaper,
+      'template'  => :json,  # AWS CloudFormation template
+      'tmproj'    => :xml,
+      'xaml'      => :xml,
+      'xhtml'     => :html,
+      'xml'       => :xml,
+      'yaml'      => :yaml,
+      'yml'       => :yaml,
     }
     for cpp_alias in %w[cc cpp cp cxx c++ C hh hpp h++ cu]
       TypeFromExt[cpp_alias] = :cpp
@@ -137,6 +141,9 @@ module CodeRay
       'Rakefile' => :ruby,
       'Rantfile' => :ruby,
       'Gemfile'  => :ruby,
+      'Guardfile' => :ruby,
+      'Vagrantfile' => :ruby,
+      'Appraisals' => :ruby
     }
     
   end
diff --git a/lib/coderay/helpers/gzip.rb b/lib/coderay/helpers/gzip.rb
deleted file mode 100644
index 245014a..0000000
--- a/lib/coderay/helpers/gzip.rb
+++ /dev/null
@@ -1,41 +0,0 @@
-module CodeRay
-  
-  # A simplified interface to the gzip library +zlib+ (from the Ruby Standard Library.)
-  module GZip
-    
-    require 'zlib'
-    
-    # The default zipping level. 7 zips good and fast.
-    DEFAULT_GZIP_LEVEL = 7
-    
-    # Unzips the given string +s+.
-    #
-    # Example:
-    #   require 'gzip_simple'
-    #   print GZip.gunzip(File.read('adresses.gz'))
-    def GZip.gunzip s
-      Zlib::Inflate.inflate s
-    end
-    
-    # Zips the given string +s+.
-    #
-    # Example:
-    #   require 'gzip_simple'
-    #   File.open('adresses.gz', 'w') do |file
-    #     file.write GZip.gzip('Mum: 0123 456 789', 9)
-    #   end
-    #
-    # If you provide a +level+, you can control how strong
-    # the string is compressed:
-    # - 0: no compression, only convert to gzip format
-    # - 1: compress fast
-    # - 7: compress more, but still fast (default)
-    # - 8: compress more, slower
-    # - 9: compress best, very slow
-    def GZip.gzip s, level = DEFAULT_GZIP_LEVEL
-      Zlib::Deflate.new(level).deflate s, Zlib::FINISH
-    end
-    
-  end
-  
-end
diff --git a/lib/coderay/helpers/plugin.rb b/lib/coderay/helpers/plugin.rb
index d14c5a9..9a724ff 100644
--- a/lib/coderay/helpers/plugin.rb
+++ b/lib/coderay/helpers/plugin.rb
@@ -30,7 +30,7 @@ module CodeRay
     # * a file could not be found
     # * the requested Plugin is not registered
     PluginNotFound = Class.new LoadError
-    HostNotFound = Class.new LoadError
+    HostNotFound   = Class.new LoadError
     
     PLUGIN_HOSTS = []
     PLUGIN_HOSTS_BY_ID = {}  # dummy hash
@@ -49,8 +49,8 @@ module CodeRay
     def [] id, *args, &blk
       plugin = validate_id(id)
       begin
-        plugin = plugin_hash.[] plugin, *args, &blk
-      end while plugin.is_a? Symbol
+        plugin = plugin_hash.[](plugin, *args, &blk)
+      end while plugin.is_a? String
       plugin
     end
     
@@ -95,7 +95,7 @@ module CodeRay
     def map hash
       for from, to in hash
         from = validate_id from
-        to = validate_id to
+        to   = validate_id to
         plugin_hash[from] = to unless plugin_hash.has_key? from
       end
     end
@@ -197,22 +197,22 @@ module CodeRay
       File.join plugin_path, "#{plugin_id}.rb"
     end
     
-    # Converts +id+ to a Symbol if it is a String,
-    # or returns +id+ if it already is a Symbol.
+    # Converts +id+ to a valid plugin ID String, or returns +nil+.
     #
     # Raises +ArgumentError+ for all other objects, or if the
     # given String includes non-alphanumeric characters (\W).
     def validate_id id
-      if id.is_a? Symbol or id.nil?
-        id
-      elsif id.is_a? String
+      case id
+      when Symbol
+        id.to_s
+      when String
         if id[/\w+/] == id
-          id.downcase.to_sym
+          id.downcase
         else
           raise ArgumentError, "Invalid id given: #{id}"
         end
       else
-        raise ArgumentError, "String or Symbol expected, but #{id.class} given."
+        raise ArgumentError, "Symbol or String expected, but #{id.class} given."
       end
     end
     
diff --git a/lib/coderay/scanners/css.rb b/lib/coderay/scanners/css.rb
index 732f9c5..55d5239 100644
--- a/lib/coderay/scanners/css.rb
+++ b/lib/coderay/scanners/css.rb
@@ -25,7 +25,7 @@ module Scanners
       
       HexColor = /#(?:#{Hex}{6}|#{Hex}{3})/
       
-      Num = /-?(?:[0-9]*\.[0-9]+|[0-9]+)/
+      Num = /-?(?:[0-9]*\.[0-9]+|[0-9]+)n?/
       Name = /#{NMChar}+/
       Ident = /-?#{NMStart}#{NMChar}*/
       AtKeyword = /@#{Ident}/
@@ -53,7 +53,7 @@ module Scanners
     end
     
     def scan_tokens encoder, options
-      states = Array(options[:state] || @state)
+      states = Array(options[:state] || @state).dup
       value_expected = @value_expected
       
       until eos?
@@ -145,10 +145,10 @@ module Scanners
           start = match[/^\w+\(/]
           encoder.text_token start, :delimiter
           if match[-1] == ?)
-            encoder.text_token match[start.size..-2], :content
+            encoder.text_token match[start.size..-2], :content if match.size > start.size + 1
             encoder.text_token ')', :delimiter
           else
-            encoder.text_token match[start.size..-1], :content
+            encoder.text_token match[start.size..-1], :content if match.size > start.size
           end
           encoder.end_group :function
           
diff --git a/lib/coderay/scanners/debug.rb b/lib/coderay/scanners/debug.rb
index 566bfa7..83ede9a 100644
--- a/lib/coderay/scanners/debug.rb
+++ b/lib/coderay/scanners/debug.rb
@@ -1,9 +1,11 @@
+require 'set'
+
 module CodeRay
 module Scanners
   
   # = Debug Scanner
   # 
-  # Interprets the output of the Encoders::Debug encoder.
+  # Interprets the output of the Encoders::Debug encoder (basically the inverse function).
   class Debug < Scanner
     
     register_for :debug
@@ -11,6 +13,11 @@ module Scanners
     
   protected
     
+    def setup
+      super
+      @known_token_kinds = TokenKinds.keys.map(&:to_s).to_set
+    end
+    
     def scan_tokens encoder, options
       
       opened_tokens = []
@@ -21,16 +28,19 @@ module Scanners
           encoder.text_token match, :space
           
         elsif match = scan(/ (\w+) \( ( [^\)\\]* ( \\. [^\)\\]* )* ) \)? /x)
-          kind = self[1].to_sym
-          match = self[2].gsub(/\\(.)/m, '\1')
-          unless TokenKinds.has_key? kind
-            kind = :error
-            match = matched
+          if @known_token_kinds.include? self[1]
+            encoder.text_token self[2].gsub(/\\(.)/m, '\1'), self[1].to_sym
+          else
+            encoder.text_token matched, :unknown
           end
-          encoder.text_token match, kind
           
         elsif match = scan(/ (\w+) ([<\[]) /x)
-          kind = self[1].to_sym
+          if @known_token_kinds.include? self[1]
+            kind = self[1].to_sym
+          else
+            kind = :unknown
+          end
+          
           opened_tokens << kind
           case self[2]
           when '<'
diff --git a/lib/coderay/scanners/diff.rb b/lib/coderay/scanners/diff.rb
index af0f755..74a6c27 100644
--- a/lib/coderay/scanners/diff.rb
+++ b/lib/coderay/scanners/diff.rb
@@ -69,7 +69,7 @@ module Scanners
             state = :added
           elsif match = scan(/\\ .*/)
             encoder.text_token match, :comment
-          elsif match = scan(/@@(?>[^@\n]*)@@/)
+          elsif match = scan(/@@(?>[^@\n]+)@@/)
             content_scanner.state = :initial unless match?(/\n\+/)
             content_scanner_entry_state = nil
             if check(/\n|$/)
@@ -100,7 +100,7 @@ module Scanners
             next
           elsif match = scan(/-/)
             deleted_lines_count += 1
-            if options[:inline_diff] && deleted_lines_count == 1 && (changed_lines_count = 1 + check(/.*(?:\n\-.*)*/).count("\n")) && match?(/(?>.*(?:\n\-.*){#{changed_lines_count - 1}}(?:\n\+.*){#{changed_lines_count}})$(?!\n\+)/)
+            if options[:inline_diff] && deleted_lines_count == 1 && (changed_lines_count = 1 + check(/.*(?:\n\-.*)*/).count("\n")) && changed_lines_count <= 100_000 && match?(/(?>.*(?:\n\-.*){#{changed_lines_count - 1}}(?:\n\+.*){#{changed_lines_count}})$(?!\n\+)/)
               deleted_lines  = Array.new(changed_lines_count) { |i| skip(/\n\-/) if i > 0; scan(/.*/) }
               inserted_lines = Array.new(changed_lines_count) { |i| skip(/\n\+/)         ; scan(/.*/) }
               
diff --git a/lib/coderay/scanners/go.rb b/lib/coderay/scanners/go.rb
new file mode 100644
index 0000000..99fdd63
--- /dev/null
+++ b/lib/coderay/scanners/go.rb
@@ -0,0 +1,208 @@
+module CodeRay
+module Scanners
+  
+  class Go < Scanner
+    
+    register_for :go
+    file_extension 'go'
+    
+    # http://golang.org/ref/spec#Keywords
+    KEYWORDS = [
+      'break', 'default', 'func', 'interface', 'select',
+      'case', 'defer', 'go', 'map', 'struct',
+      'chan', 'else', 'goto', 'package', 'switch',
+      'const', 'fallthrough', 'if', 'range', 'type',
+      'continue', 'for', 'import', 'return', 'var',
+    ]  # :nodoc:
+    
+    # http://golang.org/ref/spec#Types
+    PREDEFINED_TYPES = [
+      'bool',
+      'uint8', 'uint16', 'uint32', 'uint64',
+      'int8', 'int16', 'int32', 'int64',
+      'float32', 'float64',
+      'complex64', 'complex128',
+      'byte', 'rune', 'string', 'error',
+      'uint', 'int', 'uintptr',
+    ]  # :nodoc:
+    
+    PREDEFINED_CONSTANTS = [
+      'nil', 'iota',
+      'true', 'false',
+    ]  # :nodoc:
+    
+    PREDEFINED_FUNCTIONS = %w[
+      append cap close complex copy delete imag len
+      make new panic print println real recover
+    ] # :nodoc:
+    
+    IDENT_KIND = WordList.new(:ident).
+      add(KEYWORDS, :keyword).
+      add(PREDEFINED_TYPES, :predefined_type).
+      add(PREDEFINED_CONSTANTS, :predefined_constant).
+      add(PREDEFINED_FUNCTIONS, :predefined)  # :nodoc:
+    
+    ESCAPE = / [rbfntv\n\\'"] | x[a-fA-F0-9]{1,2} | [0-7]{1,3} /x  # :nodoc:
+    UNICODE_ESCAPE = / u[a-fA-F0-9]{4} | U[a-fA-F0-9]{8} /x  # :nodoc:
+    
+    protected
+    
+    def scan_tokens encoder, options
+      
+      state = :initial
+      label_expected = true
+      case_expected = false
+      label_expected_before_preproc_line = nil
+      in_preproc_line = false
+      
+      until eos?
+        
+        case state
+        
+        when :initial
+          
+          if match = scan(/ \s+ | \\\n /x)
+            if in_preproc_line && match != "\\\n" && match.index(?\n)
+              in_preproc_line = false
+              case_expected = false
+              label_expected = label_expected_before_preproc_line
+            end
+            encoder.text_token match, :space
+          
+          elsif match = scan(%r! // [^\n\\]* (?: \\. [^\n\\]* )* | /\* (?: .*? \*/ | .* ) !mx)
+            encoder.text_token match, :comment
+          
+          elsif match = scan(/ <?- (?![\d.]) | [+*=<>?:;,!&^|()\[\]{}~%]+ | \/=? | \.(?!\d) /x)
+            if case_expected
+              label_expected = true if match == ':'
+              case_expected = false
+            end
+            encoder.text_token match, :operator
+          
+          elsif match = scan(/ [A-Za-z_][A-Za-z_0-9]* /x)
+            kind = IDENT_KIND[match]
+            if kind == :ident && label_expected && !in_preproc_line && scan(/:(?!:)/)
+              kind = :label
+              label_expected = false
+              match << matched
+            else
+              label_expected = false
+              if kind == :keyword
+                case match
+                when 'case', 'default'
+                  case_expected = true
+                end
+              end
+            end
+            encoder.text_token match, kind
+          
+          elsif match = scan(/L?"/)
+            encoder.begin_group :string
+            if match[0] == ?L
+              encoder.text_token 'L', :modifier
+              match = '"'
+            end
+            encoder.text_token match, :delimiter
+            state = :string
+          
+          elsif match = scan(/ ` ([^`]+)? (`)? /x)
+            encoder.begin_group :shell
+            encoder.text_token '`', :delimiter
+            encoder.text_token self[1], :content if self[1]
+            encoder.text_token self[2], :delimiter if self[2]
+            encoder.end_group :shell
+          
+          elsif match = scan(/ \# \s* if \s* 0 /x)
+            match << scan_until(/ ^\# (?:elif|else|endif) .*? $ | \z /xm) unless eos?
+            encoder.text_token match, :comment
+          
+          elsif match = scan(/#[ \t]*(\w*)/)
+            encoder.text_token match, :preprocessor
+            in_preproc_line = true
+            label_expected_before_preproc_line = label_expected
+            state = :include_expected if self[1] == 'include'
+          
+          elsif match = scan(/ L?' (?: [^\'\n\\] | \\ (?: #{ESCAPE} | #{UNICODE_ESCAPE} ) )? '? /ox)
+            label_expected = false
+            encoder.text_token match, :char
+          
+          elsif match = scan(/\$/)
+            encoder.text_token match, :ident
+          
+          elsif match = scan(/-?\d*(\.\d*)?([eE][+-]?\d+)?i/)
+            label_expected = false
+            encoder.text_token match, :imaginary
+          
+          elsif match = scan(/-?0[xX][0-9A-Fa-f]+/)
+            label_expected = false
+            encoder.text_token match, :hex
+          
+          elsif match = scan(/-?(?:0[0-7]+)(?![89.eEfF])/)
+            label_expected = false
+            encoder.text_token match, :octal
+          
+          elsif match = scan(/-?(?:\d*\.\d+|\d+\.)(?:[eE][+-]?\d+)?|\d+[eE][+-]?\d+/)
+            label_expected = false
+            encoder.text_token match, :float
+          
+          elsif match = scan(/-?(?:\d+)(?![.eEfF])L?L?/)
+            label_expected = false
+            encoder.text_token match, :integer
+          
+          else
+            encoder.text_token getch, :error
+          
+          end
+        
+        when :string
+          if match = scan(/[^\\\n"]+/)
+            encoder.text_token match, :content
+          elsif match = scan(/"/)
+            encoder.text_token match, :delimiter
+            encoder.end_group :string
+            state = :initial
+            label_expected = false
+          elsif match = scan(/ \\ (?: #{ESCAPE} | #{UNICODE_ESCAPE} ) /mox)
+            encoder.text_token match, :char
+          elsif match = scan(/ \\ /x)
+            encoder.text_token match, :error
+          elsif match = scan(/$/)
+            encoder.end_group :string
+            state = :initial
+            label_expected = false
+          else
+            raise_inspect "else case \" reached; %p not handled." % peek(1), encoder
+          end
+        
+        when :include_expected
+          if match = scan(/<[^>\n]+>?|"[^"\n\\]*(?:\\.[^"\n\\]*)*"?/)
+            encoder.text_token match, :include
+            state = :initial
+          
+          elsif match = scan(/\s+/)
+            encoder.text_token match, :space
+            state = :initial if match.index ?\n
+          
+          else
+            state = :initial
+          
+          end
+        
+        else
+          raise_inspect 'Unknown state', encoder
+        
+        end
+        
+      end
+      
+      if state == :string
+        encoder.end_group :string
+      end
+      
+      encoder
+    end
+    
+  end
+  
+end
+end
diff --git a/lib/coderay/scanners/groovy.rb b/lib/coderay/scanners/groovy.rb
index cf55daf..c64454f 100644
--- a/lib/coderay/scanners/groovy.rb
+++ b/lib/coderay/scanners/groovy.rb
@@ -36,9 +36,12 @@ module Scanners
     
   protected
     
+    def setup
+      @state = :initial
+    end
+    
     def scan_tokens encoder, options
-      
-      state = :initial
+      state = options[:state] || @state
       inline_block_stack = []
       inline_block_paren_depth = nil
       string_delimiter = nil
@@ -223,7 +226,7 @@ module Scanners
             encoder.text_token match, :content  # TODO: Shouldn't this be :error?
             
           elsif match = scan(/ \\ | \n /x)
-            encoder.end_group state
+            encoder.end_group state == :regexp ? :regexp : :string
             encoder.text_token match, :error
             after_def = value_expected = false
             state = :initial
@@ -243,7 +246,17 @@ module Scanners
       end
       
       if [:multiline_string, :string, :regexp].include? state
-        encoder.end_group state
+        encoder.end_group state == :regexp ? :regexp : :string
+      end
+      
+      if options[:keep_state]
+        @state = state
+      end
+      
+      until inline_block_stack.empty?
+        state, = *inline_block_stack.pop
+        encoder.end_group :inline
+        encoder.end_group state == :regexp ? :regexp : :string
       end
       
       encoder
diff --git a/lib/coderay/scanners/html.rb b/lib/coderay/scanners/html.rb
index 3ba3b79..ebe7b01 100644
--- a/lib/coderay/scanners/html.rb
+++ b/lib/coderay/scanners/html.rb
@@ -1,13 +1,13 @@
 module CodeRay
 module Scanners
-
+  
   # HTML Scanner
   # 
   # Alias: +xhtml+
   # 
   # See also: Scanners::XML
   class HTML < Scanner
-
+    
     register_for :html
     
     KINDS_NOT_LOC = [
@@ -33,7 +33,8 @@ module Scanners
     )
     
     IN_ATTRIBUTE = WordList::CaseIgnoring.new(nil).
-      add(EVENT_ATTRIBUTES, :script)
+      add(EVENT_ATTRIBUTES, :script).
+      add(['style'], :style)
     
     ATTR_NAME = /[\w.:-]+/  # :nodoc:
     TAG_END = /\/?>/  # :nodoc:
@@ -75,9 +76,14 @@ module Scanners
     def scan_java_script encoder, code
       if code && !code.empty?
         @java_script_scanner ||= Scanners::JavaScript.new '', :keep_tokens => true
-        # encoder.begin_group :inline
         @java_script_scanner.tokenize code, :tokens => encoder
-        # encoder.end_group :inline
+      end
+    end
+    
+    def scan_css encoder, code, state = [:initial]
+      if code && !code.empty?
+        @css_scanner ||= Scanners::CSS.new '', :keep_tokens => true
+        @css_scanner.tokenize code, :tokens => encoder, :state => state
       end
     end
     
@@ -99,7 +105,15 @@ module Scanners
           case state
           
           when :initial
-            if match = scan(/<!--(?:.*?-->|.*)/m)
+            if match = scan(/<!\[CDATA\[/)
+              encoder.text_token match, :inline_delimiter
+              if match = scan(/.*?\]\]>/m)
+                encoder.text_token match[0..-4], :plain
+                encoder.text_token ']]>', :inline_delimiter
+              elsif match = scan(/.+/)
+                encoder.text_token match, :error
+              end
+            elsif match = scan(/<!--(?:.*?-->|.*)/m)
               encoder.text_token match, :comment
             elsif match = scan(/<!(\w+)(?:.*?>|.*)|\]>/m)
               encoder.text_token match, :doctype
@@ -110,7 +124,7 @@ module Scanners
             elsif match = scan(/<\/[-\w.:]*>?/m)
               in_tag = nil
               encoder.text_token match, :tag
-            elsif match = scan(/<(?:(script)|[-\w.:]+)(>)?/m)
+            elsif match = scan(/<(?:(script|style)|[-\w.:]+)(>)?/m)
               encoder.text_token match, :tag
               in_tag = self[1]
               if self[2]
@@ -161,17 +175,21 @@ module Scanners
               encoder.text_token match, :attribute_value
               state = :attribute
             elsif match = scan(/["']/)
-              if in_attribute == :script
-                encoder.begin_group :inline
-                encoder.text_token match, :inline_delimiter
+              if in_attribute == :script || in_attribute == :style
+                encoder.begin_group :string
+                encoder.text_token match, :delimiter
                 if scan(/javascript:[ \t]*/)
                   encoder.text_token matched, :comment
                 end
                 code = scan_until(match == '"' ? /(?="|\z)/ : /(?='|\z)/)
-                scan_java_script encoder, code
+                if in_attribute == :script
+                  scan_java_script encoder, code
+                else
+                  scan_css encoder, code, [:block]
+                end
                 match = scan(/["']/)
-                encoder.text_token match, :inline_delimiter if match
-                encoder.end_group :inline
+                encoder.text_token match, :delimiter if match
+                encoder.end_group :string
                 state = :attribute
                 in_attribute = nil
               else
@@ -206,19 +224,23 @@ module Scanners
             
           when :in_special_tag
             case in_tag
-            when 'script'
+            when 'script', 'style'
               encoder.text_token match, :space if match = scan(/[ \t]*\n/)
               if scan(/(\s*<!--)(?:(.*?)(-->)|(.*))/m)
                 code = self[2] || self[4]
                 closing = self[3]
                 encoder.text_token self[1], :comment
               else
-                code = scan_until(/(?=(?:\n\s*)?<\/script>)|\z/)
+                code = scan_until(/(?=(?:\n\s*)?<\/#{in_tag}>)|\z/)
                 closing = false
               end
               unless code.empty?
                 encoder.begin_group :inline
-                scan_java_script encoder, code
+                if in_tag == 'script'
+                  scan_java_script encoder, code
+                else
+                  scan_css encoder, code
+                end
                 encoder.end_group :inline
               end
               encoder.text_token closing, :comment if closing
diff --git a/lib/coderay/scanners/json.rb b/lib/coderay/scanners/json.rb
index 4e0f462..b09970c 100644
--- a/lib/coderay/scanners/json.rb
+++ b/lib/coderay/scanners/json.rb
@@ -14,15 +14,21 @@ module Scanners
     
     ESCAPE = / [bfnrt\\"\/] /x  # :nodoc:
     UNICODE_ESCAPE = / u[a-fA-F0-9]{4} /x  # :nodoc:
+    KEY = / (?> (?: [^\\"]+ | \\. )* ) " \s* : /x
     
   protected
     
+    def setup
+      @state = :initial
+    end
+    
     # See http://json.org/ for a definition of the JSON lexic/grammar.
     def scan_tokens encoder, options
+      state = options[:state] || @state
       
-      state = :initial
-      stack = []
-      key_expected = false
+      if [:string, :key].include? state
+        encoder.begin_group state
+      end
       
       until eos?
         
@@ -32,18 +38,11 @@ module Scanners
           if match = scan(/ \s+ /x)
             encoder.text_token match, :space
           elsif match = scan(/"/)
-            state = key_expected ? :key : :string
+            state = check(/#{KEY}/o) ? :key : :string
             encoder.begin_group state
             encoder.text_token match, :delimiter
           elsif match = scan(/ [:,\[{\]}] /x)
             encoder.text_token match, :operator
-            case match
-            when ':' then key_expected = false
-            when ',' then key_expected = true if stack.last == :object
-            when '{' then stack << :object; key_expected = true
-            when '[' then stack << :array
-            when '}', ']' then stack.pop  # no error recovery, but works for valid JSON
-            end
           elsif match = scan(/ true | false | null /x)
             encoder.text_token match, :value
           elsif match = scan(/ -? (?: 0 | [1-9]\d* ) /x)
@@ -82,6 +81,10 @@ module Scanners
         end
       end
       
+      if options[:keep_state]
+        @state = state
+      end
+      
       if [:string, :key].include? state
         encoder.end_group state
       end
diff --git a/lib/coderay/scanners/lua.rb b/lib/coderay/scanners/lua.rb
new file mode 100644
index 0000000..fb1e45a
--- /dev/null
+++ b/lib/coderay/scanners/lua.rb
@@ -0,0 +1,280 @@
+# encoding: utf-8
+
+module CodeRay
+module Scanners
+
+  # Scanner for the Lua[http://lua.org] programming lanuage.
+  #
+  # The language’s complete syntax is defined in
+  # {the Lua manual}[http://www.lua.org/manual/5.2/manual.html],
+  # which is what this scanner tries to conform to.
+  class Lua < Scanner
+    
+    register_for :lua
+    file_extension 'lua'
+    title 'Lua'
+    
+    # Keywords used in Lua.
+    KEYWORDS = %w[and break do else elseif end
+      for function goto if in
+      local not or repeat return
+      then until while
+    ]
+    
+    # Constants set by the Lua core.
+    PREDEFINED_CONSTANTS = %w[false true nil]
+    
+    # The expressions contained in this array are parts of Lua’s `basic'
+    # library. Although it’s not entirely necessary to load that library,
+    # it is highly recommended and one would have to provide own implementations
+    # of some of these expressions if one does not do so. They however aren’t
+    # keywords, neither are they constants, but nearly predefined, so they
+    # get tagged as `predefined' rather than anything else.
+    #
+    # This list excludes values of form `_UPPERCASE' because the Lua manual
+    # requires such identifiers to be reserved by Lua anyway and they are
+    # highlighted directly accordingly, without the need for specific
+    # identifiers to be listed here.
+    PREDEFINED_EXPRESSIONS = %w[
+      assert collectgarbage dofile error getmetatable
+      ipairs load loadfile next pairs pcall print
+      rawequal rawget rawlen rawset select setmetatable
+      tonumber tostring type xpcall
+    ]
+    
+    # Automatic token kind selection for normal words.
+    IDENT_KIND = CodeRay::WordList.new(:ident).
+      add(KEYWORDS, :keyword).
+      add(PREDEFINED_CONSTANTS, :predefined_constant).
+      add(PREDEFINED_EXPRESSIONS, :predefined)
+    
+    protected
+    
+    # Scanner initialization.
+    def setup
+      @state = :initial
+      @brace_depth = 0
+    end
+    
+    # CodeRay entry hook. Starts parsing.
+    def scan_tokens(encoder, options)
+      state = options[:state] || @state
+      brace_depth = @brace_depth
+      num_equals = nil
+      
+      until eos?
+        case state
+        
+        when :initial
+          if match = scan(/\-\-\[\=*\[/)   #--[[ long (possibly multiline) comment ]]
+            num_equals = match.count("=") # Number must match for comment end
+            encoder.begin_group(:comment)
+            encoder.text_token(match, :delimiter)
+            state = :long_comment
+          
+          elsif match = scan(/--.*$/) # --Lua comment
+            encoder.text_token(match, :comment)
+          
+          elsif match = scan(/\[=*\[/)     # [[ long (possibly multiline) string ]]
+            num_equals = match.count("=") # Number must match for comment end
+            encoder.begin_group(:string)
+            encoder.text_token(match, :delimiter)
+            state = :long_string
+          
+          elsif match = scan(/::\s*[a-zA-Z_][a-zA-Z0-9_]+\s*::/) # ::goto_label::
+            encoder.text_token(match, :label)
+          
+          elsif match = scan(/_[A-Z]+/) # _UPPERCASE are names reserved for Lua
+            encoder.text_token(match, :predefined)
+          
+          elsif match = scan(/[a-zA-Z_][a-zA-Z0-9_]*/) # Normal letters (or letters followed by digits)
+            kind = IDENT_KIND[match]
+            
+            # Extra highlighting for entities following certain keywords
+            if kind == :keyword and match == "function"
+              state = :function_expected
+            elsif kind == :keyword and match == "goto"
+              state = :goto_label_expected
+            elsif kind == :keyword and match == "local"
+              state = :local_var_expected
+            end
+            
+            encoder.text_token(match, kind)
+          
+          elsif match = scan(/\{/) # Opening table brace {
+            encoder.begin_group(:map)
+            encoder.text_token(match, brace_depth >= 1 ? :inline_delimiter : :delimiter)
+            brace_depth += 1
+            state        = :map
+          
+          elsif match = scan(/\}/) # Closing table brace }
+            if brace_depth == 1
+              brace_depth = 0
+              encoder.text_token(match, :delimiter)
+              encoder.end_group(:map)
+            elsif brace_depth == 0 # Mismatched brace
+              encoder.text_token(match, :error)
+            else
+              brace_depth -= 1
+              encoder.text_token(match, :inline_delimiter)
+              encoder.end_group(:map)
+              state = :map
+            end
+          
+          elsif match = scan(/["']/) # String delimiters " and '
+            encoder.begin_group(:string)
+            encoder.text_token(match, :delimiter)
+            start_delim = match
+            state       = :string
+          
+                            # ↓Prefix                hex number ←|→ decimal number
+          elsif match = scan(/-? (?:0x\h* \. \h+ (?:p[+\-]?\d+)? | \d*\.\d+ (?:e[+\-]?\d+)?)/ix) # hexadecimal constants have no E power, decimal ones no P power
+            encoder.text_token(match, :float)
+          
+                            # ↓Prefix         hex number ←|→ decimal number
+          elsif match = scan(/-? (?:0x\h+ (?:p[+\-]?\d+)? | \d+ (?:e[+\-]?\d+)?)/ix) # hexadecimal constants have no E power, decimal ones no P power
+            encoder.text_token(match, :integer)
+          
+          elsif match = scan(/[\+\-\*\/%^\#=~<>\(\)\[\]:;,] | \.(?!\d)/x) # Operators
+            encoder.text_token(match, :operator)
+          
+          elsif match = scan(/\s+/) # Space
+            encoder.text_token(match, :space)
+          
+          else # Invalid stuff. Note that Lua doesn’t accept multibyte chars outside of strings, hence these are also errors.
+            encoder.text_token(getch, :error)
+          end
+          
+          # It may be that we’re scanning a full-blown subexpression of a table
+          # (tables can contain full expressions in parts).
+          # If this is the case, return to :map scanning state.
+          state = :map if state == :initial && brace_depth >= 1
+        
+        when :function_expected
+          if match = scan(/\(.*?\)/m) # x = function() # "Anonymous" function without explicit name
+            encoder.text_token(match, :operator)
+            state = :initial
+          elsif match = scan(/[a-zA-Z_] (?:[a-zA-Z0-9_\.] (?!\.\d))* [\.\:]/x) # function tbl.subtbl.foo() | function tbl:foo() # Colon only allowed as last separator
+            encoder.text_token(match, :ident)
+          elsif match = scan(/[a-zA-Z_][a-zA-Z0-9_]*/) # function foo()
+            encoder.text_token(match, :function)
+            state = :initial
+          elsif match = scan(/\s+/) # Between the `function' keyword and the ident may be any amount of whitespace
+            encoder.text_token(match, :space)
+          else
+            encoder.text_token(getch, :error)
+            state = :initial
+          end
+        
+        when :goto_label_expected
+          if match = scan(/[a-zA-Z_][a-zA-Z0-9_]*/)
+            encoder.text_token(match, :label)
+            state = :initial
+          elsif match = scan(/\s+/) # Between the `goto' keyword and the label may be any amount of whitespace
+            encoder.text_token(match, :space)
+          else
+            encoder.text_token(getch, :error)
+          end
+        
+        when :local_var_expected
+          if match = scan(/function/) # local function ...
+            encoder.text_token(match, :keyword)
+            state = :function_expected
+          elsif match = scan(/[a-zA-Z_][a-zA-Z0-9_]*/)
+            encoder.text_token(match, :local_variable)
+          elsif match = scan(/,/)
+            encoder.text_token(match, :operator)
+          elsif match = scan(/\=/)
+            encoder.text_token(match, :operator)
+            # After encountering the equal sign, arbitrary expressions are
+            # allowed again, so just return to the main state for further
+            # parsing.
+            state = :initial
+          elsif match = scan(/\n/)
+            encoder.text_token(match, :space)
+            state = :initial
+          elsif match = scan(/\s+/)
+            encoder.text_token(match, :space)
+          else
+            encoder.text_token(getch, :error)
+          end
+        
+        when :long_comment
+          if match = scan(/.*?(?=\]={#{num_equals}}\])/m)
+            encoder.text_token(match, :content)
+            
+            delim = scan(/\]={#{num_equals}}\]/)
+            encoder.text_token(delim, :delimiter)
+          else # No terminator found till EOF
+            encoder.text_token(rest, :error)
+            terminate
+          end
+          encoder.end_group(:comment)
+          state = :initial
+        
+        when :long_string
+          if match = scan(/.*?(?=\]={#{num_equals}}\])/m) # Long strings do not interpret any escape sequences
+            encoder.text_token(match, :content)
+            
+            delim = scan(/\]={#{num_equals}}\]/)
+            encoder.text_token(delim, :delimiter)
+          else # No terminator found till EOF
+            encoder.text_token(rest, :error)
+            terminate
+          end
+          encoder.end_group(:string)
+          state = :initial
+        
+        when :string
+          if match = scan(/[^\\#{start_delim}\n]+/) # Everything except \ and the start delimiter character is string content (newlines are only allowed if preceeded by \ or \z)
+            encoder.text_token(match, :content)
+          elsif match = scan(/\\(?:['"abfnrtv\\]|z\s*|x\h\h|\d{1,3}|\n)/m)
+            encoder.text_token(match, :char)
+          elsif match = scan(Regexp.compile(start_delim))
+            encoder.text_token(match, :delimiter)
+            encoder.end_group(:string)
+            state = :initial
+          elsif match = scan(/\n/) # Lua forbids unescaped newlines in normal non-long strings
+            encoder.text_token("\\n\n", :error) # Visually appealing error indicator--otherwise users may wonder whether the highlighter cannot highlight multine strings
+            encoder.end_group(:string)
+            state = :initial
+          else
+            encoder.text_token(getch, :error)
+          end
+        
+        when :map
+          if match = scan(/[,;]/)
+            encoder.text_token(match, :operator)
+          elsif match = scan(/[a-zA-Z_][a-zA-Z0-9_]* (?=\s*=)/x)
+            encoder.text_token(match, :key)
+            encoder.text_token(scan(/\s+/), :space) if check(/\s+/)
+            encoder.text_token(scan(/\=/), :operator)
+            state = :initial
+          elsif match = scan(/\s+/m)
+            encoder.text_token(match, :space)
+          else
+            # Note this clause doesn’t advance the scan pointer, it’s a kind of
+            # "retry with other options" (the :initial state then of course
+            # advances the pointer).
+            state = :initial
+          end
+        else
+          raise
+        end
+        
+      end
+      
+      if options[:keep_state]
+        @state = state
+      end
+      
+      encoder.end_group :string if [:string].include? state
+      brace_depth.times { encoder.end_group :map }
+      
+      encoder
+    end
+    
+  end
+  
+end
+end
diff --git a/lib/coderay/scanners/php.rb b/lib/coderay/scanners/php.rb
index 6c68834..7a8d75d 100644
--- a/lib/coderay/scanners/php.rb
+++ b/lib/coderay/scanners/php.rb
@@ -265,7 +265,7 @@ module Scanners
             @html_scanner.tokenize match unless match.empty?
           end
         
-        when :php
+        when :php, :php_inline
           if match = scan(/\s+/)
             encoder.text_token match, :space
           
@@ -332,7 +332,7 @@ module Scanners
             if states.size == 1
               encoder.text_token match, :error
             else
-              states.pop
+              state = states.pop
               if states.last.is_a?(::Array)
                 delimiter = states.last[1]
                 states[-1] = states.last[0]
@@ -340,6 +340,7 @@ module Scanners
                 encoder.end_group :inline
               else
                 encoder.text_token match, :operator
+                encoder.end_group :inline if state == :php_inline
                 label_expected = true
               end
             end
@@ -350,7 +351,14 @@ module Scanners
           
           elsif match = scan(RE::PHP_END)
             encoder.text_token match, :inline_delimiter
-            states = [:initial]
+            while state = states.pop
+              encoder.end_group :string if [:sqstring, :dqstring].include? state
+              if state.is_a? Array
+                encoder.end_group :inline
+                encoder.end_group :string if [:sqstring, :dqstring].include? state.first
+              end
+            end
+            states << :initial
           
           elsif match = scan(/<<<(?:(#{RE::IDENTIFIER})|"(#{RE::IDENTIFIER})"|'(#{RE::IDENTIFIER})')/o)
             encoder.begin_group :string
@@ -400,6 +408,7 @@ module Scanners
           elsif match = scan(/\\/)
             encoder.text_token match, :error
           else
+            encoder.end_group :string
             states.pop
           end
         
@@ -459,7 +468,7 @@ module Scanners
               encoder.begin_group :inline
               states[-1] = [states.last, delimiter]
               delimiter = nil
-              states.push :php
+              states.push :php_inline
               encoder.text_token match, :delimiter
             else
               encoder.text_token match, :content
@@ -469,6 +478,7 @@ module Scanners
           elsif match = scan(/\$/)
             encoder.text_token match, :content
           else
+            encoder.end_group :string
             states.pop
           end
         
@@ -500,6 +510,14 @@ module Scanners
         
       end
       
+      while state = states.pop
+        encoder.end_group :string if [:sqstring, :dqstring].include? state
+        if state.is_a? Array
+          encoder.end_group :inline
+          encoder.end_group :string if [:sqstring, :dqstring].include? state.first
+        end
+      end
+      
       encoder
     end
     
diff --git a/lib/coderay/scanners/python.rb b/lib/coderay/scanners/python.rb
index a9492ab..09c8b6e 100644
--- a/lib/coderay/scanners/python.rb
+++ b/lib/coderay/scanners/python.rb
@@ -157,12 +157,12 @@ module Scanners
             encoder.text_token match, :operator
           
           elsif match = scan(/(u?r?|b)?("""|"|'''|')/i)
+            modifiers = self[1]
             string_delimiter = self[2]
-            string_type = docstring_coming ? :docstring : :string
+            string_type = docstring_coming ? :docstring : (modifiers == 'b' ? :binary : :string)
             docstring_coming = false if docstring_coming
             encoder.begin_group string_type
             string_raw = false
-            modifiers = self[1]
             unless modifiers.empty?
               string_raw = !!modifiers.index(?r)
               encoder.text_token modifiers, :modifier
diff --git a/lib/coderay/scanners/raydebug.rb b/lib/coderay/scanners/raydebug.rb
index 7a21354..1effdc8 100644
--- a/lib/coderay/scanners/raydebug.rb
+++ b/lib/coderay/scanners/raydebug.rb
@@ -1,23 +1,30 @@
+require 'set'
+
 module CodeRay
 module Scanners
-
-  # = Debug Scanner
+  
+  # = Raydebug Scanner
   # 
-  # Parses the output of the Encoders::Debug encoder.
+  # Highlights the output of the Encoders::Debug encoder.
   class Raydebug < Scanner
-
+    
     register_for :raydebug
     file_extension 'raydebug'
     title 'CodeRay Token Dump'
     
   protected
     
+    def setup
+      super
+      @known_token_kinds = TokenKinds.keys.map(&:to_s).to_set
+    end
+    
     def scan_tokens encoder, options
-
+      
       opened_tokens = []
-
+      
       until eos?
-
+        
         if match = scan(/\s+/)
           encoder.text_token match, :space
           
@@ -26,20 +33,22 @@ module Scanners
           encoder.text_token kind, :class
           encoder.text_token '(', :operator
           match = self[2]
-          encoder.text_token match, kind.to_sym
+          unless match.empty?
+            if @known_token_kinds.include? kind
+              encoder.text_token match, kind.to_sym
+            else
+              encoder.text_token match, :plain
+            end
+          end
           encoder.text_token match, :operator if match = scan(/\)/)
           
         elsif match = scan(/ (\w+) ([<\[]) /x)
-          kind = self[1]
-          case self[2]
-          when '<'
-            encoder.text_token kind, :class
-          when '['
-            encoder.text_token kind, :class
+          encoder.text_token self[1], :class
+          if @known_token_kinds.include? self[1]
+            kind = self[1].to_sym
           else
-            raise 'CodeRay bug: This case should not be reached.'
+            kind = :unknown
           end
-          kind = kind.to_sym
           opened_tokens << kind
           encoder.begin_group kind
           encoder.text_token self[2], :operator
@@ -59,8 +68,8 @@ module Scanners
       
       encoder
     end
-
+    
   end
-
+  
 end
 end
diff --git a/lib/coderay/scanners/ruby.rb b/lib/coderay/scanners/ruby.rb
index c282f31..5b8de42 100644
--- a/lib/coderay/scanners/ruby.rb
+++ b/lib/coderay/scanners/ruby.rb
@@ -164,15 +164,19 @@ module Scanners
               end
               
             elsif match = scan(/ ' (?:(?>[^'\\]*) ')? | " (?:(?>[^"\\\#]*) ")? /mx)
-              encoder.begin_group :string
               if match.size == 1
+                kind = check(self.class::StringState.simple_key_pattern(match)) ? :key : :string
+                encoder.begin_group kind
                 encoder.text_token match, :delimiter
-                state = self.class::StringState.new :string, match == '"', match  # important for streaming
+                state = self.class::StringState.new kind, match == '"', match  # important for streaming
               else
+                kind = value_expected == true && scan(/:/) ? :key : :string
+                encoder.begin_group kind
                 encoder.text_token match[0,1], :delimiter
                 encoder.text_token match[1..-2], :content if match.size > 2
                 encoder.text_token match[-1,1], :delimiter
-                encoder.end_group :string
+                encoder.end_group kind
+                encoder.text_token ':', :operator if kind == :key
                 value_expected = false
               end
               
@@ -191,11 +195,14 @@ module Scanners
                 encoder.text_token match, :error
                 method_call_expected = false
               else
-                encoder.text_token match, self[1] ? :float : :integer  # TODO: send :hex/:octal/:binary
+                kind = self[1] ? :float : :integer  # TODO: send :hex/:octal/:binary
+                match << 'r' if match !~ /e/i && scan(/r/)
+                match << 'i' if scan(/i/)
+                encoder.text_token match, kind
               end
               value_expected = false
               
-            elsif match = scan(/ [-+!~^\/]=? | [:;] | [*|&]{1,2}=? | >>? /x)
+            elsif match = scan(/ [-+!~^\/]=? | [:;] | &\. | [*|&]{1,2}=? | >>? /x)
               value_expected = true
               encoder.text_token match, :operator
               
@@ -208,7 +215,7 @@ module Scanners
               encoder.end_group kind
               heredocs ||= []  # create heredocs if empty
               heredocs << self.class::StringState.new(kind, quote != "'", delim,
-                self[1] == '-' ? :indented : :linestart)
+                self[1] ? :indented : :linestart)
               value_expected = false
               
             elsif value_expected && match = scan(/#{patterns::FANCY_STRING_START}/o)
@@ -269,7 +276,7 @@ module Scanners
             end
             
             if last_state
-              state = last_state
+              state = last_state unless state.is_a?(StringState)  # otherwise, a simple 'def"' results in unclosed tokens
               last_state = nil
             end
             
diff --git a/lib/coderay/scanners/ruby/patterns.rb b/lib/coderay/scanners/ruby/patterns.rb
index ed071d2..e5a156d 100644
--- a/lib/coderay/scanners/ruby/patterns.rb
+++ b/lib/coderay/scanners/ruby/patterns.rb
@@ -114,7 +114,7 @@ module Scanners
     # NOTE: This is not completely correct, but
     # nobody needs heredoc delimiters ending with \n.
     HEREDOC_OPEN = /
-      << (-)?              # $1 = float
+      << ([-~])?           # $1 = float
       (?:
         ( [A-Za-z_0-9]+ )  # $2 = delim
       |
@@ -157,13 +157,16 @@ module Scanners
       yield
     ])
     
-    FANCY_STRING_START = / % ( [QqrsWwx] | (?![a-zA-Z0-9]) ) ([^a-zA-Z0-9]) /x
+    FANCY_STRING_START = / % ( [iIqQrswWx] | (?![a-zA-Z0-9]) ) ([^a-zA-Z0-9]) /x
     FANCY_STRING_KIND = Hash.new(:string).merge({
+      'i' => :symbol,
+      'I' => :symbol,
       'r' => :regexp,
       's' => :symbol,
       'x' => :shell,
     })
     FANCY_STRING_INTERPRETED = Hash.new(true).merge({
+      'i' => false,
       'q' => false,
       's' => false,
       'w' => false,
diff --git a/lib/coderay/scanners/ruby/string_state.rb b/lib/coderay/scanners/ruby/string_state.rb
index 2f398d1..95f1e83 100644
--- a/lib/coderay/scanners/ruby/string_state.rb
+++ b/lib/coderay/scanners/ruby/string_state.rb
@@ -16,7 +16,6 @@ module Scanners
       
       STRING_PATTERN = Hash.new do |h, k|
         delim, interpreted = *k
-        # delim = delim.dup  # workaround for old Ruby
         delim_pattern = Regexp.escape(delim)
         if closing_paren = CLOSING_PAREN[delim]
           delim_pattern << Regexp.escape(closing_paren)
@@ -29,12 +28,21 @@ module Scanners
         #     '| [|?*+(){}\[\].^$]'
         #   end
         
-        h[k] =
-          if interpreted && delim != '#'
-            / (?= [#{delim_pattern}] | \# [{$@] ) /mx
-          else
-            / (?= [#{delim_pattern}] ) /mx
-          end
+        if interpreted && delim != '#'
+          / (?= [#{delim_pattern}] | \# [{$@] ) /mx
+        else
+          / (?= [#{delim_pattern}] ) /mx
+        end.tap do |pattern|
+          h[k] = pattern if (delim.respond_to?(:ord) ? delim.ord : delim[0]) < 256
+        end
+      end
+      
+      def self.simple_key_pattern delim
+        if delim == "'"
+          / (?> (?: [^\\']+ | \\. )* ) ' : /mx
+        else
+          / (?> (?: [^\\"\#]+ | \\. | \#\$[\\"] | \#\{[^\{\}]+\} | \#(?!\{) )* ) " : /mx
+        end
       end
       
       def initialize kind, interpreted, delim, heredoc = false
diff --git a/lib/coderay/scanners/sass.rb b/lib/coderay/scanners/sass.rb
index 167051d..e3296b9 100644
--- a/lib/coderay/scanners/sass.rb
+++ b/lib/coderay/scanners/sass.rb
@@ -7,11 +7,6 @@ module Scanners
     register_for :sass
     file_extension 'sass'
     
-    STRING_CONTENT_PATTERN = {
-      "'" => /(?:[^\n\'\#]+|\\\n|#{RE::Escape}|#(?!\{))+/,
-      '"' => /(?:[^\n\"\#]+|\\\n|#{RE::Escape}|#(?!\{))+/,
-    }
-    
   protected
     
     def setup
@@ -19,8 +14,9 @@ module Scanners
     end
     
     def scan_tokens encoder, options
-      states = Array(options[:state] || @state)
-      string_delimiter = nil
+      states = Array(options[:state] || @state).dup
+      
+      encoder.begin_group :string if states.last == :sqstring || states.last == :dqstring
       
       until eos?
         
@@ -48,7 +44,7 @@ module Scanners
         elsif case states.last
           when :initial, :media, :sass_inline
             if match = scan(/(?>#{RE::Ident})(?!\()/ox)
-              encoder.text_token match, value_expected ? :value : (check(/.*:/) ? :key : :tag)
+              encoder.text_token match, value_expected ? :value : (check(/.*:(?![a-z])/) ? :key : :tag)
               next
             elsif !value_expected && (match = scan(/\*/))
               encoder.text_token match, :tag
@@ -91,24 +87,23 @@ module Scanners
               next
             end
             
-          when :string
-            if match = scan(STRING_CONTENT_PATTERN[string_delimiter])
+          when :sqstring, :dqstring
+            if match = scan(states.last == :sqstring ? /(?:[^\n\'\#]+|\\\n|#{RE::Escape}|#(?!\{))+/o : /(?:[^\n\"\#]+|\\\n|#{RE::Escape}|#(?!\{))+/o)
               encoder.text_token match, :content
             elsif match = scan(/['"]/)
               encoder.text_token match, :delimiter
               encoder.end_group :string
-              string_delimiter = nil
               states.pop
             elsif match = scan(/#\{/)
               encoder.begin_group :inline
               encoder.text_token match, :inline_delimiter
               states.push :sass_inline
             elsif match = scan(/ \\ | $ /x)
-              encoder.end_group :string
+              encoder.end_group states.last
               encoder.text_token match, :error unless match.empty?
               states.pop
             else
-              raise_inspect "else case #{string_delimiter} reached; %p not handled." % peek(1), encoder
+              raise_inspect "else case #{states.last} reached; %p not handled." % peek(1), encoder
             end
           
           when :include
@@ -119,7 +114,7 @@ module Scanners
           
           else
             #:nocov:
-            raise_inspect 'Unknown state', encoder
+            raise_inspect 'Unknown state: %p' % [states.last], encoder
             #:nocov:
             
           end
@@ -157,15 +152,15 @@ module Scanners
           
         elsif match = scan(/['"]/)
           encoder.begin_group :string
-          string_delimiter = match
           encoder.text_token match, :delimiter
           if states.include? :sass_inline
-            content = scan_until(/(?=#{string_delimiter}|\}|\z)/)
+            # no nesting, just scan the string until delimiter
+            content = scan_until(/(?=#{match}|\}|\z)/)
             encoder.text_token content, :content unless content.empty?
-            encoder.text_token string_delimiter, :delimiter if scan(/#{string_delimiter}/)
+            encoder.text_token match, :delimiter if scan(/#{match}/)
             encoder.end_group :string
           else
-            states.push :string
+            states.push match == "'" ? :sqstring : :dqstring
           end
           
         elsif match = scan(/#{RE::Function}/o)
@@ -176,7 +171,7 @@ module Scanners
             encoder.text_token match[start.size..-2], :content
             encoder.text_token ')', :delimiter
           else
-            encoder.text_token match[start.size..-1], :content
+            encoder.text_token match[start.size..-1], :content if start.size < match.size
           end
           encoder.end_group :function
           
@@ -195,7 +190,7 @@ module Scanners
         elsif match = scan(/(?:rgb|hsl)a?\([^()\n]*\)?/)
           encoder.text_token match, :color
           
-        elsif match = scan(/@else if\b|#{RE::AtKeyword}/)
+        elsif match = scan(/@else if\b|#{RE::AtKeyword}/o)
           encoder.text_token match, :directive
           value_expected = true
           
@@ -214,8 +209,18 @@ module Scanners
         
       end
       
+      states.pop if states.last == :include
+      
       if options[:keep_state]
-        @state = states
+        @state = states.dup
+      end
+      
+      while state = states.pop
+        if state == :sass_inline
+          encoder.end_group :inline
+        elsif state == :sqstring || state == :dqstring
+          encoder.end_group :string
+        end
       end
       
       encoder
diff --git a/lib/coderay/scanners/sql.rb b/lib/coderay/scanners/sql.rb
index b757278..7d57f77 100644
--- a/lib/coderay/scanners/sql.rb
+++ b/lib/coderay/scanners/sql.rb
@@ -1,8 +1,9 @@
-module CodeRay module Scanners
+module CodeRay
+module Scanners
   
   # by Josh Goebel
   class SQL < Scanner
-
+    
     register_for :sql
     
     KEYWORDS = %w(
@@ -56,6 +57,12 @@ module CodeRay module Scanners
     
     STRING_PREFIXES = /[xnb]|_\w+/i
     
+    STRING_CONTENT_PATTERN = {
+      '"' => / (?: [^\\"] | "" )+ /x,
+      "'" => / (?: [^\\'] | '' )+ /x,
+      '`' => / (?: [^\\`] | `` )+ /x,
+    }
+    
     def scan_tokens encoder, options
       
       state = :initial
@@ -89,7 +96,7 @@ module CodeRay module Scanners
             state = :string
             encoder.text_token match, :delimiter
             
-          elsif match = scan(/ @? [A-Za-z_][A-Za-z_0-9]* /x)
+          elsif match = scan(/ @? [A-Za-z_][A-Za-z_0-9\$]* /x)
             encoder.text_token match, name_expected ? :ident : (match[0] == ?@ ? :variable : IDENT_KIND[match])
             name_expected = false
             
@@ -114,41 +121,28 @@ module CodeRay module Scanners
           end
           
         elsif state == :string
-          if match = scan(/[^\\"'`]+/)
-            string_content << match
-            next
+          if match = scan(STRING_CONTENT_PATTERN[string_type])
+            encoder.text_token match, :content
           elsif match = scan(/["'`]/)
             if string_type == match
               if peek(1) == string_type  # doubling means escape
-                string_content << string_type << getch
-                next
-              end
-              unless string_content.empty?
-                encoder.text_token string_content, :content
-                string_content = ''
+                encoder.text_token match + getch, :content
+              else
+                encoder.text_token match, :delimiter
+                encoder.end_group :string
+                state = :initial
+                string_type = nil
               end
-              encoder.text_token match, :delimiter
-              encoder.end_group :string
-              state = :initial
-              string_type = nil
             else
-              string_content << match
+              encoder.text_token match, :content
             end
           elsif match = scan(/ \\ (?: #{ESCAPE} | #{UNICODE_ESCAPE} ) /mox)
-            unless string_content.empty?
-              encoder.text_token string_content, :content
-              string_content = ''
-            end
             encoder.text_token match, :char
           elsif match = scan(/ \\ . /mox)
-            string_content << match
-            next
+            encoder.text_token match, :content
           elsif match = scan(/ \\ | $ /x)
-            unless string_content.empty?
-              encoder.text_token string_content, :content
-              string_content = ''
-            end
             encoder.text_token match, :error unless match.empty?
+            encoder.end_group :string
             state = :initial
           else
             raise "else case \" reached; %p not handled." % peek(1), encoder
@@ -171,4 +165,5 @@ module CodeRay module Scanners
     
   end
   
-end end
\ No newline at end of file
+end
+end
diff --git a/lib/coderay/scanners/taskpaper.rb b/lib/coderay/scanners/taskpaper.rb
new file mode 100644
index 0000000..42670bc
--- /dev/null
+++ b/lib/coderay/scanners/taskpaper.rb
@@ -0,0 +1,36 @@
+module CodeRay
+module Scanners
+  
+  class Taskpaper < Scanner
+    
+    register_for :taskpaper
+    file_extension 'taskpaper'
+    
+  protected
+    
+    def scan_tokens encoder, options
+      until eos?
+        if match = scan(/\S.*:.*$/)                  # project
+          encoder.text_token(match, :namespace)
+        elsif match = scan(/-.+@done.*/)             # completed task
+          encoder.text_token(match, :done)
+        elsif match = scan(/-(?:[^@\n]+|@(?!due))*/) # task
+          encoder.text_token(match, :plain)
+        elsif match = scan(/@due.*/)                 # comment
+          encoder.text_token(match, :important)
+        elsif match = scan(/.+/)                     # comment
+          encoder.text_token(match, :comment)
+        elsif match = scan(/\s+/)                    # space
+          encoder.text_token(match, :space)
+        else                                         # other
+          encoder.text_token getch, :error
+        end
+      end
+      
+      encoder
+    end
+    
+  end
+  
+end
+end
diff --git a/lib/coderay/scanners/yaml.rb b/lib/coderay/scanners/yaml.rb
index 96f4e93..32c8e2c 100644
--- a/lib/coderay/scanners/yaml.rb
+++ b/lib/coderay/scanners/yaml.rb
@@ -47,7 +47,7 @@ module Scanners
           when !check(/(?:"[^"]*")(?=: |:$)/) && match = scan(/"/)
             encoder.begin_group :string
             encoder.text_token match, :delimiter
-            encoder.text_token match, :content if match = scan(/ [^"\\]* (?: \\. [^"\\]* )* /mx)
+            encoder.text_token match, :content if (match = scan(/ [^"\\]* (?: \\. [^"\\]* )* /mx)) && !match.empty?
             encoder.text_token match, :delimiter if match = scan(/"/)
             encoder.end_group :string
             next
@@ -84,7 +84,7 @@ module Scanners
           when match = scan(/(?:"[^"\n]*"|'[^'\n]*')(?= *:(?: |$))/)
             encoder.begin_group :key
             encoder.text_token match[0,1], :delimiter
-            encoder.text_token match[1..-2], :content
+            encoder.text_token match[1..-2], :content if match.size > 2
             encoder.text_token match[-1,1], :delimiter
             encoder.end_group :key
             key_indent = column(pos - match.size) - 1
diff --git a/lib/coderay/styles/alpha.rb b/lib/coderay/styles/alpha.rb
index a21fbf1..d304dc4 100644
--- a/lib/coderay/styles/alpha.rb
+++ b/lib/coderay/styles/alpha.rb
@@ -3,14 +3,14 @@ module Styles
   
   # A colorful theme using CSS 3 colors (with alpha channel).
   class Alpha < Style
-
+    
     register_for :alpha
-
+    
     code_background = 'hsl(0,0%,95%)'
     numbers_background = 'hsl(180,65%,90%)'
     border_color = 'silver'
     normal_color = 'black'
-
+    
     CSS_MAIN_STYLES = <<-MAIN  # :nodoc:
 .CodeRay {
   background-color: #{code_background};
@@ -56,25 +56,26 @@ table.CodeRay td { padding: 2px 4px; vertical-align: top; }
 .annotation { color:#007 }
 .attribute-name { color:#b48 }
 .attribute-value { color:#700 }
-.binary { color:#509 }
+.binary { color:#549 }
+.binary .char { color:#325 }
+.binary .delimiter { color:#325 }
+.char { color:#D20 }
 .char .content { color:#D20 }
 .char .delimiter { color:#710 }
-.char { color:#D20 }
 .class { color:#B06; font-weight:bold }
 .class-variable { color:#369 }
 .color { color:#0A0 }
 .comment { color:#777 }
 .comment .char { color:#444 }
 .comment .delimiter { color:#444 }
-.complex { color:#A08 }
 .constant { color:#036; font-weight:bold }
 .decorator { color:#B0B }
 .definition { color:#099; font-weight:bold }
 .delimiter { color:black }
 .directive { color:#088; font-weight:bold }
-.doc { color:#970 }
-.doc-string { color:#D42; font-weight:bold }
+.docstring { color:#D42; }
 .doctype { color:#34b }
+.done { text-decoration: line-through; color: gray }
 .entity { color:#800; font-weight:bold }
 .error { color:#F00; background-color:#FAA }
 .escape  { color:#666 }
@@ -85,51 +86,54 @@ table.CodeRay td { padding: 2px 4px; vertical-align: top; }
 .global-variable { color:#d70 }
 .hex { color:#02b }
 .id  { color:#33D; font-weight:bold }
-.imaginary { color:#f00 }
 .include { color:#B44; font-weight:bold }
 .inline { background-color: hsla(0,0%,0%,0.07); color: black }
 .inline-delimiter { font-weight: bold; color: #666 }
 .instance-variable { color:#33B }
 .integer  { color:#00D }
+.imaginary { color:#f00 }
 .important { color:#D00 }
+.key { color: #606 }
 .key .char { color: #60f }
 .key .delimiter { color: #404 }
-.key { color: #606 }
 .keyword { color:#080; font-weight:bold }
 .label { color:#970; font-weight:bold }
-.local-variable { color:#963 }
+.local-variable { color:#950 }
+.map .content { color:#808 }
+.map .delimiter { color:#40A}
+.map { background-color:hsla(200,100%,50%,0.06); }
 .namespace { color:#707; font-weight:bold }
 .octal { color:#40E }
 .operator { }
 .predefined { color:#369; font-weight:bold }
 .predefined-constant { color:#069 }
-.predefined-type { color:#0a5; font-weight:bold }
+.predefined-type { color:#0a8; font-weight:bold }
 .preprocessor { color:#579 }
 .pseudo-class { color:#00C; font-weight:bold }
+.regexp { background-color:hsla(300,100%,50%,0.06); }
 .regexp .content { color:#808 }
 .regexp .delimiter { color:#404 }
 .regexp .modifier { color:#C2C }
-.regexp { background-color:hsla(300,100%,50%,0.06); }
 .reserved { color:#080; font-weight:bold }
+.shell { background-color:hsla(120,100%,50%,0.06); }
 .shell .content { color:#2B2 }
 .shell .delimiter { color:#161 }
-.shell { background-color:hsla(120,100%,50%,0.06); }
+.string { background-color:hsla(0,100%,50%,0.05); }
 .string .char { color: #b0b }
 .string .content { color: #D20 }
 .string .delimiter { color: #710 }
 .string .modifier { color: #E40 }
-.string { background-color:hsla(0,100%,50%,0.05); }
-.symbol .content { color:#A60 }
-.symbol .delimiter { color:#630 }
 .symbol { color:#A60 }
-.tag { color:#070 }
+.symbol .content { color:#A60 }
+.symbol .delimiter { color:#740 }
+.tag { color:#070; font-weight:bold }
 .type { color:#339; font-weight:bold }
-.value { color: #088; }
-.variable  { color:#037 }
+.value { color: #088 }
+.variable { color:#037 }
 
 .insert { background: hsla(120,100%,50%,0.12) }
 .delete { background: hsla(0,100%,50%,0.12) }
-.change { color: #bbf; background: #007; }
+.change { color: #bbf; background: #007 }
 .head { color: #f8f; background: #505 }
 .head .filename { color: white; }
 
@@ -141,8 +145,8 @@ table.CodeRay td { padding: 2px 4px; vertical-align: top; }
 .change .change { color: #88f }
 .head .head { color: #f4f }
     TOKENS
-
+    
   end
-
+  
 end
 end
diff --git a/lib/coderay/token_kinds.rb b/lib/coderay/token_kinds.rb
index bd8fd6c..f911862 100755
--- a/lib/coderay/token_kinds.rb
+++ b/lib/coderay/token_kinds.rb
@@ -1,89 +1,85 @@
 module CodeRay
   
   # A Hash of all known token kinds and their associated CSS classes.
-  TokenKinds = Hash.new do |h, k|
-    warn 'Undefined Token kind: %p' % [k] if $CODERAY_DEBUG
-    false
-  end
+  TokenKinds = Hash.new(false)
   
   # speedup
   TokenKinds.compare_by_identity if TokenKinds.respond_to? :compare_by_identity
   
   TokenKinds.update(  # :nodoc:
-    :annotation          => 'annotation',
-    :attribute_name      => 'attribute-name',
-    :attribute_value     => 'attribute-value',
-    :binary              => 'bin',
-    :char                => 'char',
-    :class               => 'class',
-    :class_variable      => 'class-variable',
-    :color               => 'color',
-    :comment             => 'comment',
-    :complex             => 'complex',
-    :constant            => 'constant',
-    :content             => 'content',
-    :debug               => 'debug',
-    :decorator           => 'decorator',
-    :definition          => 'definition',
-    :delimiter           => 'delimiter',
-    :directive           => 'directive',
-    :doc                 => 'doc',
-    :doctype             => 'doctype',
-    :doc_string          => 'doc-string',
-    :entity              => 'entity',
-    :error               => 'error',
-    :escape              => 'escape',
-    :exception           => 'exception',
-    :filename            => 'filename',
-    :float               => 'float',
-    :function            => 'function',
-    :global_variable     => 'global-variable',
-    :hex                 => 'hex',
-    :id                  => 'id',
-    :imaginary           => 'imaginary',
-    :important           => 'important',
-    :include             => 'include',
-    :inline              => 'inline',
-    :inline_delimiter    => 'inline-delimiter',
-    :instance_variable   => 'instance-variable',
-    :integer             => 'integer',
-    :key                 => 'key',
-    :keyword             => 'keyword',
-    :label               => 'label',
-    :local_variable      => 'local-variable',
-    :modifier            => 'modifier',
-    :namespace           => 'namespace',
-    :octal               => 'octal',
-    :predefined          => 'predefined',
-    :predefined_constant => 'predefined-constant',
-    :predefined_type     => 'predefined-type',
-    :preprocessor        => 'preprocessor',
-    :pseudo_class        => 'pseudo-class',
-    :regexp              => 'regexp',
-    :reserved            => 'reserved',
-    :shell               => 'shell',
-    :string              => 'string',
-    :symbol              => 'symbol',
-    :tag                 => 'tag',
-    :type                => 'type',
-    :value               => 'value',
-    :variable            => 'variable',
+    :debug               => 'debug',              # highlight for debugging (white on blue background)
     
-    :change              => 'change',
-    :delete              => 'delete',
-    :head                => 'head',
-    :insert              => 'insert',
+    :annotation          => 'annotation',         # Groovy, Java
+    :attribute_name      => 'attribute-name',     # HTML, CSS
+    :attribute_value     => 'attribute-value',    # HTML
+    :binary              => 'binary',             # Python, Ruby
+    :char                => 'char',               # most scanners, also inside of strings
+    :class               => 'class',              # lots of scanners, for different purposes also in CSS
+    :class_variable      => 'class-variable',     # Ruby, YAML
+    :color               => 'color',              # CSS
+    :comment             => 'comment',            # most scanners
+    :constant            => 'constant',           # PHP, Ruby
+    :content             => 'content',            # inside of strings, most scanners
+    :decorator           => 'decorator',          # Python
+    :definition          => 'definition',         # CSS
+    :delimiter           => 'delimiter',          # inside strings, comments and other types
+    :directive           => 'directive',          # lots of scanners
+    :doctype             => 'doctype',            # Goorvy, HTML, Ruby, YAML
+    :docstring           => 'docstring',          # Python
+    :done                => 'done',               # Taskpaper
+    :entity              => 'entity',             # HTML
+    :error               => 'error',              # invalid token, most scanners
+    :escape              => 'escape',             # Ruby (string inline variables like #$foo, #@bar)
+    :exception           => 'exception',          # Java, PHP, Python
+    :filename            => 'filename',           # Diff
+    :float               => 'float',              # most scanners
+    :function            => 'function',           # CSS, JavaScript, PHP
+    :global_variable     => 'global-variable',    # Ruby, YAML
+    :hex                 => 'hex',                # hexadecimal number; lots of scanners
+    :id                  => 'id',                 # CSS
+    :imaginary           => 'imaginary',          # Python
+    :important           => 'important',          # CSS, Taskpaper
+    :include             => 'include',            # C, Groovy, Java, Python, Sass
+    :inline              => 'inline',             # nested code, eg. inline string evaluation; lots of scanners
+    :inline_delimiter    => 'inline-delimiter',   # used instead of :inline > :delimiter FIXME: Why use inline_delimiter?
+    :instance_variable   => 'instance-variable',  # Ruby
+    :integer             => 'integer',            # most scanners
+    :key                 => 'key',                # lots of scanners, used together with :value
+    :keyword             => 'keyword',            # reserved word that's actually implemented; most scanners
+    :label               => 'label',              # C, PHP
+    :local_variable      => 'local-variable',     # local and magic variables; some scanners
+    :map                 => 'map',                # Lua tables
+    :modifier            => 'modifier',           # used inside on strings; lots of scanners
+    :namespace           => 'namespace',          # Clojure, Java, Taskpaper
+    :octal               => 'octal',              # lots of scanners
+    :predefined          => 'predefined',         # predefined function: lots of scanners
+    :predefined_constant => 'predefined-constant',# lots of scanners
+    :predefined_type     => 'predefined-type',    # C, Java, PHP
+    :preprocessor        => 'preprocessor',       # C, Delphi, HTML
+    :pseudo_class        => 'pseudo-class',       # CSS
+    :regexp              => 'regexp',             # Groovy, JavaScript, Ruby
+    :reserved            => 'reserved',           # most scanners
+    :shell               => 'shell',              # Ruby
+    :string              => 'string',             # most scanners
+    :symbol              => 'symbol',             # Clojure, Ruby, YAML
+    :tag                 => 'tag',                # CSS, HTML
+    :type                => 'type',               # CSS, Java, SQL, YAML
+    :value               => 'value',              # used together with :key; CSS, JSON, YAML
+    :variable            => 'variable',           # Sass, SQL, YAML
     
-    :eyecatcher          => 'eyecatcher',
+    :change              => 'change',             # Diff
+    :delete              => 'delete',             # Diff
+    :head                => 'head',               # Diff, YAML
+    :insert              => 'insert',             # Diff
+    :eyecatcher          => 'eyecatcher',         # Diff
     
-    :ident               => false,
-    :operator            => false,
+    :ident               => false,                # almost all scanners
+    :operator            => false,                # almost all scanners
     
-    :space               => false,
-    :plain               => false
+    :space               => false,                # almost all scanners
+    :plain               => false                 # almost all scanners
   )
   
-  TokenKinds[:method]    = TokenKinds[:function]
-  TokenKinds[:escape]    = TokenKinds[:delimiter]
-  TokenKinds[:docstring] = TokenKinds[:comment]
+  TokenKinds[:method]  = TokenKinds[:function]
+  TokenKinds[:unknown] = TokenKinds[:plain]
 end
diff --git a/lib/coderay/tokens.rb b/lib/coderay/tokens.rb
index 6957d69..e7bffce 100644
--- a/lib/coderay/tokens.rb
+++ b/lib/coderay/tokens.rb
@@ -1,55 +1,43 @@
 module CodeRay
   
-  # GZip library for writing and reading token dumps.
-  autoload :GZip, coderay_path('helpers', 'gzip')
-  
-  # = Tokens  TODO: Rewrite!
-  #
-  # The Tokens class represents a list of tokens returnd from
-  # a Scanner.
+  # The Tokens class represents a list of tokens returned from
+  # a Scanner. It's actually just an Array with a few helper methods.
   #
-  # A token is not a special object, just a two-element Array
-  # consisting of
+  # A token itself is not a special object, just two elements in an Array:
   # * the _token_ _text_ (the original source of the token in a String) or
   #   a _token_ _action_ (begin_group, end_group, begin_line, end_line)
   # * the _token_ _kind_ (a Symbol representing the type of the token)
   #
-  # A token looks like this:
+  # It looks like this:
   #
-  #   ['# It looks like this', :comment]
-  #   ['3.1415926', :float]
-  #   ['$^', :error]
+  #   ..., '# It looks like this', :comment, ...
+  #   ..., '3.1415926', :float, ...
+  #   ..., '$^', :error, ...
   #
   # Some scanners also yield sub-tokens, represented by special
-  # token actions, namely begin_group and end_group.
+  # token actions, for example :begin_group and :end_group.
   #
   # The Ruby scanner, for example, splits "a string" into:
   #
   #  [
-  #   [:begin_group, :string],
-  #   ['"', :delimiter],
-  #   ['a string', :content],
-  #   ['"', :delimiter],
-  #   [:end_group, :string]
+  #   :begin_group, :string,
+  #   '"',          :delimiter,
+  #   'a string',   :content,
+  #   '"',          :delimiter,
+  #   :end_group,   :string
   #  ]
   #
-  # Tokens is the interface between Scanners and Encoders:
-  # The input is split and saved into a Tokens object. The Encoder
-  # then builds the output from this object.
-  #
-  # Thus, the syntax below becomes clear:
+  # Tokens can be used to save the output of a Scanners in a simple
+  # Ruby object that can be send to an Encoder later:
   #
-  #   CodeRay.scan('price = 2.59', :ruby).html
-  #   # the Tokens object is here -------^
-  #
-  # See how small it is? ;)
+  #   tokens = CodeRay.scan('price = 2.59', :ruby).tokens
+  #   tokens.encode(:html)
+  #   tokens.html
+  #   CodeRay.encoder(:html).encode_tokens(tokens)
   #
   # Tokens gives you the power to handle pre-scanned code very easily:
-  # You can convert it to a webpage, a YAML file, or dump it into a gzip'ed string
-  # that you put in your DB.
-  # 
-  # It also allows you to generate tokens directly (without using a scanner),
-  # to load them from a file, and still use any Encoder that CodeRay provides.
+  # You can serialize it to a JSON string and store it in a database, pass it
+  # around to encode it more than once, send it to other algorithms...
   class Tokens < Array
     
     # The Scanner instance that created the tokens.
@@ -58,8 +46,7 @@ module CodeRay
     # Encode the tokens using encoder.
     #
     # encoder can be
-    # * a symbol like :html oder :statistic
-    # * an Encoder class
+    # * a plugin name like :html oder 'statistic'
     # * an Encoder object
     #
     # options are passed to the encoder.
@@ -157,53 +144,11 @@ module CodeRay
       parts
     end
     
-    # Dumps the object into a String that can be saved
-    # in files or databases.
-    #
-    # The dump is created with Marshal.dump;
-    # In addition, it is gzipped using GZip.gzip.
-    #
-    # The returned String object includes Undumping
-    # so it has an #undump method. See Tokens.load.
-    #
-    # You can configure the level of compression,
-    # but the default value 7 should be what you want
-    # in most cases as it is a good compromise between
-    # speed and compression rate.
-    #
-    # See GZip module.
-    def dump gzip_level = 7
-      dump = Marshal.dump self
-      dump = GZip.gzip dump, gzip_level
-      dump.extend Undumping
-    end
-    
     # Return the actual number of tokens.
     def count
       size / 2
     end
     
-    # Include this module to give an object an #undump
-    # method.
-    #
-    # The string returned by Tokens.dump includes Undumping.
-    module Undumping
-      # Calls Tokens.load with itself.
-      def undump
-        Tokens.load self
-      end
-    end
-    
-    # Undump the object using Marshal.load, then
-    # unzip it using GZip.gunzip.
-    #
-    # The result is commonly a Tokens object, but
-    # this is not guaranteed.
-    def Tokens.load dump
-      dump = GZip.gunzip dump
-      @dump = Marshal.load dump
-    end
-    
     alias text_token push
     def begin_group kind; push :begin_group, kind end
     def end_group kind; push :end_group, kind end
diff --git a/lib/coderay/version.rb b/lib/coderay/version.rb
index 4b4f085..7ea3f70 100644
--- a/lib/coderay/version.rb
+++ b/lib/coderay/version.rb
@@ -1,3 +1,3 @@
 module CodeRay
-  VERSION = '1.1.0'
+  VERSION = '1.1.1'
 end
diff --git a/rake_helpers/ca.rb b/rake_helpers/ca.rb
deleted file mode 100644
index ba6fe1e..0000000
--- a/rake_helpers/ca.rb
+++ /dev/null
@@ -1,40 +0,0 @@
-#!c:/ruby/bin/rubyw

-# Hy-Ca 0.2 by murphy

-

-module Hy

-  def self.ca str

-    str.gsub! %r-^(\s*)(?://)(.*)?-, '\1/*\2*/'

-    str.gsub! %r-\s*/\*.*?\*/\n?-m, ''

-    str.gsub!(/<<(.*?)>>/m) do

-      begin

-        eval $1

-        ''

-      rescue Exception => boom

-        "<<\n#{boom}\n>>"

-      end

-    end

-    

-    str.gsub!(/\$([\w_]+)/m) do

-      begin

-        eval $1

-      rescue

-        ''

-      end

-    end

-

-    str

-  end

-end

-

-begin

-  if file = ENV['PATH_TRANSLATED']

-    puts "Content-Type: text/css"

-    puts

-    ca = File.read file

-  else

-    ca = ARGF.read

-  end

-  print Hy.ca(ca)

-rescue => boom

-  p boom

-end if __FILE__ == $0

diff --git a/rake_helpers/coderay_rdoc_template.rb b/rake_helpers/coderay_rdoc_template.rb
deleted file mode 100644
index 21a3231..0000000
--- a/rake_helpers/coderay_rdoc_template.rb
+++ /dev/null
@@ -1,636 +0,0 @@
-# This goes to /lib/ruby/1.8/rdoc/generators/template/html/
-module RDoc::Page
-
-FONTS = "Tahoma, Verdana, sans-serif"
-
-require 'rake_helpers/ca.rb'
-
-Hy.ca <<CA
-<<
-	BACKGROUND = '#322'
-
-	GOLD = '#e92'
-	DARK_GOLD = '#630'
-	FIRE = '#fc3'
-	MORE_FIRE = '#fe0'
-	LIGHT_BLUE = '#CCF'
-	DARK_BLUE = '#004'
-	FLARE = LIGHT_BLUE
-	DARK_GRAY = '#444'
-	PURPLE = '#d8f'
-	DARK_PURPLE = '#303'
-
-	LINK = LIGHT_BLUE
-	VISITED = '#ccc'
-	HOVER = '#f44'
-
-	BORDER = '3px double gray'
->>
-CA
-
-require 'coderay'
-
-STYLE = Hy.ca <<CSS + CodeRay::Encoders[:html]::CSS.new.stylesheet
-a { text-decoration: none; }
-a:link { color: $LINK }
-a:visited { color: $VISITED }
-a:hover, a:active, a:focus { color: $HOVER }
-
-body {
-  background-color: $BACKGROUND;
-  color: $GOLD;
-  margin: 0px;
-  font-family: %fonts%;
-}
-
-img { border: $BORDER; }
-
-hr {
-	background-color: none; color: none;
-	height: 3px;
-	border: 1px solid gray;
-	margin: 0.5em 0em;
-	padding: 0px;
-}
-
-tt { color: $LIGHT_BLUE; font-size: 1em; }
-
-.sectiontitle {
-	font-size: 1.2em;
-	font-weight: bold;
-	font-color: white;
-	border: $BORDER;
-	padding: 0.2em;
-	margin: 1em auto; margin-top: 3em;
-	text-align: center;
-	width: 75%;
-	background-color: $DARK_BLUE;
-}
-
-.attr-rw {
-  padding-left: 1em;
-  padding-right: 1em;
-  text-align: center;
-  color: silver;
-}
-
-.attr-name {
-  font-weight: bold;
-}
-
-.attr-desc {
-}
-
-.attr-value {
-  font-family: monospace;
-	color: $LIGHT_BLUE;
-	font-size: 1em;
-}
-
-.banner {
-	border-collapse: collapse;
-  font-size: small;
-  background: $DARK_PURPLE;
-  color: silver;
-  border: 0px;
-  border-bottom: $BORDER;
-  padding: 0.5em;
-  margin-bottom: 1em;
-}
-.xbanner table { border-collapse: collapse; }
-
-.banner td {
-  color: silver;
-	background-color: transparent;
-	padding: 0.2em 0.5em;
-}
-.banner td.ruby-chan {
-	vertical-align: bottom;
-	padding: 0px;
-	width: 1px;
-}
-
-.file-title-prefix { }
-
-td.file-title {
-  font-size: 140%;
-  font-weight: bold;
-  color: $PURPLE;
-}
-
-.dyn-source {
-  display: none;
-  margin: 0.5em;
-}
-
-.method {
-  margin-left: 1em;
-  margin-right: 1em;
-  margin-bottom: 1em;
-  border: 1px solid white;
-	color: $MORE_FIRE;
-	background: $DARK_PURPLE;
-}
-
-.description pre {
-	border: 1px solid gray;
-	background: $DARK_BLUE;
-	color: white;
-	padding: 0.5em;
-}
-
-.method .title {
-  font-family: monospace;
-  font-size: larger;
-  color: $PURPLE;
-  background: $DARK_GRAY;
-  border-bottom: $BORDER;
-  margin: 0px; padding: 0.1em 0.5em;
-}
-
-.method .description, .method .sourcecode {
-  margin: 0.2em 1em;
-}
-.method p {
-	color: $GOLD;
-}
-
-.description p, .sourcecode p {
-  margin-bottom: 0.5em;
-}
-
-.method .sourcecode p.source-link {
-  margin-top: 0.5em;
-  font-style: normal;
-}
-.arrow { font-size: larger; }
-
-.method .aka {
-  margin-top: 0.3em;
-  margin-left: 1em;
-  color: $FIRE;
-}
-
-#content {
-  margin: 2em; margin-top: 0px;
-}
-
-#description p {
-  margin-bottom: 0.5em;
-}
-
-h1 {
-	font-size: 1.5em;
-	font-weight: bold;
-	color: $FLARE;
-	border: $BORDER;
-	padding: 0.25em;
-	margin: 1em auto;
-	text-align: center;
-	width: 33%;
-	background-color: $DARK_BLUE;
-}
-h1 a:link, h1 a:visited { color: $FLARE }
-
-h2 {
-	margin-bottom: 0.5em;
-	margin-top: 2em;
-  font-size: 1.2em;
-  font-weight: bold;
-  color: $FIRE;
-}
-
-h3, h4, h5, h6 {
-	margin-bottom: 0.4em;
-	margin-top: 1.5em;
-	padding: 0px;
-  border: 0px;
-  color: $FIRE;
-  font-size: 1em;
-}
-
-.sourcecode > pre {
-	border: 1px solid silver;
-	background: #112;
-	color: white;
-	padding: 0.5em;
-}
-CSS
-
-XHTML_FRAMESET_PREAMBLE = #<?xml version="1.0" encoding="%charset%"?>
-%{<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Frameset//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-frameset.dtd">
-}
-
-XHTML_PREAMBLE =
-%{<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-}
-
-HEADER = XHTML_PREAMBLE + <<ENDHEADER
-<html>
-  <head>
-    <title>%title%</title>
-    <meta http-equiv="Content-Type" content="text/html; charset=%charset%" />
-    <link rel="stylesheet" href="%style_url%" type="text/css" media="screen" />
-
-    <script language="JavaScript" type="text/javascript">
-    // <![CDATA[
-
-        function toggleSource( id )
-        {
-          var elem
-          var link
-
-          if( document.getElementById )
-          {
-            elem = document.getElementById( id )
-            link = document.getElementById( "l_" + id )
-          }
-          else if ( document.all )
-          {
-            elem = eval( "document.all." + id )
-            link = eval( "document.all.l_" + id )
-          }
-          else
-            return false;
-
-          if( elem.style.display == "block" )
-          {
-            elem.style.display = "none"
-            link.innerHTML = "show source"
-          }
-          else
-          {
-            elem.style.display = "block"
-            link.innerHTML = "hide source"
-          }
-        }
-
-        function openCode( url )
-        {
-          window.open( url, "SOURCE_CODE", "width=400,height=400,scrollbars=yes" )
-        }
-      // ]]>
-    </script>
-  </head>
-
-  <body>
-ENDHEADER
-
-FILE_PAGE = <<HTML
-<table width="100%" border="0" cellpadding="0" cellspacing="0" class="banner">
-  <tr><td>
-    <table width="100%" border="0" cellpadding="0" cellspacing="0">
-    <tr>
-      <td class="file-title" colspan="2"><span class="file-title-prefix">File</span> %short_name%</td>
-		</tr>
-    <tr>
-    	<td>
-        <table border="0" cellspacing="0" cellpadding="2">
-          <tr>
-            <td>Path:</td>
-            <td>%full_path%
-IF:cvsurl
-				&nbsp;(<a href="%cvsurl%">CVS</a>)
-ENDIF:cvsurl
-            </td>
-          </tr>
-          <tr>
-            <td>Modified:</td>
-            <td>%dtm_modified%</td>
-          </tr>
-        </table>
-      </td></tr>
-    </table>
-  </td>
-  <td class="ruby-chan">
-	  <a href="http://rubychan.de"><img src="http://cycnus.de/ruby/pics/ruby-doc-chan.gif" alt="Ruby-Chan" style="border: 0px" /></a>
-  </td>
- </tr>
-</table><br />
-HTML
-
-###################################################################
-
-CLASS_PAGE = <<HTML
-<table width="100%" border="0" cellpadding="0" cellspacing="0" class="banner">
-	<tr>
-  	<td class="file-title"><span class="file-title-prefix">%classmod%</span> %full_name%</td>
-		<td rowspan="2" class="ruby-chan">
-	  <a href="http://rubychan.de"><img src="http://cycnus.de/ruby/pics/ruby-doc-chan.gif" alt="Ruby-Chan" style="border: 0px" /></a>
-		</td>
-	</tr>
-	<tr>
- 	 <td>
-    <table cellspacing="0" cellpadding="2">
-      <tr valign="top">
-        <td>In:</td>
-        <td>
-START:infiles
-HREF:full_path_url:full_path:
-IF:cvsurl
-&nbsp;(<a href="%cvsurl%">CVS</a>)
-ENDIF:cvsurl
-END:infiles
-        </td>
-      </tr>
-IF:parent
-    <tr>
-      <td>Parent:</td>
-      <td>
-IF:par_url
-        <a href="%par_url%">
-ENDIF:par_url
-%parent%
-IF:par_url
-         </a>
-ENDIF:par_url
-     </td>
-   </tr>
-ENDIF:parent
-         </table>
-        </td>
-        </tr>
-      </table>
-HTML
-
-###################################################################
-
-METHOD_LIST = <<HTML
-  <div id="content">
-IF:diagram
-  <table cellpadding="0" cellspacing="0" border="0" width="100%"><tr><td align="center">
-    %diagram%
-  </td></tr></table>
-ENDIF:diagram
-
-IF:description
-  <div class="description">%description%</div>
-ENDIF:description
-
-IF:requires
-  <div class="sectiontitle">Required Files</div>
-  <ul>
-START:requires
-  <li>HREF:aref:name:</li>
-END:requires
-  </ul>
-ENDIF:requires
-
-IF:toc
-  <div class="sectiontitle">Contents</div>
-  <ul>
-START:toc
-  <li><a href="#%href%">%secname%</a></li>
-END:toc
-  </ul>
-ENDIF:toc
-
-IF:methods
-  <div class="sectiontitle">Methods</div>
-  <ul>
-START:methods
-  <li>HREF:aref:name:</li>
-END:methods
-  </ul>
-ENDIF:methods
-
-IF:includes
-<div class="sectiontitle">Included Modules</div>
-<ul>
-START:includes
-  <li>HREF:aref:name:</li>
-END:includes
-</ul>
-ENDIF:includes
-
-START:sections
-IF:sectitle
-<div class="sectiontitle"><a nem="%secsequence%">%sectitle%</a></div>
-IF:seccomment
-<div class="description">
-%seccomment%
-</div>
-ENDIF:seccomment
-ENDIF:sectitle
-
-IF:classlist
-  <div class="sectiontitle">Classes and Modules</div>
-  %classlist%
-ENDIF:classlist
-
-IF:constants
-  <div class="sectiontitle">Constants</div>
-  <table border="0" cellpadding="5">
-START:constants
-  <tr valign="top">
-    <td class="attr-name">%name%</td>
-    <td>=</td>
-    <td class="attr-value">%value%</td>
-  </tr>
-IF:desc
-  <tr valign="top">
-    <td>&nbsp;</td>
-    <td colspan="2" class="attr-desc">%desc%</td>
-  </tr>
-ENDIF:desc
-END:constants
-  </table>
-ENDIF:constants
-
-IF:attributes
-  <div class="sectiontitle">Attributes</div>
-  <table border="0" cellpadding="5">
-START:attributes
-  <tr valign="top">
-    <td class="attr-rw">
-IF:rw
-[%rw%]
-ENDIF:rw
-    </td>
-    <td class="attr-name">%name%</td>
-    <td class="attr-desc">%a_desc%</td>
-  </tr>
-END:attributes
-  </table>
-ENDIF:attributes
-
-IF:method_list
-START:method_list
-IF:methods
-<div class="sectiontitle">%type% %category% methods</div>
-START:methods
-<div class="method">
-  <div class="title">
-IF:callseq
-    <a name="%aref%"></a><b>%callseq%</b>
-ENDIF:callseq
-IFNOT:callseq
-    <a name="%aref%"></a><b>%name%</b>%params%
-ENDIF:callseq
-IF:codeurl
-[ <a href="javascript:openCode("%codeurl%")">source</a> ]
-ENDIF:codeurl
-  </div>
-IF:m_desc
-  <div class="description">
-  %m_desc%
-  </div>
-ENDIF:m_desc
-IF:aka
-<div class="aka">
-  --- This method is also aliased as
-START:aka
-  <a href="%aref%">%name%</a>
-END:aka
-  ---
-</div>
-ENDIF:aka
-IF:sourcecode
-<div class="sourcecode">
-  <p class="source-link"><span class="arrow">&rarr;</span> <a href="javascript:toggleSource('%aref%_source')" id="l_%aref%_source">show source</a></p>
-  <div id="%aref%_source" class="dyn-source">
-%sourcecode%
-  </div>
-</div>
-ENDIF:sourcecode
-</div>
-END:methods
-ENDIF:methods
-END:method_list
-ENDIF:method_list
-END:sections
-</div>
-HTML
-
-FOOTER = <<ENDFOOTER
-  </body>
-</html>
-ENDFOOTER
-
-BODY = HEADER + <<ENDBODY
-  !INCLUDE! <!-- banner header -->
-
-  <div id="bodyContent">
-    #{METHOD_LIST}
-  </div>
-
-  #{FOOTER}
-ENDBODY
-
-########################## Source code ##########################
-
-SRC_PAGE = XHTML_PREAMBLE + <<HTML
-<html>
-<head><title>%title%</title>
-<meta http-equiv="Content-Type" content="text/html; charset=%charset%" />
-<style type="text/css">
-.ruby-comment    { color: green; font-style: italic }
-.ruby-constant   { color: #4433aa; font-weight: bold; }
-.ruby-identifier { color: #222222;  }
-.ruby-ivar       { color: #2233dd; }
-.ruby-keyword    { color: #3333FF; font-weight: bold }
-.ruby-node       { color: #777777; }
-.ruby-operator   { color: #111111;  }
-.ruby-regexp     { color: #662222; }
-.ruby-value      { color: #662222; font-style: italic }
-  .kw { color: #3333FF; font-weight: bold }
-  .cmt { color: green; font-style: italic }
-  .str { color: #662222; font-style: italic }
-  .re  { color: #662222; }
-</style>
-</head>
-<body bgcolor="white">
-<pre>%code%</pre>
-</body>
-</html>
-HTML
-
-########################## Index ################################
-
-FR_INDEX_BODY = <<HTML
-!INCLUDE!
-HTML
-
-FILE_INDEX = XHTML_PREAMBLE + <<HTML
-<html>
-<head><title>List</title>
-<meta http-equiv="Content-Type" content="text/html; charset=%charset%" />
-<style type="text/css">
-<!--
-#{ Hy.ca <<CA
-  body {
-    background-color: $DARK_PURPLE;
-    font-family: #{FONTS};
-    color: white;
-    margin: 0px;
-  }
-  .banner {
-    background: $DARK_BLUE;
-    color: $GOLD;
-    padding: 0em 0.2em;
-    border-bottom: $BORDER;
-    font-size: smaller;
-    font-weight: bold;
-    text-align: center;
-  }
-  .entries {
-    margin: 0.25em 0em 0em 0.5em;
-    font-size: 75%;
-  }
-  a { text-decoration: none; white-space: nowrap; }
-  a:link { color: $LINK; }
-  a:visited { color: $VISITED; }
-  a:hover, a:active, a:focus { color: $HOVER; }
-CA
-}
--->
-</style>
-<base target="docwin" />
-</head>
-<body>
-<div class="banner">%list_title%</div>
-<div class="entries">
-START:entries
-<a href="%href%">%name%</a><br />
-END:entries
-</div>
-</body></html>
-HTML
-
-CLASS_INDEX = FILE_INDEX
-METHOD_INDEX = FILE_INDEX
-
-INDEX = XHTML_FRAMESET_PREAMBLE + <<HTML
-<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
-<head>
-  <title>%title%</title>
-  <meta http-equiv="Content-Type" content="text/html; charset=%charset%" />
-</head>
-
-<frameset cols="20%,*">
-    <frameset rows="15%,35%,50%">
-        <frame src="fr_file_index.html"   title="Files" name="Files" />
-        <frame src="fr_class_index.html"  name="Classes" />
-        <frame src="fr_method_index.html" name="Methods" />
-    </frameset>
-IF:inline_source
-      <frame  src="%initial_page%" name="docwin" />
-ENDIF:inline_source
-IFNOT:inline_source
-    <frameset rows="80%,20%">
-      <frame  src="%initial_page%" name="docwin" />
-      <frame  src="blank.html" name="source" />
-    </frameset>
-ENDIF:inline_source
-    <noframes>
-          <body bgcolor="white">
-            Click <a href="html/index.html">here</a> for a non-frames
-            version of this page.
-          </body>
-    </noframes>
-</frameset>
-
-</html>
-HTML
-
-end
diff --git a/rake_helpers/html_coderay_generator.rb b/rake_helpers/html_coderay_generator.rb
deleted file mode 100644
index 3c77705..0000000
--- a/rake_helpers/html_coderay_generator.rb
+++ /dev/null
@@ -1,1517 +0,0 @@
-# We're responsible for generating all the HTML files
-# from the object tree defined in code_objects.rb. We
-# generate:
-#
-# [files]   an html file for each input file given. These
-#           input files appear as objects of class
-#           TopLevel
-#
-# [classes] an html file for each class or module encountered.
-#           These classes are not grouped by file: if a file
-#           contains four classes, we'll generate an html
-#           file for the file itself, and four html files 
-#           for the individual classes. 
-#
-# [indices] we generate three indices for files, classes,
-#           and methods. These are displayed in a browser
-#           like window with three index panes across the
-#           top and the selected description below
-#
-# Method descriptions appear in whatever entity (file, class,
-# or module) that contains them.
-#
-# We generate files in a structure below a specified subdirectory,
-# normally +doc+.
-#
-#  opdir
-#     |
-#     |___ files
-#     |       |__  per file summaries
-#     |
-#     |___ classes
-#             |__ per class/module descriptions
-#
-# HTML is generated using the Template class.
-#
-
-require 'ftools'
-
-require 'rdoc/options'
-require 'rdoc/template'
-require 'rdoc/markup/simple_markup'
-require 'rdoc/markup/simple_markup/to_html'
-require 'cgi'
-
-module Generators
-
-  # Name of sub-direcories that hold file and class/module descriptions
-
-  FILE_DIR  = "files"
-  CLASS_DIR = "classes"
-  CSS_NAME  = "rdoc-style.css"
-  
-
-  ##
-  # Build a hash of all items that can be cross-referenced.
-  # This is used when we output required and included names: 
-  # if the names appear in this hash, we can generate
-  # an html cross reference to the appropriate description.
-  # We also use this when parsing comment blocks: any decorated 
-  # words matching an entry in this list are hyperlinked.
-
-  class AllReferences
-    @@refs = {}
-    
-    def AllReferences::reset
-      @@refs = {}
-    end
-
-    def AllReferences.add(name, html_class)
-      @@refs[name] = html_class
-    end
-
-    def AllReferences.[](name)
-      @@refs[name]
-    end
-
-    def AllReferences.keys
-      @@refs.keys
-    end
-  end
-
-
-  ##
-  # Subclass of the SM::ToHtml class that supports looking
-  # up words in the AllReferences list. Those that are
-  # found (like AllReferences in this comment) will
-  # be hyperlinked
-
-  class HyperlinkHtml < SM::ToHtml
-    # We need to record the html path of our caller so we can generate
-    # correct relative paths for any hyperlinks that we find
-    def initialize(from_path, context)
-      super()
-      @from_path = from_path
-
-      @parent_name = context.parent_name
-      @parent_name += "::" if @parent_name
-      @context = context
-    end
-
-    # We're invoked when any text matches the CROSSREF pattern
-    # (defined in MarkUp). If we fine the corresponding reference,
-    # generate a hyperlink. If the name we're looking for contains
-    # no punctuation, we look for it up the module/class chain. For
-    # example, HyperlinkHtml is found, even without the Generators::
-    # prefix, because we look for it in module Generators first.
-
-    def handle_special_CROSSREF(special)
-      name = special.text
-      if name[0,1] == '#'
-        lookup = name[1..-1]
-        name = lookup unless Options.instance.show_hash
-      else
-        lookup = name
-      end
-
-      if /([A-Z].*)[.\#](.*)/ =~ lookup
-        container = $1
-        method = $2
-        ref = @context.find_symbol(container, method)
-      else
-        ref = @context.find_symbol(lookup)
-      end
-
-      if ref and ref.document_self
-        "<a href=\"#{ref.as_href(@from_path)}\">#{name}</a>"
-      else
-        name
-      end
-    end
-
-
-    # Generate a hyperlink for url, labeled with text. Handle the
-    # special cases for img: and link: described under handle_special_HYPEDLINK
-    def gen_url(url, text)
-      if url =~ /([A-Za-z]+):(.*)/
-        type = $1
-        path = $2
-      else
-        type = "http"
-        path = url
-        url  = "http://#{url}"
-      end
-
-      if type == "link"
-        if path[0,1] == '#'     # is this meaningful?
-          url = path
-        else
-          url = HTMLGenerator.gen_url(@from_path, path)
-        end
-      end
-
-      if (type == "http" || type == "link") && 
-          url =~ /\.(gif|png|jpg|jpeg|bmp)$/
-
-        "<img src=\"#{url}\">"
-      else
-        "<a href=\"#{url}\">#{text.sub(%r{^#{type}:/*}, '')}</a>"
-      end
-    end
-
-    # And we're invoked with a potential external hyperlink mailto:
-    # just gets inserted. http: links are checked to see if they
-    # reference an image. If so, that image gets inserted using an
-    # <img> tag. Otherwise a conventional <a href> is used.  We also
-    # support a special type of hyperlink, link:, which is a reference
-    # to a local file whose path is relative to the --op directory.
-
-    def handle_special_HYPERLINK(special)
-      url = special.text
-      gen_url(url, url)
-    end
-
-    # HEre's a hypedlink where the label is different to the URL
-    #  <label>[url]
-    #
-    
-    def handle_special_TIDYLINK(special)
-      text = special.text
-#      unless text =~ /(\S+)\[(.*?)\]/
-      unless text =~ /\{(.*?)\}\[(.*?)\]/ or text =~ /(\S+)\[(.*?)\]/ 
-        return text
-      end
-      label = $1
-      url   = $2
-      gen_url(url, label)
-    end
-
-  end
-
-
-  
-  #####################################################################
-  #
-  # Handle common markup tasks for the various Html classes
-  #
-
-  module MarkUp
-
-    # Convert a string in markup format into HTML. We keep a cached
-    # SimpleMarkup object lying around after the first time we're
-    # called per object.
-
-    def markup(str, remove_para=false)
-      return '' unless str
-      unless defined? @markup
-        @markup = SM::SimpleMarkup.new
-
-        # class names, variable names, file names, or instance variables
-        @markup.add_special(/(
-                               \b([A-Z]\w*(::\w+)*[.\#]\w+)  #    A::B.meth
-                             | \b([A-Z]\w+(::\w+)*)       #    A::B..
-                             | \#\w+[!?=]?                #    #meth_name 
-                             | \b\w+([_\/\.]+\w+)+[!?=]?  #    meth_name
-                             )/x, 
-                            :CROSSREF)
-
-        # external hyperlinks
-        @markup.add_special(/((link:|https?:|mailto:|ftp:|www\.)\S+\w)/, :HYPERLINK)
-
-        # and links of the form  <text>[<url>]
-        @markup.add_special(/(((\{.*?\})|\b\S+?)\[\S+?\.\S+?\])/, :TIDYLINK)
-#        @markup.add_special(/\b(\S+?\[\S+?\.\S+?\])/, :TIDYLINK)
-
-      end
-      unless defined? @html_formatter
-        @html_formatter = HyperlinkHtml.new(self.path, self)
-      end
-
-      # Convert leading comment markers to spaces, but only
-      # if all non-blank lines have them
-
-      if str =~ /^(?>\s*)[^\#]/
-        content = str
-      else
-        content = str.gsub(/^\s*(#+)/)  { $1.tr('#',' ') }
-      end
-
-      res = @markup.convert(content, @html_formatter)
-      if remove_para
-        res.sub!(/^<p>/, '')
-        res.sub!(/<\/p>$/, '')
-      end
-      res
-    end
-
-    # Qualify a stylesheet URL; if if +css_name+ does not begin with '/' or
-    # 'http[s]://', prepend a prefix relative to +path+. Otherwise, return it
-    # unmodified.
-
-    def style_url(path, css_name=nil)
-#      $stderr.puts "style_url( #{path.inspect}, #{css_name.inspect} )"
-      css_name ||= CSS_NAME
-      if %r{^(https?:/)?/} =~ css_name
-        return css_name
-      else
-        return HTMLGenerator.gen_url(path, css_name)
-      end
-    end
-
-    # Build a webcvs URL with the given 'url' argument. URLs with a '%s' in them
-    # get the file's path sprintfed into them; otherwise they're just catenated
-    # together.
-
-    def cvs_url(url, full_path)
-      if /%s/ =~ url
-        return sprintf( url, full_path )
-      else
-        return url + full_path
-      end
-    end
-  end
-
-
-  #####################################################################
-  #
-  # A Context is built by the parser to represent a container: contexts
-  # hold classes, modules, methods, require lists and include lists.
-  # ClassModule and TopLevel are the context objects we process here
-  # 
-  class ContextUser
-
-    include MarkUp
-
-    attr_reader :context
-    
-    def initialize(context, options)
-      @context = context
-      @options = options
-    end
-      
-    # convenience method to build a hyperlink
-    def href(link, cls, name)
-      %{<a href="#{link}" class="#{cls}">#{name}</a>} #"
-    end
-
-    # return a reference to outselves to be used as an href=
-    # the form depends on whether we're all in one file
-    # or in multiple files
-
-    def as_href(from_path)
-      if @options.all_one_file
-        "#" + path
-      else
-        HTMLGenerator.gen_url(from_path, path)
-      end
-    end
-
-    # Create a list of HtmlMethod objects for each method
-    # in the corresponding context object. If the @options.show_all
-    # variable is set (corresponding to the <tt>--all</tt> option,
-    # we include all methods, otherwise just the public ones.
-
-    def collect_methods
-      list = @context.method_list
-      unless @options.show_all
-        list = list.find_all {|m| m.visibility == :public || m.force_documentation }
-      end
-      @methods = list.collect {|m| HtmlMethod.new(m, self, @options) }
-    end
-
-    # Build a summary list of all the methods in this context
-    def build_method_summary_list(path_prefix="")
-      collect_methods unless @methods
-      meths = @methods.sort
-      res = []
-      meths.each do |meth|
-        res << {
-          "name" => CGI.escapeHTML(meth.name),
-          "aref" => "#{path_prefix}\##{meth.aref}" 
-        }
-      end
-      res
-    end
-
-
-    # Build a list of aliases for which we couldn't find a
-    # corresponding method
-    def build_alias_summary_list(section)
-      values = []
-      @context.aliases.each do |al|
-        next unless al.section == section
-        res = {
-          'old_name' => al.old_name,
-          'new_name' => al.new_name,
-        }
-        if al.comment && !al.comment.empty?
-          res['desc'] = markup(al.comment, true)
-        end
-        values << res
-      end
-      values
-    end
-    
-    # Build a list of constants
-    def build_constants_summary_list(section)
-      values = []
-      @context.constants.each do |co|
-        next unless co.section == section
-        res = {
-          'name'  => co.name,
-          'value' => CGI.escapeHTML(co.value)
-        }
-        res['desc'] = markup(co.comment, true) if co.comment && !co.comment.empty?
-        values << res
-      end
-      values
-    end
-    
-    def build_requires_list(context)
-      potentially_referenced_list(context.requires) {|fn| [fn + ".rb"] }
-    end
-
-    def build_include_list(context)
-      potentially_referenced_list(context.includes)
-    end
-
-    # Build a list from an array of <i>Htmlxxx</i> items. Look up each
-    # in the AllReferences hash: if we find a corresponding entry,
-    # we generate a hyperlink to it, otherwise just output the name.
-    # However, some names potentially need massaging. For example,
-    # you may require a Ruby file without the .rb extension,
-    # but the file names we know about may have it. To deal with
-    # this, we pass in a block which performs the massaging,
-    # returning an array of alternative names to match
-
-    def potentially_referenced_list(array)
-      res = []
-      array.each do |i|
-        ref = AllReferences[i.name] 
-#         if !ref
-#           container = @context.parent
-#           while !ref && container
-#             name = container.name + "::" + i.name
-#             ref = AllReferences[name] 
-#             container = container.parent
-#           end
-#         end
-
-        ref = @context.find_symbol(i.name)
-        ref = ref.viewer if ref
-
-        if !ref && block_given?
-          possibles = yield(i.name)
-          while !ref and !possibles.empty?
-            ref = AllReferences[possibles.shift]
-          end
-        end
-        h_name = CGI.escapeHTML(i.name)
-        if ref and ref.document_self
-          path = url(ref.path)
-          res << { "name" => h_name, "aref" => path }
-        else
-          res << { "name" => h_name }
-        end
-      end
-      res
-    end
-
-    # Build an array of arrays of method details. The outer array has up
-    # to six entries, public, private, and protected for both class
-    # methods, the other for instance methods. The inner arrays contain
-    # a hash for each method
-
-    def build_method_detail_list(section)
-      outer = []
-
-      methods = @methods.sort
-      for singleton in [true, false]
-        for vis in [ :public, :protected, :private ] 
-          res = []
-          methods.each do |m|
-            if m.section == section and
-                m.document_self and 
-                m.visibility == vis and 
-                m.singleton == singleton
-              row = {}
-              if m.call_seq
-                row["callseq"] = m.call_seq.gsub(/->/, '&rarr;')
-              else
-                row["name"]        = CGI.escapeHTML(m.name)
-                row["params"]      = m.params
-              end
-              desc = m.description.strip
-              row["m_desc"]      = desc unless desc.empty?
-              row["aref"]        = m.aref
-              row["visibility"]  = m.visibility.to_s
-
-              alias_names = []
-              m.aliases.each do |other|
-                if other.viewer   # won't be if the alias is private
-                  alias_names << {
-                    'name' => other.name,
-                    'aref'  => other.viewer.as_href(path)
-                  } 
-                end
-              end
-              unless alias_names.empty?
-                row["aka"] = alias_names
-              end
-
-              if @options.inline_source
-                code = m.source_code
-                row["sourcecode"] = code if code
-              else
-                code = m.src_url
-                if code
-                  row["codeurl"] = code
-                  row["imgurl"]  = m.img_url
-                end
-              end
-              res << row
-            end
-          end
-          if res.size > 0 
-            outer << {
-              "type"    => vis.to_s.capitalize,
-              "category"    => singleton ? "Class" : "Instance",
-              "methods" => res
-            }
-          end
-        end
-      end
-      outer
-    end
-
-    # Build the structured list of classes and modules contained
-    # in this context. 
-
-    def build_class_list(level, from, section, infile=nil)
-      res = ""
-      prefix = "&nbsp;&nbsp;::" * level;
-
-      from.modules.sort.each do |mod|
-        next unless mod.section == section
-        next if infile && !mod.defined_in?(infile)
-        if mod.document_self
-          res << 
-            prefix <<
-            "Module " <<
-            href(url(mod.viewer.path), "link", mod.full_name) <<
-            "<br />\n" <<
-            build_class_list(level + 1, mod, section, infile)
-        end
-      end
-
-      from.classes.sort.each do |cls|
-        next unless cls.section == section
-        next if infile && !cls.defined_in?(infile)
-        if cls.document_self
-          res      <<
-            prefix << 
-            "Class " <<
-            href(url(cls.viewer.path), "link", cls.full_name) <<
-            "<br />\n" <<
-            build_class_list(level + 1, cls, section, infile)
-        end
-      end
-
-      res
-    end
-    
-    def url(target)
-      HTMLGenerator.gen_url(path, target)
-    end
-
-    def aref_to(target)
-      if @options.all_one_file
-        "#" + target
-      else
-        url(target)
-      end
-    end
-
-    def document_self
-      @context.document_self
-    end
-
-    def diagram_reference(diagram)
-      res = diagram.gsub(/((?:src|href)=")(.*?)"/) {
-        $1 + url($2) + '"'
-      }
-      res
-    end
-
-
-    # Find a symbol in ourselves or our parent
-    def find_symbol(symbol, method=nil)
-      res = @context.find_symbol(symbol, method)
-      if res
-        res = res.viewer
-      end
-      res
-    end
-
-    # create table of contents if we contain sections
-      
-    def add_table_of_sections
-      toc = []
-      @context.sections.each do |section|
-        if section.title
-          toc << {
-            'secname' => section.title,
-            'href'    => section.sequence
-          }
-        end
-      end
-      
-      @values['toc'] = toc unless toc.empty?
-    end
-
-
-  end
-
-  #####################################################################
-  #
-  # Wrap a ClassModule context
-
-  class HtmlClass < ContextUser
-
-    attr_reader :path
-
-    def initialize(context, html_file, prefix, options)
-      super(context, options)
-
-      @html_file = html_file
-      @is_module = context.is_module?
-      @values    = {}
-
-      context.viewer = self
-
-      if options.all_one_file
-        @path = context.full_name
-      else
-        @path = http_url(context.full_name, prefix)
-      end
-
-      collect_methods
-
-      AllReferences.add(name, self)
-    end
-
-    # return the relative file name to store this class in,
-    # which is also its url
-    def http_url(full_name, prefix)
-      path = full_name.dup
-      if path['<<']
-        path.gsub!(/<<\s*(\w*)/) { "from-#$1" }
-      end
-      File.join(prefix, path.split("::")) + ".html"
-    end
-
-
-    def name
-      @context.full_name
-    end
-
-    def parent_name
-      @context.parent.full_name
-    end
-
-    def index_name
-      name
-    end
-
-    def write_on(f)
-      value_hash
-      template = TemplatePage.new(RDoc::Page::BODY,
-                                      RDoc::Page::CLASS_PAGE,
-                                      RDoc::Page::METHOD_LIST)
-      template.write_html_on(f, @values)
-    end
-
-    def value_hash
-      class_attribute_values
-      add_table_of_sections
-
-      @values["charset"] = @options.charset
-      @values["style_url"] = style_url(path, @options.css)
-
-      d = markup(@context.comment)
-      @values["description"] = d unless d.empty?
-
-      ml = build_method_summary_list
-      @values["methods"] = ml unless ml.empty?
-
-      il = build_include_list(@context)
-      @values["includes"] = il unless il.empty?
-
-      @values["sections"] = @context.sections.map do |section|
-
-        secdata = {
-          "sectitle" => section.title,
-          "secsequence" => section.sequence,
-          "seccomment" => markup(section.comment)
-        }
-
-        al = build_alias_summary_list(section)
-        secdata["aliases"] = al unless al.empty?
-        
-        co = build_constants_summary_list(section)
-        secdata["constants"] = co unless co.empty?
-        
-        al = build_attribute_list(section)
-        secdata["attributes"] = al unless al.empty?
-        
-        cl = build_class_list(0, @context, section)
-        secdata["classlist"] = cl unless cl.empty?
-        
-        mdl = build_method_detail_list(section)
-        secdata["method_list"] = mdl unless mdl.empty?
-
-        secdata
-      end
-
-      @values
-    end
-
-    def build_attribute_list(section)
-      atts = @context.attributes.sort
-      res = []
-      atts.each do |att|
-        next unless att.section == section
-        if att.visibility == :public || @options.show_all
-          entry = {
-            "name"   => CGI.escapeHTML(att.name), 
-            "rw"     => att.rw, 
-            "a_desc" => markup(att.comment, true)
-          }
-          unless att.visibility == :public
-            entry["rw"] << "-"
-          end
-          res << entry
-        end
-      end
-      res
-    end
-
-    def class_attribute_values
-      h_name = CGI.escapeHTML(name)
-
-      @values["classmod"]  = @is_module ? "Module" : "Class"
-      @values["title"]     = "#{@values['classmod']}: #{h_name}"
-
-      c = @context
-      c = c.parent while c and !c.diagram
-      if c && c.diagram
-        @values["diagram"] = diagram_reference(c.diagram)
-      end
-
-      @values["full_name"] = h_name
-
-      parent_class = @context.superclass
-
-      if parent_class
-        @values["parent"] = CGI.escapeHTML(parent_class)
-
-        if parent_name
-          lookup = parent_name + "::" + parent_class
-        else
-          lookup = parent_class
-        end
-
-        parent_url = AllReferences[lookup] || AllReferences[parent_class]
-
-        if parent_url and parent_url.document_self
-          @values["par_url"] = aref_to(parent_url.path)
-        end
-      end
-
-      files = []
-      @context.in_files.each do |f|
-        res = {}
-        full_path = CGI.escapeHTML(f.file_absolute_name)
-
-        res["full_path"]     = full_path
-        res["full_path_url"] = aref_to(f.viewer.path) if f.document_self
-
-        if @options.webcvs
-          res["cvsurl"] = cvs_url( @options.webcvs, full_path )
-        end
-
-        files << res
-      end
-
-      @values['infiles'] = files
-    end
-
-    def <=>(other)
-      self.name <=> other.name
-    end
-
-  end
-
-  #####################################################################
-  #
-  # Handles the mapping of a file's information to HTML. In reality,
-  # a file corresponds to a +TopLevel+ object, containing modules,
-  # classes, and top-level methods. In theory it _could_ contain
-  # attributes and aliases, but we ignore these for now.
-
-  class HtmlFile < ContextUser
-
-    attr_reader :path
-    attr_reader :name
-
-    def initialize(context, options, file_dir)
-      super(context, options)
-
-      @values = {}
-
-      if options.all_one_file
-        @path = filename_to_label
-      else
-        @path = http_url(file_dir)
-      end
-
-      @name = @context.file_relative_name
-
-      collect_methods
-      AllReferences.add(name, self)
-      context.viewer = self
-    end
-
-    def http_url(file_dir)
-      File.join(file_dir, @context.file_relative_name.tr('.', '_')) +
-        ".html"
-    end
-
-    def filename_to_label
-      @context.file_relative_name.gsub(/%|\/|\?|\#/) {|s| '%' + ("%x" % s[0]) }
-    end
-
-    def index_name
-      name
-    end
-
-    def parent_name
-      nil
-    end
-
-    def value_hash
-      file_attribute_values
-      add_table_of_sections
-
-      @values["charset"]   = @options.charset
-      @values["href"]      = path
-      @values["style_url"] = style_url(path, @options.css)
-
-      if @context.comment
-        d = markup(@context.comment)
-        @values["description"] = d if d.size > 0
-      end
-
-      ml = build_method_summary_list
-      @values["methods"] = ml unless ml.empty?
-
-      il = build_include_list(@context)
-      @values["includes"] = il unless il.empty?
-
-      rl = build_requires_list(@context)
-      @values["requires"] = rl unless rl.empty?
-
-      if @options.promiscuous
-        file_context = nil
-      else
-        file_context = @context
-      end
-
-
-      @values["sections"] = @context.sections.map do |section|
-
-        secdata = {
-          "sectitle" => section.title,
-          "secsequence" => section.sequence,
-          "seccomment" => markup(section.comment)
-        }
-
-        cl = build_class_list(0, @context, section, file_context)
-        @values["classlist"] = cl unless cl.empty?
-
-        mdl = build_method_detail_list(section)
-        secdata["method_list"] = mdl unless mdl.empty?
-
-        al = build_alias_summary_list(section)
-        secdata["aliases"] = al unless al.empty?
-        
-        co = build_constants_summary_list(section)
-        @values["constants"] = co unless co.empty?
-
-        secdata
-      end
-      
-      @values
-    end
-    
-    def write_on(f)
-      value_hash
-      template = TemplatePage.new(RDoc::Page::BODY,
-                                  RDoc::Page::FILE_PAGE,
-                                  RDoc::Page::METHOD_LIST)
-      template.write_html_on(f, @values)
-    end
-
-    def file_attribute_values
-      full_path = @context.file_absolute_name
-      short_name = File.basename(full_path)
-      
-      @values["title"] = CGI.escapeHTML("File: #{short_name}")
-
-      if @context.diagram
-        @values["diagram"] = diagram_reference(@context.diagram)
-      end
-
-      @values["short_name"]   = CGI.escapeHTML(short_name)
-      @values["full_path"]    = CGI.escapeHTML(full_path)
-      @values["dtm_modified"] = @context.file_stat.mtime.to_s
-
-      if @options.webcvs
-        @values["cvsurl"] = cvs_url( @options.webcvs, @values["full_path"] )
-      end
-    end
-
-    def <=>(other)
-      self.name <=> other.name
-    end
-  end
-
-  #####################################################################
-
-  class HtmlMethod
-    include MarkUp
-
-    attr_reader :context
-    attr_reader :src_url
-    attr_reader :img_url
-    attr_reader :source_code
-
-    @@seq = "M000000"
-
-    @@all_methods = []
-
-    def HtmlMethod::reset
-      @@all_methods = []
-    end
-
-    def initialize(context, html_class, options)
-      @context    = context
-      @html_class = html_class
-      @options    = options
-      @@seq       = @@seq.succ
-      @seq        = @@seq
-      @@all_methods << self
-
-      context.viewer = self
-
-      if (ts = @context.token_stream)
-        @source_code = markup_code(ts)
-        unless @options.inline_source
-          @src_url = create_source_code_file(@source_code)
-          @img_url = HTMLGenerator.gen_url(path, 'source.png')
-        end
-      end
-
-      AllReferences.add(name, self)
-    end
-    
-    # return a reference to outselves to be used as an href=
-    # the form depends on whether we're all in one file
-    # or in multiple files
-
-    def as_href(from_path)
-      if @options.all_one_file
-        "#" + path
-      else
-        HTMLGenerator.gen_url(from_path, path)
-      end
-    end
-
-    def name
-      @context.name
-    end
-
-    def section
-      @context.section
-    end
-
-    def index_name
-      "#{@context.name} (#{@html_class.name})"
-    end
-
-    def parent_name
-      if @context.parent.parent
-        @context.parent.parent.full_name
-      else
-        nil
-      end
-    end
-
-    def aref
-      @seq
-    end
-
-    def path
-      if @options.all_one_file
-        aref
-      else
-        @html_class.path + "#" + aref
-      end
-    end
-
-    def description
-      markup(@context.comment)
-    end
-
-    def visibility
-      @context.visibility
-    end
-
-    def singleton
-      @context.singleton
-    end
-
-    def call_seq
-      cs = @context.call_seq
-      if cs
-        cs.gsub(/\n/, "<br />\n")
-      else
-        nil
-      end
-    end
-
-    def params
-      # params coming from a call-seq in 'C' will start with the
-      # method name
-      p = @context.params
-      if p !~ /^\w/
-        p = @context.params.gsub(/\s*\#.*/, '')
-        p = p.tr("\n", " ").squeeze(" ")
-        p = "(" + p + ")" unless p[0] == ?(
-        
-        if (block = @context.block_params)
-         # If this method has explicit block parameters, remove any
-         # explicit &block
-
-         p.sub!(/,?\s*&\w+/, '')
-
-          block.gsub!(/\s*\#.*/, '')
-          block = block.tr("\n", " ").squeeze(" ")
-          if block[0] == ?(
-            block.sub!(/^\(/, '').sub!(/\)/, '')
-          end
-          p << " {|#{block.strip}| ...}"
-        end
-      end
-      CGI.escapeHTML(p)
-    end
-    
-    def create_source_code_file(code_body)
-      meth_path = @html_class.path.sub(/\.html$/, '.src')
-      File.makedirs(meth_path)
-      file_path = File.join(meth_path, @seq) + ".html"
-
-      template = TemplatePage.new(RDoc::Page::SRC_PAGE)
-      File.open(file_path, "w") do |f|
-        values = {
-          'title'     => CGI.escapeHTML(index_name),
-          'code'      => code_body,
-          'style_url' => style_url(file_path, @options.css),
-          'charset'   => @options.charset
-        }
-        template.write_html_on(f, values)
-      end
-      HTMLGenerator.gen_url(path, file_path)
-    end
-
-    def HtmlMethod.all_methods
-      @@all_methods
-    end
-
-    def <=>(other)
-      @context <=> other.context
-    end
-
-    ##
-    # Given a sequence of source tokens, mark up the source code
-    # to make it look purty.
-    def old_markup_code(tokens)
-      src = ""
-      tokens.each do |t|
-        next unless t
-        #    p t.class
-#        style = STYLE_MAP[t.class]
-        style = case t
-                when RubyToken::TkCONSTANT then "ruby-constant"
-                when RubyToken::TkKW       then "ruby-keyword kw"
-                when RubyToken::TkIVAR     then "ruby-ivar"
-                when RubyToken::TkOp       then "ruby-operator"
-                when RubyToken::TkId       then "ruby-identifier"
-                when RubyToken::TkNode     then "ruby-node"
-                when RubyToken::TkCOMMENT  then "ruby-comment cmt"
-                when RubyToken::TkREGEXP   then "ruby-regexp re"
-                when RubyToken::TkSTRING   then "ruby-value str"
-                when RubyToken::TkVal      then "ruby-value"
-                else
-                    nil
-                end
-
-        text = CGI.escapeHTML(t.text)
-
-        if style
-          src << "<span class=\"#{style}\">#{text}</span>"
-        else
-          src << text
-        end
-      end
-
-      add_line_numbers(src) if Options.instance.include_line_numbers
-      src
-    end
-
-    require 'coderay'
-    CodeRay::Scanners.load_all
-    CodeRay::Encoders.load_all
-    CodeRay::Styles.load_all
-
-    def markup_code(tokens)
-      code = tokens.map { |t| t.text }.join
-      options = {
-        :css => :class,
-        :line_numbers_start => code[/\A.*?, line (\d+)/,1].to_i - 1,
-        :bold_every => :no_bolding,
-      }
-      options[:line_numbers] = nil unless Options.instance.include_line_numbers
-      CodeRay.scan(code, :ruby).div(options)
-    end
-
-    # we rely on the fact that the first line of a source code
-    # listing has 
-    #    # File xxxxx, line dddd
-
-    def add_line_numbers(src)
-      if src =~ /\A.*, line (\d+)/
-        first = $1.to_i - 1
-        last  = first + src.count("\n")
-        size = last.to_s.length
-        real_fmt = "%#{size}d: "
-        fmt = " " * (size+2)
-        src.gsub!(/^/) do
-          res = sprintf(fmt, first) 
-          first += 1
-          fmt = real_fmt
-          res
-        end
-      end
-    end
-
-    def document_self
-      @context.document_self
-    end
-
-    def aliases
-      @context.aliases
-    end
-
-    def find_symbol(symbol, method=nil)
-      res = @context.parent.find_symbol(symbol, method)
-      if res
-        res = res.viewer
-      end
-      res
-    end
-  end
-
-  #####################################################################
-        
-  class HTMLGenerator
-
-    include MarkUp
-
-    ##
-    # convert a target url to one that is relative to a given
-    # path
-    
-    def HTMLGenerator.gen_url(path, target)
-      from          = File.dirname(path)
-      to, to_file   = File.split(target)
-      
-      from = from.split("/")
-      to   = to.split("/")
-      
-      while from.size > 0 and to.size > 0 and from[0] == to[0]
-        from.shift
-        to.shift
-      end
-      
-      from.fill("..")
-      from.concat(to)
-      from << to_file
-      File.join(*from)
-    end
-
-    # Generators may need to return specific subclasses depending
-    # on the options they are passed. Because of this
-    # we create them using a factory
-
-    def HTMLGenerator.for(options)
-      AllReferences::reset
-      HtmlMethod::reset
-
-      if options.all_one_file
-        HTMLGeneratorInOne.new(options)
-      else
-        HTMLGenerator.new(options)
-      end
-    end
-
-    class <<self
-      protected :new
-    end
-
-    # Set up a new HTML generator. Basically all we do here is load
-    # up the correct output temlate
-
-    def initialize(options) #:not-new:
-      @options    = options
-      load_html_template
-    end
-
-
-    ##
-    # Build the initial indices and output objects
-    # based on an array of TopLevel objects containing
-    # the extracted information. 
-
-    def generate(toplevels)
-      @toplevels  = toplevels
-      @files      = []
-      @classes    = []
-
-      write_style_sheet
-      gen_sub_directories()
-      build_indices
-      generate_html
-    end
-
-    private
-
-    ##
-    # Load up the HTML template specified in the options.
-    # If the template name contains a slash, use it literally
-    #
-    def load_html_template
-      template = @options.template
-      unless template =~ %r{/|\\}
-        template = File.join("rdoc/generators/template",
-                             @options.generator.key, template)
-      end
-      require template
-      extend RDoc::Page
-    rescue LoadError
-      $stderr.puts "Could not find HTML template '#{template}'"
-      exit 99
-    end
-
-    ##
-    # Write out the style sheet used by the main frames
-    #
-    
-    def write_style_sheet
-      template = TemplatePage.new(RDoc::Page::STYLE)
-      unless @options.css
-        File.open(CSS_NAME, "w") do |f|
-          values = { "fonts" => RDoc::Page::FONTS }
-          template.write_html_on(f, values)
-        end
-      end
-    end
-
-    ##
-    # See the comments at the top for a description of the
-    # directory structure
-
-    def gen_sub_directories
-      File.makedirs(FILE_DIR, CLASS_DIR)
-    rescue 
-      $stderr.puts $!.message
-      exit 1
-    end
-
-    ##
-    # Generate:
-    #
-    # * a list of HtmlFile objects for each TopLevel object.
-    # * a list of HtmlClass objects for each first level
-    #   class or module in the TopLevel objects
-    # * a complete list of all hyperlinkable terms (file,
-    #   class, module, and method names)
-
-    def build_indices
-
-      @toplevels.each do |toplevel|
-        @files << HtmlFile.new(toplevel, @options, FILE_DIR)
-      end
-
-      RDoc::TopLevel.all_classes_and_modules.each do |cls|
-        build_class_list(cls, @files[0], CLASS_DIR)
-      end
-    end
-
-    def build_class_list(from, html_file, class_dir)
-      @classes << HtmlClass.new(from, html_file, class_dir, @options)
-      from.each_classmodule do |mod|
-        build_class_list(mod, html_file, class_dir)
-      end
-    end
-
-    ##
-    # Generate all the HTML
-    #
-    def generate_html
-      # the individual descriptions for files and classes
-      gen_into(@files)
-      gen_into(@classes)
-      # and the index files
-      gen_file_index
-      gen_class_index
-      gen_method_index
-      gen_main_index
-      
-      # this method is defined in the template file
-      write_extra_pages if defined? write_extra_pages
-    end
-
-    def gen_into(list)
-      list.each do |item|
-        if item.document_self
-          op_file = item.path
-          File.makedirs(File.dirname(op_file))
-          File.open(op_file, "w") { |file| item.write_on(file) }
-        end
-      end
-
-    end
-
-    def gen_file_index
-      gen_an_index(@files, 'Files', 
-                   RDoc::Page::FILE_INDEX, 
-                   "fr_file_index.html")
-    end
-
-    def gen_class_index
-      gen_an_index(@classes, 'Classes',
-                   RDoc::Page::CLASS_INDEX,
-                   "fr_class_index.html")
-    end
-
-    def gen_method_index
-      gen_an_index(HtmlMethod.all_methods, 'Methods', 
-                   RDoc::Page::METHOD_INDEX,
-                   "fr_method_index.html")
-    end
-
-    
-    def gen_an_index(collection, title, template, filename)
-      template = TemplatePage.new(RDoc::Page::FR_INDEX_BODY, template)
-      res = []
-      collection.sort.each do |f|
-        if f.document_self
-          res << { "href" => f.path, "name" => f.index_name }
-        end
-      end
-
-      values = {
-        "entries"    => res,
-        'list_title' => CGI.escapeHTML(title),
-        'index_url'  => main_url,
-        'charset'    => @options.charset,
-        'style_url'  => style_url('', @options.css),
-      }
-
-      File.open(filename, "w") do |f|
-        template.write_html_on(f, values)
-      end
-    end
-
-    # The main index page is mostly a template frameset, but includes
-    # the initial page. If the <tt>--main</tt> option was given,
-    # we use this as our main page, otherwise we use the
-    # first file specified on the command line.
-
-    def gen_main_index
-      template = TemplatePage.new(RDoc::Page::INDEX)
-      File.open("index.html", "w") do |f|
-        values = {
-          "initial_page" => main_url,
-          'title'        => CGI.escapeHTML(@options.title),
-          'charset'      => @options.charset
-        }
-        if @options.inline_source
-          values['inline_source'] = true
-        end
-        template.write_html_on(f, values)
-      end
-    end
-
-    # return the url of the main page
-    def main_url
-      main_page = @options.main_page
-      ref = nil
-      if main_page
-        ref = AllReferences[main_page]
-        if ref
-          ref = ref.path
-        else
-          $stderr.puts "Could not find main page #{main_page}"
-        end
-      end
-
-      unless ref
-        for file in @files
-          if file.document_self
-            ref = file.path 
-            break
-          end
-        end
-      end
-
-      unless ref
-        $stderr.puts "Couldn't find anything to document"
-        $stderr.puts "Perhaps you've used :stopdoc: in all classes"
-        exit(1)
-      end
-
-      ref
-    end
-
-
-  end
-  HTML_CODERAYGenerator = HTMLGenerator
-
-
-  ######################################################################
-
-
-  class HTMLGeneratorInOne < HTMLGenerator
-
-    def initialize(*args)
-      super
-    end
-
-    ##
-    # Build the initial indices and output objects
-    # based on an array of TopLevel objects containing
-    # the extracted information. 
-
-    def generate(info)
-      @toplevels  = info
-      @files      = []
-      @classes    = []
-      @hyperlinks = {}
-
-      build_indices
-      generate_xml
-    end
-
-
-    ##
-    # Generate:
-    #
-    # * a list of HtmlFile objects for each TopLevel object.
-    # * a list of HtmlClass objects for each first level
-    #   class or module in the TopLevel objects
-    # * a complete list of all hyperlinkable terms (file,
-    #   class, module, and method names)
-
-    def build_indices
-
-      @toplevels.each do |toplevel|
-        @files << HtmlFile.new(toplevel, @options, FILE_DIR)
-      end
-
-      RDoc::TopLevel.all_classes_and_modules.each do |cls|
-        build_class_list(cls, @files[0], CLASS_DIR)
-      end
-    end
-
-    def build_class_list(from, html_file, class_dir)
-      @classes << HtmlClass.new(from, html_file, class_dir, @options)
-      from.each_classmodule do |mod|
-        build_class_list(mod, html_file, class_dir)
-      end
-    end
-
-    ##
-    # Generate all the HTML. For the one-file case, we generate
-    # all the information in to one big hash
-    #
-    def generate_xml
-      values = { 
-        'charset' => @options.charset,
-        'files'   => gen_into(@files),
-        'classes' => gen_into(@classes),
-        'title'        => CGI.escapeHTML(@options.title),
-      }
-      
-      # this method is defined in the template file
-      write_extra_pages if defined? write_extra_pages
-
-      template = TemplatePage.new(RDoc::Page::ONE_PAGE)
-
-      if @options.op_name
-        opfile = File.open(@options.op_name, "w")
-      else
-        opfile = $stdout
-      end
-      template.write_html_on(opfile, values)
-    end
-
-    def gen_into(list)
-      res = []
-      list.each do |item|
-        res << item.value_hash
-      end
-      res
-    end
-
-    def gen_file_index
-      gen_an_index(@files, 'Files')
-    end
-
-    def gen_class_index
-      gen_an_index(@classes, 'Classes')
-    end
-
-    def gen_method_index
-      gen_an_index(HtmlMethod.all_methods, 'Methods')
-    end
-
-    
-    def gen_an_index(collection, title)
-      res = []
-      collection.sort.each do |f|
-        if f.document_self
-          res << { "href" => f.path, "name" => f.index_name }
-        end
-      end
-
-      return {
-        "entries" => res,
-        'list_title' => title,
-        'index_url'  => main_url,
-      }
-    end
-
-  end
-end
diff --git a/rake_tasks/benchmark.rake b/rake_tasks/benchmark.rake
index 040951b..2e38b57 100644
--- a/rake_tasks/benchmark.rake
+++ b/rake_tasks/benchmark.rake
@@ -1,7 +1,6 @@
 desc 'Do a benchmark'
 task :benchmark do
-  ruby "-v"
-  ruby "-wIlib bench/bench.rb ruby div 3000 N5"
+  ruby 'bench/bench.rb ruby html 3000'
 end
 
 task :bench => :benchmark
diff --git a/rake_tasks/bundler.rake b/rake_tasks/bundler.rake
deleted file mode 100644
index 8de149d..0000000
--- a/rake_tasks/bundler.rake
+++ /dev/null
@@ -1,6 +0,0 @@
-begin

-  require 'bundler'

-  Bundler::GemHelper.install_tasks

-rescue LoadError

-  puts 'Please gem install bundler.'

-end

diff --git a/rake_helpers/code_statistics.rb b/rake_tasks/code_statistics.rb
index 0a2016b..0a2016b 100644
--- a/rake_helpers/code_statistics.rb
+++ b/rake_tasks/code_statistics.rb
diff --git a/rake_tasks/diff.rake b/rake_tasks/diff.rake
deleted file mode 100644
index f0af55a..0000000
--- a/rake_tasks/diff.rake
+++ /dev/null
@@ -1,93 +0,0 @@
-# A simple differ using svn. Handles externals.
-class Differ < Hash
-  
-  include Rake::DSL if defined? Rake::DSL
-  
-  def initialize path
-    @path = path
-    super 0
-  end
-  
-  def count key, value
-    self[key] += value
-    value
-  end
-  
-  FORMAT = '  %-30s %8d lines, %3d changes in %2d files'
-  
-  def scan(path = @path)
-    Dir.chdir path do
-      diff_file_name = 'diff'
-      if File.directory? 'diff'
-        diff_file_name = 'diff.diff'
-      end
-      system "svn diff > #{diff_file_name}"
-      if File.size? diff_file_name
-        puts FORMAT %
-          [
-            path,
-            count(:LOC, `wc -l #{diff_file_name}`.to_i),
-            count(:changes, `grep ^@@ #{diff_file_name} | wc -l`.to_i),
-            count(:files, `grep ^Index #{diff_file_name} | wc -l`.to_i),
-          ]
-      else
-        rm diff_file_name
-      end
-    end
-  end
-  
-  def scan_with_externals(path = @path)
-    scan path
-    `svn status`.scan(/^X\s*(.*)/) do |external,|
-      scan external
-    end
-  end
-  
-  def clean(path = @path)
-    Dir.chdir path do
-      rm 'diff' if File.file? 'diff'
-      rm 'diff.diff' if File.file? 'diff.diff'
-    end
-  end
-  
-  def clean_with_externals(path = @path)
-    clean path
-    `svn status`.scan(/^X\s*(.*)/) do |external,|
-      clean external
-    end
-  end
-  
-  def differences?
-    self[:LOC] > 0
-  end
-  
-  def inspect
-    FORMAT %
-      [ 'Total', self[:LOC], self[:changes], self[:files] ]
-  end
-  
-end
-
-namespace :diff do
-  
-  desc 'Make a diff and print a summary'
-  task :summary do
-    differ = Differ.new '.'
-    differ.scan_with_externals
-    if differ.empty?
-      puts 'No differences found.'
-    else
-      p differ
-    end
-  end
-  
-  desc 'Remove all diffs'
-  task :clean do
-    differ = Differ.new '.'
-    differ.clean_with_externals
-  end
-  
-end
-
-desc 'Make a diff and print a summary'
-task :diff => 'diff:summary'
diff --git a/rake_tasks/documentation.rake b/rake_tasks/documentation.rake
index d555022..4f7cef7 100644
--- a/rake_tasks/documentation.rake
+++ b/rake_tasks/documentation.rake
@@ -15,18 +15,9 @@ Rake::RDocTask.new :doc do |rd|
   rd.title = 'CodeRay Documentation'

   

   rd.options << '--line-numbers' << '--tab-width' << '2'

-  # rd.options << '--fmt' << ENV.fetch('format', 'html_coderay')

-  # require 'pathname'

-  # template = File.join ROOT, 'rake_helpers', 'coderay_rdoc_template.rb'

-  # rd.template = Pathname.new(template).expand_path.to_s

   

   rd.main = 'README_INDEX.rdoc'

   rd.rdoc_files.add 'README_INDEX.rdoc'

   rd.rdoc_files.add Dir['lib']

   rd.rdoc_dir = 'doc'

 end if defined? Rake::RDocTask

-

-desc 'Copy the documentation over to the CodeRay website'

-task :copy_doc do

-  cp_r 'doc/.', '../../rails/coderay/public/doc'

-end

diff --git a/rake_tasks/ruby-versions.rake b/rake_tasks/ruby-versions.rake
deleted file mode 100644
index af408ff..0000000
--- a/rake_tasks/ruby-versions.rake
+++ /dev/null
@@ -1,10 +0,0 @@
-task 'ruby:version' do
-  puts
-  if defined? RUBY_DESCRIPTION
-    ruby_version = RUBY_DESCRIPTION
-  else
-    ruby_version = "ruby #{RUBY_VERSION} (#{RUBY_RELEASE_DATE} patchlevel #{RUBY_PATCHLEVEL}) [#{RUBY_PLATFORM}]"
-  end
-  require './test/lib/term/ansicolor'
-  puts Term::ANSIColor.bold(Term::ANSIColor.green(ruby_version))
-end
\ No newline at end of file
diff --git a/rake_tasks/statistic.rake b/rake_tasks/statistic.rake
index 99de378..d30e9b1 100644
--- a/rake_tasks/statistic.rake
+++ b/rake_tasks/statistic.rake
@@ -1,6 +1,6 @@
 desc 'Report code statistics (LOC) from the application'

 task :stats do

-  require 'rake_helpers/code_statistics'

+  require './rake_tasks/code_statistics'

   CodeStatistics.new(

     ['Main', 'lib', /coderay.rb$/],

     ['CodeRay', 'lib/coderay/'],

diff --git a/rake_tasks/test.rake b/rake_tasks/test.rake
index a60699d..b15b999 100644
--- a/rake_tasks/test.rake
+++ b/rake_tasks/test.rake
@@ -1,9 +1,4 @@
 namespace :test do
-  desc 'run all sample tests'
-  task :samples do
-    ruby './sample/suite.rb'
-  end
-  
   desc 'run functional tests'
   task :functional do
     ruby './test/functional/suite.rb'
@@ -85,4 +80,3 @@ Please rename or remove it and run again to use the GitHub repository:
 end
 
 task :test => %w(test:functional test:units test:exe)
-task :samples => 'test:samples'
\ No newline at end of file
diff --git a/sample/README b/sample/README
deleted file mode 100644
index b0ab6e4..0000000
--- a/sample/README
+++ /dev/null
@@ -1 +0,0 @@
-These demos rely on Ruby 1.8.5, so the tests might fail on other versions.
\ No newline at end of file
diff --git a/sample/cache.expected b/sample/cache.expected
deleted file mode 100644
index f815e88..0000000
--- a/sample/cache.expected
+++ /dev/null
@@ -1,2 +0,0 @@
-test &lt;test&gt;
-test <span class="ta">&lt;test&gt;</span>
diff --git a/sample/cache.rb b/sample/cache.rb
deleted file mode 100644
index 0c0b847..0000000
--- a/sample/cache.rb
+++ /dev/null
@@ -1,12 +0,0 @@
-require 'coderay'

-

-html_encoder = CodeRay.encoder :html

-

-scanner = Hash.new do |h, lang|

-	h[lang] = CodeRay.scanner lang

-end

-

-for lang in [:ruby, :html]

-	tokens = scanner[lang].tokenize 'test <test>'

-	puts html_encoder.encode_tokens(tokens)

-end

diff --git a/sample/count.expected b/sample/count.expected
deleted file mode 100644
index 7f493b6..0000000
--- a/sample/count.expected
+++ /dev/null
@@ -1 +0,0 @@
-2 out of 4 tokens have the kind :integer.
diff --git a/sample/count.rb b/sample/count.rb
deleted file mode 100644
index bcb7c2d..0000000
--- a/sample/count.rb
+++ /dev/null
@@ -1,10 +0,0 @@
-require 'coderay'

-

-stats = CodeRay.encoder(:statistic)

-stats.encode("puts 17 + 4\n", :ruby)

-

-puts '%d out of %d tokens have the kind :integer.' % [

-	stats.type_stats[:integer].count,

-	stats.real_token_count

-]

-#-> 2 out of 4 tokens have the kind :integer.

diff --git a/sample/css.expected b/sample/css.expected
deleted file mode 100644
index 09709ff..0000000
--- a/sample/css.expected
+++ /dev/null
@@ -1,130 +0,0 @@
-.CodeRay {
-  background-color: #f8f8f8;
-  border: 1px solid silver;
-  font-family: 'Courier New', 'Terminal', monospace;
-  color: #000;
-}
-.CodeRay pre { margin: 0px }
-
-div.CodeRay { }
-
-span.CodeRay { white-space: pre; border: 0px; padding: 2px }
-
-table.CodeRay { border-collapse: collapse; width: 100%; padding: 2px }
-table.CodeRay td { padding: 2px 4px; vertical-align: top }
-
-.CodeRay .line_numbers, .CodeRay .no {
-  background-color: #def;
-  color: gray;
-  text-align: right;
-}
-.CodeRay .line_numbers tt { font-weight: bold }
-.CodeRay .line_numbers .highlighted { color: red }
-.CodeRay .no { padding: 0px 4px }
-.CodeRay .code { width: 100% }
-
-ol.CodeRay { font-size: 10pt }
-ol.CodeRay li { white-space: pre }
-
-.CodeRay .code pre { overflow: auto }
-
-.CodeRay .debug { color:white ! important; background:blue ! important; }
-
-.CodeRay .af { color:#00C }
-.CodeRay .an { color:#007 }
-.CodeRay .at { color:#f08 }
-.CodeRay .av { color:#700 }
-.CodeRay .aw { color:#C00 }
-.CodeRay .bi { color:#509; font-weight:bold }
-.CodeRay .c  { color:#888; }
-
-.CodeRay .ch { color:#04D }
-.CodeRay .ch .k { color:#04D }
-.CodeRay .ch .dl { color:#039 }
-
-.CodeRay .cl { color:#B06; font-weight:bold }
-.CodeRay .cm { color:#A08; font-weight:bold }
-.CodeRay .co { color:#036; font-weight:bold }
-.CodeRay .cr { color:#0A0 }
-.CodeRay .cv { color:#369 }
-.CodeRay .de { color:#B0B; }
-.CodeRay .df { color:#099; font-weight:bold }
-.CodeRay .di { color:#088; font-weight:bold }
-.CodeRay .dl { color:black }
-.CodeRay .do { color:#970 }
-.CodeRay .dt { color:#34b }
-.CodeRay .ds { color:#D42; font-weight:bold }
-.CodeRay .e  { color:#666; font-weight:bold }
-.CodeRay .en { color:#800; font-weight:bold }
-.CodeRay .er { color:#F00; background-color:#FAA }
-.CodeRay .ex { color:#C00; font-weight:bold }
-.CodeRay .fl { color:#60E; font-weight:bold }
-.CodeRay .fu { color:#06B; font-weight:bold }
-.CodeRay .gv { color:#d70; font-weight:bold }
-.CodeRay .hx { color:#058; font-weight:bold }
-.CodeRay .i  { color:#00D; font-weight:bold }
-.CodeRay .ic { color:#B44; font-weight:bold }
-
-.CodeRay .il { background: #ddd; color: black }
-.CodeRay .il .il { background: #ccc }
-.CodeRay .il .il .il { background: #bbb }
-.CodeRay .il .idl { background: #ddd; font-weight: bold; color: #666 }
-.CodeRay .idl { background-color: #bbb; font-weight: bold; color: #666; }
-
-.CodeRay .im { color:#f00; }
-.CodeRay .in { color:#B2B; font-weight:bold }
-.CodeRay .iv { color:#33B }
-.CodeRay .la { color:#970; font-weight:bold }
-.CodeRay .lv { color:#963 }
-.CodeRay .oc { color:#40E; font-weight:bold }
-.CodeRay .of { color:#000; font-weight:bold }
-.CodeRay .op { }
-.CodeRay .pc { color:#038; font-weight:bold }
-.CodeRay .pd { color:#369; font-weight:bold }
-.CodeRay .pp { color:#579; }
-.CodeRay .ps { color:#00C; font-weight:bold }
-.CodeRay .pt { color:#074; font-weight:bold }
-.CodeRay .r, .kw  { color:#080; font-weight:bold }
-
-.CodeRay .ke { color: #808; }
-.CodeRay .ke .dl { color: #606; }
-.CodeRay .ke .ch { color: #80f; }
-.CodeRay .vl { color: #088; }
-
-.CodeRay .rx { background-color:#fff0ff }
-.CodeRay .rx .k { color:#808 }
-.CodeRay .rx .dl { color:#404 }
-.CodeRay .rx .mod { color:#C2C }
-.CodeRay .rx .fu  { color:#404; font-weight: bold }
-
-.CodeRay .s { background-color:#fff0f0; color: #D20; }
-.CodeRay .s .s { background-color:#ffe0e0 }
-.CodeRay .s .s  .s { background-color:#ffd0d0 }
-.CodeRay .s .k { }
-.CodeRay .s .ch { color: #b0b; }
-.CodeRay .s .dl { color: #710; }
-
-.CodeRay .sh { background-color:#f0fff0; color:#2B2 }
-.CodeRay .sh .k { }
-.CodeRay .sh .dl { color:#161 }
-
-.CodeRay .sy { color:#A60 }
-.CodeRay .sy .k { color:#A60 }
-.CodeRay .sy .dl { color:#630 }
-
-.CodeRay .ta { color:#070 }
-.CodeRay .tf { color:#070; font-weight:bold }
-.CodeRay .ts { color:#D70; font-weight:bold }
-.CodeRay .ty { color:#339; font-weight:bold }
-.CodeRay .v  { color:#036 }
-.CodeRay .xt { color:#444 }
-
-.CodeRay .ins { background: #afa; }
-.CodeRay .del { background: #faa; }
-.CodeRay .chg { color: #aaf; background: #007; }
-.CodeRay .head { color: #f8f; background: #505 }
-
-.CodeRay .ins .ins { color: #080; font-weight:bold }
-.CodeRay .del .del { color: #800; font-weight:bold }
-.CodeRay .chg .chg { color: #66f; }
-.CodeRay .head .head { color: #f4f; }
diff --git a/sample/css.rb b/sample/css.rb
deleted file mode 100644
index 52e4bcc..0000000
--- a/sample/css.rb
+++ /dev/null
@@ -1,4 +0,0 @@
-require 'coderay'

-

-# print the default stylesheet for HTML codes

-puts CodeRay::Encoders[:html]::CSS.new.stylesheet

diff --git a/sample/div.expected b/sample/div.expected
deleted file mode 100644
index f28ede3..0000000
--- a/sample/div.expected
+++ /dev/null
@@ -1,17 +0,0 @@
-<div class="CodeRay">
-  <div class="code"><pre><span style="color:#080;font-weight:bold">for</span> a <span style="color:#080;font-weight:bold">in</span> <span style="color:#00D;font-weight:bold">0</span>..<span style="color:#00D;font-weight:bold">255</span>
-        a = a.chr
-        <span style="color:#080;font-weight:bold">begin</span>
-                x = eval(<span style="background-color:#fff0f0;color:#D20"><span style="color:#710">&quot;</span><span style="">?</span><span style="color:#b0b">\\</span><span style="background:#ddd;color:black"><span style="background:#ddd;font-weight:bold;color:#666">#{</span>a<span style="background:#ddd;font-weight:bold;color:#666">}</span></span><span style="color:#710">&quot;</span></span>)
-                <span style="color:#080;font-weight:bold">if</span> x == a[<span style="color:#00D;font-weight:bold">0</span>]
-                        <span style="color:#080;font-weight:bold">next</span>
-                <span style="color:#080;font-weight:bold">else</span>
-                        print <span style="background-color:#fff0f0;color:#D20"><span style="color:#710">&quot;</span><span style="background:#ddd;color:black"><span style="background:#ddd;font-weight:bold;color:#666">#{</span>a<span style="background:#ddd;font-weight:bold;color:#666">}</span></span><span style="">: </span><span style="background:#ddd;color:black"><span style="background:#ddd;font-weight:bold;color:#666">#{</span>x<span style="background:#ddd;font-weight:bold;color:#666">}</span></span><span style="color:#710">&quot;</span></span>
-                <span style="color:#080;font-weight:bold">end</span>
-        <span style="color:#080;font-weight:bold">rescue</span> <span style="color:#036;font-weight:bold">SyntaxError</span> =&gt; boom
-                print <span style="background-color:#fff0f0;color:#D20"><span style="color:#710">&quot;</span><span style="background:#ddd;color:black"><span style="background:#ddd;font-weight:bold;color:#666">#{</span>a<span style="background:#ddd;font-weight:bold;color:#666">}</span></span><span style="">: error</span><span style="color:#710">&quot;</span></span>
-        <span style="color:#080;font-weight:bold">end</span>
-        puts
-<span style="color:#080;font-weight:bold">end</span>
-</pre></div>
-</div>
diff --git a/sample/div.rb b/sample/div.rb
deleted file mode 100644
index 27b6f32..0000000
--- a/sample/div.rb
+++ /dev/null
@@ -1,19 +0,0 @@
-require 'coderay'
-
-puts CodeRay.scan(DATA.read, :ruby).div
-
-__END__
-for a in 0..255
-	a = a.chr
-	begin
-		x = eval("?\\#{a}")
-		if x == a[0]
-			next
-		else
-			print "#{a}: #{x}"
-		end
-	rescue SyntaxError => boom
-		print "#{a}: error"
-	end
-	puts
-end
diff --git a/sample/dump.expected b/sample/dump.expected
deleted file mode 100644
index a451686..0000000
--- a/sample/dump.expected
+++ /dev/null
@@ -1,21 +0,0 @@
-YAML: 2358 bytes
-Dump: 1109 bytes
-undumped:
-<div class="CodeRay">
-  <div class="code"><pre>require <span class="s"><span class="dl">'</span><span class="k">coderay</span><span class="dl">'</span></span>
-
-<span class="c"># scan some code</span>
-tokens = <span class="co">CodeRay</span>.scan(<span class="co">File</span>.read(<span class="gv">$0</span>), <span class="sy">:ruby</span>)
-
-<span class="c"># dump using YAML</span>
-yaml = tokens.yaml
-puts <span class="s"><span class="dl">'</span><span class="k">YAML: %4d bytes</span><span class="dl">'</span></span> % yaml.size
-
-<span class="c"># dump using Marshal</span>
-dump = tokens.dump(<span class="i">0</span>)
-puts <span class="s"><span class="dl">'</span><span class="k">Dump: %4d bytes</span><span class="dl">'</span></span> % dump.size
-
-<span class="c"># undump and encode</span>
-puts <span class="s"><span class="dl">'</span><span class="k">undumped:</span><span class="dl">'</span></span>, dump.undump.div(<span class="sy">:css</span> =&gt; <span class="sy">:class</span>)
-</pre></div>
-</div>
diff --git a/sample/dump.rb b/sample/dump.rb
deleted file mode 100644
index cd68dc8..0000000
--- a/sample/dump.rb
+++ /dev/null
@@ -1,15 +0,0 @@
-require 'coderay'

-

-# scan some code

-tokens = CodeRay.scan(File.read($0), :ruby)

-

-# dump using YAML

-yaml = tokens.yaml

-puts 'YAML: %4d bytes' % yaml.size

-

-# dump using Marshal

-dump = tokens.dump(0)

-puts 'Dump: %4d bytes' % dump.size

-

-# undump and encode

-puts 'undumped:', dump.undump.div(:css => :class)

diff --git a/sample/encoder.expected b/sample/encoder.expected
deleted file mode 100644
index 438032a..0000000
--- a/sample/encoder.expected
+++ /dev/null
@@ -1,65 +0,0 @@
-Encoders Demo: puts 17 + 4
-
-Statistic:
-
-Code Statistics
-
-Tokens                   8
-  Non-Whitespace         4
-Bytes Total             12
-
-Token Types (4):
-  type                     count     ratio    size (average)
--------------------------------------------------------------
-  TOTAL                        8  100.00 %     1.5
-  space                        4   50.00 %     1.0
-  integer                      2   25.00 %     1.5
-  ident                        1   12.50 %     4.0
-  operator                     1   12.50 %     1.0
-
-
-Original text:
-[{"type":"text","text":"puts","kind":"ident"},{"type":"text","text":" ","kind":"space"},{"type":"text","text":"17","kind":"integer"},{"type":"text","text":" ","kind":"space"},{"type":"text","text":"+","kind":"operator"},{"type":"text","text":" ","kind":"space"},{"type":"text","text":"4","kind":"integer"},{"type":"text","text":"\n","kind":"space"}]
-
-YAML:
---- 
-- - puts
-  - :ident
-- - " "
-  - :space
-- - "17"
-  - :integer
-- - " "
-  - :space
-- - +
-  - :operator
-- - " "
-  - :space
-- - "4"
-  - :integer
-- - |
-    
-
-  - :space
-
-Dump:
-"x\332\355\330\273\n\302@\020\005PQIL4\235\245E\260\265\022\004a\266\021\002B\332\250U\2525\031$\210\233\260\273)\202?o\036\370\370\006\271\325\354\314\345\334\017\330\351,\216h\031\2259'\262!:\227wV&\035\207\223\324]{Um\r\371E\316\312\266\253\023\222o*\231q\373v\267{Z\024\312\362\215u\037\t\267\e\e\n\312\212\265\264\345\357u'f\335\360\373\255/\025\3167n\253\206\374\335!<XXXXXXXXXXXXXXXXXXXXX\330\277\267\016\005\a\223I\245X\307\027Z}\276\352O\303\315\020%\365\265y:47V\263|\034G/\301K\315)"
-compressed: 188 byte < 1200 byte
-
-Undump:
-
-Code Statistics
-
-Tokens                 800
-  Non-Whitespace       400
-Bytes Total           1200
-
-Token Types (4):
-  type                     count     ratio    size (average)
--------------------------------------------------------------
-  TOTAL                      800  100.00 %     1.5
-  space                      400   50.00 %     1.0
-  integer                    200   25.00 %     1.5
-  ident                      100   12.50 %     4.0
-  operator                   100   12.50 %     1.0
-
diff --git a/sample/encoder.rb b/sample/encoder.rb
deleted file mode 100644
index 22eaa22..0000000
--- a/sample/encoder.rb
+++ /dev/null
@@ -1,39 +0,0 @@
-require 'coderay'

-

-SAMPLE = "puts 17 + 4\n"

-puts 'Encoders Demo: ' + SAMPLE

-scanner = CodeRay::Scanners[:ruby].new SAMPLE

-encoder = CodeRay::Encoders[:statistic].new

-

-tokens = scanner.tokenize

-stats = encoder.encode_tokens tokens

-

-puts

-puts 'Statistic:'

-puts stats

-

-# alternative 1

-tokens = CodeRay.scan SAMPLE, :ruby

-encoder = CodeRay.encoder(:json)

-textual = encoder.encode_tokens tokens

-puts

-puts 'Original text:'

-puts textual

-

-# alternative 2

-yaml = CodeRay.encoder(:yaml).encode SAMPLE, :ruby

-puts

-puts 'YAML:'

-puts yaml

-

-# alternative 3

-BIGSAMPLE = SAMPLE * 100

-dump = CodeRay.scan(BIGSAMPLE, :ruby).dump

-puts

-puts 'Dump:'

-p dump

-puts 'compressed: %d byte < %d byte' % [dump.size, BIGSAMPLE.size]

-

-puts

-puts 'Undump:'

-puts dump.undump.statistic

diff --git a/sample/global_vars.expected b/sample/global_vars.expected
deleted file mode 100644
index 0dc13c8..0000000
--- a/sample/global_vars.expected
+++ /dev/null
@@ -1,3 +0,0 @@
-<--$IE-->.TEXT_FIELD(:NAME, "PANFRAGE OHNE $GV UND MIT #{<--$GV-->}").SET ARTIKEL
-ODER
-TEXT = <--$BLA-->.TEST(...) 
\ No newline at end of file
diff --git a/sample/global_vars.rb b/sample/global_vars.rb
deleted file mode 100644
index 8066d67..0000000
--- a/sample/global_vars.rb
+++ /dev/null
@@ -1,13 +0,0 @@
-code = <<'CODE'

-$ie.text_field(:name, "pAnfrage ohne $gV und mit #{$gv}").set artikel

-oder

-text = $bla.test(...) 

-CODE

-

-require 'coderay'

-

-tokens = CodeRay.scan code, :ruby

-tokens.each_text_token { |text, kind| text.upcase! }

-tokens.each(:global_variable) { |text, kind| text.replace '<--%s-->' % text }

-

-print tokens

diff --git a/sample/global_vars2.expected b/sample/global_vars2.expected
deleted file mode 100644
index 964cf50..0000000
--- a/sample/global_vars2.expected
+++ /dev/null
@@ -1,10 +0,0 @@
-<html>
-<head>
-<style>span.glob-var { color: green; font-weight: bold; }</style>
-</head>
-<body>
-<span class="glob-var">$ie</span>.text_field(:name, &quot;pAnfrage ohne $gV und mit #{<span class="glob-var">$gv</span>}&quot;).set artikel
-oder
-text = <span class="glob-var">$bla</span>.test(...) 
-</body>
-</html>
diff --git a/sample/global_vars2.rb b/sample/global_vars2.rb
deleted file mode 100644
index 7646890..0000000
--- a/sample/global_vars2.rb
+++ /dev/null
@@ -1,28 +0,0 @@
-require 'coderay'

-require 'erb'

-include ERB::Util

-

-code = <<'CODE'

-$ie.text_field(:name, "pAnfrage ohne $gV und mit #{$gv}").set artikel

-oder

-text = $bla.test(...) 

-CODE

-puts <<HTML

-<html>

-<head>

-<style>span.glob-var { color: green; font-weight: bold; }</style>

-</head>

-<body>

-HTML

-

-CodeRay.scan_stream code, :ruby do |text, kind|

-	next if text.is_a? Symbol

-	text = h(text)

-	text = '<span class="glob-var">%s</span>' % text if kind == :global_variable

-	print text

-end

-

-puts <<HTML

-</body>

-</html>

-HTML

diff --git a/sample/highlight.expected b/sample/highlight.expected
deleted file mode 100644
index 6a9b278..0000000
--- a/sample/highlight.expected
+++ /dev/null
@@ -1,175 +0,0 @@
-<div class="CodeRay">
-  <div class="code"><pre>puts <span class="s"><span class="dl">&quot;</span><span class="k">Hello, World!</span><span class="dl">&quot;</span></span></pre></div>
-</div>
-<html>
-<head>
-<style type="text/css">
-.CodeRay {
-  background-color: #f8f8f8;
-  border: 1px solid silver;
-  font-family: 'Courier New', 'Terminal', monospace;
-  color: #000;
-}
-.CodeRay pre { margin: 0px }
-
-div.CodeRay { }
-
-span.CodeRay { white-space: pre; border: 0px; padding: 2px }
-
-table.CodeRay { border-collapse: collapse; width: 100%; padding: 2px }
-table.CodeRay td { padding: 2px 4px; vertical-align: top }
-
-.CodeRay .line_numbers, .CodeRay .no {
-  background-color: #def;
-  color: gray;
-  text-align: right;
-}
-.CodeRay .line_numbers tt { font-weight: bold }
-.CodeRay .line_numbers .highlighted { color: red }
-.CodeRay .no { padding: 0px 4px }
-.CodeRay .code { width: 100% }
-
-ol.CodeRay { font-size: 10pt }
-ol.CodeRay li { white-space: pre }
-
-.CodeRay .code pre { overflow: auto }
-
-.CodeRay .debug { color:white ! important; background:blue ! important; }
-
-.CodeRay .af { color:#00C }
-.CodeRay .an { color:#007 }
-.CodeRay .at { color:#f08 }
-.CodeRay .av { color:#700 }
-.CodeRay .aw { color:#C00 }
-.CodeRay .bi { color:#509; font-weight:bold }
-.CodeRay .c  { color:#888; }
-
-.CodeRay .ch { color:#04D }
-.CodeRay .ch .k { color:#04D }
-.CodeRay .ch .dl { color:#039 }
-
-.CodeRay .cl { color:#B06; font-weight:bold }
-.CodeRay .cm { color:#A08; font-weight:bold }
-.CodeRay .co { color:#036; font-weight:bold }
-.CodeRay .cr { color:#0A0 }
-.CodeRay .cv { color:#369 }
-.CodeRay .de { color:#B0B; }
-.CodeRay .df { color:#099; font-weight:bold }
-.CodeRay .di { color:#088; font-weight:bold }
-.CodeRay .dl { color:black }
-.CodeRay .do { color:#970 }
-.CodeRay .dt { color:#34b }
-.CodeRay .ds { color:#D42; font-weight:bold }
-.CodeRay .e  { color:#666; font-weight:bold }
-.CodeRay .en { color:#800; font-weight:bold }
-.CodeRay .er { color:#F00; background-color:#FAA }
-.CodeRay .ex { color:#C00; font-weight:bold }
-.CodeRay .fl { color:#60E; font-weight:bold }
-.CodeRay .fu { color:#06B; font-weight:bold }
-.CodeRay .gv { color:#d70; font-weight:bold }
-.CodeRay .hx { color:#058; font-weight:bold }
-.CodeRay .i  { color:#00D; font-weight:bold }
-.CodeRay .ic { color:#B44; font-weight:bold }
-
-.CodeRay .il { background: #ddd; color: black }
-.CodeRay .il .il { background: #ccc }
-.CodeRay .il .il .il { background: #bbb }
-.CodeRay .il .idl { background: #ddd; font-weight: bold; color: #666 }
-.CodeRay .idl { background-color: #bbb; font-weight: bold; color: #666; }
-
-.CodeRay .im { color:#f00; }
-.CodeRay .in { color:#B2B; font-weight:bold }
-.CodeRay .iv { color:#33B }
-.CodeRay .la { color:#970; font-weight:bold }
-.CodeRay .lv { color:#963 }
-.CodeRay .oc { color:#40E; font-weight:bold }
-.CodeRay .of { color:#000; font-weight:bold }
-.CodeRay .op { }
-.CodeRay .pc { color:#038; font-weight:bold }
-.CodeRay .pd { color:#369; font-weight:bold }
-.CodeRay .pp { color:#579; }
-.CodeRay .ps { color:#00C; font-weight:bold }
-.CodeRay .pt { color:#074; font-weight:bold }
-.CodeRay .r, .kw  { color:#080; font-weight:bold }
-
-.CodeRay .ke { color: #808; }
-.CodeRay .ke .dl { color: #606; }
-.CodeRay .ke .ch { color: #80f; }
-.CodeRay .vl { color: #088; }
-
-.CodeRay .rx { background-color:#fff0ff }
-.CodeRay .rx .k { color:#808 }
-.CodeRay .rx .dl { color:#404 }
-.CodeRay .rx .mod { color:#C2C }
-.CodeRay .rx .fu  { color:#404; font-weight: bold }
-
-.CodeRay .s { background-color:#fff0f0; color: #D20; }
-.CodeRay .s .s { background-color:#ffe0e0 }
-.CodeRay .s .s  .s { background-color:#ffd0d0 }
-.CodeRay .s .k { }
-.CodeRay .s .ch { color: #b0b; }
-.CodeRay .s .dl { color: #710; }
-
-.CodeRay .sh { background-color:#f0fff0; color:#2B2 }
-.CodeRay .sh .k { }
-.CodeRay .sh .dl { color:#161 }
-
-.CodeRay .sy { color:#A60 }
-.CodeRay .sy .k { color:#A60 }
-.CodeRay .sy .dl { color:#630 }
-
-.CodeRay .ta { color:#070 }
-.CodeRay .tf { color:#070; font-weight:bold }
-.CodeRay .ts { color:#D70; font-weight:bold }
-.CodeRay .ty { color:#339; font-weight:bold }
-.CodeRay .v  { color:#036 }
-.CodeRay .xt { color:#444 }
-
-.CodeRay .ins { background: #afa; }
-.CodeRay .del { background: #faa; }
-.CodeRay .chg { color: #aaf; background: #007; }
-.CodeRay .head { color: #f8f; background: #505 }
-
-.CodeRay .ins .ins { color: #080; font-weight:bold }
-.CodeRay .del .del { color: #800; font-weight:bold }
-.CodeRay .chg .chg { color: #66f; }
-.CodeRay .head .head { color: #f4f; }
-
-</style>
-
-<body>
-<table class="CodeRay"><tr>
-  <td class="line_numbers" title="click to toggle" onclick="with (this.firstChild.style) { display = (display == '') ? 'none' : '' }"><pre>1<tt>
-</tt>2<tt>
-</tt>3<tt>
-</tt>4<tt>
-</tt>5<tt>
-</tt>6<tt>
-</tt>7<tt>
-</tt>8<tt>
-</tt>9<tt>
-</tt><strong>10</strong><tt>
-</tt>11<tt>
-</tt>12<tt>
-</tt>13<tt>
-</tt>14<tt>
-</tt></pre></td>
-  <td class="code"><pre ondblclick="with (this.style) { overflow = (overflow == 'auto' || overflow == '') ? 'visible' : 'auto' }">require <span style="background-color:#fff0f0;color:#D20"><span style="color:#710">'</span><span style="">coderay</span><span style="color:#710">'</span></span><tt>
-</tt><tt>
-</tt>puts <span style="color:#036;font-weight:bold">CodeRay</span>.highlight(<span style="background-color:#fff0f0;color:#D20"><span style="color:#710">'</span><span style="">puts &quot;Hello, World!&quot;</span><span style="color:#710">'</span></span>, <span style="color:#A60">:ruby</span>)<tt>
-</tt><tt>
-</tt>output = <span style="color:#036;font-weight:bold">CodeRay</span>.highlight_file(<span style="color:#d70;font-weight:bold">$0</span>, <span style="color:#A60">:line_numbers</span> =&gt; <span style="color:#A60">:table</span>)<tt>
-</tt>puts <span style="background-color:#fff0f0;color:#D20"><span style="color:#710">&lt;&lt;HTML</span></span><span style="background-color:#fff0f0;color:#D20"><span style=""><tt>
-</tt>&lt;html&gt;<tt>
-</tt>&lt;head&gt;<tt>
-</tt></span><span style="background:#ddd;color:black"><span style="background:#ddd;font-weight:bold;color:#666">#{</span>output.stylesheet <span style="color:#038;font-weight:bold">true</span><span style="background:#ddd;font-weight:bold;color:#666">}</span></span><span style=""><tt>
-</tt>&lt;body&gt;<tt>
-</tt></span><span style="background:#ddd;color:black"><span style="background:#ddd;font-weight:bold;color:#666">#{</span>output<span style="background:#ddd;font-weight:bold;color:#666">}</span></span><span style=""><tt>
-</tt>&lt;/body&gt;<tt>
-</tt>&lt;/html&gt;</span><span style="color:#710"><tt>
-</tt>HTML</span></span><tt>
-</tt></pre></td>
-</tr></table>
-
-</body>
-</html>
diff --git a/sample/highlight.rb b/sample/highlight.rb
deleted file mode 100644
index 846efa4..0000000
--- a/sample/highlight.rb
+++ /dev/null
@@ -1,14 +0,0 @@
-require 'coderay'

-

-puts CodeRay.highlight('puts "Hello, World!"', :ruby)

-

-output = CodeRay.highlight_file($0, :line_numbers => :table)

-puts <<HTML

-<html>

-<head>

-#{output.stylesheet true}

-<body>

-#{output}

-</body>

-</html>

-HTML

diff --git a/sample/html.expected b/sample/html.expected
deleted file mode 100644
index e98d589..0000000
--- a/sample/html.expected
+++ /dev/null
@@ -1,919 +0,0 @@
-<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
-  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
-<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="de">
-<head>
-  <meta http-equiv="content-type" content="text/html; charset=utf-8" />
-  <title>CodeRay HTML Encoder Example</title>
-  <style type="text/css">
-.CodeRay {
-  background-color: #f8f8f8;
-  border: 1px solid silver;
-  font-family: 'Courier New', 'Terminal', monospace;
-  color: #000;
-}
-.CodeRay pre { margin: 0px }
-
-div.CodeRay { }
-
-span.CodeRay { white-space: pre; border: 0px; padding: 2px }
-
-table.CodeRay { border-collapse: collapse; width: 100%; padding: 2px }
-table.CodeRay td { padding: 2px 4px; vertical-align: top }
-
-.CodeRay .line_numbers, .CodeRay .no {
-  background-color: #def;
-  color: gray;
-  text-align: right;
-}
-.CodeRay .line_numbers tt { font-weight: bold }
-.CodeRay .line_numbers .highlighted { color: red }
-.CodeRay .no { padding: 0px 4px }
-.CodeRay .code { width: 100% }
-
-ol.CodeRay { font-size: 10pt }
-ol.CodeRay li { white-space: pre }
-
-.CodeRay .code pre { overflow: auto }
-
-.CodeRay .debug { color:white ! important; background:blue ! important; }
-
-.CodeRay .af { color:#00C }
-.CodeRay .an { color:#007 }
-.CodeRay .at { color:#f08 }
-.CodeRay .av { color:#700 }
-.CodeRay .aw { color:#C00 }
-.CodeRay .bi { color:#509; font-weight:bold }
-.CodeRay .c  { color:#888; }
-
-.CodeRay .ch { color:#04D }
-.CodeRay .ch .k { color:#04D }
-.CodeRay .ch .dl { color:#039 }
-
-.CodeRay .cl { color:#B06; font-weight:bold }
-.CodeRay .cm { color:#A08; font-weight:bold }
-.CodeRay .co { color:#036; font-weight:bold }
-.CodeRay .cr { color:#0A0 }
-.CodeRay .cv { color:#369 }
-.CodeRay .de { color:#B0B; }
-.CodeRay .df { color:#099; font-weight:bold }
-.CodeRay .di { color:#088; font-weight:bold }
-.CodeRay .dl { color:black }
-.CodeRay .do { color:#970 }
-.CodeRay .dt { color:#34b }
-.CodeRay .ds { color:#D42; font-weight:bold }
-.CodeRay .e  { color:#666; font-weight:bold }
-.CodeRay .en { color:#800; font-weight:bold }
-.CodeRay .er { color:#F00; background-color:#FAA }
-.CodeRay .ex { color:#C00; font-weight:bold }
-.CodeRay .fl { color:#60E; font-weight:bold }
-.CodeRay .fu { color:#06B; font-weight:bold }
-.CodeRay .gv { color:#d70; font-weight:bold }
-.CodeRay .hx { color:#058; font-weight:bold }
-.CodeRay .i  { color:#00D; font-weight:bold }
-.CodeRay .ic { color:#B44; font-weight:bold }
-
-.CodeRay .il { background: #ddd; color: black }
-.CodeRay .il .il { background: #ccc }
-.CodeRay .il .il .il { background: #bbb }
-.CodeRay .il .idl { background: #ddd; font-weight: bold; color: #666 }
-.CodeRay .idl { background-color: #bbb; font-weight: bold; color: #666; }
-
-.CodeRay .im { color:#f00; }
-.CodeRay .in { color:#B2B; font-weight:bold }
-.CodeRay .iv { color:#33B }
-.CodeRay .la { color:#970; font-weight:bold }
-.CodeRay .lv { color:#963 }
-.CodeRay .oc { color:#40E; font-weight:bold }
-.CodeRay .of { color:#000; font-weight:bold }
-.CodeRay .op { }
-.CodeRay .pc { color:#038; font-weight:bold }
-.CodeRay .pd { color:#369; font-weight:bold }
-.CodeRay .pp { color:#579; }
-.CodeRay .ps { color:#00C; font-weight:bold }
-.CodeRay .pt { color:#074; font-weight:bold }
-.CodeRay .r, .kw  { color:#080; font-weight:bold }
-
-.CodeRay .ke { color: #808; }
-.CodeRay .ke .dl { color: #606; }
-.CodeRay .ke .ch { color: #80f; }
-.CodeRay .vl { color: #088; }
-
-.CodeRay .rx { background-color:#fff0ff }
-.CodeRay .rx .k { color:#808 }
-.CodeRay .rx .dl { color:#404 }
-.CodeRay .rx .mod { color:#C2C }
-.CodeRay .rx .fu  { color:#404; font-weight: bold }
-
-.CodeRay .s { background-color:#fff0f0; color: #D20; }
-.CodeRay .s .s { background-color:#ffe0e0 }
-.CodeRay .s .s  .s { background-color:#ffd0d0 }
-.CodeRay .s .k { }
-.CodeRay .s .ch { color: #b0b; }
-.CodeRay .s .dl { color: #710; }
-
-.CodeRay .sh { background-color:#f0fff0; color:#2B2 }
-.CodeRay .sh .k { }
-.CodeRay .sh .dl { color:#161 }
-
-.CodeRay .sy { color:#A60 }
-.CodeRay .sy .k { color:#A60 }
-.CodeRay .sy .dl { color:#630 }
-
-.CodeRay .ta { color:#070 }
-.CodeRay .tf { color:#070; font-weight:bold }
-.CodeRay .ts { color:#D70; font-weight:bold }
-.CodeRay .ty { color:#339; font-weight:bold }
-.CodeRay .v  { color:#036 }
-.CodeRay .xt { color:#444 }
-
-.CodeRay .ins { background: #afa; }
-.CodeRay .del { background: #faa; }
-.CodeRay .chg { color: #aaf; background: #007; }
-.CodeRay .head { color: #f8f; background: #505 }
-
-.CodeRay .ins .ins { color: #080; font-weight:bold }
-.CodeRay .del .del { color: #800; font-weight:bold }
-.CodeRay .chg .chg { color: #66f; }
-.CodeRay .head .head { color: #f4f; }
-
-  </style>
-</head>
-<body style="background-color: white;">
-
-<table class="CodeRay"><tr>
-  <td class="line_numbers" title="click to toggle" onclick="with (this.firstChild.style) { display = (display == '') ? 'none' : '' }"><pre>1<tt>
-</tt>2<tt>
-</tt>3<tt>
-</tt>4<tt>
-</tt>5<tt>
-</tt>6<tt>
-</tt>7<tt>
-</tt>8<tt>
-</tt>9<tt>
-</tt><strong>10</strong><tt>
-</tt>11<tt>
-</tt>12<tt>
-</tt>13<tt>
-</tt>14<tt>
-</tt>15<tt>
-</tt>16<tt>
-</tt>17<tt>
-</tt>18<tt>
-</tt>19<tt>
-</tt><strong>20</strong><tt>
-</tt>21<tt>
-</tt>22<tt>
-</tt>23<tt>
-</tt>24<tt>
-</tt>25<tt>
-</tt>26<tt>
-</tt>27<tt>
-</tt>28<tt>
-</tt>29<tt>
-</tt><strong>30</strong><tt>
-</tt>31<tt>
-</tt>32<tt>
-</tt>33<tt>
-</tt>34<tt>
-</tt>35<tt>
-</tt>36<tt>
-</tt>37<tt>
-</tt>38<tt>
-</tt>39<tt>
-</tt><strong>40</strong><tt>
-</tt>41<tt>
-</tt>42<tt>
-</tt>43<tt>
-</tt>44<tt>
-</tt>45<tt>
-</tt>46<tt>
-</tt>47<tt>
-</tt>48<tt>
-</tt>49<tt>
-</tt><strong>50</strong><tt>
-</tt>51<tt>
-</tt>52<tt>
-</tt>53<tt>
-</tt>54<tt>
-</tt>55<tt>
-</tt>56<tt>
-</tt>57<tt>
-</tt>58<tt>
-</tt>59<tt>
-</tt><strong>60</strong><tt>
-</tt>61<tt>
-</tt>62<tt>
-</tt>63<tt>
-</tt>64<tt>
-</tt>65<tt>
-</tt>66<tt>
-</tt>67<tt>
-</tt>68<tt>
-</tt>69<tt>
-</tt><strong>70</strong><tt>
-</tt>71<tt>
-</tt>72<tt>
-</tt>73<tt>
-</tt>74<tt>
-</tt>75<tt>
-</tt>76<tt>
-</tt>77<tt>
-</tt>78<tt>
-</tt>79<tt>
-</tt><strong>80</strong><tt>
-</tt>81<tt>
-</tt>82<tt>
-</tt>83<tt>
-</tt>84<tt>
-</tt>85<tt>
-</tt>86<tt>
-</tt>87<tt>
-</tt>88<tt>
-</tt>89<tt>
-</tt><strong>90</strong><tt>
-</tt>91<tt>
-</tt>92<tt>
-</tt>93<tt>
-</tt>94<tt>
-</tt>95<tt>
-</tt>96<tt>
-</tt>97<tt>
-</tt>98<tt>
-</tt>99<tt>
-</tt><strong>100</strong><tt>
-</tt>101<tt>
-</tt>102<tt>
-</tt>103<tt>
-</tt>104<tt>
-</tt>105<tt>
-</tt>106<tt>
-</tt>107<tt>
-</tt>108<tt>
-</tt>109<tt>
-</tt><strong>110</strong><tt>
-</tt>111<tt>
-</tt>112<tt>
-</tt>113<tt>
-</tt>114<tt>
-</tt>115<tt>
-</tt>116<tt>
-</tt>117<tt>
-</tt>118<tt>
-</tt>119<tt>
-</tt><strong>120</strong><tt>
-</tt>121<tt>
-</tt>122<tt>
-</tt>123<tt>
-</tt>124<tt>
-</tt>125<tt>
-</tt>126<tt>
-</tt>127<tt>
-</tt>128<tt>
-</tt>129<tt>
-</tt><strong>130</strong><tt>
-</tt>131<tt>
-</tt>132<tt>
-</tt>133<tt>
-</tt>134<tt>
-</tt>135<tt>
-</tt>136<tt>
-</tt>137<tt>
-</tt>138<tt>
-</tt>139<tt>
-</tt><strong>140</strong><tt>
-</tt>141<tt>
-</tt>142<tt>
-</tt>143<tt>
-</tt>144<tt>
-</tt>145<tt>
-</tt>146<tt>
-</tt>147<tt>
-</tt>148<tt>
-</tt>149<tt>
-</tt><strong>150</strong><tt>
-</tt>151<tt>
-</tt>152<tt>
-</tt>153<tt>
-</tt>154<tt>
-</tt>155<tt>
-</tt>156<tt>
-</tt>157<tt>
-</tt>158<tt>
-</tt>159<tt>
-</tt><strong>160</strong><tt>
-</tt>161<tt>
-</tt>162<tt>
-</tt>163<tt>
-</tt>164<tt>
-</tt>165<tt>
-</tt>166<tt>
-</tt>167<tt>
-</tt>168<tt>
-</tt>169<tt>
-</tt><strong>170</strong><tt>
-</tt>171<tt>
-</tt>172<tt>
-</tt>173<tt>
-</tt>174<tt>
-</tt>175<tt>
-</tt>176<tt>
-</tt>177<tt>
-</tt>178<tt>
-</tt>179<tt>
-</tt><strong>180</strong><tt>
-</tt>181<tt>
-</tt>182<tt>
-</tt>183<tt>
-</tt>184<tt>
-</tt>185<tt>
-</tt>186<tt>
-</tt>187<tt>
-</tt>188<tt>
-</tt>189<tt>
-</tt><strong>190</strong><tt>
-</tt>191<tt>
-</tt>192<tt>
-</tt>193<tt>
-</tt>194<tt>
-</tt>195<tt>
-</tt>196<tt>
-</tt>197<tt>
-</tt>198<tt>
-</tt>199<tt>
-</tt><strong>200</strong><tt>
-</tt>201<tt>
-</tt>202<tt>
-</tt>203<tt>
-</tt>204<tt>
-</tt>205<tt>
-</tt>206<tt>
-</tt>207<tt>
-</tt>208<tt>
-</tt>209<tt>
-</tt><strong>210</strong><tt>
-</tt>211<tt>
-</tt>212<tt>
-</tt>213<tt>
-</tt>214<tt>
-</tt>215<tt>
-</tt>216<tt>
-</tt>217<tt>
-</tt>218<tt>
-</tt>219<tt>
-</tt><strong>220</strong><tt>
-</tt>221<tt>
-</tt>222<tt>
-</tt>223<tt>
-</tt>224<tt>
-</tt>225<tt>
-</tt>226<tt>
-</tt>227<tt>
-</tt>228<tt>
-</tt>229<tt>
-</tt><strong>230</strong><tt>
-</tt>231<tt>
-</tt>232<tt>
-</tt>233<tt>
-</tt>234<tt>
-</tt>235<tt>
-</tt>236<tt>
-</tt>237<tt>
-</tt>238<tt>
-</tt>239<tt>
-</tt><strong>240</strong><tt>
-</tt>241<tt>
-</tt>242<tt>
-</tt>243<tt>
-</tt>244<tt>
-</tt>245<tt>
-</tt>246<tt>
-</tt>247<tt>
-</tt>248<tt>
-</tt>249<tt>
-</tt><strong>250</strong><tt>
-</tt>251<tt>
-</tt>252<tt>
-</tt>253<tt>
-</tt>254<tt>
-</tt>255<tt>
-</tt>256<tt>
-</tt>257<tt>
-</tt>258<tt>
-</tt>259<tt>
-</tt><strong>260</strong><tt>
-</tt>261<tt>
-</tt>262<tt>
-</tt>263<tt>
-</tt>264<tt>
-</tt>265<tt>
-</tt>266<tt>
-</tt>267<tt>
-</tt>268<tt>
-</tt>269<tt>
-</tt><strong>270</strong><tt>
-</tt>271<tt>
-</tt>272<tt>
-</tt>273<tt>
-</tt>274<tt>
-</tt>275<tt>
-</tt>276<tt>
-</tt>277<tt>
-</tt>278<tt>
-</tt>279<tt>
-</tt><strong>280</strong><tt>
-</tt>281<tt>
-</tt>282<tt>
-</tt>283<tt>
-</tt>284<tt>
-</tt>285<tt>
-</tt>286<tt>
-</tt>287<tt>
-</tt>288<tt>
-</tt>289<tt>
-</tt><strong>290</strong><tt>
-</tt>291<tt>
-</tt>292<tt>
-</tt>293<tt>
-</tt>294<tt>
-</tt>295<tt>
-</tt>296<tt>
-</tt>297<tt>
-</tt>298<tt>
-</tt>299<tt>
-</tt><strong>300</strong><tt>
-</tt>301<tt>
-</tt>302<tt>
-</tt>303<tt>
-</tt>304<tt>
-</tt>305<tt>
-</tt>306<tt>
-</tt>307<tt>
-</tt>308<tt>
-</tt>309<tt>
-</tt><strong>310</strong><tt>
-</tt>311<tt>
-</tt>312<tt>
-</tt>313<tt>
-</tt>314<tt>
-</tt>315<tt>
-</tt>316<tt>
-</tt>317<tt>
-</tt>318<tt>
-</tt>319<tt>
-</tt><strong>320</strong><tt>
-</tt>321<tt>
-</tt>322<tt>
-</tt>323<tt>
-</tt>324<tt>
-</tt>325<tt>
-</tt>326<tt>
-</tt>327<tt>
-</tt>328<tt>
-</tt>329<tt>
-</tt><strong>330</strong><tt>
-</tt>331<tt>
-</tt>332<tt>
-</tt>333<tt>
-</tt>334<tt>
-</tt>335<tt>
-</tt>336<tt>
-</tt>337<tt>
-</tt>338<tt>
-</tt>339<tt>
-</tt><strong>340</strong><tt>
-</tt>341<tt>
-</tt>342<tt>
-</tt>343<tt>
-</tt>344<tt>
-</tt>345<tt>
-</tt>346<tt>
-</tt>347<tt>
-</tt>348<tt>
-</tt>349<tt>
-</tt><strong>350</strong><tt>
-</tt>351<tt>
-</tt>352<tt>
-</tt>353<tt>
-</tt>354<tt>
-</tt>355<tt>
-</tt>356<tt>
-</tt>357<tt>
-</tt>358<tt>
-</tt>359<tt>
-</tt><strong>360</strong><tt>
-</tt>361<tt>
-</tt>362<tt>
-</tt>363<tt>
-</tt>364<tt>
-</tt>365<tt>
-</tt>366<tt>
-</tt>367<tt>
-</tt>368<tt>
-</tt>369<tt>
-</tt><strong>370</strong><tt>
-</tt>371<tt>
-</tt>372<tt>
-</tt>373<tt>
-</tt>374<tt>
-</tt>375<tt>
-</tt>376<tt>
-</tt>377<tt>
-</tt>378<tt>
-</tt>379<tt>
-</tt><strong>380</strong><tt>
-</tt>381<tt>
-</tt>382<tt>
-</tt>383<tt>
-</tt>384<tt>
-</tt>385<tt>
-</tt></pre></td>
-  <td class="code"><pre ondblclick="with (this.style) { overflow = (overflow == 'auto' || overflow == '') ? 'visible' : 'auto' }">require <span class="s"><span class="dl">'</span><span class="k">scanner</span><span class="dl">'</span></span><tt>
-</tt><tt>
-</tt><span class="r">module</span> <span class="cl">CodeRay</span><tt>
-</tt>  <tt>
-</tt>  <span class="r">class</span> <span class="cl">RubyScanner</span> &lt; <span class="co">Scanner</span><tt>
-</tt>    <tt>
-</tt>    <span class="co">RESERVED_WORDS</span> = [<tt>
-</tt>      <span class="s"><span class="dl">'</span><span class="k">and</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">def</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">end</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">in</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">or</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">unless</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">begin</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="s"><span class="dl">'</span><span class="k">defined?</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">ensure</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">module</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">redo</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">super</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">until</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="s"><span class="dl">'</span><span class="k">BEGIN</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">break</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">do</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">next</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">rescue</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">then</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="s"><span class="dl">'</span><span class="k">when</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">END</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">case</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">else</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">for</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">retry</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="s"><span class="dl">'</span><span class="k">while</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">alias</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">class</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">elsif</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">if</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">not</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">return</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="s"><span class="dl">'</span><span class="k">undef</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">yield</span><span class="dl">'</span></span>,<tt>
-</tt>    ]<tt>
-</tt><tt>
-</tt>    <span class="co">DEF_KEYWORDS</span> = [<span class="s"><span class="dl">'</span><span class="k">def</span><span class="dl">'</span></span>]<tt>
-</tt>    <span class="co">MODULE_KEYWORDS</span> = [<span class="s"><span class="dl">'</span><span class="k">class</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">module</span><span class="dl">'</span></span>]<tt>
-</tt>    <span class="co">DEF_NEW_STATE</span> = <span class="co">WordList</span>.new(<span class="sy">:initial</span>).<tt>
-</tt>      add(<span class="co">DEF_KEYWORDS</span>, <span class="sy">:def_expected</span>).<tt>
-</tt>      add(<span class="co">MODULE_KEYWORDS</span>, <span class="sy">:module_expected</span>)<tt>
-</tt><tt>
-</tt>    <span class="co">WORDS_ALLOWING_REGEXP</span> = [<tt>
-</tt>      <span class="s"><span class="dl">'</span><span class="k">and</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">or</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">not</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">while</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">until</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">unless</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">if</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">elsif</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">when</span><span class="dl">'</span></span><tt>
-</tt>    ]<tt>
-</tt>    <span class="co">REGEXP_ALLOWED</span> = <span class="co">WordList</span>.new(<span class="pc">false</span>).<tt>
-</tt>      add(<span class="co">WORDS_ALLOWING_REGEXP</span>, <span class="sy">:set</span>)<tt>
-</tt>    <tt>
-</tt>    <span class="co">PREDEFINED_CONSTANTS</span> = [<tt>
-</tt>      <span class="s"><span class="dl">'</span><span class="k">nil</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">true</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">false</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">self</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="s"><span class="dl">'</span><span class="k">DATA</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">ARGV</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">ARGF</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">__FILE__</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">__LINE__</span><span class="dl">'</span></span>,<tt>
-</tt>    ]<tt>
-</tt><tt>
-</tt>    <span class="co">IDENT_KIND</span> = <span class="co">WordList</span>.new(<span class="sy">:ident</span>).<tt>
-</tt>      add(<span class="co">RESERVED_WORDS</span>, <span class="sy">:reserved</span>).<tt>
-</tt>      add(<span class="co">PREDEFINED_CONSTANTS</span>, <span class="sy">:pre_constant</span>)<tt>
-</tt><tt>
-</tt>    <span class="co">METHOD_NAME</span> = <span class="rx"><span class="dl">/</span><span class="k"> </span><span class="il"><span class="idl">#{</span><span class="co">IDENT</span><span class="idl">}</span></span><span class="k"> [?!]? </span><span class="dl">/</span><span class="mod">xo</span></span><tt>
-</tt>    <span class="co">METHOD_NAME_EX</span> = <span class="rx"><span class="dl">/</span><span class="k"><tt>
-</tt>     </span><span class="il"><span class="idl">#{</span><span class="co">METHOD_NAME</span><span class="idl">}</span></span><span class="k">  # common methods: split, foo=, empty?, gsub!<tt>
-</tt>     | </span><span class="ch">\*</span><span class="ch">\*</span><span class="k">?         # multiplication and power<tt>
-</tt>     | [-+~]@?       # plus, minus<tt>
-</tt>     | [</span><span class="ch">\/</span><span class="k">%&amp;|^`]     # division, modulo or format strings, &amp;and, |or, ^xor, `system`<tt>
-</tt>     | </span><span class="ch">\[</span><span class="ch">\]</span><span class="k">=?        # array getter and setter<tt>
-</tt>     | &lt;=?&gt;? | &gt;=?   # comparison, rocket operator<tt>
-</tt>     | &lt;&lt; | &gt;&gt;       # append or shift left, shift right<tt>
-</tt>     | ===?          # simple equality and case equality<tt>
-</tt>    </span><span class="dl">/</span><span class="mod">ox</span></span><tt>
-</tt>    <span class="co">GLOBAL_VARIABLE</span> = <span class="rx"><span class="dl">/</span><span class="k"> </span><span class="ch">\$</span><span class="k"> (?: </span><span class="il"><span class="idl">#{</span><span class="co">IDENT</span><span class="idl">}</span></span><span class="k"> | </span><span class="ch">\d</span><span class="k">+ | [~&amp;+`'=</span><span class="ch">\/</span><span class="k">,;_.&lt;&gt;!@0$?*&quot;:F</span><span class="ch">\\</span><span class="k">] | -[a-zA-Z_0-9] ) </span><span class="dl">/</span><span class="mod">ox</span></span><tt>
-</tt><tt>
-</tt>    <span class="co">DOUBLEQ</span> = <span class="rx"><span class="dl">/</span><span class="k"> &quot;  [^&quot;</span><span class="ch">\#</span><span class="ch">\\</span><span class="k">]*  (?: (?: </span><span class="ch">\#</span><span class="ch">\{</span><span class="k">.*?</span><span class="ch">\}</span><span class="k"> | </span><span class="ch">\#</span><span class="k">(?:$&quot;)?  | </span><span class="ch">\\</span><span class="k">. ) [^&quot;</span><span class="ch">\#</span><span class="ch">\\</span><span class="k">]*  )* &quot;?  </span><span class="dl">/</span><span class="mod">ox</span></span><tt>
-</tt>    <span class="co">SINGLEQ</span> = <span class="rx"><span class="dl">/</span><span class="k"> '  [^'</span><span class="ch">\\</span><span class="k">]*    (?:                              </span><span class="ch">\\</span><span class="k">.   [^'</span><span class="ch">\\</span><span class="k">]*    )* '?  </span><span class="dl">/</span><span class="mod">ox</span></span><tt>
-</tt>    <span class="co">STRING</span>  = <span class="rx"><span class="dl">/</span><span class="k"> </span><span class="il"><span class="idl">#{</span><span class="co">SINGLEQ</span><span class="idl">}</span></span><span class="k"> | </span><span class="il"><span class="idl">#{</span><span class="co">DOUBLEQ</span><span class="idl">}</span></span><span class="k"> </span><span class="dl">/</span><span class="mod">ox</span></span><tt>
-</tt>    <span class="co">SHELL</span>   = <span class="rx"><span class="dl">/</span><span class="k"> `  [^`</span><span class="ch">\#</span><span class="ch">\\</span><span class="k">]*  (?: (?: </span><span class="ch">\#</span><span class="ch">\{</span><span class="k">.*?</span><span class="ch">\}</span><span class="k"> | </span><span class="ch">\#</span><span class="k">(?:$`)?  | </span><span class="ch">\\</span><span class="k">. ) [^`</span><span class="ch">\#</span><span class="ch">\\</span><span class="k">]*  )* `?  </span><span class="dl">/</span><span class="mod">ox</span></span><tt>
-</tt>    <span class="co">REGEXP</span>  = <span class="rx"><span class="dl">/</span><span class="k"> </span><span class="ch">\/</span><span class="k"> [^</span><span class="ch">\/</span><span class="ch">\#</span><span class="ch">\\</span><span class="k">]* (?: (?: </span><span class="ch">\#</span><span class="ch">\{</span><span class="k">.*?</span><span class="ch">\}</span><span class="k"> | </span><span class="ch">\#</span><span class="k">(?:$</span><span class="ch">\/</span><span class="k">)? | </span><span class="ch">\\</span><span class="k">. ) [^</span><span class="ch">\/</span><span class="ch">\#</span><span class="ch">\\</span><span class="k">]* )* </span><span class="ch">\/</span><span class="k">? </span><span class="dl">/</span><span class="mod">ox</span></span><tt>
-</tt>    <tt>
-</tt>    <span class="co">DECIMAL</span> = <span class="rx"><span class="dl">/</span><span class="ch">\d</span><span class="k">+(?:_</span><span class="ch">\d</span><span class="k">+)*</span><span class="dl">/</span></span>  <span class="c"># doesn't recognize 09 as octal error</span><tt>
-</tt>    <span class="co">OCTAL</span> = <span class="rx"><span class="dl">/</span><span class="k">0_?[0-7]+(?:_[0-7]+)*</span><span class="dl">/</span></span><tt>
-</tt>    <span class="co">HEXADECIMAL</span> = <span class="rx"><span class="dl">/</span><span class="k">0x[0-9A-Fa-f]+(?:_[0-9A-Fa-f]+)*</span><span class="dl">/</span></span><tt>
-</tt>    <span class="co">BINARY</span> = <span class="rx"><span class="dl">/</span><span class="k">0b[01]+(?:_[01]+)*</span><span class="dl">/</span></span><tt>
-</tt><tt>
-</tt>    <span class="co">EXPONENT</span> = <span class="rx"><span class="dl">/</span><span class="k"> [eE] [+-]? </span><span class="il"><span class="idl">#{</span><span class="co">DECIMAL</span><span class="idl">}</span></span><span class="k"> </span><span class="dl">/</span><span class="mod">ox</span></span><tt>
-</tt>    <span class="co">FLOAT</span> = <span class="rx"><span class="dl">/</span><span class="k"> </span><span class="il"><span class="idl">#{</span><span class="co">DECIMAL</span><span class="idl">}</span></span><span class="k"> (?: </span><span class="il"><span class="idl">#{</span><span class="co">EXPONENT</span><span class="idl">}</span></span><span class="k"> | </span><span class="ch">\.</span><span class="k"> </span><span class="il"><span class="idl">#{</span><span class="co">DECIMAL</span><span class="idl">}</span></span><span class="k"> </span><span class="il"><span class="idl">#{</span><span class="co">EXPONENT</span><span class="idl">}</span></span><span class="k">? ) </span><span class="dl">/</span></span><tt>
-</tt>    <span class="co">INTEGER</span> = <span class="rx"><span class="dl">/</span><span class="il"><span class="idl">#{</span><span class="co">OCTAL</span><span class="idl">}</span></span><span class="k">|</span><span class="il"><span class="idl">#{</span><span class="co">HEXADECIMAL</span><span class="idl">}</span></span><span class="k">|</span><span class="il"><span class="idl">#{</span><span class="co">BINARY</span><span class="idl">}</span></span><span class="k">|</span><span class="il"><span class="idl">#{</span><span class="co">DECIMAL</span><span class="idl">}</span></span><span class="dl">/</span></span><tt>
-</tt>    <tt>
-</tt>    <span class="r">def</span> <span class="fu">reset</span><tt>
-</tt>      <span class="r">super</span><tt>
-</tt>      <span class="iv">@regexp_allowed</span> = <span class="pc">false</span><tt>
-</tt>    <span class="r">end</span><tt>
-</tt>    <tt>
-</tt>    <span class="r">def</span> <span class="fu">next_token</span><tt>
-</tt>      <span class="r">return</span> <span class="r">if</span> <span class="iv">@scanner</span>.eos?<tt>
-</tt><tt>
-</tt>      kind = <span class="sy">:error</span><tt>
-</tt>      <span class="r">if</span> <span class="iv">@scanner</span>.scan(<span class="rx"><span class="dl">/</span><span class="ch">\s</span><span class="k">+</span><span class="dl">/</span></span>)  <span class="c"># in every state</span><tt>
-</tt>        kind = <span class="sy">:space</span><tt>
-</tt>        <span class="iv">@regexp_allowed</span> = <span class="sy">:set</span> <span class="r">if</span> <span class="iv">@regexp_allowed</span> <span class="r">or</span> <span class="iv">@scanner</span>.matched.index(<span class="i">?\n</span>)  <span class="c"># delayed flag setting</span><tt>
-</tt><tt>
-</tt>      <span class="r">elsif</span> <span class="iv">@state</span> == <span class="sy">:def_expected</span><tt>
-</tt>        <span class="r">if</span> <span class="iv">@scanner</span>.scan(<span class="rx"><span class="dl">/</span><span class="k"> (?: (?:</span><span class="il"><span class="idl">#{</span><span class="co">IDENT</span><span class="idl">}</span></span><span class="k">(?:</span><span class="ch">\.</span><span class="k">|::))* | (?:@@?|$)? </span><span class="il"><span class="idl">#{</span><span class="co">IDENT</span><span class="idl">}</span></span><span class="k">(?:</span><span class="ch">\.</span><span class="k">|::) ) </span><span class="il"><span class="idl">#{</span><span class="co">METHOD_NAME_EX</span><span class="idl">}</span></span><span class="k"> </span><span class="dl">/</span><span class="mod">ox</span></span>)<tt>
-</tt>          kind = <span class="sy">:method</span><tt>
-</tt>          <span class="iv">@state</span> = <span class="sy">:initial</span><tt>
-</tt>        <span class="r">else</span><tt>
-</tt>          <span class="iv">@scanner</span>.scan(<span class="rx"><span class="dl">/</span><span class="k">.</span><span class="dl">/</span></span>)<tt>
-</tt>          kind = <span class="sy">:error</span><tt>
-</tt>        <span class="r">end</span><tt>
-</tt>        <span class="iv">@state</span> = <span class="sy">:initial</span><tt>
-</tt>        <tt>
-</tt>      <span class="r">elsif</span> <span class="iv">@state</span> == <span class="sy">:module_expected</span><tt>
-</tt>        <span class="r">if</span> <span class="iv">@scanner</span>.scan(<span class="rx"><span class="dl">/</span><span class="k">&lt;&lt;</span><span class="dl">/</span></span>)<tt>
-</tt>          kind = <span class="sy">:operator</span><tt>
-</tt>        <span class="r">else</span><tt>
-</tt>          <span class="r">if</span> <span class="iv">@scanner</span>.scan(<span class="rx"><span class="dl">/</span><span class="k"> (?: </span><span class="il"><span class="idl">#{</span><span class="co">IDENT</span><span class="idl">}</span></span><span class="k"> (?:</span><span class="ch">\.</span><span class="k">|::))* </span><span class="il"><span class="idl">#{</span><span class="co">IDENT</span><span class="idl">}</span></span><span class="k"> </span><span class="dl">/</span><span class="mod">ox</span></span>)<tt>
-</tt>            kind = <span class="sy">:method</span><tt>
-</tt>          <span class="r">else</span><tt>
-</tt>            <span class="iv">@scanner</span>.scan(<span class="rx"><span class="dl">/</span><span class="k">.</span><span class="dl">/</span></span>)<tt>
-</tt>            kind = <span class="sy">:error</span><tt>
-</tt>          <span class="r">end</span><tt>
-</tt>          <span class="iv">@state</span> = <span class="sy">:initial</span><tt>
-</tt>        <span class="r">end</span><tt>
-</tt>        <tt>
-</tt>      <span class="r">elsif</span> <span class="c"># state == :initial</span><tt>
-</tt>        <span class="c"># IDENTIFIERS, KEYWORDS</span><tt>
-</tt>        <span class="r">if</span> <span class="iv">@scanner</span>.scan(<span class="co">GLOBAL_VARIABLE</span>)<tt>
-</tt>          kind = <span class="sy">:global_variable</span><tt>
-</tt>        <span class="r">elsif</span> <span class="iv">@scanner</span>.scan(<span class="rx"><span class="dl">/</span><span class="k"> @@ </span><span class="il"><span class="idl">#{</span><span class="co">IDENT</span><span class="idl">}</span></span><span class="k"> </span><span class="dl">/</span><span class="mod">ox</span></span>)<tt>
-</tt>          kind = <span class="sy">:class_variable</span><tt>
-</tt>        <span class="r">elsif</span> <span class="iv">@scanner</span>.scan(<span class="rx"><span class="dl">/</span><span class="k"> @ </span><span class="il"><span class="idl">#{</span><span class="co">IDENT</span><span class="idl">}</span></span><span class="k"> </span><span class="dl">/</span><span class="mod">ox</span></span>)<tt>
-</tt>          kind = <span class="sy">:instance_variable</span><tt>
-</tt>        <span class="r">elsif</span> <span class="iv">@scanner</span>.scan(<span class="rx"><span class="dl">/</span><span class="k"> __END__</span><span class="ch">\n</span><span class="k"> ( (?!</span><span class="ch">\#</span><span class="k">CODE</span><span class="ch">\#</span><span class="k">) .* )? | </span><span class="ch">\#</span><span class="k">[^</span><span class="ch">\n</span><span class="k">]* | =begin(?=</span><span class="ch">\s</span><span class="k">).*? </span><span class="ch">\n</span><span class="k">=end(?=</span><span class="ch">\s</span><span class="k">|</span><span class="ch">\z</span><span class="k">)(?:[^</span><span class="ch">\n</span><span class="k">]*)? </span><span class="dl">/</span><span class="mod">x</span></span>)<tt>
-</tt>          kind = <span class="sy">:comment</span><tt>
-</tt>        <span class="r">elsif</span> <span class="iv">@scanner</span>.scan(<span class="co">METHOD_NAME</span>)<tt>
-</tt>          <span class="r">if</span> <span class="iv">@last_token_dot</span><tt>
-</tt>            kind = <span class="sy">:ident</span><tt>
-</tt>          <span class="r">else</span><tt>
-</tt>            matched = <span class="iv">@scanner</span>.matched<tt>
-</tt>            kind = <span class="co">IDENT_KIND</span>[matched]<tt>
-</tt>            <span class="r">if</span> kind == <span class="sy">:ident</span> <span class="r">and</span> matched =~ <span class="rx"><span class="dl">/</span><span class="k">^[A-Z]</span><span class="dl">/</span></span><tt>
-</tt>              kind = <span class="sy">:constant</span><tt>
-</tt>            <span class="r">elsif</span> kind == <span class="sy">:reserved</span><tt>
-</tt>              <span class="iv">@state</span> = <span class="co">DEF_NEW_STATE</span>[matched]<tt>
-</tt>              <span class="iv">@regexp_allowed</span> = <span class="co">REGEXP_ALLOWED</span>[matched]<tt>
-</tt>            <span class="r">end</span><tt>
-</tt>          <span class="r">end</span><tt>
-</tt>          <tt>
-</tt>        <span class="r">elsif</span> <span class="iv">@scanner</span>.scan(<span class="co">STRING</span>)<tt>
-</tt>          kind = <span class="sy">:string</span><tt>
-</tt>        <span class="r">elsif</span> <span class="iv">@scanner</span>.scan(<span class="co">SHELL</span>)<tt>
-</tt>          kind = <span class="sy">:shell</span><tt>
-</tt>        <span class="c">## HEREDOCS</span><tt>
-</tt>        <span class="r">elsif</span> <span class="iv">@scanner</span>.scan(<span class="rx"><span class="dl">/</span><span class="ch">\/</span><span class="dl">/</span></span>) <span class="r">and</span> <span class="iv">@regexp_allowed</span><tt>
-</tt>           <span class="iv">@scanner</span>.unscan<tt>
-</tt>           <span class="iv">@scanner</span>.scan(<span class="co">REGEXP</span>)<tt>
-</tt>          kind = <span class="sy">:regexp</span><tt>
-</tt>        <span class="c">## %strings</span><tt>
-</tt>        <span class="r">elsif</span> <span class="iv">@scanner</span>.scan(<span class="rx"><span class="dl">/</span><span class="k">:(?:</span><span class="il"><span class="idl">#{</span><span class="co">GLOBAL_VARIABLE</span><span class="idl">}</span></span><span class="k">|</span><span class="il"><span class="idl">#{</span><span class="co">METHOD_NAME_EX</span><span class="idl">}</span></span><span class="k">|</span><span class="il"><span class="idl">#{</span><span class="co">STRING</span><span class="idl">}</span></span><span class="k">)</span><span class="dl">/</span><span class="mod">ox</span></span>)<tt>
-</tt>          kind = <span class="sy">:global_variable</span><tt>
-</tt>        <span class="r">elsif</span> <span class="iv">@scanner</span>.scan(<span class="rx"><span class="dl">/</span><span class="k"><tt>
-</tt>          </span><span class="ch">\?</span><span class="k"> (?:<tt>
-</tt>            [^</span><span class="ch">\s</span><span class="ch">\\</span><span class="k">]<tt>
-</tt>          | <tt>
-</tt>            </span><span class="ch">\\</span><span class="k"> (?:M-</span><span class="ch">\\</span><span class="k">C-|C-</span><span class="ch">\\</span><span class="k">M-|M-</span><span class="ch">\\</span><span class="k">c|c</span><span class="ch">\\</span><span class="k">M-|c|C-|M-))? (?: </span><span class="ch">\\</span><span class="k"> (?: . | [0-7]{3} | x[0-9A-Fa-f][0-9A-Fa-f] )<tt>
-</tt>          )<tt>
-</tt>        </span><span class="dl">/</span><span class="mod">ox</span></span>)<tt>
-</tt>          kind = <span class="sy">:integer</span><tt>
-</tt>          <tt>
-</tt>        <span class="r">elsif</span> <span class="iv">@scanner</span>.scan(<span class="rx"><span class="dl">/</span><span class="k"> [-+*</span><span class="ch">\/</span><span class="k">%=&lt;&gt;;,|&amp;!()</span><span class="ch">\[</span><span class="ch">\]</span><span class="k">{}~?] | </span><span class="ch">\.</span><span class="ch">\.</span><span class="k">?</span><span class="ch">\.</span><span class="k">? | ::? </span><span class="dl">/</span><span class="mod">x</span></span>)<tt>
-</tt>          kind = <span class="sy">:operator</span><tt>
-</tt>          <span class="iv">@regexp_allowed</span> = <span class="sy">:set</span> <span class="r">if</span> <span class="iv">@scanner</span>.matched[<span class="i">-1</span>,<span class="i">1</span>] =~ <span class="rx"><span class="dl">/</span><span class="k">[~=!&lt;&gt;|&amp;^,</span><span class="ch">\(</span><span class="ch">\[</span><span class="k">+</span><span class="ch">\-</span><span class="ch">\/</span><span class="ch">\*</span><span class="k">%]</span><span class="ch">\z</span><span class="dl">/</span></span><tt>
-</tt>        <span class="r">elsif</span> <span class="iv">@scanner</span>.scan(<span class="co">FLOAT</span>)<tt>
-</tt>          kind = <span class="sy">:float</span><tt>
-</tt>        <span class="r">elsif</span> <span class="iv">@scanner</span>.scan(<span class="co">INTEGER</span>)<tt>
-</tt>          kind = <span class="sy">:integer</span><tt>
-</tt>        <span class="r">elsif</span> <span class="iv">@scanner</span>.scan(<span class="rx"><span class="dl">/</span><span class="k">:(?:</span><span class="il"><span class="idl">#{</span><span class="co">GLOBAL_VARIABLE</span><span class="idl">}</span></span><span class="k">|</span><span class="il"><span class="idl">#{</span><span class="co">METHOD_NAME_EX</span><span class="idl">}</span></span><span class="k">|</span><span class="il"><span class="idl">#{</span><span class="co">STRING</span><span class="idl">}</span></span><span class="k">)</span><span class="dl">/</span><span class="mod">ox</span></span>)<tt>
-</tt>          kind = <span class="sy">:global_variable</span><tt>
-</tt>        <span class="r">else</span><tt>
-</tt>          <span class="iv">@scanner</span>.scan(<span class="rx"><span class="dl">/</span><span class="k">.</span><span class="dl">/</span><span class="mod">m</span></span>)<tt>
-</tt>        <span class="r">end</span><tt>
-</tt>      <span class="r">end</span><tt>
-</tt>      <tt>
-</tt>      token = <span class="co">Token</span>.new <span class="iv">@scanner</span>.matched, kind<tt>
-</tt><tt>
-</tt>      <span class="r">if</span> kind == <span class="sy">:regexp</span><tt>
-</tt>        token.text &lt;&lt; <span class="iv">@scanner</span>.scan(<span class="rx"><span class="dl">/</span><span class="k">[eimnosux]*</span><span class="dl">/</span></span>)<tt>
-</tt>      <span class="r">end</span><tt>
-</tt>      <tt>
-</tt>      <span class="iv">@regexp_allowed</span> = (<span class="iv">@regexp_allowed</span> == <span class="sy">:set</span>)  <span class="c"># delayed flag setting</span><tt>
-</tt><tt>
-</tt>      token<tt>
-</tt>    <span class="r">end</span><tt>
-</tt>  <span class="r">end</span><tt>
-</tt>  <tt>
-</tt>  <span class="co">ScannerList</span>.register <span class="co">RubyScanner</span>, <span class="s"><span class="dl">'</span><span class="k">ruby</span><span class="dl">'</span></span><tt>
-</tt><tt>
-</tt><span class="r">end</span><tt>
-</tt><tt>
-</tt><span class="r">module</span> <span class="cl">CodeRay</span><tt>
-</tt>  require <span class="s"><span class="dl">'</span><span class="k">scanner</span><span class="dl">'</span></span><tt>
-</tt><tt>
-</tt>  <span class="r">class</span> <span class="cl">Highlighter</span><tt>
-</tt><tt>
-</tt>    <span class="r">def</span> <span class="fu">initialize</span> lang<tt>
-</tt>      <span class="iv">@scanner</span> = <span class="co">Scanner</span>[lang].new<tt>
-</tt>    <span class="r">end</span><tt>
-</tt><tt>
-</tt>    <span class="r">def</span> <span class="fu">highlight</span> code<tt>
-</tt>      <span class="iv">@scanner</span>.feed code<tt>
-</tt>      <span class="iv">@scanner</span>.all_tokens.map { |t| t.inspect }.join <span class="s"><span class="dl">&quot;</span><span class="ch">\n</span><span class="dl">&quot;</span></span><tt>
-</tt>    <span class="r">end</span><tt>
-</tt><tt>
-</tt>  <span class="r">end</span><tt>
-</tt><tt>
-</tt>  <span class="r">class</span> <span class="cl">HTMLHighlighter</span> &lt; <span class="co">Highlighter</span><tt>
-</tt>    <tt>
-</tt>    <span class="co">ClassOfKind</span> = {<tt>
-</tt>      <span class="sy">:attribute_name</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">an</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:attribute_name_fat</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">af</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:attribute_value</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">av</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:attribute_value_fat</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">aw</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:bin</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">bi</span><span class="dl">'</span></span>,<tt>
-</tt>       <span class="sy">:char</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">ch</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:class</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">cl</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:class_variable</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">cv</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:color</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">cr</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:comment</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">c</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:constant</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">co</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:definition</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">df</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:directive</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">di</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:doc</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">do</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:doc_string</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">ds</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:exception</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">ex</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:error</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">er</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:float</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">fl</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:function</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">fu</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:global_variable</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">gv</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:hex</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">hx</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:include</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">ic</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:instance_variable</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">iv</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:integer</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">i</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:interpreted</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">in</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:label</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">la</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:local_variable</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">lv</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:oct</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">oc</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:operator_name</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">on</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:pre_constant</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">pc</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:pre_type</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">pt</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:predefined</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">pd</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:preprocessor</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">pp</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:regexp</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">rx</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:reserved</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">r</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:shell</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">sh</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:string</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">s</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:symbol</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">sy</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:tag</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">ta</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:tag_fat</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">tf</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:tag_special</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">ts</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:type</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">ty</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:variable</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">v</span><span class="dl">'</span></span>,<tt>
-</tt>      <span class="sy">:xml_text</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">xt</span><span class="dl">'</span></span>,<tt>
-</tt><tt>
-</tt>      <span class="sy">:ident</span> =&gt; <span class="sy">:NO_HIGHLIGHT</span>,<tt>
-</tt>      <span class="sy">:operator</span> =&gt; <span class="sy">:NO_HIGHLIGHT</span>,<tt>
-</tt>      <span class="sy">:space</span> =&gt; <span class="sy">:NO_HIGHLIGHT</span>,<tt>
-</tt>    }<tt>
-</tt>    <span class="co">ClassOfKind</span>[<span class="sy">:procedure</span>] = <span class="co">ClassOfKind</span>[<span class="sy">:method</span>] = <span class="co">ClassOfKind</span>[<span class="sy">:function</span>]<tt>
-</tt>    <span class="co">ClassOfKind</span>.default = <span class="co">ClassOfKind</span>[<span class="sy">:error</span>] <span class="r">or</span> raise <span class="s"><span class="dl">'</span><span class="k">no class found for :error!</span><span class="dl">'</span></span><tt>
-</tt>    <tt>
-</tt>    <span class="r">def</span> <span class="fu">initialize</span> lang, options = {}<tt>
-</tt>      <span class="r">super</span> lang<tt>
-</tt>      <tt>
-</tt>      <span class="iv">@HTML_TAB</span> = <span class="s"><span class="dl">'</span><span class="k"> </span><span class="dl">'</span></span> * options.fetch(<span class="sy">:tabs2space</span>, <span class="i">8</span>)<tt>
-</tt>      <span class="r">case</span> level = options.fetch(<span class="sy">:level</span>, <span class="s"><span class="dl">'</span><span class="k">xhtml</span><span class="dl">'</span></span>)<tt>
-</tt>        <span class="r">when</span> <span class="s"><span class="dl">'</span><span class="k">html</span><span class="dl">'</span></span><tt>
-</tt>          <span class="iv">@HTML_BR</span> = <span class="s"><span class="dl">&quot;</span><span class="k">&lt;BR&gt;</span><span class="ch">\n</span><span class="dl">&quot;</span></span><tt>
-</tt>        <span class="r">when</span> <span class="s"><span class="dl">'</span><span class="k">xhtml</span><span class="dl">'</span></span><tt>
-</tt>          <span class="iv">@HTML_BR</span> = <span class="s"><span class="dl">&quot;</span><span class="k">&lt;br /&gt;</span><span class="ch">\n</span><span class="dl">&quot;</span></span><tt>
-</tt>      <span class="r">else</span><tt>
-</tt>        raise <span class="s"><span class="dl">&quot;</span><span class="k">Unknown HTML level: </span><span class="il"><span class="idl">#{</span>level<span class="idl">}</span></span><span class="dl">&quot;</span></span><tt>
-</tt>      <span class="r">end</span><tt>
-</tt>    <span class="r">end</span><tt>
-</tt><tt>
-</tt>    <span class="r">def</span> <span class="fu">highlight</span> code<tt>
-</tt>      <span class="iv">@scanner</span>.feed code<tt>
-</tt>      <tt>
-</tt>      out = <span class="s"><span class="dl">'</span><span class="dl">'</span></span><tt>
-</tt>      <span class="r">while</span> t = <span class="iv">@scanner</span>.next_token<tt>
-</tt>        warn t.inspect <span class="r">if</span> t.text.nil?<tt>
-</tt>        out &lt;&lt; to_html(t)<tt>
-</tt>      <span class="r">end</span><tt>
-</tt>      <span class="co">TEMPLATE</span> =~ <span class="rx"><span class="dl">/</span><span class="k">&lt;%CONTENT%&gt;</span><span class="dl">/</span></span><tt>
-</tt>      <span class="gv">$`</span> + out + <span class="gv">$'</span><tt>
-</tt>    <span class="r">end</span><tt>
-</tt>    <tt>
-</tt>  private<tt>
-</tt>    <span class="r">def</span> <span class="fu">to_html</span> token<tt>
-</tt>      css_class = <span class="co">ClassOfKind</span>[token.kind]<tt>
-</tt>      <span class="r">if</span> <span class="r">defined?</span> ::<span class="co">DEBUG</span> <span class="r">and</span> <span class="r">not</span> <span class="co">ClassOfKind</span>.has_key? token.kind<tt>
-</tt>        warn <span class="s"><span class="dl">&quot;</span><span class="k">no token class found for :</span><span class="il"><span class="idl">#{</span>token.kind<span class="idl">}</span></span><span class="dl">&quot;</span></span><tt>
-</tt>      <span class="r">end</span><tt>
-</tt>        <tt>
-</tt>      text = text_to_html token.text<tt>
-</tt>      <span class="r">if</span> css_class == <span class="sy">:NO_HIGHLIGHT</span><tt>
-</tt>        text<tt>
-</tt>      <span class="r">else</span><tt>
-</tt>        <span class="s"><span class="dl">&quot;</span><span class="k">&lt;span class=</span><span class="ch">\&quot;</span><span class="il"><span class="idl">#{</span>css_class<span class="idl">}</span></span><span class="ch">\&quot;</span><span class="k">&gt;</span><span class="il"><span class="idl">#{</span>text<span class="idl">}</span></span><span class="k">&lt;/span&gt;</span><span class="dl">&quot;</span></span><tt>
-</tt>      <span class="r">end</span><tt>
-</tt>    <span class="r">end</span><tt>
-</tt>    <tt>
-</tt>    <span class="r">def</span> <span class="fu">text_to_html</span> text<tt>
-</tt>      <span class="r">return</span> <span class="s"><span class="dl">'</span><span class="dl">'</span></span> <span class="r">if</span> text.empty?<tt>
-</tt>      text = text.dup  <span class="c"># important</span><tt>
-</tt>      <span class="r">if</span> text.index(<span class="rx"><span class="dl">/</span><span class="k">[&quot;&gt;&lt;&amp;]</span><span class="dl">/</span></span>)<tt>
-</tt>        text.gsub!(<span class="s"><span class="dl">'</span><span class="k">&amp;</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">&amp;amp;</span><span class="dl">'</span></span>)<tt>
-</tt>        text.gsub!(<span class="s"><span class="dl">'</span><span class="k">&quot;</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">&amp;quot;</span><span class="dl">'</span></span>)<tt>
-</tt>        text.gsub!(<span class="s"><span class="dl">'</span><span class="k">&gt;</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">&amp;gt;</span><span class="dl">'</span></span>)<tt>
-</tt>        text.gsub!(<span class="s"><span class="dl">'</span><span class="k">&lt;</span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k">&amp;lt;</span><span class="dl">'</span></span>)<tt>
-</tt>      <span class="r">end</span><tt>
-</tt>      <span class="r">if</span> text.index(<span class="rx"><span class="dl">/</span><span class="ch">\s</span><span class="dl">/</span></span>)<tt>
-</tt>        text.gsub!(<span class="s"><span class="dl">&quot;</span><span class="ch">\n</span><span class="dl">&quot;</span></span>, <span class="iv">@HTML_BR</span>)<tt>
-</tt>        text.gsub!(<span class="s"><span class="dl">&quot;</span><span class="ch">\t</span><span class="dl">&quot;</span></span>, <span class="iv">@HTML_TAB</span>)<tt>
-</tt>        text.gsub!(<span class="rx"><span class="dl">/</span><span class="k">^ </span><span class="dl">/</span></span>, <span class="s"><span class="dl">'</span><span class="k">&amp;nbsp;</span><span class="dl">'</span></span>)<tt>
-</tt>        text.gsub!(<span class="s"><span class="dl">'</span><span class="k">  </span><span class="dl">'</span></span>, <span class="s"><span class="dl">'</span><span class="k"> &amp;nbsp;</span><span class="dl">'</span></span>)<tt>
-</tt>      <span class="r">end</span><tt>
-</tt>      text<tt>
-</tt>    <span class="r">end</span><tt>
-</tt>    <tt>
-</tt>    <span class="co">TEMPLATE</span> = <span class="s"><span class="dl">&lt;&lt;-'TEMPLATE'</span></span><span class="s"><span class="k"><tt>
-</tt>&lt;!DOCTYPE HTML PUBLIC &quot;-//W3C//DTD HTML 4.01//EN&quot; &quot;http://www.w3.org/TR/html4/strict.dtd&quot;&gt;<tt>
-</tt>&lt;html dir=&quot;ltr&quot;&gt;<tt>
-</tt>&lt;head&gt;<tt>
-</tt>&lt;meta http-equiv=&quot;Content-Type&quot; content=&quot;text/html; charset=iso-8859-1&quot;&gt;<tt>
-</tt>&lt;meta http-equiv=&quot;Content-Style-Type&quot; content=&quot;text/css&quot;&gt;<tt>
-</tt><tt>
-</tt>&lt;title&gt;RubyBB BBCode&lt;/title&gt;<tt>
-</tt>&lt;style type=&quot;text/css&quot;&gt;<tt>
-</tt>.code {<tt>
-</tt>  width: 100%;<tt>
-</tt>  background-color: #FAFAFA;<tt>
-</tt>  border: 1px solid #D1D7DC;<tt>
-</tt>  font-family: 'Courier New', 'Terminal', monospace;<tt>
-</tt>  font-size: 10pt;<tt>
-</tt>  color: black;<tt>
-</tt>  vertical-align: top;<tt>
-</tt>  text-align: left;<tt>
-</tt>}<tt>
-</tt>.code .af { color:#00C; }<tt>
-</tt>.code .an { color:#007; }<tt>
-</tt>.code .av { color:#700; }<tt>
-</tt>.code .aw { color:#C00; }<tt>
-</tt>.code .bi { color:#509; font-weight:bold; }<tt>
-</tt>.code .c  { color:#888; }<tt>
-</tt>.code .ch { color:#C28; font-weight:bold; }<tt>
-</tt>.code .cl { color:#B06; font-weight:bold; }<tt>
-</tt>.code .co { color:#036; font-weight:bold; }<tt>
-</tt>.code .cr { color:#0A0; }<tt>
-</tt>.code .cv { color:#369; }<tt>
-</tt>.code .df { color:#099; font-weight:bold; }<tt>
-</tt>.code .di { color:#088; font-weight:bold; }<tt>
-</tt>.code .do { color:#970; }<tt>
-</tt>.code .ds { color:#D42; font-weight:bold; }<tt>
-</tt>.code .er { color:#F00; background-color:#FAA; }<tt>
-</tt>.code .ex { color:#F00; font-weight:bold; }<tt>
-</tt>.code .fl { color:#60E; font-weight:bold; }<tt>
-</tt>.code .fu { color:#06B; font-weight:bold; }<tt>
-</tt>.code .gv { color:#800; font-weight:bold; }<tt>
-</tt>.code .hx { color:#058; font-weight:bold; }<tt>
-</tt>.code .i  { color:#00D; font-weight:bold; }<tt>
-</tt>.code .ic { color:#B44; font-weight:bold; }<tt>
-</tt>.code .in { color:#B2B; font-weight:bold; }<tt>
-</tt>.code .iv { color:#33B; }<tt>
-</tt>.code .la { color:#970; font-weight:bold; }<tt>
-</tt>.code .lv { color:#963; }<tt>
-</tt>.code .oc { color:#40E; font-weight:bold; }<tt>
-</tt>.code .on { color:#000; font-weight:bold; }<tt>
-</tt>.code .pc { color:#038; font-weight:bold; }<tt>
-</tt>.code .pd { color:#369; font-weight:bold; }<tt>
-</tt>.code .pp { color:#579; }<tt>
-</tt>.code .pt { color:#339; font-weight:bold; }<tt>
-</tt>.code .r  { color:#080; font-weight:bold; }<tt>
-</tt>.code .rx { color:#927; font-weight:bold; }<tt>
-</tt>.code .s  { color:#D42; font-weight:bold; }<tt>
-</tt>.code .sh { color:#B2B; font-weight:bold; }<tt>
-</tt>.code .sy { color:#A60; }<tt>
-</tt>.code .ta { color:#070; }<tt>
-</tt>.code .tf { color:#070; font-weight:bold; }<tt>
-</tt>.code .ts { color:#D70; font-weight:bold; }<tt>
-</tt>.code .ty { color:#339; font-weight:bold; }<tt>
-</tt>.code .v  { color:#036; }<tt>
-</tt>.code .xt { color:#444; }<tt>
-</tt>&lt;/style&gt;<tt>
-</tt>&lt;/head&gt;<tt>
-</tt>&lt;body&gt;<tt>
-</tt>&lt;div class=&quot;code&quot;&gt;<tt>
-</tt>&lt;%CONTENT%&gt;<tt>
-</tt>&lt;/div&gt;<tt>
-</tt>&lt;div class=&quot;validators&quot;&gt;<tt>
-</tt>&lt;a href=&quot;http://validator.w3.org/check?uri=referer&quot;&gt;&lt;img src=&quot;http://www.w3.org/Icons/valid-html401&quot; alt=&quot;Valid HTML 4.01!&quot; height=&quot;31&quot; width=&quot;88&quot; style=&quot;border:none;&quot;&gt;&lt;/a&gt;<tt>
-</tt>&lt;img style=&quot;border:0&quot; src=&quot;http://jigsaw.w3.org/css-validator/images/vcss&quot; alt=&quot;Valid CSS!&quot; &gt;<tt>
-</tt>&lt;/div&gt;    <tt>
-</tt>&lt;/body&gt;<tt>
-</tt>&lt;/html&gt;</span><span class="dl"><tt>
-</tt>    TEMPLATE</span></span><tt>
-</tt><tt>
-</tt>  <span class="r">end</span><tt>
-</tt><tt>
-</tt><span class="r">end</span><tt>
-</tt></pre></td>
-</tr></table>
-
-</body>
-</html>
diff --git a/sample/html.rb b/sample/html.rb
deleted file mode 100644
index c18284a..0000000
--- a/sample/html.rb
+++ /dev/null
@@ -1,394 +0,0 @@
-$: << '..'
-require 'coderay'
-
-tokens = CodeRay.scan DATA.read, :ruby
-html = tokens.page(:tab_width => 2, :line_numbers => :table, :title => 'CodeRay HTML Encoder Example')
-
-puts html
-
-__END__
-require 'scanner'
-
-module CodeRay
-	
-	class RubyScanner < Scanner
-		
-		RESERVED_WORDS = [
-			'and', 'def', 'end', 'in', 'or', 'unless', 'begin',
-			'defined?', 'ensure', 'module', 'redo', 'super', 'until',
-			'BEGIN', 'break', 'do', 'next', 'rescue', 'then',
-			'when', 'END', 'case', 'else', 'for', 'retry',
-			'while', 'alias', 'class', 'elsif', 'if', 'not', 'return',
-			'undef', 'yield',
-		]
-
-		DEF_KEYWORDS = ['def']
-		MODULE_KEYWORDS = ['class', 'module']
-		DEF_NEW_STATE = WordList.new(:initial).
-			add(DEF_KEYWORDS, :def_expected).
-			add(MODULE_KEYWORDS, :module_expected)
-
-		WORDS_ALLOWING_REGEXP = [
-			'and', 'or', 'not', 'while', 'until', 'unless', 'if', 'elsif', 'when'
-		]
-		REGEXP_ALLOWED = WordList.new(false).
-			add(WORDS_ALLOWING_REGEXP, :set)
-		
-		PREDEFINED_CONSTANTS = [
-			'nil', 'true', 'false', 'self',
-			'DATA', 'ARGV', 'ARGF', '__FILE__', '__LINE__',
-		]
-
-		IDENT_KIND = WordList.new(:ident).
-			add(RESERVED_WORDS, :reserved).
-			add(PREDEFINED_CONSTANTS, :pre_constant)
-
-		METHOD_NAME = / #{IDENT} [?!]? /xo
-		METHOD_NAME_EX = /
-		 #{METHOD_NAME}  # common methods: split, foo=, empty?, gsub!
-		 | \*\*?         # multiplication and power
-		 | [-+~]@?       # plus, minus
-		 | [\/%&|^`]     # division, modulo or format strings, &and, |or, ^xor, `system`
-		 | \[\]=?        # array getter and setter
-		 | <=?>? | >=?   # comparison, rocket operator
-		 | << | >>       # append or shift left, shift right
-		 | ===?          # simple equality and case equality
-		/ox
-		GLOBAL_VARIABLE = / \$ (?: #{IDENT} | \d+ | [~&+`'=\/,;_.<>!@0$?*":F\\] | -[a-zA-Z_0-9] ) /ox
-
-		DOUBLEQ = / "  [^"\#\\]*  (?: (?: \#\{.*?\} | \#(?:$")?  | \\. ) [^"\#\\]*  )* "?  /ox
-		SINGLEQ = / '  [^'\\]*    (?:                              \\.   [^'\\]*    )* '?  /ox
-		STRING  = / #{SINGLEQ} | #{DOUBLEQ} /ox
-		SHELL   = / `  [^`\#\\]*  (?: (?: \#\{.*?\} | \#(?:$`)?  | \\. ) [^`\#\\]*  )* `?  /ox
-		REGEXP  = / \/ [^\/\#\\]* (?: (?: \#\{.*?\} | \#(?:$\/)? | \\. ) [^\/\#\\]* )* \/? /ox
-		
-		DECIMAL = /\d+(?:_\d+)*/  # doesn't recognize 09 as octal error
-		OCTAL = /0_?[0-7]+(?:_[0-7]+)*/
-		HEXADECIMAL = /0x[0-9A-Fa-f]+(?:_[0-9A-Fa-f]+)*/
-		BINARY = /0b[01]+(?:_[01]+)*/
-
-		EXPONENT = / [eE] [+-]? #{DECIMAL} /ox
-		FLOAT = / #{DECIMAL} (?: #{EXPONENT} | \. #{DECIMAL} #{EXPONENT}? ) /
-		INTEGER = /#{OCTAL}|#{HEXADECIMAL}|#{BINARY}|#{DECIMAL}/
-		
-		def reset
-			super
-			@regexp_allowed = false
-		end
-		
-		def next_token
-			return if @scanner.eos?
-
-			kind = :error
-			if @scanner.scan(/\s+/)  # in every state
-				kind = :space
-				@regexp_allowed = :set if @regexp_allowed or @scanner.matched.index(?\n)  # delayed flag setting
-
-			elsif @state == :def_expected
-				if @scanner.scan(/ (?: (?:#{IDENT}(?:\.|::))* | (?:@@?|$)? #{IDENT}(?:\.|::) ) #{METHOD_NAME_EX} /ox)
-					kind = :method
-					@state = :initial
-				else
-					@scanner.scan(/./)
-					kind = :error
-				end
-				@state = :initial
-				
-			elsif @state == :module_expected
-				if @scanner.scan(/<</)
-					kind = :operator
-				else
-					if @scanner.scan(/ (?: #{IDENT} (?:\.|::))* #{IDENT} /ox)
-						kind = :method
-					else
-						@scanner.scan(/./)
-						kind = :error
-					end
-					@state = :initial
-				end
-				
-			elsif # state == :initial
-				# IDENTIFIERS, KEYWORDS
-				if @scanner.scan(GLOBAL_VARIABLE)
-					kind = :global_variable
-				elsif @scanner.scan(/ @@ #{IDENT} /ox)
-					kind = :class_variable
-				elsif @scanner.scan(/ @ #{IDENT} /ox)
-					kind = :instance_variable
-				elsif @scanner.scan(/ __END__\n ( (?!\#CODE\#) .* )? | \#[^\n]* | =begin(?=\s).*? \n=end(?=\s|\z)(?:[^\n]*)? /x)
-					kind = :comment
-				elsif @scanner.scan(METHOD_NAME)
-					if @last_token_dot
-						kind = :ident
-					else
-						matched = @scanner.matched
-						kind = IDENT_KIND[matched]
-						if kind == :ident and matched =~ /^[A-Z]/
-							kind = :constant
-						elsif kind == :reserved
-							@state = DEF_NEW_STATE[matched]
-							@regexp_allowed = REGEXP_ALLOWED[matched]
-						end
-					end
-					
-				elsif @scanner.scan(STRING)
-					kind = :string
-				elsif @scanner.scan(SHELL)
-					kind = :shell
-				## HEREDOCS
-				elsif @scanner.scan(/\//) and @regexp_allowed
-				 	@scanner.unscan
-				 	@scanner.scan(REGEXP)
-					kind = :regexp
-				## %strings
-				elsif @scanner.scan(/:(?:#{GLOBAL_VARIABLE}|#{METHOD_NAME_EX}|#{STRING})/ox)
-					kind = :global_variable
-				elsif @scanner.scan(/
-				  \? (?:
-				    [^\s\\]
-				  | 
-				    \\ (?:M-\\C-|C-\\M-|M-\\c|c\\M-|c|C-|M-))? (?: \\ (?: . | [0-7]{3} | x[0-9A-Fa-f][0-9A-Fa-f] )
-				  )
-				/ox)
-					kind = :integer
-					
-				elsif @scanner.scan(/ [-+*\/%=<>;,|&!()\[\]{}~?] | \.\.?\.? | ::? /x)
-					kind = :operator
-					@regexp_allowed = :set if @scanner.matched[-1,1] =~ /[~=!<>|&^,\(\[+\-\/\*%]\z/
-				elsif @scanner.scan(FLOAT)
-					kind = :float
-				elsif @scanner.scan(INTEGER)
-					kind = :integer
-				elsif @scanner.scan(/:(?:#{GLOBAL_VARIABLE}|#{METHOD_NAME_EX}|#{STRING})/ox)
-					kind = :global_variable
-				else
-					@scanner.scan(/./m)
-				end
-			end
-			
-			token = Token.new @scanner.matched, kind
-
-			if kind == :regexp
-				token.text << @scanner.scan(/[eimnosux]*/)
-			end
-			
-			@regexp_allowed = (@regexp_allowed == :set)  # delayed flag setting
-
-			token
-		end
-	end
-	
-	ScannerList.register RubyScanner, 'ruby'
-
-end
-
-module CodeRay
-	require 'scanner'
-
-	class Highlighter
-
-		def initialize lang
-			@scanner = Scanner[lang].new
-		end
-
-		def highlight code
-			@scanner.feed code
-			@scanner.all_tokens.map { |t| t.inspect }.join "\n"
-		end
-
-	end
-
-	class HTMLHighlighter < Highlighter
-		
-		ClassOfKind = {
-			:attribute_name => 'an',
-			:attribute_name_fat => 'af',
-			:attribute_value => 'av',
-			:attribute_value_fat => 'aw',
-			:bin => 'bi',
-	 		:char => 'ch',
-			:class => 'cl',
-			:class_variable => 'cv',
-			:color => 'cr',
-			:comment => 'c',
-			:constant => 'co',
-			:definition => 'df',
-			:directive => 'di',
-			:doc => 'do',
-			:doc_string => 'ds',
-			:exception => 'ex',
-			:error => 'er',
-			:float => 'fl',
-			:function => 'fu',
-			:global_variable => 'gv',
-			:hex => 'hx',
-			:include => 'ic',
-			:instance_variable => 'iv',
-			:integer => 'i',
-			:interpreted => 'in',
-			:label => 'la',
-			:local_variable => 'lv',
-			:oct => 'oc',
-			:operator_name => 'on',
-			:pre_constant => 'pc',
-			:pre_type => 'pt',
-			:predefined => 'pd',
-			:preprocessor => 'pp',
-			:regexp => 'rx',
-			:reserved => 'r',
-			:shell => 'sh',
-			:string => 's',
-			:symbol => 'sy',
-			:tag => 'ta',
-			:tag_fat => 'tf',
-			:tag_special => 'ts',
-			:type => 'ty',
-			:variable => 'v',
-			:xml_text => 'xt',
-
-			:ident => :NO_HIGHLIGHT,
-			:operator => :NO_HIGHLIGHT,
-			:space => :NO_HIGHLIGHT,
-		}
-		ClassOfKind[:procedure] = ClassOfKind[:method] = ClassOfKind[:function]
-		ClassOfKind.default = ClassOfKind[:error] or raise 'no class found for :error!'
-		
-		def initialize lang, options = {}
-			super lang
-			
-			@HTML_TAB = ' ' * options.fetch(:tabs2space, 8)
-			case level = options.fetch(:level, 'xhtml')
-				when 'html'
-					@HTML_BR = "<BR>\n"
-				when 'xhtml'
-					@HTML_BR = "<br />\n"
-			else
-				raise "Unknown HTML level: #{level}"
-			end
-		end
-
-		def highlight code
-			@scanner.feed code
-			
-			out = ''
-			while t = @scanner.next_token
-				warn t.inspect if t.text.nil?
-				out << to_html(t)
-			end
-			TEMPLATE =~ /<%CONTENT%>/
-			$` + out + $'
-		end
-		
-	private
-		def to_html token
-			css_class = ClassOfKind[token.kind]
-			if defined? ::DEBUG and not ClassOfKind.has_key? token.kind
-				warn "no token class found for :#{token.kind}"
-			end
-				
-			text = text_to_html token.text
-			if css_class == :NO_HIGHLIGHT
-				text
-			else
-				"<span class=\"#{css_class}\">#{text}</span>"
-			end
-		end
-		
-		def text_to_html text
-			return '' if text.empty?
-			text = text.dup  # important
-			if text.index(/["><&]/)
-				text.gsub!('&', '&amp;')
-				text.gsub!('"', '&quot;')
-				text.gsub!('>', '&gt;')
-				text.gsub!('<', '&lt;')
-			end
-			if text.index(/\s/)
-				text.gsub!("\n", @HTML_BR)
-				text.gsub!("\t", @HTML_TAB)
-				text.gsub!(/^ /, '&nbsp;')
-				text.gsub!('  ', ' &nbsp;')
-			end
-			text
-		end
-		
-		TEMPLATE = <<-'TEMPLATE'
-<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
-<html dir="ltr">
-<head>
-<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
-<meta http-equiv="Content-Style-Type" content="text/css">
-
-<title>RubyBB BBCode</title>
-<style type="text/css">
-.code {
-	width: 100%;
-	background-color: #FAFAFA;
-	border: 1px solid #D1D7DC;
-	font-family: 'Courier New', 'Terminal', monospace;
-	font-size: 10pt;
-	color: black;
-	vertical-align: top;
-	text-align: left;
-}
-.code .af { color:#00C; }
-.code .an { color:#007; }
-.code .av { color:#700; }
-.code .aw { color:#C00; }
-.code .bi { color:#509; font-weight:bold; }
-.code .c  { color:#888; }
-.code .ch { color:#C28; font-weight:bold; }
-.code .cl { color:#B06; font-weight:bold; }
-.code .co { color:#036; font-weight:bold; }
-.code .cr { color:#0A0; }
-.code .cv { color:#369; }
-.code .df { color:#099; font-weight:bold; }
-.code .di { color:#088; font-weight:bold; }
-.code .do { color:#970; }
-.code .ds { color:#D42; font-weight:bold; }
-.code .er { color:#F00; background-color:#FAA; }
-.code .ex { color:#F00; font-weight:bold; }
-.code .fl { color:#60E; font-weight:bold; }
-.code .fu { color:#06B; font-weight:bold; }
-.code .gv { color:#800; font-weight:bold; }
-.code .hx { color:#058; font-weight:bold; }
-.code .i  { color:#00D; font-weight:bold; }
-.code .ic { color:#B44; font-weight:bold; }
-.code .in { color:#B2B; font-weight:bold; }
-.code .iv { color:#33B; }
-.code .la { color:#970; font-weight:bold; }
-.code .lv { color:#963; }
-.code .oc { color:#40E; font-weight:bold; }
-.code .on { color:#000; font-weight:bold; }
-.code .pc { color:#038; font-weight:bold; }
-.code .pd { color:#369; font-weight:bold; }
-.code .pp { color:#579; }
-.code .pt { color:#339; font-weight:bold; }
-.code .r  { color:#080; font-weight:bold; }
-.code .rx { color:#927; font-weight:bold; }
-.code .s  { color:#D42; font-weight:bold; }
-.code .sh { color:#B2B; font-weight:bold; }
-.code .sy { color:#A60; }
-.code .ta { color:#070; }
-.code .tf { color:#070; font-weight:bold; }
-.code .ts { color:#D70; font-weight:bold; }
-.code .ty { color:#339; font-weight:bold; }
-.code .v  { color:#036; }
-.code .xt { color:#444; }
-</style>
-</head>
-<body>
-<div class="code">
-<%CONTENT%>
-</div>
-<div class="validators">
-<a href="http://validator.w3.org/check?uri=referer"><img src="http://www.w3.org/Icons/valid-html401" alt="Valid HTML 4.01!" height="31" width="88" style="border:none;"></a>
-<img style="border:0" src="http://jigsaw.w3.org/css-validator/images/vcss" alt="Valid CSS!" >
-</div>    
-</body>
-</html>
-		TEMPLATE
-
-	end
-
-end
diff --git a/sample/html2.expected b/sample/html2.expected
deleted file mode 100644
index c8ae56a..0000000
--- a/sample/html2.expected
+++ /dev/null
@@ -1,185 +0,0 @@
-<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
-  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
-<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="de">
-<head>
-  <meta http-equiv="content-type" content="text/html; charset=utf-8" />
-  <title>CodeRay HTML Encoder Example</title>
-  <style type="text/css">
-.CodeRay {
-  background-color: #f8f8f8;
-  border: 1px solid silver;
-  font-family: 'Courier New', 'Terminal', monospace;
-  color: #000;
-}
-.CodeRay pre { margin: 0px }
-
-div.CodeRay { }
-
-span.CodeRay { white-space: pre; border: 0px; padding: 2px }
-
-table.CodeRay { border-collapse: collapse; width: 100%; padding: 2px }
-table.CodeRay td { padding: 2px 4px; vertical-align: top }
-
-.CodeRay .line_numbers, .CodeRay .no {
-  background-color: #def;
-  color: gray;
-  text-align: right;
-}
-.CodeRay .line_numbers tt { font-weight: bold }
-.CodeRay .line_numbers .highlighted { color: red }
-.CodeRay .no { padding: 0px 4px }
-.CodeRay .code { width: 100% }
-
-ol.CodeRay { font-size: 10pt }
-ol.CodeRay li { white-space: pre }
-
-.CodeRay .code pre { overflow: auto }
-
-.CodeRay .debug { color:white ! important; background:blue ! important; }
-
-.CodeRay .af { color:#00C }
-.CodeRay .an { color:#007 }
-.CodeRay .at { color:#f08 }
-.CodeRay .av { color:#700 }
-.CodeRay .aw { color:#C00 }
-.CodeRay .bi { color:#509; font-weight:bold }
-.CodeRay .c  { color:#888; }
-
-.CodeRay .ch { color:#04D }
-.CodeRay .ch .k { color:#04D }
-.CodeRay .ch .dl { color:#039 }
-
-.CodeRay .cl { color:#B06; font-weight:bold }
-.CodeRay .cm { color:#A08; font-weight:bold }
-.CodeRay .co { color:#036; font-weight:bold }
-.CodeRay .cr { color:#0A0 }
-.CodeRay .cv { color:#369 }
-.CodeRay .de { color:#B0B; }
-.CodeRay .df { color:#099; font-weight:bold }
-.CodeRay .di { color:#088; font-weight:bold }
-.CodeRay .dl { color:black }
-.CodeRay .do { color:#970 }
-.CodeRay .dt { color:#34b }
-.CodeRay .ds { color:#D42; font-weight:bold }
-.CodeRay .e  { color:#666; font-weight:bold }
-.CodeRay .en { color:#800; font-weight:bold }
-.CodeRay .er { color:#F00; background-color:#FAA }
-.CodeRay .ex { color:#C00; font-weight:bold }
-.CodeRay .fl { color:#60E; font-weight:bold }
-.CodeRay .fu { color:#06B; font-weight:bold }
-.CodeRay .gv { color:#d70; font-weight:bold }
-.CodeRay .hx { color:#058; font-weight:bold }
-.CodeRay .i  { color:#00D; font-weight:bold }
-.CodeRay .ic { color:#B44; font-weight:bold }
-
-.CodeRay .il { background: #ddd; color: black }
-.CodeRay .il .il { background: #ccc }
-.CodeRay .il .il .il { background: #bbb }
-.CodeRay .il .idl { background: #ddd; font-weight: bold; color: #666 }
-.CodeRay .idl { background-color: #bbb; font-weight: bold; color: #666; }
-
-.CodeRay .im { color:#f00; }
-.CodeRay .in { color:#B2B; font-weight:bold }
-.CodeRay .iv { color:#33B }
-.CodeRay .la { color:#970; font-weight:bold }
-.CodeRay .lv { color:#963 }
-.CodeRay .oc { color:#40E; font-weight:bold }
-.CodeRay .of { color:#000; font-weight:bold }
-.CodeRay .op { }
-.CodeRay .pc { color:#038; font-weight:bold }
-.CodeRay .pd { color:#369; font-weight:bold }
-.CodeRay .pp { color:#579; }
-.CodeRay .ps { color:#00C; font-weight:bold }
-.CodeRay .pt { color:#074; font-weight:bold }
-.CodeRay .r, .kw  { color:#080; font-weight:bold }
-
-.CodeRay .ke { color: #808; }
-.CodeRay .ke .dl { color: #606; }
-.CodeRay .ke .ch { color: #80f; }
-.CodeRay .vl { color: #088; }
-
-.CodeRay .rx { background-color:#fff0ff }
-.CodeRay .rx .k { color:#808 }
-.CodeRay .rx .dl { color:#404 }
-.CodeRay .rx .mod { color:#C2C }
-.CodeRay .rx .fu  { color:#404; font-weight: bold }
-
-.CodeRay .s { background-color:#fff0f0; color: #D20; }
-.CodeRay .s .s { background-color:#ffe0e0 }
-.CodeRay .s .s  .s { background-color:#ffd0d0 }
-.CodeRay .s .k { }
-.CodeRay .s .ch { color: #b0b; }
-.CodeRay .s .dl { color: #710; }
-
-.CodeRay .sh { background-color:#f0fff0; color:#2B2 }
-.CodeRay .sh .k { }
-.CodeRay .sh .dl { color:#161 }
-
-.CodeRay .sy { color:#A60 }
-.CodeRay .sy .k { color:#A60 }
-.CodeRay .sy .dl { color:#630 }
-
-.CodeRay .ta { color:#070 }
-.CodeRay .tf { color:#070; font-weight:bold }
-.CodeRay .ts { color:#D70; font-weight:bold }
-.CodeRay .ty { color:#339; font-weight:bold }
-.CodeRay .v  { color:#036 }
-.CodeRay .xt { color:#444 }
-
-.CodeRay .ins { background: #afa; }
-.CodeRay .del { background: #faa; }
-.CodeRay .chg { color: #aaf; background: #007; }
-.CodeRay .head { color: #f8f; background: #505 }
-
-.CodeRay .ins .ins { color: #080; font-weight:bold }
-.CodeRay .del .del { color: #800; font-weight:bold }
-.CodeRay .chg .chg { color: #66f; }
-.CodeRay .head .head { color: #f4f; }
-
-  </style>
-</head>
-<body style="background-color: white;">
-
-<table class="CodeRay"><tr>
-  <td class="line_numbers" title="click to toggle" onclick="with (this.firstChild.style) { display = (display == '') ? 'none' : '' }"><pre>1<tt>
-</tt>2<tt>
-</tt>3<tt>
-</tt>4<tt>
-</tt>5<tt>
-</tt>6<tt>
-</tt>7<tt>
-</tt>8<tt>
-</tt>9<tt>
-</tt><strong>10</strong><tt>
-</tt>11<tt>
-</tt></pre></td>
-  <td class="code"><pre ondblclick="with (this.style) { overflow = (overflow == 'auto' || overflow == '') ? 'visible' : 'auto' }">require <span style="background-color:#fff0f0;color:#D20"><span style="color:#710">'</span><span style="">coderay</span><span style="color:#710">'</span></span><tt>
-</tt><tt>
-</tt><span style="color:#888"># scan this file</span><tt>
-</tt>tokens = <span style="color:#036;font-weight:bold">CodeRay</span>.scan(<span style="color:#036;font-weight:bold">File</span>.read(<span style="color:#d70;font-weight:bold">$0</span>) * <span style="color:#00D;font-weight:bold">1</span>, <span style="color:#A60">:ruby</span>)<tt>
-</tt><tt>
-</tt><span style="color:#888"># output it with two styles of line numbers</span><tt>
-</tt>out = tokens.div(<span style="color:#A60">:line_numbers</span> =&gt; <span style="color:#A60">:table</span>)<tt>
-</tt>out &lt;&lt; <span style="background-color:#fff0f0;color:#D20"><span style="color:#710">'</span><span style="">&lt;hr /&gt;</span><span style="color:#710">'</span></span><tt>
-</tt>out &lt;&lt; tokens.div(<span style="color:#A60">:line_numbers</span> =&gt; <span style="color:#A60">:inline</span>, <span style="color:#A60">:line_number_start</span> =&gt; <span style="color:#00D;font-weight:bold">8</span>)<tt>
-</tt><tt>
-</tt>puts out.page(<span style="color:#A60">:title</span> =&gt; <span style="background-color:#fff0f0;color:#D20"><span style="color:#710">'</span><span style="">CodeRay HTML Encoder Example</span><span style="color:#710">'</span></span>)<tt>
-</tt></pre></td>
-</tr></table>
-<hr /><div class="CodeRay">
-  <div class="code"><pre><span class="no"> 8</span> require <span style="background-color:#fff0f0;color:#D20"><span style="color:#710">'</span><span style="">coderay</span><span style="color:#710">'</span></span>
-<span class="no"> 9</span> 
-<span class="no"><strong>10</strong></span> <span style="color:#888"># scan this file</span>
-<span class="no">11</span> tokens = <span style="color:#036;font-weight:bold">CodeRay</span>.scan(<span style="color:#036;font-weight:bold">File</span>.read(<span style="color:#d70;font-weight:bold">$0</span>) * <span style="color:#00D;font-weight:bold">1</span>, <span style="color:#A60">:ruby</span>)
-<span class="no">12</span> 
-<span class="no">13</span> <span style="color:#888"># output it with two styles of line numbers</span>
-<span class="no">14</span> out = tokens.div(<span style="color:#A60">:line_numbers</span> =&gt; <span style="color:#A60">:table</span>)
-<span class="no">15</span> out &lt;&lt; <span style="background-color:#fff0f0;color:#D20"><span style="color:#710">'</span><span style="">&lt;hr /&gt;</span><span style="color:#710">'</span></span>
-<span class="no">16</span> out &lt;&lt; tokens.div(<span style="color:#A60">:line_numbers</span> =&gt; <span style="color:#A60">:inline</span>, <span style="color:#A60">:line_number_start</span> =&gt; <span style="color:#00D;font-weight:bold">8</span>)
-<span class="no">17</span> 
-<span class="no">18</span> puts out.page(<span style="color:#A60">:title</span> =&gt; <span style="background-color:#fff0f0;color:#D20"><span style="color:#710">'</span><span style="">CodeRay HTML Encoder Example</span><span style="color:#710">'</span></span>)
-</pre></div>
-</div>
-
-</body>
-</html>
diff --git a/sample/html2.rb b/sample/html2.rb
deleted file mode 100644
index 618d168..0000000
--- a/sample/html2.rb
+++ /dev/null
@@ -1,11 +0,0 @@
-require 'coderay'
-
-# scan this file
-tokens = CodeRay.scan(File.read($0) * 1, :ruby)
-
-# output it with two styles of line numbers
-out = tokens.div(:line_numbers => :table)
-out << '<hr />'
-out << tokens.div(:line_numbers => :inline, :line_number_start => 8)
-
-puts out.page(:title => 'CodeRay HTML Encoder Example')
diff --git a/sample/html_list.expected b/sample/html_list.expected
deleted file mode 100644
index a4092c8..0000000
--- a/sample/html_list.expected
+++ /dev/null
@@ -1,160 +0,0 @@
-<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
-  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
-<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="de">
-<head>
-  <meta http-equiv="content-type" content="text/html; charset=utf-8" />
-  <title>CodeRay HTML Encoder Example</title>
-  <style type="text/css">
-.CodeRay {
-  background-color: #f8f8f8;
-  border: 1px solid silver;
-  font-family: 'Courier New', 'Terminal', monospace;
-  color: #000;
-}
-.CodeRay pre { margin: 0px }
-
-div.CodeRay { }
-
-span.CodeRay { white-space: pre; border: 0px; padding: 2px }
-
-table.CodeRay { border-collapse: collapse; width: 100%; padding: 2px }
-table.CodeRay td { padding: 2px 4px; vertical-align: top }
-
-.CodeRay .line_numbers, .CodeRay .no {
-  background-color: #def;
-  color: gray;
-  text-align: right;
-}
-.CodeRay .line_numbers tt { font-weight: bold }
-.CodeRay .line_numbers .highlighted { color: red }
-.CodeRay .no { padding: 0px 4px }
-.CodeRay .code { width: 100% }
-
-ol.CodeRay { font-size: 10pt }
-ol.CodeRay li { white-space: pre }
-
-.CodeRay .code pre { overflow: auto }
-
-.CodeRay .debug { color:white ! important; background:blue ! important; }
-
-.CodeRay .af { color:#00C }
-.CodeRay .an { color:#007 }
-.CodeRay .at { color:#f08 }
-.CodeRay .av { color:#700 }
-.CodeRay .aw { color:#C00 }
-.CodeRay .bi { color:#509; font-weight:bold }
-.CodeRay .c  { color:#888; }
-
-.CodeRay .ch { color:#04D }
-.CodeRay .ch .k { color:#04D }
-.CodeRay .ch .dl { color:#039 }
-
-.CodeRay .cl { color:#B06; font-weight:bold }
-.CodeRay .cm { color:#A08; font-weight:bold }
-.CodeRay .co { color:#036; font-weight:bold }
-.CodeRay .cr { color:#0A0 }
-.CodeRay .cv { color:#369 }
-.CodeRay .de { color:#B0B; }
-.CodeRay .df { color:#099; font-weight:bold }
-.CodeRay .di { color:#088; font-weight:bold }
-.CodeRay .dl { color:black }
-.CodeRay .do { color:#970 }
-.CodeRay .dt { color:#34b }
-.CodeRay .ds { color:#D42; font-weight:bold }
-.CodeRay .e  { color:#666; font-weight:bold }
-.CodeRay .en { color:#800; font-weight:bold }
-.CodeRay .er { color:#F00; background-color:#FAA }
-.CodeRay .ex { color:#C00; font-weight:bold }
-.CodeRay .fl { color:#60E; font-weight:bold }
-.CodeRay .fu { color:#06B; font-weight:bold }
-.CodeRay .gv { color:#d70; font-weight:bold }
-.CodeRay .hx { color:#058; font-weight:bold }
-.CodeRay .i  { color:#00D; font-weight:bold }
-.CodeRay .ic { color:#B44; font-weight:bold }
-
-.CodeRay .il { background: #ddd; color: black }
-.CodeRay .il .il { background: #ccc }
-.CodeRay .il .il .il { background: #bbb }
-.CodeRay .il .idl { background: #ddd; font-weight: bold; color: #666 }
-.CodeRay .idl { background-color: #bbb; font-weight: bold; color: #666; }
-
-.CodeRay .im { color:#f00; }
-.CodeRay .in { color:#B2B; font-weight:bold }
-.CodeRay .iv { color:#33B }
-.CodeRay .la { color:#970; font-weight:bold }
-.CodeRay .lv { color:#963 }
-.CodeRay .oc { color:#40E; font-weight:bold }
-.CodeRay .of { color:#000; font-weight:bold }
-.CodeRay .op { }
-.CodeRay .pc { color:#038; font-weight:bold }
-.CodeRay .pd { color:#369; font-weight:bold }
-.CodeRay .pp { color:#579; }
-.CodeRay .ps { color:#00C; font-weight:bold }
-.CodeRay .pt { color:#074; font-weight:bold }
-.CodeRay .r, .kw  { color:#080; font-weight:bold }
-
-.CodeRay .ke { color: #808; }
-.CodeRay .ke .dl { color: #606; }
-.CodeRay .ke .ch { color: #80f; }
-.CodeRay .vl { color: #088; }
-
-.CodeRay .rx { background-color:#fff0ff }
-.CodeRay .rx .k { color:#808 }
-.CodeRay .rx .dl { color:#404 }
-.CodeRay .rx .mod { color:#C2C }
-.CodeRay .rx .fu  { color:#404; font-weight: bold }
-
-.CodeRay .s { background-color:#fff0f0; color: #D20; }
-.CodeRay .s .s { background-color:#ffe0e0 }
-.CodeRay .s .s  .s { background-color:#ffd0d0 }
-.CodeRay .s .k { }
-.CodeRay .s .ch { color: #b0b; }
-.CodeRay .s .dl { color: #710; }
-
-.CodeRay .sh { background-color:#f0fff0; color:#2B2 }
-.CodeRay .sh .k { }
-.CodeRay .sh .dl { color:#161 }
-
-.CodeRay .sy { color:#A60 }
-.CodeRay .sy .k { color:#A60 }
-.CodeRay .sy .dl { color:#630 }
-
-.CodeRay .ta { color:#070 }
-.CodeRay .tf { color:#070; font-weight:bold }
-.CodeRay .ts { color:#D70; font-weight:bold }
-.CodeRay .ty { color:#339; font-weight:bold }
-.CodeRay .v  { color:#036 }
-.CodeRay .xt { color:#444 }
-
-.CodeRay .ins { background: #afa; }
-.CodeRay .del { background: #faa; }
-.CodeRay .chg { color: #aaf; background: #007; }
-.CodeRay .head { color: #f8f; background: #505 }
-
-.CodeRay .ins .ins { color: #080; font-weight:bold }
-.CodeRay .del .del { color: #800; font-weight:bold }
-.CodeRay .chg .chg { color: #66f; }
-.CodeRay .head .head { color: #f4f; }
-
-  </style>
-</head>
-<body style="background-color: white;">
-
-<div class="CodeRay">
-  <div class="code"><pre><span class="no">-1</span> <span class="gv">$:</span> &lt;&lt; <span class="s"><span class="dl">'</span><span class="k">..</span><span class="dl">'</span></span>
-<span class="no"> <strong>0</strong></span> require <span class="s"><span class="dl">'</span><span class="k">coderay</span><span class="dl">'</span></span>
-<span class="no"> 1</span> 
-<span class="no"> 2</span> tokens = <span class="co">CodeRay</span>.scan <span class="co">File</span>.read(<span class="pc">__FILE__</span>), <span class="sy">:ruby</span>
-<span class="no"> 3</span> html = tokens.html(<span class="sy">:tab_width</span> =&gt; <span class="i">2</span>, <span class="sy">:line_numbers</span> =&gt; <span class="sy">:inline</span>, <span class="sy">:line_number_start</span> =&gt; <span class="i">-1</span>)
-<span class="no"> 4</span> 
-<span class="no"> 5</span> puts html.page(<span class="sy">:title</span> =&gt; <span class="s"><span class="dl">'</span><span class="k">CodeRay HTML Encoder Example</span><span class="dl">'</span></span>)
-<span class="no"> 6</span> 
-<span class="no"> 7</span> commment = <span class="s"><span class="dl">&lt;&lt;_</span></span><span class="s"><span class="k">
-<span class="no"> 8</span> This code must be &gt; 10 lines
-<span class="no"> 9</span> because I want to test the correct adjustment of the line numbers.</span><span class="dl">
-<span class="no"><strong>10</strong></span> _</span></span>
-</pre></div>
-</div>
-
-</body>
-</html>
diff --git a/sample/html_list.rb b/sample/html_list.rb
deleted file mode 100644
index fdfa512..0000000
--- a/sample/html_list.rb
+++ /dev/null
@@ -1,12 +0,0 @@
-$: << '..'

-require 'coderay'

-

-tokens = CodeRay.scan File.read(__FILE__), :ruby

-html = tokens.html(:tab_width => 2, :line_numbers => :inline, :line_number_start => -1)

-

-puts html.page(:title => 'CodeRay HTML Encoder Example')

-

-commment = <<_

-This code must be > 10 lines

-because I want to test the correct adjustment of the line numbers.

-_

diff --git a/sample/load_encoder.expected b/sample/load_encoder.expected
deleted file mode 100644
index 1cff356..0000000
--- a/sample/load_encoder.expected
+++ /dev/null
@@ -1,8 +0,0 @@
-CodeRay::Encoders::YAML is not defined; you must load it first.
-Now it is loaded: CodeRay::Encoders::YAML
-See?
-Require is also possible: CodeRay::Encoders::Tokens
-See?
-Now load some mapped encoders: stats and plain.
-Require all Encoders:
-[[:count, CodeRay::Encoders::Count], [:debug, CodeRay::Encoders::Debug], [:div, CodeRay::Encoders::Div], [:html, CodeRay::Encoders::HTML], [:null, CodeRay::Encoders::Null], [:page, CodeRay::Encoders::Page], [:plain, :text], [:span, CodeRay::Encoders::Span], [:statistic, CodeRay::Encoders::Statistic], [:stats, CodeRay::Encoders::Statistic], [:text, CodeRay::Encoders::Text], [:tokens, CodeRay::Encoders::Tokens], [:xml, CodeRay::Encoders::XML], [:yaml, CodeRay::Encoders::YAML]]
diff --git a/sample/load_encoder.rb b/sample/load_encoder.rb
deleted file mode 100644
index 9594bfa..0000000
--- a/sample/load_encoder.rb
+++ /dev/null
@@ -1,25 +0,0 @@
-require 'coderay'

-

-begin

-	CodeRay::Encoders::YAML

-rescue

-	puts 'CodeRay::Encoders::YAML is not defined; you must load it first.'

-end

-

-yaml_encoder = CodeRay::Encoders[:yaml] 

-print 'Now it is loaded: '

-p yaml_encoder

-puts 'See?'

-

-tokens_encoder = CodeRay.require_plugin 'CodeRay::Encoders/tokens'

-print 'Require is also possible: '

-p tokens_encoder

-puts 'See?'

-

-puts 'Now load some mapped encoders: stats and plain.'

-CodeRay.require_plugin 'CodeRay::Encoders/stats'

-CodeRay.require_plugin 'CodeRay::Encoders/plain'

-

-puts 'Require all Encoders:'

-CodeRay::Encoders.load_all

-p CodeRay::Encoders.plugin_hash.sort_by { |k,v| k.to_s }

diff --git a/sample/load_scanner.expected b/sample/load_scanner.expected
deleted file mode 100644
index a2d200d..0000000
--- a/sample/load_scanner.expected
+++ /dev/null
@@ -1,8 +0,0 @@
-CodeRay::Encoders::Ruby is not defined; you must load it first.
-Now it is loaded: CodeRay::Scanners::Ruby
-See?
-Require is also possible: CodeRay::Scanners::C
-See?
-Now load some mapped scanners: cpp and plain.
-Require all Scanners:
-[[nil, :plain], [:c, CodeRay::Scanners::C], [:cpp, :c], [:delphi, CodeRay::Scanners::Delphi], [:html, CodeRay::Scanners::HTML], [:irb, :ruby], [:nitro, :nitro_xhtml], [:nitro_xhtml, CodeRay::Scanners::NitroXHTML], [:pascal, :delphi], [:plain, CodeRay::Scanners::Plaintext], [:plaintext, CodeRay::Scanners::Plaintext], [:rhtml, CodeRay::Scanners::RHTML], [:ruby, CodeRay::Scanners::Ruby], [:xhtml, :nitro_xhtml], [:xml, :html]]
diff --git a/sample/load_scanner.rb b/sample/load_scanner.rb
deleted file mode 100644
index 23be8a2..0000000
--- a/sample/load_scanner.rb
+++ /dev/null
@@ -1,25 +0,0 @@
-require 'coderay'

-

-begin

-	CodeRay::Scanners::Ruby

-rescue

-	puts 'CodeRay::Encoders::Ruby is not defined; you must load it first.'

-end

-

-ruby_scanner = CodeRay::Scanners[:ruby]

-print 'Now it is loaded: '

-p ruby_scanner

-puts 'See?'

-

-c_scanner = CodeRay.require_plugin 'CodeRay::Scanners/c'

-print 'Require is also possible: '

-p c_scanner

-puts 'See?'

-

-puts 'Now load some mapped scanners: cpp and plain.'

-CodeRay.require_plugin 'CodeRay::Scanners/cpp'

-CodeRay.require_plugin 'CodeRay::Scanners/plain'

-

-puts 'Require all Scanners:'

-CodeRay::Scanners.load_all

-p CodeRay::Scanners.plugin_hash.sort_by { |k,v| k.to_s }

diff --git a/sample/more.expected b/sample/more.expected
deleted file mode 100644
index 196904d..0000000
--- a/sample/more.expected
+++ /dev/null
@@ -1,2 +0,0 @@
-Input: 4983B, Output: 23484B
-Take a look with your browser.
diff --git a/sample/more.rb b/sample/more.rb
deleted file mode 100644
index 0db7ba4..0000000
--- a/sample/more.rb
+++ /dev/null
@@ -1,205 +0,0 @@
-require 'coderay'
-
-c, ruby = DATA.read.split(/^---$/)
-DATA.rewind
-me = DATA.read[/.*^__END__$/m]
-$input = c + ruby + me
-
-require 'benchmark'
-time = Benchmark.realtime do
-
-	# here CodeRay comes to play
-	hl = CodeRay.encoder(:html, :tab_width => 2, :line_numbers => :table, :wrap => :div)
-	c = hl.highlight c, :c
-	ruby = hl.highlight ruby, :ruby
-	me = hl.highlight me, :ruby
-
-	body = %w[C Ruby Genereated\ by].zip([c, ruby, me]).map do |title, code|
-		"<h1>#{title}</h1>\n#{code}"
-	end.join
-	body = hl.class::Output.new(body, hl.css, :div).page!
-
-	# CodeRay also provides a simple page generator
-	$output = body #hl.class.wrap_in_page body
-end
-
-File.open('test.html', 'w') do |f|
-	f.write $output
-end
-puts 'Input: %dB, Output: %dB' % [$input.size, $output.size]
-#puts 'Created "test.html" in %0.3f seconds (%d KB/s).' % [time, $input.size / 1024.0 / time]
-puts 'Take a look with your browser.'
-
-__END__
-/**********************************************************************
-
-  version.c -
-
-  $Author: nobu $
-  $Date: 2004/03/25 12:01:40 $
-  created at: Thu Sep 30 20:08:01 JST 1993
-
-  Copyright (C) 1993-2003 Yukihiro Matsumoto
-
-**********************************************************************/
-
-#include "ruby.h"
-#include "version.h"
-#include <stdio.h>
-
-const char ruby_version[] = RUBY_VERSION;
-const char ruby_release_date[] = RUBY_RELEASE_DATE;
-const char ruby_platform[] = RUBY_PLATFORM;
-
-void
-Init_version()
-{
-    VALUE v = rb_obj_freeze(rb_str_new2(ruby_version));
-    VALUE d = rb_obj_freeze(rb_str_new2(ruby_release_date));
-    VALUE p = rb_obj_freeze(rb_str_new2(ruby_platform));
-
-    rb_define_global_const("RUBY_VERSION", v);
-    rb_define_global_const("RUBY_RELEASE_DATE", d);
-    rb_define_global_const("RUBY_PLATFORM", p);
-}
-
-void
-ruby_show_version()
-{
-    printf("ruby %s (%s) [%s]\n", RUBY_VERSION, RUBY_RELEASE_DATE, RUBY_PLATFORM);
-}
-
-void
-ruby_show_copyright()
-{
-    printf("ruby - Copyright (C) 1993-%d Yukihiro Matsumoto\n", RUBY_RELEASE_YEAR);
-    exit(0);
-}
----
-#
-# = ostruct.rb: OpenStruct implementation
-#
-# Author:: Yukihiro Matsumoto
-# Documentation:: Gavin Sinclair
-#
-# OpenStruct allows the creation of data objects with arbitrary attributes.
-# See OpenStruct for an example.
-#
-
-#
-# OpenStruct allows you to create data objects and set arbitrary attributes.
-# For example:
-#
-#   require 'ostruct' 
-#
-#   record = OpenStruct.new
-#   record.name    = "John Smith"
-#   record.age     = 70
-#   record.pension = 300
-#   
-#   puts record.name     # -> "John Smith"
-#   puts record.address  # -> nil
-#
-# It is like a hash with a different way to access the data.  In fact, it is
-# implemented with a hash, and you can initialize it with one.
-#
-#   hash = { "country" => "Australia", :population => 20_000_000 }
-#   data = OpenStruct.new(hash)
-#
-#   p data        # -> <OpenStruct country="Australia" population=20000000>
-#
-class OpenStruct
-  #
-  # Create a new OpenStruct object.  The optional +hash+, if given, will
-  # generate attributes and values.  For example.
-  #
-  #   require 'ostruct'
-  #   hash = { "country" => "Australia", :population => 20_000_000 }
-  #   data = OpenStruct.new(hash)
-  #
-  #   p data        # -> <OpenStruct country="Australia" population=20000000>
-  #
-  # By default, the resulting OpenStruct object will have no attributes. 
-  #
-  def initialize(hash=nil)
-    @table = {}
-    if hash
-      for k,v in hash
-	@table[k.to_sym] = v
-        new_ostruct_member(k)
-      end
-    end
-  end
-
-  # Duplicate an OpenStruct object members. 
-  def initialize_copy(orig)
-    super
-    @table = @table.dup
-  end
-
-  def marshal_dump
-    @table
-  end
-  def marshal_load(x)
-    @table = x
-    @table.each_key{|key| new_ostruct_member(key)}
-  end
-
-  def new_ostruct_member(name)
-    unless self.respond_to?(name)
-      self.instance_eval %{
-        def #{name}; @table[:#{name}]; end
-        def #{name}=(x); @table[:#{name}] = x; end
-      }
-    end
-  end
-
-  def method_missing(mid, *args) # :nodoc:
-    mname = mid.id2name
-    len = args.length
-    if mname =~ /=$/
-      if len != 1
-	raise ArgumentError, "wrong number of arguments (#{len} for 1)", caller(1)
-      end
-      if self.frozen?
-	raise TypeError, "can't modify frozen #{self.class}", caller(1)
-      end
-      mname.chop!
-      @table[mname.intern] = args[0]
-      self.new_ostruct_member(mname)
-    elsif len == 0
-      @table[mid]
-    else
-      raise NoMethodError, "undefined method `#{mname}' for #{self}", caller(1)
-    end
-  end
-
-  #
-  # Remove the named field from the object.
-  #
-  def delete_field(name)
-    @table.delete name.to_sym
-  end
-
-  #
-  # Returns a string containing a detailed summary of the keys and values.
-  #
-  def inspect
-    str = "<#{self.class}"
-    for k,v in @table
-      str << " #{k}=#{v.inspect}"
-    end
-    str << ">"
-  end
-
-  attr_reader :table # :nodoc:
-  protected :table
-
-  #
-  # Compare this object and +other+ for equality.
-  #
-  def ==(other)
-    return false unless(other.kind_of?(OpenStruct))
-    return @table == other.table
-  end
-end
diff --git a/sample/scanner.expected b/sample/scanner.expected
deleted file mode 100644
index 5015168..0000000
--- a/sample/scanner.expected
+++ /dev/null
@@ -1,16 +0,0 @@
-C Code: if (*p == '{') nest++;
-
-> print only operators:
-(*==)++;
-------------------------------
-
-Ruby Code: ruby_code(:can, BE, %r[q[ui]te #{ /comple/x },] => $-s, &?\xee)
-
-> has a string?
-false
-
-> number of regexps?
-2
-
-> has a string?
-"ruby_code" (ident), "(" (operator), ":can" (symbol), "," (operator), " " (space), "BE" (constant), "," (operator), " " (space), "%r[" (delimiter), "q" (content), "[" (nesting_delimiter), "ui" (content), "]" (nesting_delimiter), "te " (content), "#{" (inline_delimiter), " " (space), "/" (delimiter), "comple" (content), "/" (delimiter), "x" (modifier), " " (space), "}" (inline_delimiter), "," (content), "]" (delimiter), " " (space), "=" (operator), ">" (operator), " " (space), "$-s" (global_variable), "," (operator), " " (space), "&" (operator), "?\xee" (integer), ")" (operator)
diff --git a/sample/scanner.rb b/sample/scanner.rb
deleted file mode 100644
index 6a0245e..0000000
--- a/sample/scanner.rb
+++ /dev/null
@@ -1,36 +0,0 @@
-require 'coderay'

-

-c_code = "if (*p == '{') nest++;"

-puts 'C Code: ' + c_code

-puts

-

-c_scanner = CodeRay::Scanners[:c].new c_code

-

-puts '> print only operators:'

-for text, kind in c_scanner

-	print text if kind == :operator

-end

-puts

-puts '-' * 30

-puts

-

-ruby_code = %q!ruby_code(:can, BE, %r[q[ui]te #{ /comple/x },] => $-s, &?\xee)!

-puts 'Ruby Code: ' + ruby_code

-puts

-

-ruby_scanner = CodeRay::Scanners[:ruby].new ruby_code

-

-puts '> has a string?'

-puts ruby_scanner.

-	any? { |text, kind| kind == :string }

-puts

-

-puts '> number of regexps?'

-puts ruby_scanner.

-	select { |token| token == [:open, :regexp] }.size

-puts

-

-puts '> has a string?'

-puts ruby_scanner.

-	reject { |text, kind| not text.is_a? String }.

-	map { |text, kind| %("#{text}" (#{kind})) }.join(', ')

diff --git a/sample/server.rb b/sample/server.rb
deleted file mode 100644
index ccdff32..0000000
--- a/sample/server.rb
+++ /dev/null
@@ -1,110 +0,0 @@
-# CodeRay dynamic highlighter

-

-unless ARGV.grep(/-[hv]|--(help|version)/).empty?

-	puts <<-USAGE

-CodeRay Server 0.5

-$Id: demo_server.rb 113 2006-03-15 23:24:37Z murphy $

-

-Usage:

-	1) Start this and your browser.

-	2) Go to http://localhost:2468/?<path to the file>

-	   and you should get the highlighted version.

-

-Parameters:

-	-d     Debug mode; reload CodeRay engine for every file.

-	       (prepare for MANY "already initialized" and "method redefined"

-	       messages - ingore it.)

-	

-	...    More to come.

-	USAGE

-	exit

-end

-

-require 'webrick'

-require 'pathname'

-

-class << File

-	alias dir? directory?

-end

-

-require 'erb'

-include ERB::Util

-def url_decode s

-	s.to_s.gsub(/%([0-9a-f]{2})/i) { [$1.hex].pack 'C' }

-end

-

-class String

-	def to_link name = File.basename(self)

-		"<a href=\"?path=#{url_encode self}\">#{name}</a>"

-	end

-end

-

-require 'coderay'

-class CodeRayServlet < WEBrick::HTTPServlet::AbstractServlet

-

-	STYLE = 'style="font-family: sans-serif; color: navy;"'

-	BANNER = '<p><img src="http://rd.cYcnus.de/coderay/coderay-banner" style="border: 0" alt="Highlighted by CodeRay"/></p>'

-

-	def do_GET req, res

-		q = req.query_string || ''

-		args = Hash[*q.scan(/(.*?)=(.*?)(?:&|$)/).flatten].each_value { |v| v.replace url_decode(v) }

-		path = args.fetch 'path', '.'

-		

-		backlinks = '<p>current path: %s<br />' % html_escape(path) +

-			(Pathname.new(path) + '..').cleanpath.to_s.to_link('up') + ' - ' +

-			'.'.to_link('current') + '</p>'

-		

-		res.body = 

-			if File.dir? path

-				path = Pathname.new(path).cleanpath.to_s

-				dirs, files = Dir[File.join(path, '*')].sort.partition { |p| File.dir? p }

-

-				page = "<html><head></head><body #{STYLE}>"

-				page << backlinks

-				

-				page << '<dl>'

-				page << "<dt>Directories</dt>\n" + dirs.map do |p|

-					"<dd>#{p.to_link}</dd>\n"

-				end.join << "\n"

-				page << "<dt>Files</dt>\n" + files.map do |p|

-					"<dd>#{p.to_link}</dd>\n"

-				end.join << "\n"

-				page << "</dl>\n"

-				page << "#{BANNER}</body></html>"

-			

-			elsif File.exist? path

-				if $DEBUG

-					$".delete_if { |f| f =~ /coderay/ }

-					require 'coderay'

-				end

-				div = CodeRay.scan_file(path).html :tab_width => 8, :wrap => :div, :hint => :info

-				div.replace <<-DIV

-	<div #{STYLE}>

-		#{backlinks}

-#{div}

-	</div>

-	#{BANNER}

-				DIV

-				div.page

-			end

-

-		res['Content-Type'] = 'text/html'

-	end

-end

-

-# This port is taken by "qip_msgd" - I don't know that. Do you?

-module CodeRay

-	PORT = 0xC0DE / 20

-end

-

-server = WEBrick::HTTPServer.new :Port => CodeRay::PORT

-

-server.mount '/', CodeRayServlet

-

-server.mount_proc '/version' do |req, res|

-	res.body = 'CodeRay::Version = ' + CodeRay::Version

-	res['Content-Type'] = "text/plain"

-end

-

-trap("INT") { server.shutdown }

-server.start

diff --git a/sample/simple.expected b/sample/simple.expected
deleted file mode 100644
index b3d7875..0000000
--- a/sample/simple.expected
+++ /dev/null
@@ -1 +0,0 @@
-<span class="CodeRay">puts <span style="background-color:#fff0f0;color:#D20"><span style="color:#710">'</span><span style="">Hello, world!</span><span style="color:#710">'</span></span></span>
diff --git a/sample/simple.rb b/sample/simple.rb
deleted file mode 100644
index a3129b0..0000000
--- a/sample/simple.rb
+++ /dev/null
@@ -1,10 +0,0 @@
-

-# Load CodeRay

-# If this doesn't work, try ruby -rubygems.

-require 'coderay'

-

-# Generate HTML page for Ruby code.

-page = CodeRay.scan("puts 'Hello, world!'", :ruby).span

-

-# Print it

-puts page

diff --git a/sample/stream.rb b/sample/stream.rb
deleted file mode 100644
index 7ed8a22..0000000
--- a/sample/stream.rb
+++ /dev/null
@@ -1,25 +0,0 @@
-require 'coderay'

-

-code = File.read($0) * 500

-puts "Size of code: %d KB" % [code.size / 1024]

-

-puts "Use your system's memory tracker to see how much RAM this takes."

-print 'Press some key to continue...'; gets

-

-require 'benchmark'

-e = CodeRay.encoder(:div)

-for do_stream in [true, false]

-	puts "Scanning and encoding in %s mode, please wait..." %

-		[do_stream ? 'streaming' : 'normal']

-	output = ''

-	time = Benchmark.realtime do

-		if do_stream

-			output = e.encode_stream(code, :ruby)

-		else

-			output = e.encode_tokens(t = CodeRay.scan(code, :ruby))

-		end

-	end

-	puts 'Finished after %4.2f seconds.' % time

-	puts "Size of output: %d KB" % [output.size / 1024]

-	print 'Press some key to continue...'; gets

-end

diff --git a/sample/stream2.expected b/sample/stream2.expected
deleted file mode 100644
index 83aee98..0000000
--- a/sample/stream2.expected
+++ /dev/null
@@ -1,2 +0,0 @@
-kind: regexp, text size: 5.
-kind: space, text size: 1.
diff --git a/sample/stream2.rb b/sample/stream2.rb
deleted file mode 100644
index d43cc9a..0000000
--- a/sample/stream2.rb
+++ /dev/null
@@ -1,8 +0,0 @@
-require 'coderay'

-

-token_stream = CodeRay::TokenStream.new do |kind, text|

-  puts 'kind: %s, text size: %d.' % [kind, text.size]

-end

-

-token_stream << [:regexp, '/\d+/'] << [:space, "\n"]

-#-> kind: rexpexp, text size: 5.

diff --git a/sample/suite.rb b/sample/suite.rb
deleted file mode 100644
index fa24114..0000000
--- a/sample/suite.rb
+++ /dev/null
@@ -1,86 +0,0 @@
-mydir = File.dirname(__FILE__)
-$:.unshift mydir + '/../lib/'
-
-$VERBOSE = true
-
-require 'test/unit'
-include Test::Unit
-
-class CodeRaySuite < TestCase
-
-  def self.dir &block
-    @dir ||= File.dirname(__FILE__)
-    if block
-      Dir.chdir @dir, &block
-    end
-    @dir
-  end
-
-  def dir &block
-    self.class.dir(&block)
-  end
-
-  def test_ALL
-    dir do
-      for input in Dir["*.rb"] - %w(server.rb stream.rb suite.rb)
-        next if input[/^load_/]
-        puts "[ testing #{input}... ]"
-        name = File.basename(input, ".rb")
-        output = name + '.expected'
-        code = File.open(input, 'rb') { |f| break f.read }
-        
-        result = `ruby -wI../lib #{input}`
-        
-        diff = output.sub '.expected', '.diff'
-        File.delete diff if File.exist? diff
-        computed = output.sub '.expected', '.actual'
-        if File.exist? output
-          expected = File.read output
-          ok = expected == result
-          unless ok
-            File.open(computed, 'w') { |f| f.write result }
-            `diff #{output} #{computed} > #{diff}` if $DEBUG
-            puts "Test failed; output written to #{diff}."
-          end
-          assert(ok, "Output error: #{computed} != #{output}") unless $DEBUG
-        else
-          File.open(output, 'w') do |f| f.write result end
-          puts "New test: #{output}"
-        end
-
-      end
-    end
-  end
-
-end
-
-require 'test/unit/testsuite'
-$suite = TestSuite.new 'CodeRay Demos Test'
-$suite << CodeRaySuite.suite
-
-def load_suite name
-  begin
-    require name + '/suite.rb'
-  rescue LoadError
-    $stderr.puts <<-ERR
-
-!! Folder #{File.split(__FILE__).first + '/' + name} not found
-
-    ERR
-    false
-  end
-end
-
-if subsuite = ARGV.find { |a| break $1 if a[/^([^-].*)/] }
-  load_suite(subsuite) or exit
-else
-  Dir[mydir + '/*/'].each { |suite| load_suite suite }
-end
-
-if ARGV.include? '-f'
-  require 'test/unit/ui/fox/testrunner'
-  UI::Fox::TestRunner.run $suite
-else
-  require 'test/unit/ui/console/testrunner'
-  UI::Console::TestRunner.run $suite
-end
diff --git a/sample/tokens.expected b/sample/tokens.expected
deleted file mode 100644
index 747904e..0000000
--- a/sample/tokens.expected
+++ /dev/null
@@ -1 +0,0 @@
-[["puts", :ident], [" ", :space], ["3", :integer], [" ", :space], ["+", :operator], [" ", :space], ["4", :integer], [",", :operator], [" ", :space], [:open, :string], ["'", :delimiter], ["3 + 4", :content], ["'", :delimiter], [:close, :string]]
diff --git a/sample/tokens.rb b/sample/tokens.rb
deleted file mode 100644
index 91b8abb..0000000
--- a/sample/tokens.rb
+++ /dev/null
@@ -1,3 +0,0 @@
-require 'coderay'

-

-p CodeRay.scan("puts 3 + 4, '3 + 4'", :ruby)

diff --git a/test/executable/suite.rb b/test/executable/suite.rb
index d386f4b..997405c 100644
--- a/test/executable/suite.rb
+++ b/test/executable/suite.rb
@@ -14,7 +14,7 @@ class TestCodeRayExecutable < Test::Unit::TestCase
   
   ROOT_DIR = Pathname.new(File.dirname(__FILE__)) + '..' + '..'
   EXECUTABLE = ROOT_DIR + 'bin' + 'coderay'
-  RUBY_COMMAND = RUBY_VERSION < '2.0.0' ? 'ruby -w' : 'ruby'  # Ruby 2 currently throws warnings for bundler
+  RUBY_COMMAND = 'ruby'
   EXE_COMMAND =
     if RUBY_PLATFORM === 'java' && `ruby --ng -e '' 2> /dev/null` && $?.success?
       # use Nailgun
diff --git a/test/functional/basic.rb b/test/functional/basic.rb
index 3053b54..752d4ba 100755
--- a/test/functional/basic.rb
+++ b/test/functional/basic.rb
@@ -164,9 +164,7 @@ more code  # and another comment, in-line.
       end
     end
     assert_equal 'reserved', CodeRay::TokenKinds[:reserved]
-    assert_warning 'Undefined Token kind: :shibboleet' do
-      assert_equal false, CodeRay::TokenKinds[:shibboleet]
-    end
+    assert_equal false,      CodeRay::TokenKinds[:shibboleet]
   end
   
   class Milk < CodeRay::Encoders::Encoder
diff --git a/test/functional/examples.rb b/test/functional/examples.rb
index 15f9ca3..985ef87 100755
--- a/test/functional/examples.rb
+++ b/test/functional/examples.rb
@@ -114,7 +114,7 @@ Token Types (7):
     
     # format the tokens
     term = terminal_encoder.encode_tokens(tokens)
-    assert_equal "\e[1;31mimport\e[0m \e[33mthis\e[0m;  \e[37m# The Zen of Python\e[0m", term
+    assert_equal "\e[32mimport\e[0m \e[31mthis\e[0m;  \e[1;30m# The Zen of Python\e[0m", term
     
     # re-using scanner and encoder
     ruby_highlighter = CodeRay::Duo[:ruby, :div]
diff --git a/test/unit/debug.rb b/test/unit/debug.rb
index f2b80bd..88baf56 100644
--- a/test/unit/debug.rb
+++ b/test/unit/debug.rb
@@ -18,15 +18,15 @@ class DebugEncoderTest < Test::Unit::TestCase
     [:begin_group, :string],
     ['test', :content],
     [:end_group, :string],
-    [:begin_line, :test],
+    [:begin_line, :head],
     ["\n", :space],
     ["\n  \t", :space],
     ["   \n", :space],
     ["[]", :method],
-    [:end_line, :test],
+    [:end_line, :head],
   ].flatten
   TEST_OUTPUT = <<-'DEBUG'.chomp
-integer(10)operator((\\\))string<content(test)>test[
+integer(10)operator((\\\))string<content(test)>head[
 
   	   
 method([])]
@@ -62,10 +62,10 @@ method([])]
     [:begin_group, :string],
     ['test', :content],
     [:end_group, :string],
-    [:begin_line, :test],
+    [:begin_line, :unknown],
     ["\n\n  \t   \n", :space],
     ["[]", :method],
-    [:end_line, :test],
+    [:end_line, :unknown],
   ].flatten
   
   def test_filtering_text_tokens
diff --git a/test/unit/tokens.rb b/test/unit/tokens.rb
index 86dc632..73b0fd5 100644
--- a/test/unit/tokens.rb
+++ b/test/unit/tokens.rb
@@ -18,15 +18,6 @@ class TokensTest < Test::Unit::TestCase
     assert_equal tokens.count, 4
   end
   
-  def test_dump_undump
-    tokens = make_tokens
-    tokens2 = nil
-    assert_nothing_raised do
-      tokens2 = tokens.dump.undump
-    end
-    assert_equal tokens, tokens2
-  end
-  
   def test_to_s
     assert_equal 'string()', make_tokens.to_s
   end