renaming changelogs

3 files changed, 661 insertions, 661 deletions

diff --git a/Changes-1.0.textile b/Changes-1.0.textile
deleted file mode 100644
index 5448767..0000000
--- a/Changes-1.0.textile
+++ /dev/null
@@ -1,345 +0,0 @@
-h1=. CodeRay Version History
- 
-p=. _This files lists all changes in the CodeRay library since the 0.9.8 release._
- 
-{{toc}}
- 
-h2. Changes in 1.0
- 
-CodeRay 1.0 is a major rewrite of the library, and incompatible to earlier versions.
- 
-The command line and programmer interfaces are similar to 0.9, but the internals have completely changed.
-
-h3. General changes
- 
-* *NEW*: The new Diff scanner colorizes code inside of the diff, and highlights inline changes.
-* *NEW*: Extended support and usage of HTML5 and CSS 3 features.
-* *NEW*: Direct Streaming
-* *NEW* scanners: Clojure and HAML
-* *CHANGED*: Token classes (used as CSS classes) are readable names; breaks you stylesheet!
-* *IMPROVED* documentation
-* *IMPROVED* speed: faster startup (using @autoload@), scanning, and encoding
-* *IMPROVED* Ruby 1.9 encodings support
-* *IMPROVED* Tests: There are more of them now!
-
-h3. Direct Streaming
- 
-CodeRay 1.0 introduces _Direct Streaming_ as a faster and simpler alternative to Tokens. It means that all Scanners,
-Encoders and Filters had to be rewritten, and that older scanners using the Tokens API are no longer compatible with
-this version.
- 
-The main benefits of this change are:
- 
-* more speed (benchmarks show 10% to 50% more tokens per second compared to CodeRay 0.9)
-* the ability to stream output into a pipe on the command line
-* a simpler API
-* less code
- 
-Changes related to the new tokens handling include:
-* *CHANGED*: The Scanners now call Encoders directly; tokens are not added to a Tokens array, but are send to the
-  Encoder as a method call. The Tokens representation (which can be seen as a cache now) is still present, but as a
-  special case; Tokens just encodes the given tokens into an Array for later use.
-* *CHANGED*: The token actions (@text_token@, @begin_group@ etc.) are now public methods of @Encoder@ and @Tokens@.
-* *REWRITE* of all Scanners, Encoders, Filters, and Tokens.
-* *RENAMED* @:open@ and @:close@ actions to @:begin_group@ and @:end_group@.
-* *RENAMED* @open_token@ and @close_token@ methods to @begin_group@ and @end_group@.
-* *NEW* method @#tokens@ allows to add several tokens to the stream. @Tokens@ and @Encoders::Encoder@ define this
-  method.
-* *CHANGED* The above name changes also affect the JSON, XML, and YAML encoders. CodeRay 1.0 output will be incompatible
-  with earlier versions.
-* *REMOVED* @TokenStream@ and the @Streamable@ API and all related features like @NotStreamableError@ are now obsolete
-  and have been removed.
-
-h3. Command Line
- 
-The @coderay@ executable was rewritten and has a few new features:
- 
-* *NEW* Ability to stream into a pipe; try @coderay file | more -r@
-* *NEW* help
-* *IMPROVED*: more consistent parameter handling
-* *REMOVED* @coderay_stylesheet@ executable; use @coderay stylesheet [name]@.
-
-h3. @Tokens@
- 
-* *NEW* methods @count@, @begin_group@, @end_group@, @begin_line@, and @end_line@.
-* *REMOVED* methods @#stream?@, @#each_text_token@.
-* *REMOVED* methods @#optimize@, @#fix@, @#split_into_lines@ along with their bang! variants.
-* *REMOVED* @#text@ and @#text_size@ methods. Use the @Text@ encoder instead.
-* *REMOVED* special implementation of @#each@ taking a filter parameter. Use @TokenKindFilter@ instead.
-
-h3. *RENAMED*: @TokenKinds@
- 
-Renamed from @Tokens::ClassOfKind@ (was also @Tokens::AbbreviationForKind@ for a while).
-The term "token class" is no longer used in CodeRay. Instead, tokens have _kinds_.
-See "#122":http://redmine.rubychan.de/issues/122.
- 
-* *CHANGED* all token CSS classes to readable names.
-* *ADDED* token kinds @:filename@, @:namespace@, and @:eyecatcher@.
-* *RENAMED* @:pre_constant@ and @:pre_type@ to @:predefined_constant@ and @predefined_type@.
-* *RENAMED* @:oct@ and @:bin@ to @:octal@ and @binary@.
-* *REMOVED* token kinds @:attribute_name_fat@, @:attribute_value_fat@, @:operator_fat@, @interpreted@,
-  @:tag_fat@, @tag_special@, @:xml_text@, @:nesting_delimiter@, @:open@, and @:close@.
-* *CHANGED*: Don't raise error for unknown token kinds unless in @$CODERAY_DEBUG@ mode.
-* *CHANGED* the value for a token kind that is not highlighted from
-  @:NO_HIGHLIGHT@ to @false@.
-
-h3. @Duo@
- 
-* *NEW* method @call@ for allowing code like @CodeRay::Duo[:python => :yaml].(code)@ in Ruby 1.9.
-
-h3. @Encoders::CommentFilter@
- 
-* *NEW* alias @:remove_comments@
-
-h3. @Encoders::Filter@
- 
-* *NEW* option @tokens@.
-* *CHANGED*: Now it simply delegates to the output.
-* *REMOVED* @include_text_token?@ and @include_block_token?@ methods.
-
-h3. @Encoders::HTML@
- 
-The HTML encoder was cleaned up and simplified.
- 
-* *NEW*: HTML5 and CSS 3 compatible.
-  See "#215":http://redmine.rubychan.de/issues/215.
-* *ADDED* support for @:line_number_anchors@.
-  See "#208":http://redmine.rubychan.de/issues/208.
-* *CHANGED* the default style to @:alpha@.
-* *CHANGED*: Use double click to toggle line numbers in table mode (as single
-  click jumps to an anchor.)
-* *REMOVED* support for @:line_numbers => :list@.
-* *FIXED* splitting of lines for @:line_numbers => :inline@, so that the line
-  numbers don't get colored, too.
-* *RENAMED* @Output#numerize@ to @#number@, which is an actual English word.
-
-h3. @Encoders::LinesOfCode@
- 
-* *CHANGED*: @compile@ and @finish@ methods are now protected.
-
-h3. *Renamed*: @Encoders::Terminal@ (was @Encoders::Term@)
- 
-* *RENAMED* from @Encoders::Term@, added @:term@ alias.
-* *CLEANUP*: Use @#setup@'s @super@, don't use @:procedure@ token class.
-* *CHANGED*: @#token@'s second parameter is no longer optional.
-* *REMOVED* colors for obsolete token kinds.
-* *FIXED* handling of line tokens.
-
-h3. @Encoders::Text@
- 
-* *FIXED* default behavior of stripping the trailing newline.
-
-h3. *RENAMED*: @Encoders::TokenKindFilter@ (was @Encoders::TokenClassFilter@)
- 
-* *NEW*: Handles token groups.
-  See "#223":http://redmine.rubychan.de/issues/223.
-* *RENAMED* @include_block_token?@ to @include_group?@.
-
-h3. @Encoders::Statistic@
- 
-* *CHANGED*: Tokens actions are counted separately.
-
-h3. @Scanners::Scanner@
- 
-* *NEW* methods @#file_extension@ and @#encoding@.
-* *NEW*: The @#tokenize@ method also takes an Array of Strings as source. The
-  code is highlighted as one and split into parts of the input lengths
-  after that using @Tokens#split_into_parts@.
-* *NEW* method @#binary_string@
-* *REMOVED* helper method @String#to_unix@.
-* *REMOVED* method @#streamable?@.
-* *REMOVED* @#marshal_load@ and @#marshal_dump@.
-* *RENAMED* class method @normify@ to @normalize@; it also deals with encoding now.
-* *CHANGED*: @#column@ starts counting with 1 instead of 0
-
-h3. *NEW*: @Scanners::Clojure@
- 
-Thanks to Licenser, CodeRay now supports the Clojure language.
-
-h3. @Scanners::CSS@
- 
-* *NEW*: Rudimentary support for the @attr@, @counter@, and @counters@ functions.
-  See "#224":http://redmine.rubychan.de/issues/224.
-* *NEW*: Rudimentary support for CSS 3 colors.
-* *CHANGED*: Attribute selectors are highlighted as @:attribute_name@ instead of @:string@.
-* *CHANGED*: Comments are scanned as one token instead of three.
-
-h3. @Scanners::Debug@
- 
-* *NEW*: Support for line tokens (@begin_line@ and @end_line@ represented by @[@ and @]@.)
-* *FIXED*: Don't send @:error@ and @nil@ tokens for buggy input any more.
-* *FIXED*: Closes unclosed tokens at the end of @scan_tokens@.
-* *IMPROVED*: Highlight unknown tokens as @:error@.
-* *CHANGED*: Raises an error when trying to end an invalid token group.
-
-h3. @Scanners::Delphi@
- 
-* *FIXED*: Closes open string groups.
-
-h3. @Scanners::Diff@
- 
-* *NEW*: Highlighting of code based on file names.
-  See ticket "#52":http://redmine.rubychan.de/issues/52.
-  
-  Use the @:highlight_code@ option to turn this feature off. It's enabled
-  by default.
-  
-  This is a very original feature. It enables multi-language highlighting for
-  diff files, which is especially helpful for CodeRay development itself. The
-  updated version of the scanner test suite generated .debug.diff.html files
-  using this.
-  
-  Note: This is still experimental. Tokens spanning more than one line
-  may get highlighted incorrectly. CodeRay tries to keep scanner states
-  between the lines and changes, but the quality of the results depend on
-  the scanner.
-* *NEW*: Inline change highlighting, as suggested by Eric Thomas.
-  See ticket "#227":http://redmine.rubychan.de/issues/227 for details.
-  
-  Use the @:inline_diff@ option to turn this feature off. It's enabled by
-  default.
-  
-  For single-line changes (that is, a single deleted line followed by a single
-  inserted line), this feature surrounds the changed parts with an
-  @:eyecatcher@ group which appears in a more saturated background color.
-  The implementation is quite complex, and highly experimental. The problem
-  with multi-layer tokenizing is that the tokens have to be split into parts.
-  If the inline change starts, say, in the middle of a string, then additional
-  @:end_group@ and @:begin_group@ tokens must be inserted to keep the group
-  nesting intact. The extended @Scanner#tokenize@ method and the new
-  @Tokens#split_into_parts@ method take care of this.
-* *NEW*: Highlight the file name in the change headers as @:filename@.
-* *CHANGED*: Highlight unknown lines as @:comment@ instead of @:head@.
-* *IMPROVED*: Background colors for Diff output have been optimized.
-
-h3. *RENAMED*: @Scanners::ERB@ (was @Scanners::RHTML@)
-
-h3. *NEW*: @Scanners::HAML@
- 
-It uses the new :state options of the HTML and Ruby scanners.
- 
-Some rare cases are not considered (like @#{...}@ snippets inside of :javascript blocks),
-but it highlights pretty well.
-
-h3. @Scanners::HTML@
- 
-* *FIXED*: Closes open string groups.
-
-h3. @Scanners::JavaScript@
- 
-* *IMPROVED*: Added @NaN@ and @Infinity@ to list of predefined constants.
-* *IMPROVED* recognition of RegExp literals with leading spaces.
-
-h3. @Scanners::Java@
- 
-* *NEW*: Package names are highlighted as @:namespace@.
-  See "#210":http://redmine.rubychan.de/issues/210.
-
-h3. *REMOVED*: @Scanners::NitroXHTML@
- 
-Nitro is "dead":http://www.nitrohq.com/.
-
-h3. *RENAMED*: @Scanners::Text@ (was @Scanners::Plaintext@)
- 
-* *IMPROVED*: Just returns the string without scanning (faster).
- 
- This is much faster than scanning until @/\z/@ in Ruby 1.8.
-
-h3. @Scanners::Python@
- 
-* *CHANGED*: Docstrings are highlighted as @:comment@.
-  See "#190":http://redmine.rubychan.de/issues/190.
-
-h3. *NEW*: @Scanners::Raydebug@
-
-Copied from @Scanners::Debug@, highlights the token dump instead of importing it. It also reacts to the @.raydebug@ file
-name suffix now.
-
-h3. @Scanners::Ruby@
- 
-* *ADDED* more predefined keywords (see http://murfy.de/ruby-constants).
-* *IMPROVED* support for singleton method definitions.
-  See "#147":http://redmine.rubychan.de/issues/147.
-* *FIXED*: Don't highlight methods with a capital letter as constants
-  (eg. @GL.PushMatrix@).
-* *NEW*: Highlight buggy floats (like .5) as @:error@.
-* *CLEANUP* of documentation, names of constants and variables, state handling.
-  
-  Moved @StringState@ class from @patterns.rb@ into a separate file.
-* *NEW*: Complicated rule for recognition of @foo=@ style method names.
-* *NEW*: Handles @:keep_state@ option (a bit; experimental).
-  
-  Actually, Ruby checks if there is @[~>=]@, but not @=>@ following the name.
-  
-* *REMOVED* @EncodingError@
-
-h3. *REMOVED* @Scanners::Scheme@
- 
-* It is too buggy, and nobody was using it. To be added again when it's fixed.
-  See "#59":http://redmine.rubychan.de/issues/59.
-
-h3. @Scanners::SQL@
- 
-* *IMPROVED*: Extended list of keywords and functions (thanks to
-  Joshua Galvez, Etienne Massip, and others).
- 
-  See "#221":http://redmine.rubychan.de/issues/221.
-* *FIXED*: Closes open string groups.
-* *FIXED*: Words after @.@ are always recognized as @:ident@.
-
-h3. @Scanners::YAML@
- 
-* *FIXED*: Allow spaces before colon in mappings.
- 
-  See "#231":http://redmine.rubychan.de/issues/231.
-
-h3. *NEW*: @Styles::Alpha@
-
-A style that uses transparent HSLA colors as defined in CSS 3. See "#199":http://redmine.rubychan.de/issues/199.
- 
-It also uses the CSS 3 property @user-select: none@ to keep the user from selecting the line numbers. This is especially
-nice for @:inline@ line numbers. See "#226":http://redmine.rubychan.de/issues/226.
-
-h3. @WordList@
- 
-Stripped down to 19 LOC.
-
-* *RENAMED* @CaseIgnoringWordList@ to @WordList::CaseIgnoring@.
-* *REMOVED* caching option because it creates memory leaks.
-* *REMOVED* block option.
-
-h3. @FileType@
- 
-* *NEW*: Recognizes @.gemspec@, @.rjs@, @.rpdf@ extensions, @Gemfile@, and @Capfile@ as Ruby.
-  
-  Thanks to the authors of the TextMate Ruby bundle!
-* *REMOVED* @FileType#shebang@ is a protected method now.
-
-h3. @Plugin@
- 
-* *IMPROVED*: @register_for@ sets the @plugin_id@; it can now be a @Symbol@.
-* *ADDED* @PluginHost#const_missing@ method: Plugins are loaded automatically.
-  Using @Scanners::JavaScript@ in your code loads @scanners/java_script.rb@.
-* *ADDED* @#all_plugins@ method to simplify getting 
-  information about all available plugins (suggested by bnhymn).
-* *CHANGED* the default plugin key from @nil@ to @:default@.
-
-h3. @GZip@
- 
-* *MOVED* into @CodeRay@ namespace.
-* *MOVED* file from @gzip_simple.rb@ to @gzip.rb@.
-* *REMOVED* @String@ extensions.
-
-h3. More API changes
- 
-* *FIXED* @Encoders::HTML#token@'s second parameter is no longer optional.
-* *CHANGED* @Encoders::HTML::Output@'s API.
-* *REMOVED* lots of unused methods.
- 
-The helper classes were cleaned up; see above for details.
- 
-* *CHANGED* @Plugin@ API was simplified and stripped of all unnecessary features.
-* *CHANGED* Moved @GZip@ and @FileType@ libraries into @CodeRay@; cleaned them up.
-
-
-
diff --git a/Changes-pre-1.0.textile b/Changes-pre-1.0.textile
new file mode 100644
index 0000000..cdbfdd9
--- /dev/null
+++ b/Changes-pre-1.0.textile
@@ -0,0 +1,421 @@
+h1=. CodeRay Version History
+ 
+p=. _This files lists all changes in the CodeRay library since the 0.8.4 release._
+ 
+{{toc}}
+ 
+h2. Changes in 0.9.8 "banister" [2011-05-01]
+ 
+Fixes for JRuby's 1.9 mode and minor issues.
+ 
+h3. Rake tasks
+ 
+* *REMOVED* obsolete @has_rdoc@ gem specification, fixing a warning.
+
+h3. @Scanners::Scanner@
+ 
+* *NEW* method @#scan_rest@ replaces @scan_until(/\z/)@, which is broken in JRuby 1.6 --1.9 mode.
+  See "#297":http://redmine.rubychan.de/issues/297.
+
+h3. @Scanners::CSS@
+ 
+* *FIXED* LOC counting (should be 0).
+  See "#296":http://redmine.rubychan.de/issues/296.
+
+h3. @Scanners::Ruby@
+ 
+* *FIXED* the @IDENT@ pattern not to use character properties, which are broken in JRuby 1.6 --1.9 mode.
+  See "#297":http://redmine.rubychan.de/issues/297, thanks to banister for reporting!
+
+h3. @Scanners::SQL@
+ 
+* *ADDED* more keywords: @between@, @databases@, @distinct@, @fields@, @full@, @having@, @is@, @prompt@, @tables@.
+  See "#221":http://redmine.rubychan.de/issues/221, thanks to Etienne Massip again.
+
+h3. @FileType@
+ 
+* *NEW* regonizes ColdFusion file type extensions @.cfm@ and @.cfc@ as XML.
+  See "#298":http://redmine.rubychan.de/issues/298, thanks to Emidio Stani.
+
+
+h2. Changes in 0.9.7 "Etienne" [2011-01-14]
+ 
+Fixes a dangerous JavaScript scanner bug, and a testing problem with Ruby 1.9.1.
+ 
+h3. Tests
+ 
+* *FIXED* The functional tests now load the lib directory (instead of the gem) in Ruby 1.9.1.
+
+h3. @Scanners::JavaScript@
+ 
+* *FIXED* @KEY_CHECK_PATTERN@ regexp
+  See "#264":http://redmine.rubychan.de/issues/264, thanks to Etienne Massip!
+
+
+h2. Changes in 0.9.6 "WoNáDo" [2010-11-25]
+ 
+Minor improvements to the Ruby scanner and a fix for Ruby 1.9.
+ 
+h3. @Scanners::Ruby@
+ 
+* *IMPROVED* handling of new hash syntax (keys are marked as @:key@ now,
+  colon is a separate @:operator@ token, all idents can be used as keys)
+  See "#257":http://code.licenser.net/issues/257, thanks to WoNáDo!
+* *ADDED* @__ENCODING__@ magic constant (Ruby 1.9)
+* *FIXED*: Scanner no longer tries to modify the input string on Ruby 1.9.
+  See "#260":http://code.licenser.net/issues/260, thanks to Jan Lelis!
+
+
+h2. Changes in 0.9.5 "Germany.rb" [2010-09-28]
+ 
+Support for Rubinius ("#251":http://redmine.rubychan.de/issues/251), improved mutlibyte handling, Ruby 1.9 syntax, and valid HTML.
+ 
+h3. @Encoders::HTML@
+ 
+* *FIXED*: Line tokens use @span@ with @display: block@ instead of @div@, which was invalid HTML ("#255":http://redmine.rubychan.de/issues/255).
+
+h3. @Scanner::Scanner@
+ 
+* *IMPROVED* handling of encodings in Ruby 1.9: UTF-8 and Windows-1252 are checked.
+* *NEW*: Invalid chars will be converted to @?@ in Ruby 1.9.
+* *FIXED* @string=@ method for Rubinius. See "issue 481":http://github.com/evanphx/rubinius/issues/481 on their site.
+
+h3. @Scanners::CSS@
+ 
+* *FIXED*: Don't use non-ASCII regexps.
+
+h3. @Scanners::Diff@
+ 
+* *FIXED*: Highlight unexpected lines as @:comment@.
+
+h3. @Scanners::PHP@
+ 
+* *FIXED*: Use @ASCII-8BIT@ encoding for now.
+
+h3. @Scanners::Ruby@
+ 
+* *ADDED* support for some Ruby 1.9 syntax ("#254":http://redmine.rubychan.de/issues/254):
+** the @->@ lambda shortcut
+** new Hash syntax using colons (@{ a: b }@)
+* *FIXED*: Use @UTF-8@ encoding.
+* *IMPROVED* unicode support on Ruby 1.8 ("#253":http://redmine.rubychan.de/issues/253).
+* *FIXED* recognition of non-ASCII identifiers in Ruby 1.9, JRuby, and Rubinius ("#253":http://redmine.rubychan.de/issues/253).
+* *CHANGED* heredoc recognition to ignore delimiters starting with a digit. This is incorrect, but causes less false positives.
+
+h3. @Scanners::SQL@
+ 
+* *FIXED* scanning of comments; nice catch, Rubinius!
+  ("#252":http://redmine.rubychan.de/issues/252)
+
+
+h2. Changes in 0.9.4 "Ramadan" [2010-08-31]
+ 
+Updated command line interface and minor scanner fixes for the Diff, HTML, and RHTML scanners.
+ 
+h3. @coderay@ executable
+ 
+* *FIXED*: Partly rewritten, simplified, fixed.
+  ("#244":http://redmine.rubychan.de/issues/244)
+
+h3. @Scanners::Diff@
+ 
+* *FIXED* handling of change headers with code on the same line as the @@ marker.
+  ("#247":http://redmine.rubychan.de/issues/242)
+
+h3. @Scanners::HTML@
+ 
+* *FIXED* a missing regexp modifier that slowed down the scanning.
+  ("#245":http://redmine.rubychan.de/issues/245)
+
+h3. @Scanners::RHTML@
+ 
+* *FIXED* highlighting of ERB comment blocks.
+  ("#246":http://redmine.rubychan.de/issues/246)
+
+
+h2. Changes in 0.9.3 "Eyjafjallajökull" [2010-04-18]
+ 
+* *FIXED*: Documentation of Tokens.
+  ("#218":http://redmine.rubychan.de/issues/218)
+ 
+h3. @coderay@ executable
+ 
+* *NEW*: automatic TTY detection (uses @Term@ encoder)
+* *NEW*: optional 3rd parameter for the filename
+* *FIXED*: Converted to UNIX format.
+* *FIXED*: Warn about generated files.
+* *FIXED*: Ensure line break after the output (especially for LoC counter).
+
+h3. @Scanners::JavaScript@
+ 
+* *FIXED*: Don't keep state of XML scanner between calls for E4X literals.
+
+h3. @Scanners::Java@, @Scanners::JSON@
+ 
+* *FIXED*: Close unfinished strings with the correct token kind.
+
+
+h2. Changes in 0.9.2 "Flameeyes" [2010-03-14]
+ 
+* *NEW* Basic tests and a _Rakefile_ are now included in the Gem. [Flameeyes]
+  A @doc@ task is also included.
+* *FIXED* Use @$CODERAY_DEBUG@ for debugging instead of @$DEBUG@. [Trans]
+  ("#192":http://redmine.rubychan.de/issues/192)
+* *REMOVED* @Term::Ansicolor@ was bundled under _lib/_, but not used. [Flameeyes]
+  ("#205":http://redmine.rubychan.de/issues/205)
+* *WORKAROUND* for Ruby bug 
+  "#2745":http://redmine.ruby-lang.org/issues/show/2745
+ 
+h3. @Encoders::Term@
+ 
+* *FIXED* strings are closed correctly
+  ("#138":http://redmine.rubychan.de/issues/138)
+* *FIXED* several token kinds had no associated color
+  ("#139":http://redmine.rubychan.de/issues/139)
+* *NEW* alias @terminal@
+  
+  *NOTE:* This encoder will be renamed to @Encoders::Terminal@ in the next release.
+
+h3. @Scanners::Debug@
+ 
+* *FIXED* Don't close tokens that are not open. Send @:error@ token instead.
+
+h3. @Scanners::Groovy@
+ 
+* *FIXED* token kind of closing brackets is @:operator@ instead of @nil@
+  ("#148":http://redmine.rubychan.de/issues/148)
+
+h3. @Scanners::PHP@
+ 
+* *FIXED* allow @\@ operator (namespace separator)
+  ("#209":http://redmine.rubychan.de/issues/209)
+
+h3. @Scanners::YAML@
+ 
+* *FIXED* doesn't send debug tokens when @$DEBUG@ is true [Trans]
+  ("#149":http://redmine.rubychan.de/issues/149)
+
+
+h2. Changes in 0.9.1 [2009-12-31]
+ 
+h3. Token classes
+ 
+* *NEW* token classes @:complex@, @:decorator@, @:imaginary@
+  (all for Python)
+* *REMOVED* token class @:procedure@
+  – use @:function@ or @:method@ instead.
+
+h3. @Tokens@
+ 
+* *NEW* method @#scanner@
+  
+  Stores the scanner.
+* *REMOVED* methods @.write_token@, @.read_token@, @.escape@, @.unescape@
+  
+  They were only used by the @Tokens@ encoder, which was removed also.
+
+h3. @Encoders::Encoder@
+ 
+* *REMOVED* Don't require the _stringio_ library.
+* *NEW* public methods @#open_token@, @#close_token@, @#begin_line@, @#end_line@
+  These methods are called automatically, like @#text_token@.
+* *NEW* proteced method @#append_encoded_token_to_output@
+
+h3. @Encoders::Tokens@
+ 
+* *REMOVED* – use @Tokens#dump@ and @Tokens.load@.
+
+h3. @Encoders::Filter@
+ 
+* *NEW*
+  A @Filter@ encoder has another @Tokens@ instance as output.
+
+h3. @Encoders::TokenClassFilter@
+ 
+* *NEW*
+  
+  It takes 2 options, @:exclude@ and @:include@, that specify which token classes
+  to include or exclude for the output. They can be a single token class,
+  an @Array@ of classes, or the value @:all@.
+
+h3. @Encoders::CommentFilter@
+ 
+* *NEW*
+  
+  Removes tokens of the @:comment@ class.
+
+h3. @Encoders::LinesOfCode@
+ 
+* *NEW*
+  
+  Counts the lines of code according to the @KINDS_NOT_LOC@ token class list
+  defined by the scanner. It uses the new @TokenClassFilter@.
+  
+  Alias: @:loc@, as in @tokens.loc@.
+
+h3. @Encoders::JSON@
+ 
+* *NEW*
+  
+  Outputs tokens in a simple JSON format.
+
+h3. @Encoders::Term@
+ 
+* *NEW* (beta, by Rob Aldred)
+  
+  Outputs code highlighted for a color terminal.
+
+h3. @Encoders::HTML@
+ 
+* *NEW* option @:title@ (default value is _CodeRay output_)
+  
+  Setting this changes the title of the HTML page.
+* *NEW* option @:highlight_lines@ (default: @nil@)
+  
+  Highlights the given set of line numbers.
+- *REMOVED* option @:level@
+  
+  It didn't do anything. CodeRay always outputs XHTML.
+
+h3. @Encoders::Text@
+ 
+* Uses @Encoder@ interface with @super@ and @#text_token@.
+
+h3. @Encoders::XML@
+ 
+* @FIXED@ ("#94":http://redmine.rubychan.de/issues/94)
+  
+  It didn't work at all.
+
+h3. Scanners
+ 
+* *NEW* Mapped @:h@ to @:c@, @:cplusplus@ and @:'c++'@ to @:cpp@,
+  @:ecma@, @:ecmascript@, @:ecma_script@ to @:java_script@,
+  @:pascal@ to @:delphi@, and @:plain@ to @:plaintext@.
+
+h3. @Scanners::Scanner@
+ 
+* *NEW* constant @KINDS_NOT_LOC@
+  
+  A list of all token classes not considered in LOC count.
+  Added appropriate values for scanners.
+* *NEW* method @#lang@ returns the scanner's lang, which is its @plugin_id@.
+* *FIXED* automatic, safe UTF-8 detection _[Ruby 1.9]_
+* *FIXED* column takes care of multibyte encodings _[Ruby 1.9]_
+* *FIXED* is dumpable (@Tokens@ store their scanner in an @@scanner@ variable)
+
+h3. @Scanners::Cpp@
+ 
+* *NEW* (C++)
+
+h3. @Scanners::Groovy@
+ 
+* *NEW* (beta)
+
+h3. @Scanners::Python@
+ 
+* *NEW*
+
+h3. @Scanners::PHP@
+ 
+* *NEW* (based on Stefan Walk's work)
+
+h3. @Scanners::SQL@
+ 
+* *NEW* (based on code by Josh Goebel)
+
+h3. @Scanners::C@
+ 
+* *IMPROVED* added a list of @:directive@ tokens that were @:reserved@ before
+* *IMPROVED* detection of labels
+* *IMPROVED* allow @1L@ and @1LL@ style literals
+
+h3. @Scanners::CSS@
+ 
+* *IMPROVED* element selectors are highlighted as @:type@ instead of @:keyword@
+
+h3. @Scanners::Delphi@
+ 
+* *IMPROVED* Don't cache tokens in CaseIgnoringWordList.
+
+h3. @Scanners::Java@
+ 
+* *IMPROVED* @assert@ is highlighted as a @:keyword@ now
+* *IMPROVED* @const@ and @goto@ are highlighted as @:reserved@
+* *IMPROVED* @false@, @true@, and @null@ are highlighted as @:pre_constant@
+* *IMPROVED* @threadsafe@ is no longer a @:directive@
+* *IMPROVED* @String@ is highlighted as a @:pre_type@
+* *IMPROVED* built-in classes ending with _Error_ or _Exception_ are
+  highlighted as a @:exception@ instead of @:pre_type@
+
+h3. @Scanners::JavaScript@
+ 
+* *NEW* a list of @PREDEFINED_CONSTANTS@ to be highlighted as @:pre_constant@
+* *NEW* XML literals are recognized and highlighted
+* *NEW* function name highlighting
+* *IMPROVED* @.1@ is highlighted a number
+* *FIXED* strings close with the correct kind when terminated unexpectedly
+
+h3. @Scanners::JSON@
+ 
+* *IMPROVED* constants (@true@, @false@, @nil@) are highlighted as @:value@
+
+h3. @Scanners::Ruby@
+ 
+* *IMPROVED* @Patterns::KEYWORDS_EXPECTING_VALUE@ for more accurate
+  @value_expected@ detection
+* *IMPROVED* handling of @\@ as a string delimiter
+* *IMPROVED* handling of unicode strings; automatic switching to unicode
+* *IMPROVED* highlighting of @self.method@ definitions
+* *REMOVED* @Patterns::FANCY_START_SAVE@ (obsolete)
+* *FIXED* encoding issues _[Ruby 1.9]_
+* *FIXED* a problem in early Ruby 1.8.6 patch versions with @Regexp.escape@
+
+h3. @Scanners::YAML@
+ 
+* *IMPROVED* indentation detection
+
+h3. @Styles::Cycnus@
+ 
+* changed a few colors (exceptions, inline strings, predefined types)
+
+h3. @Plugin@
+ 
+* *NEW* method @#title@
+  
+  Set and get the plugin's title. Titles can be arbitrary strings.
+* *NEW* method @#helper@ loads helpers from different plugins
+  
+  Use this syntax: @helper 'other_plugin/helper_name'@
+
+h3. @FileType@
+ 
+* *NEW* @FileType[]@ takes @Pathname@ instances
+* *NEW* regonizes @.cc@, @.cpp@, @.cp@, @.cxx@, @.c++@, @.C@, @.hh@, @.hpp@, @.h++@, @.cu@ extensions (C++)
+  
+  Thanks to Sander Cox and the TextMate C bundle.
+* *NEW* regonizes @.pas@, @.dpr@ extensions (Delphi)
+* *NEW* regonizes @.gvy@, @.groovy@ extensions (Groovy)
+* *NEW* regonizes @.php@, @.php3@, @.php4@, @.php5@ extensions (PHP)
+* *NEW* regonizes @.py@, @.py3@, @.pyw@ extensions (Python)
+* *NEW* regonizes @.rxml@ extension (Ruby)
+* *NEW* regonizes @.sql@ extension (SQL)
+* File types list was sorted alphabetically.
+
+h3. @CaseIgnoringWordList@
+ 
+* *FIXED* ("#97":http://redmine.rubychan.de/issues/97)
+  
+  The default value is no longer ignored.
+
+h3. @ForRedCloth@
+ 
+* *FIXED* for RedCloth versions 4.2.0+ ("#119":http://redmine.rubychan.de/issues/119)
+
+h3. Cleanups
+ 
+* warnings about character classes _[Ruby 1.9]_
+* encoding issues _[Ruby 1.9]_
+* documentation, code
+
+
diff --git a/Changes.textile b/Changes.textile
index cdbfdd9..5448767 100644
--- a/Changes.textile
+++ b/Changes.textile
@@ -1,421 +1,345 @@
 h1=. CodeRay Version History
  
-p=. _This files lists all changes in the CodeRay library since the 0.8.4 release._
+p=. _This files lists all changes in the CodeRay library since the 0.9.8 release._
  
 {{toc}}
  
-h2. Changes in 0.9.8 "banister" [2011-05-01]
+h2. Changes in 1.0
  
-Fixes for JRuby's 1.9 mode and minor issues.
+CodeRay 1.0 is a major rewrite of the library, and incompatible to earlier versions.
  
-h3. Rake tasks
- 
-* *REMOVED* obsolete @has_rdoc@ gem specification, fixing a warning.
+The command line and programmer interfaces are similar to 0.9, but the internals have completely changed.
 
-h3. @Scanners::Scanner@
+h3. General changes
  
-* *NEW* method @#scan_rest@ replaces @scan_until(/\z/)@, which is broken in JRuby 1.6 --1.9 mode.
-  See "#297":http://redmine.rubychan.de/issues/297.
+* *NEW*: The new Diff scanner colorizes code inside of the diff, and highlights inline changes.
+* *NEW*: Extended support and usage of HTML5 and CSS 3 features.
+* *NEW*: Direct Streaming
+* *NEW* scanners: Clojure and HAML
+* *CHANGED*: Token classes (used as CSS classes) are readable names; breaks you stylesheet!
+* *IMPROVED* documentation
+* *IMPROVED* speed: faster startup (using @autoload@), scanning, and encoding
+* *IMPROVED* Ruby 1.9 encodings support
+* *IMPROVED* Tests: There are more of them now!
 
-h3. @Scanners::CSS@
+h3. Direct Streaming
  
-* *FIXED* LOC counting (should be 0).
-  See "#296":http://redmine.rubychan.de/issues/296.
-
-h3. @Scanners::Ruby@
+CodeRay 1.0 introduces _Direct Streaming_ as a faster and simpler alternative to Tokens. It means that all Scanners,
+Encoders and Filters had to be rewritten, and that older scanners using the Tokens API are no longer compatible with
+this version.
  
-* *FIXED* the @IDENT@ pattern not to use character properties, which are broken in JRuby 1.6 --1.9 mode.
-  See "#297":http://redmine.rubychan.de/issues/297, thanks to banister for reporting!
-
-h3. @Scanners::SQL@
+The main benefits of this change are:
  
-* *ADDED* more keywords: @between@, @databases@, @distinct@, @fields@, @full@, @having@, @is@, @prompt@, @tables@.
-  See "#221":http://redmine.rubychan.de/issues/221, thanks to Etienne Massip again.
-
-h3. @FileType@
+* more speed (benchmarks show 10% to 50% more tokens per second compared to CodeRay 0.9)
+* the ability to stream output into a pipe on the command line
+* a simpler API
+* less code
  
-* *NEW* regonizes ColdFusion file type extensions @.cfm@ and @.cfc@ as XML.
-  See "#298":http://redmine.rubychan.de/issues/298, thanks to Emidio Stani.
-
+Changes related to the new tokens handling include:
+* *CHANGED*: The Scanners now call Encoders directly; tokens are not added to a Tokens array, but are send to the
+  Encoder as a method call. The Tokens representation (which can be seen as a cache now) is still present, but as a
+  special case; Tokens just encodes the given tokens into an Array for later use.
+* *CHANGED*: The token actions (@text_token@, @begin_group@ etc.) are now public methods of @Encoder@ and @Tokens@.
+* *REWRITE* of all Scanners, Encoders, Filters, and Tokens.
+* *RENAMED* @:open@ and @:close@ actions to @:begin_group@ and @:end_group@.
+* *RENAMED* @open_token@ and @close_token@ methods to @begin_group@ and @end_group@.
+* *NEW* method @#tokens@ allows to add several tokens to the stream. @Tokens@ and @Encoders::Encoder@ define this
+  method.
+* *CHANGED* The above name changes also affect the JSON, XML, and YAML encoders. CodeRay 1.0 output will be incompatible
+  with earlier versions.
+* *REMOVED* @TokenStream@ and the @Streamable@ API and all related features like @NotStreamableError@ are now obsolete
+  and have been removed.
 
-h2. Changes in 0.9.7 "Etienne" [2011-01-14]
- 
-Fixes a dangerous JavaScript scanner bug, and a testing problem with Ruby 1.9.1.
- 
-h3. Tests
+h3. Command Line
  
-* *FIXED* The functional tests now load the lib directory (instead of the gem) in Ruby 1.9.1.
-
-h3. @Scanners::JavaScript@
+The @coderay@ executable was rewritten and has a few new features:
  
-* *FIXED* @KEY_CHECK_PATTERN@ regexp
-  See "#264":http://redmine.rubychan.de/issues/264, thanks to Etienne Massip!
+* *NEW* Ability to stream into a pipe; try @coderay file | more -r@
+* *NEW* help
+* *IMPROVED*: more consistent parameter handling
+* *REMOVED* @coderay_stylesheet@ executable; use @coderay stylesheet [name]@.
 
-
-h2. Changes in 0.9.6 "WoNáDo" [2010-11-25]
- 
-Minor improvements to the Ruby scanner and a fix for Ruby 1.9.
- 
-h3. @Scanners::Ruby@
+h3. @Tokens@
  
-* *IMPROVED* handling of new hash syntax (keys are marked as @:key@ now,
-  colon is a separate @:operator@ token, all idents can be used as keys)
-  See "#257":http://code.licenser.net/issues/257, thanks to WoNáDo!
-* *ADDED* @__ENCODING__@ magic constant (Ruby 1.9)
-* *FIXED*: Scanner no longer tries to modify the input string on Ruby 1.9.
-  See "#260":http://code.licenser.net/issues/260, thanks to Jan Lelis!
-
+* *NEW* methods @count@, @begin_group@, @end_group@, @begin_line@, and @end_line@.
+* *REMOVED* methods @#stream?@, @#each_text_token@.
+* *REMOVED* methods @#optimize@, @#fix@, @#split_into_lines@ along with their bang! variants.
+* *REMOVED* @#text@ and @#text_size@ methods. Use the @Text@ encoder instead.
+* *REMOVED* special implementation of @#each@ taking a filter parameter. Use @TokenKindFilter@ instead.
 
-h2. Changes in 0.9.5 "Germany.rb" [2010-09-28]
+h3. *RENAMED*: @TokenKinds@
  
-Support for Rubinius ("#251":http://redmine.rubychan.de/issues/251), improved mutlibyte handling, Ruby 1.9 syntax, and valid HTML.
+Renamed from @Tokens::ClassOfKind@ (was also @Tokens::AbbreviationForKind@ for a while).
+The term "token class" is no longer used in CodeRay. Instead, tokens have _kinds_.
+See "#122":http://redmine.rubychan.de/issues/122.
  
-h3. @Encoders::HTML@
- 
-* *FIXED*: Line tokens use @span@ with @display: block@ instead of @div@, which was invalid HTML ("#255":http://redmine.rubychan.de/issues/255).
+* *CHANGED* all token CSS classes to readable names.
+* *ADDED* token kinds @:filename@, @:namespace@, and @:eyecatcher@.
+* *RENAMED* @:pre_constant@ and @:pre_type@ to @:predefined_constant@ and @predefined_type@.
+* *RENAMED* @:oct@ and @:bin@ to @:octal@ and @binary@.
+* *REMOVED* token kinds @:attribute_name_fat@, @:attribute_value_fat@, @:operator_fat@, @interpreted@,
+  @:tag_fat@, @tag_special@, @:xml_text@, @:nesting_delimiter@, @:open@, and @:close@.
+* *CHANGED*: Don't raise error for unknown token kinds unless in @$CODERAY_DEBUG@ mode.
+* *CHANGED* the value for a token kind that is not highlighted from
+  @:NO_HIGHLIGHT@ to @false@.
 
-h3. @Scanner::Scanner@
+h3. @Duo@
  
-* *IMPROVED* handling of encodings in Ruby 1.9: UTF-8 and Windows-1252 are checked.
-* *NEW*: Invalid chars will be converted to @?@ in Ruby 1.9.
-* *FIXED* @string=@ method for Rubinius. See "issue 481":http://github.com/evanphx/rubinius/issues/481 on their site.
+* *NEW* method @call@ for allowing code like @CodeRay::Duo[:python => :yaml].(code)@ in Ruby 1.9.
 
-h3. @Scanners::CSS@
+h3. @Encoders::CommentFilter@
  
-* *FIXED*: Don't use non-ASCII regexps.
+* *NEW* alias @:remove_comments@
 
-h3. @Scanners::Diff@
+h3. @Encoders::Filter@
  
-* *FIXED*: Highlight unexpected lines as @:comment@.
+* *NEW* option @tokens@.
+* *CHANGED*: Now it simply delegates to the output.
+* *REMOVED* @include_text_token?@ and @include_block_token?@ methods.
 
-h3. @Scanners::PHP@
+h3. @Encoders::HTML@
  
-* *FIXED*: Use @ASCII-8BIT@ encoding for now.
-
-h3. @Scanners::Ruby@
+The HTML encoder was cleaned up and simplified.
  
-* *ADDED* support for some Ruby 1.9 syntax ("#254":http://redmine.rubychan.de/issues/254):
-** the @->@ lambda shortcut
-** new Hash syntax using colons (@{ a: b }@)
-* *FIXED*: Use @UTF-8@ encoding.
-* *IMPROVED* unicode support on Ruby 1.8 ("#253":http://redmine.rubychan.de/issues/253).
-* *FIXED* recognition of non-ASCII identifiers in Ruby 1.9, JRuby, and Rubinius ("#253":http://redmine.rubychan.de/issues/253).
-* *CHANGED* heredoc recognition to ignore delimiters starting with a digit. This is incorrect, but causes less false positives.
+* *NEW*: HTML5 and CSS 3 compatible.
+  See "#215":http://redmine.rubychan.de/issues/215.
+* *ADDED* support for @:line_number_anchors@.
+  See "#208":http://redmine.rubychan.de/issues/208.
+* *CHANGED* the default style to @:alpha@.
+* *CHANGED*: Use double click to toggle line numbers in table mode (as single
+  click jumps to an anchor.)
+* *REMOVED* support for @:line_numbers => :list@.
+* *FIXED* splitting of lines for @:line_numbers => :inline@, so that the line
+  numbers don't get colored, too.
+* *RENAMED* @Output#numerize@ to @#number@, which is an actual English word.
 
-h3. @Scanners::SQL@
- 
-* *FIXED* scanning of comments; nice catch, Rubinius!
-  ("#252":http://redmine.rubychan.de/issues/252)
-
-
-h2. Changes in 0.9.4 "Ramadan" [2010-08-31]
- 
-Updated command line interface and minor scanner fixes for the Diff, HTML, and RHTML scanners.
- 
-h3. @coderay@ executable
+h3. @Encoders::LinesOfCode@
  
-* *FIXED*: Partly rewritten, simplified, fixed.
-  ("#244":http://redmine.rubychan.de/issues/244)
+* *CHANGED*: @compile@ and @finish@ methods are now protected.
 
-h3. @Scanners::Diff@
+h3. *Renamed*: @Encoders::Terminal@ (was @Encoders::Term@)
  
-* *FIXED* handling of change headers with code on the same line as the @@ marker.
-  ("#247":http://redmine.rubychan.de/issues/242)
+* *RENAMED* from @Encoders::Term@, added @:term@ alias.
+* *CLEANUP*: Use @#setup@'s @super@, don't use @:procedure@ token class.
+* *CHANGED*: @#token@'s second parameter is no longer optional.
+* *REMOVED* colors for obsolete token kinds.
+* *FIXED* handling of line tokens.
 
-h3. @Scanners::HTML@
+h3. @Encoders::Text@
  
-* *FIXED* a missing regexp modifier that slowed down the scanning.
-  ("#245":http://redmine.rubychan.de/issues/245)
+* *FIXED* default behavior of stripping the trailing newline.
 
-h3. @Scanners::RHTML@
+h3. *RENAMED*: @Encoders::TokenKindFilter@ (was @Encoders::TokenClassFilter@)
  
-* *FIXED* highlighting of ERB comment blocks.
-  ("#246":http://redmine.rubychan.de/issues/246)
+* *NEW*: Handles token groups.
+  See "#223":http://redmine.rubychan.de/issues/223.
+* *RENAMED* @include_block_token?@ to @include_group?@.
 
-
-h2. Changes in 0.9.3 "Eyjafjallajökull" [2010-04-18]
- 
-* *FIXED*: Documentation of Tokens.
-  ("#218":http://redmine.rubychan.de/issues/218)
+h3. @Encoders::Statistic@
  
-h3. @coderay@ executable
- 
-* *NEW*: automatic TTY detection (uses @Term@ encoder)
-* *NEW*: optional 3rd parameter for the filename
-* *FIXED*: Converted to UNIX format.
-* *FIXED*: Warn about generated files.
-* *FIXED*: Ensure line break after the output (especially for LoC counter).
+* *CHANGED*: Tokens actions are counted separately.
 
-h3. @Scanners::JavaScript@
+h3. @Scanners::Scanner@
  
-* *FIXED*: Don't keep state of XML scanner between calls for E4X literals.
+* *NEW* methods @#file_extension@ and @#encoding@.
+* *NEW*: The @#tokenize@ method also takes an Array of Strings as source. The
+  code is highlighted as one and split into parts of the input lengths
+  after that using @Tokens#split_into_parts@.
+* *NEW* method @#binary_string@
+* *REMOVED* helper method @String#to_unix@.
+* *REMOVED* method @#streamable?@.
+* *REMOVED* @#marshal_load@ and @#marshal_dump@.
+* *RENAMED* class method @normify@ to @normalize@; it also deals with encoding now.
+* *CHANGED*: @#column@ starts counting with 1 instead of 0
 
-h3. @Scanners::Java@, @Scanners::JSON@
+h3. *NEW*: @Scanners::Clojure@
  
-* *FIXED*: Close unfinished strings with the correct token kind.
-
+Thanks to Licenser, CodeRay now supports the Clojure language.
 
-h2. Changes in 0.9.2 "Flameeyes" [2010-03-14]
- 
-* *NEW* Basic tests and a _Rakefile_ are now included in the Gem. [Flameeyes]
-  A @doc@ task is also included.
-* *FIXED* Use @$CODERAY_DEBUG@ for debugging instead of @$DEBUG@. [Trans]
-  ("#192":http://redmine.rubychan.de/issues/192)
-* *REMOVED* @Term::Ansicolor@ was bundled under _lib/_, but not used. [Flameeyes]
-  ("#205":http://redmine.rubychan.de/issues/205)
-* *WORKAROUND* for Ruby bug 
-  "#2745":http://redmine.ruby-lang.org/issues/show/2745
- 
-h3. @Encoders::Term@
+h3. @Scanners::CSS@
  
-* *FIXED* strings are closed correctly
-  ("#138":http://redmine.rubychan.de/issues/138)
-* *FIXED* several token kinds had no associated color
-  ("#139":http://redmine.rubychan.de/issues/139)
-* *NEW* alias @terminal@
-  
-  *NOTE:* This encoder will be renamed to @Encoders::Terminal@ in the next release.
+* *NEW*: Rudimentary support for the @attr@, @counter@, and @counters@ functions.
+  See "#224":http://redmine.rubychan.de/issues/224.
+* *NEW*: Rudimentary support for CSS 3 colors.
+* *CHANGED*: Attribute selectors are highlighted as @:attribute_name@ instead of @:string@.
+* *CHANGED*: Comments are scanned as one token instead of three.
 
 h3. @Scanners::Debug@
  
-* *FIXED* Don't close tokens that are not open. Send @:error@ token instead.
-
-h3. @Scanners::Groovy@
- 
-* *FIXED* token kind of closing brackets is @:operator@ instead of @nil@
-  ("#148":http://redmine.rubychan.de/issues/148)
-
-h3. @Scanners::PHP@
- 
-* *FIXED* allow @\@ operator (namespace separator)
-  ("#209":http://redmine.rubychan.de/issues/209)
+* *NEW*: Support for line tokens (@begin_line@ and @end_line@ represented by @[@ and @]@.)
+* *FIXED*: Don't send @:error@ and @nil@ tokens for buggy input any more.
+* *FIXED*: Closes unclosed tokens at the end of @scan_tokens@.
+* *IMPROVED*: Highlight unknown tokens as @:error@.
+* *CHANGED*: Raises an error when trying to end an invalid token group.
 
-h3. @Scanners::YAML@
- 
-* *FIXED* doesn't send debug tokens when @$DEBUG@ is true [Trans]
-  ("#149":http://redmine.rubychan.de/issues/149)
-
-
-h2. Changes in 0.9.1 [2009-12-31]
- 
-h3. Token classes
+h3. @Scanners::Delphi@
  
-* *NEW* token classes @:complex@, @:decorator@, @:imaginary@
-  (all for Python)
-* *REMOVED* token class @:procedure@
-  – use @:function@ or @:method@ instead.
+* *FIXED*: Closes open string groups.
 
-h3. @Tokens@
+h3. @Scanners::Diff@
  
-* *NEW* method @#scanner@
+* *NEW*: Highlighting of code based on file names.
+  See ticket "#52":http://redmine.rubychan.de/issues/52.
   
-  Stores the scanner.
-* *REMOVED* methods @.write_token@, @.read_token@, @.escape@, @.unescape@
+  Use the @:highlight_code@ option to turn this feature off. It's enabled
+  by default.
   
-  They were only used by the @Tokens@ encoder, which was removed also.
-
-h3. @Encoders::Encoder@
- 
-* *REMOVED* Don't require the _stringio_ library.
-* *NEW* public methods @#open_token@, @#close_token@, @#begin_line@, @#end_line@
-  These methods are called automatically, like @#text_token@.
-* *NEW* proteced method @#append_encoded_token_to_output@
-
-h3. @Encoders::Tokens@
- 
-* *REMOVED* – use @Tokens#dump@ and @Tokens.load@.
-
-h3. @Encoders::Filter@
- 
-* *NEW*
-  A @Filter@ encoder has another @Tokens@ instance as output.
-
-h3. @Encoders::TokenClassFilter@
- 
-* *NEW*
-  
-  It takes 2 options, @:exclude@ and @:include@, that specify which token classes
-  to include or exclude for the output. They can be a single token class,
-  an @Array@ of classes, or the value @:all@.
-
-h3. @Encoders::CommentFilter@
- 
-* *NEW*
+  This is a very original feature. It enables multi-language highlighting for
+  diff files, which is especially helpful for CodeRay development itself. The
+  updated version of the scanner test suite generated .debug.diff.html files
+  using this.
   
-  Removes tokens of the @:comment@ class.
-
-h3. @Encoders::LinesOfCode@
- 
-* *NEW*
+  Note: This is still experimental. Tokens spanning more than one line
+  may get highlighted incorrectly. CodeRay tries to keep scanner states
+  between the lines and changes, but the quality of the results depend on
+  the scanner.
+* *NEW*: Inline change highlighting, as suggested by Eric Thomas.
+  See ticket "#227":http://redmine.rubychan.de/issues/227 for details.
   
-  Counts the lines of code according to the @KINDS_NOT_LOC@ token class list
-  defined by the scanner. It uses the new @TokenClassFilter@.
+  Use the @:inline_diff@ option to turn this feature off. It's enabled by
+  default.
   
-  Alias: @:loc@, as in @tokens.loc@.
+  For single-line changes (that is, a single deleted line followed by a single
+  inserted line), this feature surrounds the changed parts with an
+  @:eyecatcher@ group which appears in a more saturated background color.
+  The implementation is quite complex, and highly experimental. The problem
+  with multi-layer tokenizing is that the tokens have to be split into parts.
+  If the inline change starts, say, in the middle of a string, then additional
+  @:end_group@ and @:begin_group@ tokens must be inserted to keep the group
+  nesting intact. The extended @Scanner#tokenize@ method and the new
+  @Tokens#split_into_parts@ method take care of this.
+* *NEW*: Highlight the file name in the change headers as @:filename@.
+* *CHANGED*: Highlight unknown lines as @:comment@ instead of @:head@.
+* *IMPROVED*: Background colors for Diff output have been optimized.
 
-h3. @Encoders::JSON@
- 
-* *NEW*
-  
-  Outputs tokens in a simple JSON format.
+h3. *RENAMED*: @Scanners::ERB@ (was @Scanners::RHTML@)
 
-h3. @Encoders::Term@
+h3. *NEW*: @Scanners::HAML@
  
-* *NEW* (beta, by Rob Aldred)
-  
-  Outputs code highlighted for a color terminal.
-
-h3. @Encoders::HTML@
+It uses the new :state options of the HTML and Ruby scanners.
  
-* *NEW* option @:title@ (default value is _CodeRay output_)
-  
-  Setting this changes the title of the HTML page.
-* *NEW* option @:highlight_lines@ (default: @nil@)
-  
-  Highlights the given set of line numbers.
-- *REMOVED* option @:level@
-  
-  It didn't do anything. CodeRay always outputs XHTML.
+Some rare cases are not considered (like @#{...}@ snippets inside of :javascript blocks),
+but it highlights pretty well.
 
-h3. @Encoders::Text@
+h3. @Scanners::HTML@
  
-* Uses @Encoder@ interface with @super@ and @#text_token@.
+* *FIXED*: Closes open string groups.
 
-h3. @Encoders::XML@
+h3. @Scanners::JavaScript@
  
-* @FIXED@ ("#94":http://redmine.rubychan.de/issues/94)
-  
-  It didn't work at all.
+* *IMPROVED*: Added @NaN@ and @Infinity@ to list of predefined constants.
+* *IMPROVED* recognition of RegExp literals with leading spaces.
 
-h3. Scanners
+h3. @Scanners::Java@
  
-* *NEW* Mapped @:h@ to @:c@, @:cplusplus@ and @:'c++'@ to @:cpp@,
-  @:ecma@, @:ecmascript@, @:ecma_script@ to @:java_script@,
-  @:pascal@ to @:delphi@, and @:plain@ to @:plaintext@.
+* *NEW*: Package names are highlighted as @:namespace@.
+  See "#210":http://redmine.rubychan.de/issues/210.
 
-h3. @Scanners::Scanner@
+h3. *REMOVED*: @Scanners::NitroXHTML@
  
-* *NEW* constant @KINDS_NOT_LOC@
-  
-  A list of all token classes not considered in LOC count.
-  Added appropriate values for scanners.
-* *NEW* method @#lang@ returns the scanner's lang, which is its @plugin_id@.
-* *FIXED* automatic, safe UTF-8 detection _[Ruby 1.9]_
-* *FIXED* column takes care of multibyte encodings _[Ruby 1.9]_
-* *FIXED* is dumpable (@Tokens@ store their scanner in an @@scanner@ variable)
+Nitro is "dead":http://www.nitrohq.com/.
 
-h3. @Scanners::Cpp@
+h3. *RENAMED*: @Scanners::Text@ (was @Scanners::Plaintext@)
  
-* *NEW* (C++)
-
-h3. @Scanners::Groovy@
+* *IMPROVED*: Just returns the string without scanning (faster).
  
-* *NEW* (beta)
+ This is much faster than scanning until @/\z/@ in Ruby 1.8.
 
 h3. @Scanners::Python@
  
-* *NEW*
+* *CHANGED*: Docstrings are highlighted as @:comment@.
+  See "#190":http://redmine.rubychan.de/issues/190.
 
-h3. @Scanners::PHP@
- 
-* *NEW* (based on Stefan Walk's work)
+h3. *NEW*: @Scanners::Raydebug@
 
-h3. @Scanners::SQL@
- 
-* *NEW* (based on code by Josh Goebel)
+Copied from @Scanners::Debug@, highlights the token dump instead of importing it. It also reacts to the @.raydebug@ file
+name suffix now.
 
-h3. @Scanners::C@
+h3. @Scanners::Ruby@
  
-* *IMPROVED* added a list of @:directive@ tokens that were @:reserved@ before
-* *IMPROVED* detection of labels
-* *IMPROVED* allow @1L@ and @1LL@ style literals
+* *ADDED* more predefined keywords (see http://murfy.de/ruby-constants).
+* *IMPROVED* support for singleton method definitions.
+  See "#147":http://redmine.rubychan.de/issues/147.
+* *FIXED*: Don't highlight methods with a capital letter as constants
+  (eg. @GL.PushMatrix@).
+* *NEW*: Highlight buggy floats (like .5) as @:error@.
+* *CLEANUP* of documentation, names of constants and variables, state handling.
+  
+  Moved @StringState@ class from @patterns.rb@ into a separate file.
+* *NEW*: Complicated rule for recognition of @foo=@ style method names.
+* *NEW*: Handles @:keep_state@ option (a bit; experimental).
+  
+  Actually, Ruby checks if there is @[~>=]@, but not @=>@ following the name.
+  
+* *REMOVED* @EncodingError@
 
-h3. @Scanners::CSS@
+h3. *REMOVED* @Scanners::Scheme@
  
-* *IMPROVED* element selectors are highlighted as @:type@ instead of @:keyword@
+* It is too buggy, and nobody was using it. To be added again when it's fixed.
+  See "#59":http://redmine.rubychan.de/issues/59.
 
-h3. @Scanners::Delphi@
+h3. @Scanners::SQL@
  
-* *IMPROVED* Don't cache tokens in CaseIgnoringWordList.
-
-h3. @Scanners::Java@
+* *IMPROVED*: Extended list of keywords and functions (thanks to
+  Joshua Galvez, Etienne Massip, and others).
  
-* *IMPROVED* @assert@ is highlighted as a @:keyword@ now
-* *IMPROVED* @const@ and @goto@ are highlighted as @:reserved@
-* *IMPROVED* @false@, @true@, and @null@ are highlighted as @:pre_constant@
-* *IMPROVED* @threadsafe@ is no longer a @:directive@
-* *IMPROVED* @String@ is highlighted as a @:pre_type@
-* *IMPROVED* built-in classes ending with _Error_ or _Exception_ are
-  highlighted as a @:exception@ instead of @:pre_type@
+  See "#221":http://redmine.rubychan.de/issues/221.
+* *FIXED*: Closes open string groups.
+* *FIXED*: Words after @.@ are always recognized as @:ident@.
 
-h3. @Scanners::JavaScript@
+h3. @Scanners::YAML@
  
-* *NEW* a list of @PREDEFINED_CONSTANTS@ to be highlighted as @:pre_constant@
-* *NEW* XML literals are recognized and highlighted
-* *NEW* function name highlighting
-* *IMPROVED* @.1@ is highlighted a number
-* *FIXED* strings close with the correct kind when terminated unexpectedly
-
-h3. @Scanners::JSON@
+* *FIXED*: Allow spaces before colon in mappings.
  
-* *IMPROVED* constants (@true@, @false@, @nil@) are highlighted as @:value@
+  See "#231":http://redmine.rubychan.de/issues/231.
 
-h3. @Scanners::Ruby@
- 
-* *IMPROVED* @Patterns::KEYWORDS_EXPECTING_VALUE@ for more accurate
-  @value_expected@ detection
-* *IMPROVED* handling of @\@ as a string delimiter
-* *IMPROVED* handling of unicode strings; automatic switching to unicode
-* *IMPROVED* highlighting of @self.method@ definitions
-* *REMOVED* @Patterns::FANCY_START_SAVE@ (obsolete)
-* *FIXED* encoding issues _[Ruby 1.9]_
-* *FIXED* a problem in early Ruby 1.8.6 patch versions with @Regexp.escape@
+h3. *NEW*: @Styles::Alpha@
 
-h3. @Scanners::YAML@
+A style that uses transparent HSLA colors as defined in CSS 3. See "#199":http://redmine.rubychan.de/issues/199.
  
-* *IMPROVED* indentation detection
+It also uses the CSS 3 property @user-select: none@ to keep the user from selecting the line numbers. This is especially
+nice for @:inline@ line numbers. See "#226":http://redmine.rubychan.de/issues/226.
 
-h3. @Styles::Cycnus@
+h3. @WordList@
  
-* changed a few colors (exceptions, inline strings, predefined types)
+Stripped down to 19 LOC.
 
-h3. @Plugin@
- 
-* *NEW* method @#title@
-  
-  Set and get the plugin's title. Titles can be arbitrary strings.
-* *NEW* method @#helper@ loads helpers from different plugins
-  
-  Use this syntax: @helper 'other_plugin/helper_name'@
+* *RENAMED* @CaseIgnoringWordList@ to @WordList::CaseIgnoring@.
+* *REMOVED* caching option because it creates memory leaks.
+* *REMOVED* block option.
 
 h3. @FileType@
  
-* *NEW* @FileType[]@ takes @Pathname@ instances
-* *NEW* regonizes @.cc@, @.cpp@, @.cp@, @.cxx@, @.c++@, @.C@, @.hh@, @.hpp@, @.h++@, @.cu@ extensions (C++)
+* *NEW*: Recognizes @.gemspec@, @.rjs@, @.rpdf@ extensions, @Gemfile@, and @Capfile@ as Ruby.
   
-  Thanks to Sander Cox and the TextMate C bundle.
-* *NEW* regonizes @.pas@, @.dpr@ extensions (Delphi)
-* *NEW* regonizes @.gvy@, @.groovy@ extensions (Groovy)
-* *NEW* regonizes @.php@, @.php3@, @.php4@, @.php5@ extensions (PHP)
-* *NEW* regonizes @.py@, @.py3@, @.pyw@ extensions (Python)
-* *NEW* regonizes @.rxml@ extension (Ruby)
-* *NEW* regonizes @.sql@ extension (SQL)
-* File types list was sorted alphabetically.
-
-h3. @CaseIgnoringWordList@
- 
-* *FIXED* ("#97":http://redmine.rubychan.de/issues/97)
-  
-  The default value is no longer ignored.
+  Thanks to the authors of the TextMate Ruby bundle!
+* *REMOVED* @FileType#shebang@ is a protected method now.
+
+h3. @Plugin@
+ 
+* *IMPROVED*: @register_for@ sets the @plugin_id@; it can now be a @Symbol@.
+* *ADDED* @PluginHost#const_missing@ method: Plugins are loaded automatically.
+  Using @Scanners::JavaScript@ in your code loads @scanners/java_script.rb@.
+* *ADDED* @#all_plugins@ method to simplify getting 
+  information about all available plugins (suggested by bnhymn).
+* *CHANGED* the default plugin key from @nil@ to @:default@.
 
-h3. @ForRedCloth@
+h3. @GZip@
  
-* *FIXED* for RedCloth versions 4.2.0+ ("#119":http://redmine.rubychan.de/issues/119)
+* *MOVED* into @CodeRay@ namespace.
+* *MOVED* file from @gzip_simple.rb@ to @gzip.rb@.
+* *REMOVED* @String@ extensions.
 
-h3. Cleanups
+h3. More API changes
  
-* warnings about character classes _[Ruby 1.9]_
-* encoding issues _[Ruby 1.9]_
-* documentation, code
+* *FIXED* @Encoders::HTML#token@'s second parameter is no longer optional.
+* *CHANGED* @Encoders::HTML::Output@'s API.
+* *REMOVED* lots of unused methods.
+ 
+The helper classes were cleaned up; see above for details.
+ 
+* *CHANGED* @Plugin@ API was simplified and stripped of all unnecessary features.
+* *CHANGED* Moved @GZip@ and @FileType@ libraries into @CodeRay@; cleaned them up.
+