summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorKornelius Kalnbach <murphy@rubychan.de>2013-03-10 22:46:27 +0100
committerKornelius Kalnbach <murphy@rubychan.de>2013-03-10 22:46:27 +0100
commit7d2c3e59d1691bcde16fc3e51d92797bfc9a2442 (patch)
tree89f349d803d5327f12d728fbd994c6503541f713
parentec39f80865cac97a49a6fec138dedc22e5971bad (diff)
parentfc16be24d48f8a729ac987149f98f19725943e0d (diff)
downloadcoderay-7d2c3e59d1691bcde16fc3e51d92797bfc9a2442.tar.gz
Merge branch 'master' into go-scanner
-rw-r--r--.gitignore2
-rw-r--r--.travis.yml7
-rw-r--r--CREDITS.textile (renamed from README.textile)41
-rw-r--r--Changes.textile180
-rw-r--r--Gemfile2
-rw-r--r--README.markdown27
-rw-r--r--bench/bench.rb2
-rw-r--r--coderay.gemspec9
-rw-r--r--lib/coderay/encoders/html/numbering.rb41
-rw-r--r--lib/coderay/encoders/html/output.rb2
-rw-r--r--lib/coderay/scanners/diff.rb80
-rw-r--r--lib/coderay/scanners/html.rb7
-rw-r--r--lib/coderay/scanners/php.rb5
-rw-r--r--lib/coderay/scanners/ruby.rb33
-rw-r--r--lib/coderay/scanners/ruby/patterns.rb36
-rw-r--r--lib/coderay/version.rb2
-rw-r--r--rake_tasks/test.rake35
-rw-r--r--test/executable/suite.rb5
-rwxr-xr-xtest/functional/examples.rb4
19 files changed, 283 insertions, 237 deletions
diff --git a/.gitignore b/.gitignore
index c03ec75..caab594 100644
--- a/.gitignore
+++ b/.gitignore
@@ -22,4 +22,4 @@ bench/test.div.html
diff.html
etc/CodeRay.tmproj
*.swp
-etc \ No newline at end of file
+old-stuff
diff --git a/.travis.yml b/.travis.yml
index 614c836..7c29c56 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -5,12 +5,11 @@ rvm:
- jruby-18mode
- jruby-19mode
- rbx-18mode
- # - rbx-19mode # test again later
- # - ruby-head # test again later
- # - jruby-head # test again later
+ - rbx-19mode
+ - ruby-head # test again later: RedCloth not compiling
+ - jruby-head
- ree
branches:
only:
- master
- - stable
script: "rake test" # test:scanners"
diff --git a/README.textile b/CREDITS.textile
index 543dc47..4c58c54 100644
--- a/README.textile
+++ b/CREDITS.textile
@@ -1,43 +1,4 @@
-h1. CodeRay !https://secure.travis-ci.org/rubychan/coderay.png!:https://secure.travis-ci.org/rubychan/coderay
-
-h2. About
-
-CodeRay is a Ruby library for syntax highlighting.
-
-You put your code in, and you get it back colored; Keywords, strings, floats, comments - all in different colors. And with line numbers.
-
-*Syntax Highlighting*…
-
-* makes code easier to read and maintain
-* lets you detect syntax errors faster
-* helps you to understand the syntax of a language
-* looks nice
-* is what everybody wants to have on their website
-* solves all your problems and makes the girls run after you
-
-
-h2. Installation
-
-bc. gem install coderay
-
-h3. Dependencies
-
-CodeRay needs Ruby 1.8.7+ or 1.9.2+. It also runs on Rubinius and JRuby.
-
-h2. Example Usage
-
-bc.. require 'coderay'
-
-html = CodeRay.scan("puts 'Hello, world!'", :ruby).div(:line_numbers => :table)
-
-p.
-
-h2. Documentation
-
-See "http://coderay.rubychan.de/doc/":http://coderay.rubychan.de/doc/.
-
-
-h2. Credits
+h1. Credits
h3. Special Thanks to
diff --git a/Changes.textile b/Changes.textile
index d70c303..3984b83 100644
--- a/Changes.textile
+++ b/Changes.textile
@@ -1,81 +1,93 @@
h1=. CodeRay Version History
-
+
p=. _This files lists all changes in the CodeRay library since the 0.9.8 release._
-
-{{toc}}
-
-h2. Next Version
-
+
+h2. Changes in 1.1
+
+* Diff scanner: Highlight inline changes in multi-line changes [#99]
+* Remove double-click toggle handler from HTML table output
+* Display line numbers in HTML @:table@ mode even for single-line code (remove special case) [#41, thanks to Ariejan de Vroom]
+
+h2. Changes in 1.0.9
+
+* Fix Ruby scanner: Ruby 1.9 hash syntax @{ key: value }@ is highlighted correctly. [GH #106, thanks to Seth Vargo]
+* Fix HTML scanner: Accept DTDs. [GH #83]
+* Fix PHP scanner: Accept Unicode. [GH #40, thanks to Lance Li]
+
+h2. Changes in 1.0.8
+
* add @:string/:char@, remove @:regexp/:function@ color from Terminal encoder [GH #29, thanks to Kyrylo Silin]
+* allow @-@ in line number anchor prefix for HTML encoder [GH #32, thanks to shurizzle]
+* Fix HTML scanner: Don't crash if HTML in a diff contains a JavaScript tag.
h2. Changes in 1.0.7
-
+
* Changed license from LGPL to MIT. [GH-25, thanks to jessehu]
* Fix issue with plugin files not being loaded. [GH-20, thanks to Will Read]
* Fix HTML scanner bug: Don't choke on boolean attributes. [GH-26, thanks to jugglinmike]
h2. Changes in 1.0.6
-
+
* New option @:break_lines@ for the HTML encoder (splits tokens at line breaks). [GH-15, thanks to Etienne Massip]
* Improved speed of @:line_numbers => :inline@ option for the HTML encoder.
* Fixed wrong HTML file type. (was @:page@) [GH-16, thanks to Doug Hammond]
* The CSS Scanner now highlights tokens like @url(...)@ as @:function@ instead of @:string@. [GH-13, thanks to Joel Holdbrooks]
h2. Changes in 1.0.5
-
+
Fixes:
-
+
* @autoload@ calls do not depend on @coderay/lib@ being in the load path (GitHub issue #6; thanks to tvon, banister, envygeeks, and ConradIrwin)
* avoid dark blue as terminal color (GitHub issue #9; thanks to shevegen)
h2. Changes in 1.0.4
-
+
Fixes in the CSS scanner:
-
+
* understands the unit "s" (seconds)
* ignores unexpected curly braces
* code inside of diffs is highlighted correctly
h2. Changes in 1.0.3
-
+
New:
-
+
* .tmproj files are recognized as XML.
-
+
Fixes:
-
+
* Removed files are highlighted inside diffs generated by git.
h2. Changes in 1.0.2
-
+
Fixes:
-
+
* .erb files are recognized as ERB.
h2. Changes in 1.0.1
-
+
New:
-
+
* YAML scanner allows "-" and "/" in key names
-
+
Changes:
-
+
* HTML page output has no white border anymore (alpha style)
-
+
Fixes:
-
+
* fixed warning in the output of "coderay stylesheet"
* fixed additional scrollbar in code when last line contains an eyecatcher
* minor fixes in the tests (issue github-#4)
h2. Changes in 1.0
-
+
CodeRay 1.0 is a major rewrite of the library, and incompatible to earlier versions.
-
+
The command line and programmer interfaces are similar to 0.9, but the internals have completely changed.
h3. General changes
-
+
* *NEW*: The new Diff scanner colorizes code inside of the diff, and highlights inline changes.
* *NEW*: Extended support and usage of HTML5 and CSS 3 features.
* *NEW*: Direct Streaming
@@ -87,18 +99,18 @@ h3. General changes
* *IMPROVED* Tests: There are more of them now!
h3. Direct Streaming
-
+
CodeRay 1.0 introduces _Direct Streaming_ as a faster and simpler alternative to Tokens. It means that all Scanners,
Encoders and Filters had to be rewritten, and that older scanners using the Tokens API are no longer compatible with
this version.
-
+
The main benefits of this change are:
-
+
* more speed (benchmarks show 10% to 50% more tokens per second compared to CodeRay 0.9)
* the ability to stream output into a pipe on the command line
* a simpler API
* less code
-
+
Changes related to the new tokens handling include:
* *CHANGED*: The Scanners now call Encoders directly; tokens are not added to a Tokens array, but are send to the
Encoder as a method call. The Tokens representation (which can be seen as a cache now) is still present, but as a
@@ -115,16 +127,16 @@ Changes related to the new tokens handling include:
and have been removed.
h3. Command Line
-
+
The @coderay@ executable was rewritten and has a few new features:
-
+
* *NEW* Ability to stream into a pipe; try @coderay file | more -r@
* *NEW* help
* *IMPROVED*: more consistent parameter handling
* *REMOVED* @coderay_stylesheet@ executable; use @coderay stylesheet [name]@.
h3. @Tokens@
-
+
* *NEW* methods @count@, @begin_group@, @end_group@, @begin_line@, and @end_line@.
* *REMOVED* methods @#stream?@, @#each_text_token@.
* *REMOVED* methods @#optimize@, @#fix@, @#split_into_lines@ along with their bang! variants.
@@ -132,11 +144,11 @@ h3. @Tokens@
* *REMOVED* special implementation of @#each@ taking a filter parameter. Use @TokenKindFilter@ instead.
h3. *RENAMED*: @TokenKinds@
-
+
Renamed from @Tokens::ClassOfKind@ (was also @Tokens::AbbreviationForKind@ for a while).
The term "token class" is no longer used in CodeRay. Instead, tokens have _kinds_.
See "#122":http://odd-eyed-code.org/issues/122.
-
+
* *CHANGED* all token CSS classes to readable names.
* *ADDED* token kinds @:filename@, @:namespace@, and @:eyecatcher@.
* *RENAMED* @:pre_constant@ and @:pre_type@ to @:predefined_constant@ and @predefined_type@.
@@ -148,23 +160,23 @@ See "#122":http://odd-eyed-code.org/issues/122.
@:NO_HIGHLIGHT@ to @false@.
h3. @Duo@
-
+
* *NEW* method @call@ for allowing code like @CodeRay::Duo[:python => :yaml].(code)@ in Ruby 1.9.
h3. @Encoders::CommentFilter@
-
+
* *NEW* alias @:remove_comments@
h3. @Encoders::Filter@
-
+
* *NEW* option @tokens@.
* *CHANGED*: Now it simply delegates to the output.
* *REMOVED* @include_text_token?@ and @include_block_token?@ methods.
h3. @Encoders::HTML@
-
+
The HTML encoder was cleaned up and simplified.
-
+
* *NEW*: HTML5 and CSS 3 compatible.
See "#215":http://odd-eyed-code.org/issues/215.
* *ADDED* support for @:line_number_anchors@.
@@ -178,11 +190,11 @@ The HTML encoder was cleaned up and simplified.
* *RENAMED* @Output#numerize@ to @#number@, which is an actual English word.
h3. @Encoders::LinesOfCode@
-
+
* *CHANGED*: @compile@ and @finish@ methods are now protected.
h3. *Renamed*: @Encoders::Terminal@ (was @Encoders::Term@)
-
+
* *RENAMED* from @Encoders::Term@, added @:term@ alias.
* *CLEANUP*: Use @#setup@'s @super@, don't use @:procedure@ token class.
* *CHANGED*: @#token@'s second parameter is no longer optional.
@@ -190,21 +202,21 @@ h3. *Renamed*: @Encoders::Terminal@ (was @Encoders::Term@)
* *FIXED* handling of line tokens.
h3. @Encoders::Text@
-
+
* *FIXED* default behavior of stripping the trailing newline.
h3. *RENAMED*: @Encoders::TokenKindFilter@ (was @Encoders::TokenClassFilter@)
-
+
* *NEW*: Handles token groups.
See "#223":http://odd-eyed-code.org/issues/223.
* *RENAMED* @include_block_token?@ to @include_group?@.
h3. @Encoders::Statistic@
-
+
* *CHANGED*: Tokens actions are counted separately.
h3. @Scanners::Scanner@
-
+
* *NEW* methods @#file_extension@ and @#encoding@.
* *NEW*: The @#tokenize@ method also takes an Array of Strings as source. The
code is highlighted as one and split into parts of the input lengths
@@ -217,11 +229,11 @@ h3. @Scanners::Scanner@
* *CHANGED*: @#column@ starts counting with 1 instead of 0
h3. *NEW*: @Scanners::Clojure@
-
+
Thanks to Licenser, CodeRay now supports the Clojure language.
h3. @Scanners::CSS@
-
+
* *NEW*: Rudimentary support for the @attr@, @counter@, and @counters@ functions.
See "#224":http://odd-eyed-code.org/issues/224.
* *NEW*: Rudimentary support for CSS 3 colors.
@@ -229,7 +241,7 @@ h3. @Scanners::CSS@
* *CHANGED*: Comments are scanned as one token instead of three.
h3. @Scanners::Debug@
-
+
* *NEW*: Support for line tokens (@begin_line@ and @end_line@ represented by @[@ and @]@.)
* *FIXED*: Don't send @:error@ and @nil@ tokens for buggy input any more.
* *FIXED*: Closes unclosed tokens at the end of @scan_tokens@.
@@ -237,32 +249,32 @@ h3. @Scanners::Debug@
* *CHANGED*: Raises an error when trying to end an invalid token group.
h3. @Scanners::Delphi@
-
+
* *FIXED*: Closes open string groups.
h3. @Scanners::Diff@
-
+
* *NEW*: Highlighting of code based on file names.
See ticket "#52":http://odd-eyed-code.org/issues/52.
-
+
Use the @:highlight_code@ option to turn this feature off. It's enabled
by default.
-
+
This is a very original feature. It enables multi-language highlighting for
diff files, which is especially helpful for CodeRay development itself. The
updated version of the scanner test suite generated .debug.diff.html files
using this.
-
+
Note: This is still experimental. Tokens spanning more than one line
may get highlighted incorrectly. CodeRay tries to keep scanner states
between the lines and changes, but the quality of the results depend on
the scanner.
* *NEW*: Inline change highlighting, as suggested by Eric Thomas.
See ticket "#227":http://odd-eyed-code.org/issues/227 for details.
-
+
Use the @:inline_diff@ option to turn this feature off. It's enabled by
default.
-
+
For single-line changes (that is, a single deleted line followed by a single
inserted line), this feature surrounds the changed parts with an
@:eyecatcher@ group which appears in a more saturated background color.
@@ -279,38 +291,38 @@ h3. @Scanners::Diff@
h3. *RENAMED*: @Scanners::ERB@ (was @Scanners::RHTML@)
h3. *NEW*: @Scanners::HAML@
-
+
It uses the new :state options of the HTML and Ruby scanners.
-
+
Some rare cases are not considered (like @#{...}@ snippets inside of :javascript blocks),
but it highlights pretty well.
h3. @Scanners::HTML@
-
+
* *FIXED*: Closes open string groups.
h3. @Scanners::JavaScript@
-
+
* *IMPROVED*: Added @NaN@ and @Infinity@ to list of predefined constants.
* *IMPROVED* recognition of RegExp literals with leading spaces.
h3. @Scanners::Java@
-
+
* *NEW*: Package names are highlighted as @:namespace@.
See "#210":http://odd-eyed-code.org/issues/210.
h3. *REMOVED*: @Scanners::NitroXHTML@
-
+
Nitro is "dead":http://www.nitrohq.com/.
h3. *RENAMED*: @Scanners::Text@ (was @Scanners::Plaintext@)
-
+
* *IMPROVED*: Just returns the string without scanning (faster).
-
+
This is much faster than scanning until @/\z/@ in Ruby 1.8.
h3. @Scanners::Python@
-
+
* *CHANGED*: Docstrings are highlighted as @:comment@.
See "#190":http://odd-eyed-code.org/issues/190.
@@ -320,7 +332,7 @@ Copied from @Scanners::Debug@, highlights the token dump instead of importing it
name suffix now.
h3. @Scanners::Ruby@
-
+
* *ADDED* more predefined keywords (see http://murfy.de/ruby-constants).
* *IMPROVED* support for singleton method definitions.
See "#147":http://odd-eyed-code.org/issues/147.
@@ -328,44 +340,44 @@ h3. @Scanners::Ruby@
(eg. @GL.PushMatrix@).
* *NEW*: Highlight buggy floats (like .5) as @:error@.
* *CLEANUP* of documentation, names of constants and variables, state handling.
-
+
Moved @StringState@ class from @patterns.rb@ into a separate file.
* *NEW*: Complicated rule for recognition of @foo=@ style method names.
* *NEW*: Handles @:keep_state@ option (a bit; experimental).
-
+
Actually, Ruby checks if there is @[~>=]@, but not @=>@ following the name.
-
+
* *REMOVED* @EncodingError@
h3. *REMOVED* @Scanners::Scheme@
-
+
* It is too buggy, and nobody was using it. To be added again when it's fixed.
See "#59":http://odd-eyed-code.org/issues/59.
h3. @Scanners::SQL@
-
+
* *IMPROVED*: Extended list of keywords and functions (thanks to
Joshua Galvez, Etienne Massip, and others).
-
+
See "#221":http://odd-eyed-code.org/issues/221.
* *FIXED*: Closes open string groups.
* *FIXED*: Words after @.@ are always recognized as @:ident@.
h3. @Scanners::YAML@
-
+
* *FIXED*: Allow spaces before colon in mappings.
-
+
See "#231":http://odd-eyed-code.org/issues/231.
h3. *NEW*: @Styles::Alpha@
A style that uses transparent HSLA colors as defined in CSS 3. See "#199":http://odd-eyed-code.org/issues/199.
-
+
It also uses the CSS 3 property @user-select: none@ to keep the user from selecting the line numbers. This is especially
nice for @:inline@ line numbers. See "#226":http://odd-eyed-code.org/issues/226.
h3. @WordList@
-
+
Stripped down to 19 LOC.
* *RENAMED* @CaseIgnoringWordList@ to @WordList::CaseIgnoring@.
@@ -373,14 +385,14 @@ Stripped down to 19 LOC.
* *REMOVED* block option.
h3. @FileType@
-
+
* *NEW*: Recognizes @.gemspec@, @.rjs@, @.rpdf@ extensions, @Gemfile@, and @Capfile@ as Ruby.
-
+
Thanks to the authors of the TextMate Ruby bundle!
* *REMOVED* @FileType#shebang@ is a protected method now.
h3. @Plugin@
-
+
* *IMPROVED*: @register_for@ sets the @plugin_id@; it can now be a @Symbol@.
* *ADDED* @PluginHost#const_missing@ method: Plugins are loaded automatically.
Using @Scanners::JavaScript@ in your code loads @scanners/java_script.rb@.
@@ -389,19 +401,19 @@ h3. @Plugin@
* *CHANGED* the default plugin key from @nil@ to @:default@.
h3. @GZip@
-
+
* *MOVED* into @CodeRay@ namespace.
* *MOVED* file from @gzip_simple.rb@ to @gzip.rb@.
* *REMOVED* @String@ extensions.
h3. More API changes
-
+
* *FIXED* @Encoders::HTML#token@'s second parameter is no longer optional.
* *CHANGED* @Encoders::HTML::Output@'s API.
* *REMOVED* lots of unused methods.
-
+
The helper classes were cleaned up; see above for details.
-
+
* *CHANGED* @Plugin@ API was simplified and stripped of all unnecessary features.
* *CHANGED* Moved @GZip@ and @FileType@ libraries into @CodeRay@; cleaned them up.
diff --git a/Gemfile b/Gemfile
index 80fe57c..aa03288 100644
--- a/Gemfile
+++ b/Gemfile
@@ -8,7 +8,7 @@ gemspec
group :development do
gem "bundler", ">= 1.0.0"
gem "rake", "~> 0.9.2"
- gem "RedCloth", RUBY_PLATFORM == 'java' ? "= 4.2.7" : ">= 4.0.3"
+ gem "RedCloth", RUBY_PLATFORM == 'java' ? ">= 4.2.7" : ">= 4.0.3"
gem "term-ansicolor"
gem "shoulda-context", "~> 1.0.0" if RUBY_VERSION >= '1.8.7'
gem "json" unless RUBY_VERSION >= '1.9.1'
diff --git a/README.markdown b/README.markdown
new file mode 100644
index 0000000..42bdab6
--- /dev/null
+++ b/README.markdown
@@ -0,0 +1,27 @@
+# CodeRay [![Build Status](https://travis-ci.org/rubychan/coderay.png)](https://travis-ci.org/rubychan/coderay)
+
+## About
+
+CodeRay is a Ruby library for syntax highlighting.
+
+You put your code in, and you get it back colored; Keywords, strings, floats, comments - all in different colors. And with line numbers.
+
+## Installation
+
+`gem install coderay`
+
+### Dependencies
+
+CodeRay needs Ruby 1.8.7+ or 1.9.2+. It also runs on Rubinius and JRuby.
+
+## Example Usage
+
+```ruby
+require 'coderay'
+
+html = CodeRay.scan("puts 'Hello, world!'", :ruby).div(:line_numbers => :table)
+````
+
+## Documentation
+
+See [http://coderay.rubychan.de/doc/](http://coderay.rubychan.de/doc/).
diff --git a/bench/bench.rb b/bench/bench.rb
index 45dc5b0..1889eed 100644
--- a/bench/bench.rb
+++ b/bench/bench.rb
@@ -108,7 +108,7 @@ N.times do
$file_created = here('test.' +
($dump_output ? 'dump' : $hl.file_extension))
File.open($file_created, 'wb') do |f|
- f.write $o
+ # f.write $o
end
Dir.chdir(here) do
FileUtils.copy 'test.dump', 'example.dump' if $dump_output
diff --git a/coderay.gemspec b/coderay.gemspec
index 1f88318..328b94c 100644
--- a/coderay.gemspec
+++ b/coderay.gemspec
@@ -8,10 +8,7 @@ Gem::Specification.new do |s|
if ENV['RELEASE']
s.version = CodeRay::VERSION
else
- # thanks to @Argorak for this solution
- # revision = 134 + (`git log --oneline | wc -l`.to_i)
- # s.version = "#{CodeRay::VERSION}.#{revision}rc1"
- s.version = "#{CodeRay::VERSION}.rc2"
+ s.version = "#{CodeRay::VERSION}.rc#{ENV['RC'] || 1}"
end
s.authors = ['Kornelius Kalnbach']
@@ -20,12 +17,14 @@ Gem::Specification.new do |s|
s.summary = 'Fast syntax highlighting for selected languages.'
s.description = 'Fast and easy syntax highlighting for selected languages, written in Ruby. Comes with RedCloth integration and LOC counter.'
+ s.license = 'MIT'
+
s.platform = Gem::Platform::RUBY
s.required_ruby_version = '>= 1.8.6'
readme_file = 'README_INDEX.rdoc'
- s.files = `git ls-files -- lib/* test/functional/* Rakefile #{readme_file} LICENSE`.split("\n")
+ s.files = `git ls-files -- lib/* test/functional/* Rakefile #{readme_file} MIT-LICENSE`.split("\n")
s.test_files = `git ls-files -- test/functional/*`.split("\n")
s.executables = `git ls-files -- bin/*`.split("\n").map { |f| File.basename(f) }
s.require_paths = ['lib']
diff --git a/lib/coderay/encoders/html/numbering.rb b/lib/coderay/encoders/html/numbering.rb
index 8bc6259..332145b 100644
--- a/lib/coderay/encoders/html/numbering.rb
+++ b/lib/coderay/encoders/html/numbering.rb
@@ -1,15 +1,15 @@
module CodeRay
module Encoders
-
+
class HTML
-
+
module Numbering # :nodoc:
-
+
def self.number! output, mode = :table, options = {}
return self unless mode
-
+
options = DEFAULT_OPTIONS.merge options
-
+
start = options[:line_number_start]
unless start.is_a? Integer
raise ArgumentError, "Invalid value %p for :line_number_start; Integer expected." % start
@@ -17,7 +17,7 @@ module Encoders
anchor_prefix = options[:line_number_anchors]
anchor_prefix = 'line' if anchor_prefix == true
- anchor_prefix = anchor_prefix.to_s[/\w+/] if anchor_prefix
+ anchor_prefix = anchor_prefix.to_s[/[\w-]+/] if anchor_prefix
anchoring =
if anchor_prefix
proc do |line|
@@ -56,12 +56,17 @@ module Encoders
raise ArgumentError, 'Invalid value %p for :bolding; false or Integer expected.' % bold_every
end
- line_count = output.count("\n")
- position_of_last_newline = output.rindex(RUBY_VERSION >= '1.9' ? /\n/ : ?\n)
- if position_of_last_newline
+ if position_of_last_newline = output.rindex(RUBY_VERSION >= '1.9' ? /\n/ : ?\n)
after_last_newline = output[position_of_last_newline + 1 .. -1]
ends_with_newline = after_last_newline[/\A(?:<\/span>)*\z/]
- line_count += 1 if not ends_with_newline
+
+ if ends_with_newline
+ line_count = output.count("\n")
+ else
+ line_count = output.count("\n") + 1
+ end
+ else
+ line_count = 1
end
case mode
@@ -74,30 +79,30 @@ module Encoders
line_number += 1
"<span class=\"line-numbers\">#{indent}#{line_number_text}</span>#{line}"
end
-
+
when :table
line_numbers = (start ... start + line_count).map(&bolding).join("\n")
line_numbers << "\n"
line_numbers_table_template = Output::TABLE.apply('LINE_NUMBERS', line_numbers)
-
+
output.gsub!(/<\/div>\n/, '</div>')
output.wrap_in! line_numbers_table_template
output.wrapped_in = :div
-
+
when :list
raise NotImplementedError, 'The :list option is no longer available. Use :table.'
-
+
else
raise ArgumentError, 'Unknown value %p for mode: expected one of %p' %
[mode, [:table, :inline]]
end
-
+
output
end
-
+
end
-
+
end
-
+
end
end
diff --git a/lib/coderay/encoders/html/output.rb b/lib/coderay/encoders/html/output.rb
index 9132d94..de6f6ea 100644
--- a/lib/coderay/encoders/html/output.rb
+++ b/lib/coderay/encoders/html/output.rb
@@ -124,7 +124,7 @@ module Encoders
TABLE = Template.new <<-TABLE
<table class="CodeRay"><tr>
- <td class="line-numbers" title="double click to toggle" ondblclick="with (this.firstChild.style) { display = (display == '') ? 'none' : '' }"><pre><%LINE_NUMBERS%></pre></td>
+ <td class="line-numbers"><pre><%LINE_NUMBERS%></pre></td>
<td class="code"><pre><%CONTENT%></pre></td>
</tr></table>
TABLE
diff --git a/lib/coderay/scanners/diff.rb b/lib/coderay/scanners/diff.rb
index 9e899c3..38efaf4 100644
--- a/lib/coderay/scanners/diff.rb
+++ b/lib/coderay/scanners/diff.rb
@@ -20,7 +20,7 @@ module Scanners
line_kind = nil
state = :initial
- deleted_lines = 0
+ deleted_lines_count = 0
scanners = Hash.new do |h, lang|
h[lang] = Scanners[lang].new '', :keep_tokens => true, :keep_state => true
end
@@ -30,7 +30,7 @@ module Scanners
until eos?
if match = scan(/\n/)
- deleted_lines = 0 unless line_kind == :delete
+ deleted_lines_count = 0 unless line_kind == :delete
if line_kind
encoder.end_line line_kind
line_kind = nil
@@ -99,37 +99,59 @@ module Scanners
end
next
elsif match = scan(/-/)
- deleted_lines += 1
- encoder.begin_line line_kind = :delete
- encoder.text_token match, :delete
- if options[:inline_diff] && deleted_lines == 1 && check(/(?>.*)\n\+(?>.*)$(?!\n\+)/)
- content_scanner_entry_state = content_scanner.state
- skip(/(.*)\n\+(.*)$/)
- head, deletion, insertion, tail = diff self[1], self[2]
- pre, deleted, post = content_scanner.tokenize [head, deletion, tail], :tokens => Tokens.new
- encoder.tokens pre
- unless deleted.empty?
- encoder.begin_group :eyecatcher
- encoder.tokens deleted
- encoder.end_group :eyecatcher
+ deleted_lines_count += 1
+ if options[:inline_diff] && deleted_lines_count == 1 && (changed_lines_count = 1 + check(/.*(?:\n\-.*)*/).count("\n")) && match?(/(?>.*(?:\n\-.*){#{changed_lines_count - 1}}(?:\n\+.*){#{changed_lines_count}})$(?!\n\+)/)
+ deleted_lines = Array.new(changed_lines_count) { |i| skip(/\n\-/) if i > 0; scan(/.*/) }
+ inserted_lines = Array.new(changed_lines_count) { |i| skip(/\n\+/) ; scan(/.*/) }
+
+ deleted_lines_tokenized = []
+ inserted_lines_tokenized = []
+ for deleted_line, inserted_line in deleted_lines.zip(inserted_lines)
+ pre, deleted_part, inserted_part, post = diff deleted_line, inserted_line
+ content_scanner_entry_state = content_scanner.state
+ deleted_lines_tokenized << content_scanner.tokenize([pre, deleted_part, post], :tokens => Tokens.new)
+ content_scanner.state = content_scanner_entry_state || :initial
+ inserted_lines_tokenized << content_scanner.tokenize([pre, inserted_part, post], :tokens => Tokens.new)
end
- encoder.tokens post
- encoder.end_line line_kind
- encoder.text_token "\n", :space
- encoder.begin_line line_kind = :insert
- encoder.text_token '+', :insert
- content_scanner.state = content_scanner_entry_state || :initial
- pre, inserted, post = content_scanner.tokenize [head, insertion, tail], :tokens => Tokens.new
- encoder.tokens pre
- unless inserted.empty?
- encoder.begin_group :eyecatcher
- encoder.tokens inserted
- encoder.end_group :eyecatcher
+
+ for pre, deleted_part, post in deleted_lines_tokenized
+ encoder.begin_line :delete
+ encoder.text_token '-', :delete
+ encoder.tokens pre
+ unless deleted_part.empty?
+ encoder.begin_group :eyecatcher
+ encoder.tokens deleted_part
+ encoder.end_group :eyecatcher
+ end
+ encoder.tokens post
+ encoder.end_line :delete
+ encoder.text_token "\n", :space
+ end
+
+ for pre, inserted_part, post in inserted_lines_tokenized
+ encoder.begin_line :insert
+ encoder.text_token '+', :insert
+ encoder.tokens pre
+ unless inserted_part.empty?
+ encoder.begin_group :eyecatcher
+ encoder.tokens inserted_part
+ encoder.end_group :eyecatcher
+ end
+ encoder.tokens post
+ changed_lines_count -= 1
+ if changed_lines_count > 0
+ encoder.end_line :insert
+ encoder.text_token "\n", :space
+ end
end
- encoder.tokens post
+
+ line_kind = :insert
+
elsif match = scan(/.*/)
+ encoder.begin_line line_kind = :delete
+ encoder.text_token '-', :delete
if options[:highlight_code]
- if deleted_lines == 1
+ if deleted_lines_count == 1
content_scanner_entry_state = content_scanner.state
end
content_scanner.tokenize match, :tokens => encoder unless match.empty?
diff --git a/lib/coderay/scanners/html.rb b/lib/coderay/scanners/html.rb
index 733dd6f..3ba3b79 100644
--- a/lib/coderay/scanners/html.rb
+++ b/lib/coderay/scanners/html.rb
@@ -69,6 +69,7 @@ module Scanners
def setup
@state = :initial
@plain_string_content = nil
+ @in_tag = nil
end
def scan_java_script encoder, code
@@ -83,7 +84,8 @@ module Scanners
def scan_tokens encoder, options
state = options[:state] || @state
plain_string_content = @plain_string_content
- in_tag = in_attribute = nil
+ in_tag = @in_tag
+ in_attribute = nil
encoder.begin_group :string if state == :attribute_value_string
@@ -99,7 +101,7 @@ module Scanners
when :initial
if match = scan(/<!--(?:.*?-->|.*)/m)
encoder.text_token match, :comment
- elsif match = scan(/<!DOCTYPE(?:.*?>|.*)/m)
+ elsif match = scan(/<!(\w+)(?:.*?>|.*)|\]>/m)
encoder.text_token match, :doctype
elsif match = scan(/<\?xml(?:.*?\?>|.*)/m)
encoder.text_token match, :preprocessor
@@ -237,6 +239,7 @@ module Scanners
if options[:keep_state]
@state = state
@plain_string_content = plain_string_content
+ @in_tag = in_tag
end
encoder.end_group :string if state == :attribute_value_string
diff --git a/lib/coderay/scanners/php.rb b/lib/coderay/scanners/php.rb
index 8acfff5..6c68834 100644
--- a/lib/coderay/scanners/php.rb
+++ b/lib/coderay/scanners/php.rb
@@ -1,4 +1,4 @@
-# encoding: ASCII-8BIT
+# encoding: utf-8
module CodeRay
module Scanners
@@ -11,7 +11,6 @@ module Scanners
register_for :php
file_extension 'php'
- encoding 'BINARY'
KINDS_NOT_LOC = HTML::KINDS_NOT_LOC
@@ -211,7 +210,7 @@ module Scanners
HTML_INDICATOR = /<!DOCTYPE html|<(?:html|body|div|p)[> ]/i
- IDENTIFIER = /[a-z_\x7f-\xFF][a-z0-9_\x7f-\xFF]*/i
+ IDENTIFIER = 'ä'[/[[:alpha:]]/] == 'ä' ? Regexp.new('[[:alpha:]_[^\0-\177]][[:alnum:]_[^\0-\177]]*') : Regexp.new('[a-z_\x7f-\xFF][a-z0-9_\x7f-\xFF]*', true)
VARIABLE = /\$#{IDENTIFIER}/
OPERATOR = /
diff --git a/lib/coderay/scanners/ruby.rb b/lib/coderay/scanners/ruby.rb
index 2be98a6..c5cf1e2 100644
--- a/lib/coderay/scanners/ruby.rb
+++ b/lib/coderay/scanners/ruby.rb
@@ -94,18 +94,27 @@ module Scanners
if !method_call_expected &&
match = scan(unicode ? /#{patterns::METHOD_NAME}/uo :
/#{patterns::METHOD_NAME}/o)
- value_expected = false
+
kind = patterns::IDENT_KIND[match]
- if kind == :ident
- if match[/\A[A-Z]/] && !(match[/[!?]$/] || match?(/\(/))
- kind = :constant
+ if kind == :ident && value_expected != :colon_expected && scan(/:(?!:)/)
+ value_expected = true
+ encoder.text_token match, :key
+ encoder.text_token ':', :operator
+ else
+ value_expected = false
+ if kind == :ident
+ if match[/\A[A-Z]/] && !(match[/[!?]$/] || match?(/\(/))
+ kind = :constant
+ end
+ elsif kind == :keyword
+ state = patterns::KEYWORD_NEW_STATE[match]
+ if patterns::KEYWORDS_EXPECTING_VALUE[match]
+ value_expected = match == 'when' ? :colon_expected : true
+ end
end
- elsif kind == :keyword
- state = patterns::KEYWORD_NEW_STATE[match]
- value_expected = true if patterns::KEYWORDS_EXPECTING_VALUE[match]
+ value_expected = true if !value_expected && check(/#{patterns::VALUE_FOLLOWS}/o)
+ encoder.text_token match, kind
end
- value_expected = true if !value_expected && check(/#{patterns::VALUE_FOLLOWS}/o)
- encoder.text_token match, kind
elsif method_call_expected &&
match = scan(unicode ? /#{patterns::METHOD_AFTER_DOT}/uo :
@@ -119,9 +128,9 @@ module Scanners
value_expected = check(/#{patterns::VALUE_FOLLOWS}/o)
# OPERATORS #
- elsif !method_call_expected && match = scan(/ (\.(?!\.)|::) | (?: \.\.\.? | ==?=? | [,\(\[\{] )() | [\)\]\}] /x)
+ elsif !method_call_expected && match = scan(/ (\.(?!\.)|::) | ( \.\.\.? | ==?=? | [,\(\[\{] ) | [\)\]\}] /x)
method_call_expected = self[1]
- value_expected = !method_call_expected && self[2]
+ value_expected = !method_call_expected && !!self[2]
if inline_block_stack
case match
when '{'
@@ -213,7 +222,7 @@ module Scanners
encoder.text_token match, :integer
elsif match = scan(/ %=? | <(?:<|=>?)? | \? /x)
- value_expected = true
+ value_expected = match == '?' ? :colon_expected : true
encoder.text_token match, :operator
elsif match = scan(/`/)
diff --git a/lib/coderay/scanners/ruby/patterns.rb b/lib/coderay/scanners/ruby/patterns.rb
index a52198e..ed071d2 100644
--- a/lib/coderay/scanners/ruby/patterns.rb
+++ b/lib/coderay/scanners/ruby/patterns.rb
@@ -1,9 +1,9 @@
# encoding: utf-8
module CodeRay
module Scanners
-
+
module Ruby::Patterns # :nodoc: all
-
+
KEYWORDS = %w[
and def end in or unless begin
defined? ensure module redo super until
@@ -12,7 +12,7 @@ module Scanners
while alias class elsif if not return
undef yield
]
-
+
# See http://murfy.de/ruby-constants.
PREDEFINED_CONSTANTS = %w[
nil true false self
@@ -24,19 +24,19 @@ module Scanners
RUBY_PLATFORM RUBY_RELEASE_DATE RUBY_REVISION RUBY_VERSION
__FILE__ __LINE__ __ENCODING__
]
-
+
IDENT_KIND = WordList.new(:ident).
add(KEYWORDS, :keyword).
add(PREDEFINED_CONSTANTS, :predefined_constant)
-
+
KEYWORD_NEW_STATE = WordList.new(:initial).
add(%w[ def ], :def_expected).
add(%w[ undef ], :undef_expected).
add(%w[ alias ], :alias_expected).
add(%w[ class module ], :module_expected)
-
- IDENT = 'ä'[/[[:alpha:]]/] == 'ä' ? /[[:alpha:]_][[:alnum:]_]*/ : /[^\W\d]\w*/
-
+
+ IDENT = 'ä'[/[[:alpha:]]/] == 'ä' ? Regexp.new('[[:alpha:]_[^\0-\177]][[:alnum:]_[^\0-\177]]*') : /[^\W\d]\w*/
+
METHOD_NAME = / #{IDENT} [?!]? /ox
METHOD_NAME_OPERATOR = /
\*\*? # multiplication and power
@@ -57,25 +57,25 @@ module Scanners
GLOBAL_VARIABLE = / \$ (?: #{IDENT} | [1-9]\d* | 0\w* | [~&+`'=\/,;_.<>!@$?*":\\] | -[a-zA-Z_0-9] ) /ox
PREFIX_VARIABLE = / #{GLOBAL_VARIABLE} | #{OBJECT_VARIABLE} /ox
VARIABLE = / @?@? #{IDENT} | #{GLOBAL_VARIABLE} /ox
-
+
QUOTE_TO_TYPE = {
'`' => :shell,
'/'=> :regexp,
}
QUOTE_TO_TYPE.default = :string
-
+
REGEXP_MODIFIERS = /[mousenix]*/
-
+
DECIMAL = /\d+(?:_\d+)*/
OCTAL = /0_?[0-7]+(?:_[0-7]+)*/
HEXADECIMAL = /0x[0-9A-Fa-f]+(?:_[0-9A-Fa-f]+)*/
BINARY = /0b[01]+(?:_[01]+)*/
-
+
EXPONENT = / [eE] [+-]? #{DECIMAL} /ox
FLOAT_SUFFIX = / #{EXPONENT} | \. #{DECIMAL} #{EXPONENT}? /ox
FLOAT_OR_INT = / #{DECIMAL} (?: #{FLOAT_SUFFIX} () )? /ox
NUMERIC = / (?: (?=0) (?: #{OCTAL} | #{HEXADECIMAL} | #{BINARY} ) | #{FLOAT_OR_INT} ) /ox
-
+
SYMBOL = /
:
(?:
@@ -85,7 +85,7 @@ module Scanners
)
/ox
METHOD_NAME_OR_SYMBOL = / #{METHOD_NAME_EX} | #{SYMBOL} /ox
-
+
SIMPLE_ESCAPE = /
[abefnrstv]
| [0-7]{1,3}
@@ -110,7 +110,7 @@ module Scanners
| \\ #{ESCAPE}
)
/mox
-
+
# NOTE: This is not completely correct, but
# nobody needs heredoc delimiters ending with \n.
HEREDOC_OPEN = /
@@ -122,13 +122,13 @@ module Scanners
( [^\n]*? ) \3 # $4 = delim
)
/mx
-
+
RUBYDOC = /
=begin (?!\S)
.*?
(?: \Z | ^=end (?!\S) [^\n]* )
/mx
-
+
DATA = /
__END__$
.*?
@@ -136,7 +136,7 @@ module Scanners
/mx
RUBYDOC_OR_DATA = / #{RUBYDOC} | #{DATA} /xo
-
+
# Checks for a valid value to follow. This enables
# value_expected in method calls without parentheses.
VALUE_FOLLOWS = /
diff --git a/lib/coderay/version.rb b/lib/coderay/version.rb
index 620e703..bfb5f24 100644
--- a/lib/coderay/version.rb
+++ b/lib/coderay/version.rb
@@ -1,3 +1,3 @@
module CodeRay
- VERSION = '1.0.7'
+ VERSION = '1.0.9'
end
diff --git a/rake_tasks/test.rake b/rake_tasks/test.rake
index f070ccc..a60699d 100644
--- a/rake_tasks/test.rake
+++ b/rake_tasks/test.rake
@@ -1,5 +1,4 @@
namespace :test do
-
desc 'run all sample tests'
task :samples do
ruby './sample/suite.rb'
@@ -17,22 +16,33 @@ namespace :test do
end
scanner_suite = 'test/scanners/suite.rb'
- task scanner_suite do
- unless File.exist? scanner_suite
- puts 'Scanner tests not found; downloading from Subversion...'
- sh 'svn co http://svn.rubychan.de/coderay-scanner-tests/trunk/ test/scanners/'
- puts 'Finished.'
- end
- end
-
desc 'run all scanner tests'
task :scanners => :update_scanner_suite do
ruby scanner_suite
end
- desc 'update scanner test suite from SVN'
- task :update_scanner_suite => scanner_suite do
- sh "svn up #{File.dirname(scanner_suite)}"
+ desc 'update scanner test suite from GitHub'
+ task :update_scanner_suite do
+ if File.exist? scanner_suite
+ Dir.chdir File.dirname(scanner_suite) do
+ if File.directory? '.git'
+ puts 'Updating scanner test suite...'
+ sh 'git pull'
+ elsif File.directory? '.svn'
+ raise <<-ERROR
+Found the deprecated Subversion scanner test suite in ./#{File.dirname(scanner_suite)}.
+Please rename or remove it and run again to use the GitHub repository:
+
+ mv test/scanners test/scanners-old
+ ERROR
+ else
+ raise 'No scanner test suite found.'
+ end
+ end
+ else
+ puts 'Downloading scanner test suite...'
+ sh 'git clone https://github.com/rubychan/coderay-scanner-tests.git test/scanners/'
+ end
end
namespace :scanner do
@@ -72,7 +82,6 @@ namespace :test do
puts "Skipping."
end
end
-
end
task :test => %w(test:functional test:units test:exe)
diff --git a/test/executable/suite.rb b/test/executable/suite.rb
index f3495d6..d386f4b 100644
--- a/test/executable/suite.rb
+++ b/test/executable/suite.rb
@@ -14,12 +14,13 @@ class TestCodeRayExecutable < Test::Unit::TestCase
ROOT_DIR = Pathname.new(File.dirname(__FILE__)) + '..' + '..'
EXECUTABLE = ROOT_DIR + 'bin' + 'coderay'
+ RUBY_COMMAND = RUBY_VERSION < '2.0.0' ? 'ruby -w' : 'ruby' # Ruby 2 currently throws warnings for bundler
EXE_COMMAND =
if RUBY_PLATFORM === 'java' && `ruby --ng -e '' 2> /dev/null` && $?.success?
# use Nailgun
- 'ruby --ng -wI%s %s'
+ "#{RUBY_COMMAND}--ng -I%s %s"
else
- 'ruby -wI%s %s'
+ "#{RUBY_COMMAND} -I%s %s"
end % [ROOT_DIR + 'lib', EXECUTABLE]
def coderay args, options = {}
diff --git a/test/functional/examples.rb b/test/functional/examples.rb
index ff64af3..15f9ca3 100755
--- a/test/functional/examples.rb
+++ b/test/functional/examples.rb
@@ -22,7 +22,7 @@ end
CODE
assert_equal <<-DIV, div
<table class="CodeRay"><tr>
- <td class="line-numbers" title="double click to toggle" ondblclick="with (this.firstChild.style) { display = (display == '') ? 'none' : '' }"><pre><a href="#n1" name="n1">1</a>
+ <td class="line-numbers"><pre><a href="#n1" name="n1">1</a>
<a href="#n2" name="n2">2</a>
<a href="#n3" name="n3">3</a>
</pre></td>
@@ -38,7 +38,7 @@ end
<body>
<table class="CodeRay"><tr>
- <td class="line-numbers" title="double click to toggle" ondblclick="with (this.firstChild.style) { display = (display == '') ? 'none' : '' }"><pre>
+ <td class="line-numbers"><pre><a href="#n1" name="n1">1</a>
</pre></td>
<td class="code"><pre>puts <span class="string"><span class="delimiter">&quot;</span><span class="content">Hello, world!</span><span class="delimiter">&quot;</span></span></pre></td>
</tr></table>