| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
| |
Although this is of course invalid as Ruby code, allow to just parse
and tokenize.
|
| |
|
|
|
|
|
|
|
|
|
| |
The old code of IRB still uses this method. The warning is noisy on
rails console.
In principle, Ruby 3.1 deprecates nothing, so let's avoid the
deprecation for the while.
I think It is not so hard to continue to maintain it as it is a trivial
shim.
https://github.com/ruby/ruby/pull/5093
|
| |
|
|
|
|
| |
Alias `#inspect` as `#to_s` also in the new `Ripper::Lexer::Elem`
class, so that `puts Ripper::Lexer.new(code).scan` shows the
attributes.
|
| |
|
|
|
|
|
| |
Discussed in https://github.com/ruby/ruby/pull/5093#issuecomment-964426481.
> it would be enough to mimic only [] for almost all cases
This adds back the `Lexer::Elem#[]` and `Lexer::State#[]` and adds deprecation warnings for them.
|
| |
|
|
| |
The last element in the `@buf` may be either an array or an `Elem`. In the case it is an `Elem` we iterate over every element, when we do not need to. This check guards that case by ensuring that we only iterate over an array of elements.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
## Concept
I am proposing we replace the Struct implementation of data structures inside of ripper with real classes.
This will improve performance and the implementation is not meaningfully more complicated.
## Example
Struct versus class comparison:
```ruby
Elem = Struct.new(:pos, :event, :tok, :state, :message) do
def initialize(pos, event, tok, state, message = nil)
super(pos, event, tok, State.new(state), message)
end
# ...
def to_a
a = super
a.pop unless a.empty?
a
end
end
class ElemClass
attr_accessor :pos, :event, :tok, :state, :message
def initialize(pos, event, tok, state, message = nil)
@pos = pos
@event = event
@tok = tok
@state = State.new(state)
@message = message
end
def to_a
if @message
[@pos, @event, @tok, @state, @message]
else
[@pos, @event, @tok, @state]
end
end
end
# stub state class creation for now
class State; def initialize(val); end; end
```
## MicroBenchmark creation
```ruby
require 'benchmark/ips'
require 'ripper'
pos = [1, 2]
event = :on_nl
tok = "\n".freeze
state = Ripper::EXPR_BEG
Benchmark.ips do |x|
x.report("struct") { Elem.new(pos, event, tok, state) }
x.report("class ") { ElemClass.new(pos, event, tok, state) }
x.compare!
end; nil
```
Gives ~1.2x faster creation:
```
Warming up --------------------------------------
struct 263.983k i/100ms
class 303.367k i/100ms
Calculating -------------------------------------
struct 2.638M (± 5.9%) i/s - 13.199M in 5.023460s
class 3.171M (± 4.6%) i/s - 16.078M in 5.082369s
Comparison:
class : 3170690.2 i/s
struct: 2638493.5 i/s - 1.20x (± 0.00) slower
```
## MicroBenchmark `to_a` (Called by Ripper.lex for every element)
```ruby
require 'benchmark/ips'
require 'ripper'
pos = [1, 2]
event = :on_nl
tok = "\n".freeze
state = Ripper::EXPR_BEG
struct = Elem.new(pos, event, tok, state)
from_class = ElemClass.new(pos, event, tok, state)
Benchmark.ips do |x|
x.report("struct") { struct.to_a }
x.report("class ") { from_class.to_a }
x.compare!
end; nil
```
Gives 1.46x faster `to_a`:
```
Warming up --------------------------------------
struct 612.094k i/100ms
class 893.233k i/100ms
Calculating -------------------------------------
struct 6.121M (± 5.4%) i/s - 30.605M in 5.015851s
class 8.931M (± 7.9%) i/s - 44.662M in 5.039733s
Comparison:
class : 8930619.0 i/s
struct: 6121358.9 i/s - 1.46x (± 0.00) slower
```
## MicroBenchmark data access
```ruby
require 'benchmark/ips'
require 'ripper'
pos = [1, 2]
event = :on_nl
tok = "\n".freeze
state = Ripper::EXPR_BEG
struct = Elem.new(pos, event, tok, state)
from_class = ElemClass.new(pos, event, tok, state)
Benchmark.ips do |x|
x.report("struct") { struct.pos[1] }
x.report("class ") { from_class.pos[1] }
x.compare!
end; nil
```
Gives ~1.17x faster data access:
```
Warming up --------------------------------------
struct 1.694M i/100ms
class 1.868M i/100ms
Calculating -------------------------------------
struct 16.149M (± 6.8%) i/s - 81.318M in 5.060633s
class 18.886M (± 2.9%) i/s - 95.262M in 5.048359s
Comparison:
class : 18885669.6 i/s
struct: 16149255.8 i/s - 1.17x (± 0.00) slower
```
## Full benchmark integration of this inside of Ripper.lex
Inside of this repo with this commit
```
$ cd ext/ripper
$ make
$ cat test.rb
file = File.join(__dir__, "../../array.rb")
source = File.read(file)
bench = Benchmark.measure do
10_000.times.each do
Ripper.lex(source)
end
end
puts bench
```
Then execute with and without this change 50 times:
```
rm new.txt
rm old.txt
for i in {0..50}
do
`ruby -Ilib -rripper -rbenchmark ./test.rb >> new.txt`
`ruby -rripper -rbenchmark ./test.rb >> old.txt`
done
```
I used derailed benchmarks internals to compare the results:
```
dir = Pathname(".")
branch_info = {}
branch_info["old"] = { desc: "Struct lex", time: Time.now, file: dir.join("old.txt"), name: "old" }
branch_info["new"] = { desc: "Class lex", time: Time.now, file: dir.join("new.txt"), name: "new" }
stats = DerailedBenchmarks::StatsFromDir.new(branch_info)
stats.call.banner
```
Which gave us:
```
❤️ ❤️ ❤️ (Statistically Significant) ❤️ ❤️ ❤️
[new] (3.3139 seconds) "Class lex" ref: "new"
FASTER 🚀🚀🚀 by:
1.1046x [older/newer]
9.4700% [(older - newer) / older * 100]
[old] (3.6606 seconds) "Struct lex" ref: "old"
Iterations per sample:
Samples: 51
Test type: Kolmogorov Smirnov
Confidence level: 99.0 %
Is significant? (max > critical): true
D critical: 0.30049534876137013
D max: 0.9607843137254902
Histograms (time ranges are in seconds):
[new] description: [old] description:
"Class lex" "Struct lex"
┌ ┐ ┌ ┐
[3.0, 3.3) ┤▇ 1 [3.0, 3.3) ┤ 0
[3.3, 3.6) ┤▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 47 [3.3, 3.6) ┤ 0
[3.5, 3.8) ┤▇▇ 2 [3.5, 3.8) ┤▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 46
[3.8, 4.1) ┤▇ 1 [3.8, 4.1) ┤▇▇▇ 4
[4.0, 4.3) ┤ 0 [4.0, 4.3) ┤ 0
[4.3, 4.6) ┤ 0 [4.3, 4.6) ┤▇ 1
└ ┘ └ ┘
# of runs in range # of runs in range
```
To sum this up, the "new" version of this code (using real classes instead of structs) is 10% faster across 50 runs with a statistical significance confidence level of 99%. Histograms are for visual checksum.
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
sorting alias target by event arity, and setup suitable `Elem` for error.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
yet another implements [Feature #17276]
|
| | |
|
| |
|
| |
Implements [Feature #17276]
|
| |
|
|
| |
This is follow up of 1f7cb4bee9.
|
| | |
|
| |
|
|
| |
instead of <<. All the other callsites use `push`.
|
| |
|
|
| |
The target method name is a typo.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
| |
* ext/ripper/lib/ripper/lexer.rb (Ripper::Lexer#scan): parses the
code and returns the result elements including errors.
[EXPERIMENTAL]
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
| |
* ext/ripper/lib/ripper/lexer.rb (Ripper::Lexer): add ignored_sp
event which will be fired from Ripper::Lexer#on_heredoc_dedent
method. [ruby-core:91727] [Bug #15648]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67200 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
| |
|
|
| |
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66189 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Currently `Ripper.slice` raises a FrozenError
```ruby
require 'ripper'
p Ripper.slice('foo', 'ident')
```
```
/path/to/g/lib/ruby/2.6.0/ripper/lexer.rb:193:in `concat': can't modify frozen String (FrozenError)
from /path/to/g/lib/ruby/2.6.0/ripper/lexer.rb:193:in `block in compile'
from /path/to/g/lib/ruby/2.6.0/ripper/lexer.rb:190:in `scan'
from /path/to/g/lib/ruby/2.6.0/ripper/lexer.rb:190:in `compile'
from /path/to/g/lib/ruby/2.6.0/ripper/lexer.rb:169:in `initialize'
from /path/to/g/lib/ruby/2.6.0/ripper/lexer.rb:151:in `new'
from /path/to/g/lib/ruby/2.6.0/ripper/lexer.rb:151:in `token_match'
from /path/to/g/lib/ruby/2.6.0/ripper/lexer.rb:144:in `slice'
from /tmp/tmp.kb4cnhvum2/test.rb:2:in `<main>'
```
This patch will fix the problem.
[Fix GH-1837]
From: Masataka Pocke Kuwabara <kuwabara@pocke.me>
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62743 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
| |
|
|
|
|
|
|
| |
* ext/ripper/lib/ripper/lexer.rb (Ripper::Lexer): added allbits?,
anybits? and nobits? methods, as well as Integer. a patch by
aycabta. [Feature #14170]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61205 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
| |
|
|
|
|
|
|
|
|
| |
* ext/ripper/lib/ripper/lexer.rb (Ripper::Lexer::State): wrapper
of lex_state values.
* parse.y (rb_parser_lex_state_name): return shared strings. lex
state combinations are very restricted.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60665 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
| |
|
|
|
|
|
|
| |
* ext/ripper/lib/ripper/lexer.rb (on_heredoc_dedent): dedent only
strings at the beginning, not strings in middle.
[ruby-core:83343] [Bug #14027]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60212 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
| |
|
|
| |
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60069 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
| |
|
|
|
|
|
| |
* ext/ripper/lib/ripper/lexer.rb (List#inspect): splat self data
to local variables same as members of Elem.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59972 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
| |
|
|
|
|
|
|
|
|
| |
* parse.y (ripper_lex_state_name): represent lex_state as OR-ed
form.
* ext/ripper/lib/ripper/lexer.rb (Ripper::Lexer::Elem#to_a):
lex_state for inspection.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59949 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
| |
|
|
|
|
|
|
|
|
|
| |
* parse.y (ripper_state): add states of scanner to tokens from
Ripper.lex and Ripper::Filter#on_*. based on the patch by
aycabta (Code Ahss) at [ruby-core:81789]. [Feature #13686]
* ext/ripper/tools/preproc.rb (prelude, usercode): generate EXPR_*
constants from enums.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59896 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
| |
|
|
|
|
|
|
| |
* ext/ripper/lib/ripper.rb: [DOC] fix format of the License
description and the author. patched by aycabta (Code Ahss) at
[ruby-core:82376]. [Bug #13813]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59597 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
| |
|
|
|
|
|
|
| |
* ext/ripper/lib/ripper/lexer.rb (on_heredoc_dedent): replace an
empty string content because of dedentation with :on_ignored_sp.
an empty token makes the sorted order unstable.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59269 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
| |
|
|
|
|
|
| |
* parse.y (f_margs): implemented parser events for massign
formal arguments. [ruby-core:81848] [Bug #13701]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59246 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
| |
|
|
|
|
|
|
|
| |
* ext/ripper/lib/ripper/lexer.rb (on_heredoc_dedent): insert
stripped leading spaces as `on_ignored_sp` elements, so that the
original source can be reconsructed.
[ruby-core:80977] [Bug #13536]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58584 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
| |
|
|
|
|
|
|
| |
* ext/ripper/lib/ripper/lexer.rb (on_heredoc_dedent): fix for
nested indedented here documents, where `Elem`s are nested too.
[ruby-core:80977] [Bug #13536]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58545 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
| |
|
|
| |
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@57538 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
| |
|
|
|
|
|
|
| |
* ext/ripper/lib/ripper/lexer.rb (on_heredoc_dedent): Fix
Ripper.lex error in dedenting squiggly heredoc. heredoc tree is
also an array of Elem in the outer tree. [Fix GH-1234]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53722 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
| |
|
|
|
|
|
|
| |
g -L frozen_string_literal ext/**/*.rb|xargs ruby -Ka -e'ARGV.each{|fn|puts
fn;open(fn,"r+"){|f|s=f.read.sub(/\A(#!.*\n)?(#.*coding.*\n)?/,"\\&#
frozen_string_literal: false\n");f.rewind;f.write s}}'
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53143 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
| |
|
|
|
|
| |
* parse.y: add heredoc <<~ syntax. [Feature #9098]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@52916 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
| |
|
|
|
|
|
| |
* ext/ripper/lib/ripper/lexer.rb: alias same methods instead of
eval for each events.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@52776 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
| |
|
|
|
|
|
|
| |
* ext/ripper/lib/ripper/sexp.rb (SexpBuilderPP): share scanner
event handlers and most of parser event handlers with
SexpBuilder.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@52775 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
