summaryrefslogtreecommitdiff
path: root/ast.c
Commit message (Collapse)AuthorAgeFilesLines
* Add utility macros `DECIMAL_SIZE_OF` and `DECIMAL_SIZE_OF_BYTES`Nobuyoshi Nakada2023-02-141-1/+1
|
* Check if the argument is Thread::Backtrace::Location objectyui-knk2023-01-061-0/+5
| | | | [Bug #19262]
* Enhance keep_tokens option for RubyVM::AbstractSyntaxTree parsing methodsyui-knk2022-11-211-13/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Implementation for Language Server Protocol (LSP) sometimes needs token information. For example both `m(1)` and `m(1, )` has same AST structure other than node locations then it's impossible to check the existence of `,` from AST. However in later case, it might be better to suggest variables list for the second argument. Token information is important for such case. This commit adds these methods. * Add `keep_tokens` option for `RubyVM::AbstractSyntaxTree.parse`, `.parse_file` and `.of` * Add `RubyVM::AbstractSyntaxTree::Node#tokens` which returns tokens for the node including tokens for descendants nodes. * Add `RubyVM::AbstractSyntaxTree::Node#all_tokens` which returns all tokens for the input script regardless the receiver node. [Feature #19070] Impacts on memory usage and performance are below: Memory usage: ``` $ cat test.rb root = RubyVM::AbstractSyntaxTree.parse_file(File.expand_path('../test/ruby/test_keyword.rb', __FILE__), keep_tokens: true) $ /usr/bin/time -f %Mkb /usr/local/bin/ruby -v ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux] 11408kb # keep_tokens :false $ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb 17508kb # keep_tokens :true $ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb 30960kb ``` Performance: ``` $ cat ../ast_keep_tokens.yml prelude: | src = <<~SRC module M class C def m1(a, b) 1 + a + b end end end SRC benchmark: without_keep_tokens: | RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: false) with_keep_tokens: | RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: true) $ make benchmark COMPARE_RUBY="./ruby" ARGS=../ast_keep_tokens.yml /home/kaneko.y/.rbenv/shims/ruby --disable=gems -rrubygems -I../benchmark/lib ../benchmark/benchmark-driver/exe/benchmark-driver \ --executables="compare-ruby::./ruby -I.ext/common --disable-gem" \ --executables="built-ruby::./miniruby -I../lib -I. -I.ext/common ../tool/runruby.rb --extout=.ext -- --disable-gems --disable-gem" \ --output=markdown --output-compare -v ../ast_keep_tokens.yml compare-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux] built-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux] warming up.. | |compare-ruby|built-ruby| |:--------------------|-----------:|---------:| |without_keep_tokens | 21.659k| 21.303k| | | 1.02x| -| |with_keep_tokens | 6.220k| 5.691k| | | 1.09x| -| ```
* Add `node_id_for_backtrace_location` functioneileencodes2022-10-311-0/+12
| | | | | | | | | | | | | | | We want to use error highlight with eval'd code, specifically ERB templates. We're able to recover the generated code for eval'd templates and can get a parse tree for the ERB generated code, but we don't have a way to get the node id from the backtrace location. So we can't pass the right node into error highlight. This patch gives us an API to get the node id from the backtrace location so we can find the node in the AST. Error Highlight PR: https://github.com/ruby/error_highlight/pull/26 Co-authored-by: Aaron Patterson <tenderlove@ruby-lang.org>
* Move `error` from top_stmts and top_stmt to stmtyui-knk2022-10-081-0/+2
| | | | | | | | | | | | | | | | | | | By this change, syntax error is recovered smaller units. In the case below, "DEFN :bar" is same level with "CLASS :Foo" now. ``` module Z class Foo foo. end def bar end end ``` [Feature #19013]
* Add error_tolerant option to RubyVM::ASTyui-knk2022-10-081-13/+16
| | | | | | | If this option is enabled, SyntaxError is not raised and Node is returned even if passed script is broken. [Feature #19013]
* Add ISEQ_BODY macroPeter Zhu2022-03-241-2/+2
| | | | | | Use ISEQ_BODY macro to get the rb_iseq_constant_body of the ISeq. Using this macro will make it easier for us to change the allocation strategy of rb_iseq_constant_body when using Variable Width Allocation.
* Make RubyVM::AST.of work with code written in `-e` command-line optionYusuke Endoh2021-12-261-4/+7
| | | | [Bug #18434]
* Make AST.of possible even under eval when keep_script_lines is enabledYusuke Endoh2021-12-191-2/+2
| | | | | | | | | | | | | | | Now the following code works without an exception. ``` RubyVM.keep_script_lines = true eval(<<END) def foo end END p RubyVM::AbstractSyntaxTree.of(method(:foo)) ```
* Make RubyVM::AbstractSyntaxTree.of raise for backtrace location in evalYusuke Endoh2021-12-191-11/+15
| | | | | | | | | This check is needed to fix a bug of error_highlight when NameError occurred in eval'ed code. https://github.com/ruby/error_highlight/pull/16 The same check for proc/method has been already introduced since 64ac984129a7a4645efe5ac57c168ef880b479b2.
* Remove `NODE_DASGN_CURR` [Feature #18406]Nobuyoshi Nakada2021-12-131-1/+0
| | | | | | | This `NODE` type was used in pre-YARV implementation, to improve the performance of assignment to dynamic local variable defined at the innermost scope. It has no longer any actual difference with `NODE_DASGN`, except for the node dump.
* Add `nd_type_p` macroS.H2021-12-041-3/+3
|
* Refactor hacky ID tables to struct rb_ast_id_table_tYusuke Endoh2021-11-211-3/+3
| | | | | | | | | The implementation of a local variable tables was represented as `ID*`, but it was very hacky: the first element is not an ID but the size of the table, and, the last element is (sometimes) a link to the next local table only when the id tables are a linked list. This change converts the hacky implementation to a normal struct.
* ast.c: Use kept script_lines data instead of re-opening the source file (#5019)Yusuke Endoh2021-10-261-4/+5
| | | ast.c: Use kept script_lines data instead of re-open the source file
* ast.c: AST.of against C method should return nil (as Ruby 2.6--3.0)Yusuke Endoh2021-09-181-1/+1
|
* ast.c: AST.of checks if a given method object is defined in CYusuke Endoh2021-09-181-0/+3
| | | | [Bug #18178]
* Replace RBOOL macroS-H-GAMELINKS2021-09-051-2/+2
|
* ast.c: Rename "save_script_lines" to "keep_script_lines"Yusuke Endoh2021-08-201-16/+16
| | | | | | | ... as per ko1's preference. He is preparing to extend this feature to ISeq for his new debugger. He prefers "keep" to "save" for this wording. This API is internal and not included in any released version, so I change it in advance.
* Make RubyVM::AbstractSyntaxTree.of raise for method/proc created in evalJeremy Evans2021-07-291-0/+3
| | | | | | | | | | | This changes Thread::Location::Backtrace#absolute_path to return nil for methods/procs defined in eval. If the realpath of an iseq is nil, that indicates it was defined in eval, in which case you cannot use RubyVM::AbstractSyntaxTree.of. Fixes [Bug #16983] Co-authored-by: Koichi Sasada <ko1@atdot.net>
* Experimentally expose RubyVM::AST::Node#node_idYusuke Endoh2021-06-211-6/+1
| | | | | | Now ISeq#to_a includes the node_id list for each bytecode instruction. I want a way to retrieve the AST::Node instance corresponding to an instruction for a research purpose including TypeProf-based LSP server.
* Enable USE_ISEQ_NODE_ID by defaultYusuke Endoh2021-06-181-2/+2
| | | | | | | | ... which is formally called EXPERIMENTAL_ISEQ_NODE_ID. See also ff69ef27b06eed1ba750e7d9cab8322f351ed245. https://bugs.ruby-lang.org/issues/17930
* Make it possible to get AST::Node from Thread::Backtrace::LocationYusuke Endoh2021-06-181-12/+15
| | | | | | | | | RubyVM::AST.of(Thread::Backtrace::Location) returns a node that corresponds to the location. Typically, the node is a method call, but not always. This change also includes iseq's dump/load support of node_ids for each instructions.
* node.h: Reduce struct size to fit with Ruby object size (five VALUEs)Yusuke Endoh2021-06-181-1/+1
| | | | | | | | by merging `rb_ast_body_t#line_count` and `#script_lines`. Fortunately `line_count == RARRAY_LEN(script_lines)` was always satisfied. When script_lines is saved, it has an array of lines, and when not saved, it has a Fixnum that represents the old line_count.
* ast.rb: RubyVM::AST.parse and .of accepts `save_script_lines: true`Yusuke Endoh2021-06-181-16/+32
| | | | | | | This option makes the parser keep the original source as an array of the original code lines. This feature exploits the mechanism of `SCRIPT_LINES__` but records only the specified code that is passed to RubyVM::AST.of or .parse, instead of recording all parsed program texts.
* compile.c: Pass node instead of nd_line(node) to ADD_INSN* functionsYusuke Endoh2021-05-071-0/+14
| | | | | | | | | | | | | | | ... then, new_insn_core extracts nd_line(node). Also, if a macro "EXPERIMENTAL_ISEQ_NODE_ID" is defined, this changeset keeps nd_node_id(node) for each instruction. This is intended for TypeProf to identify what AST::Node corresponds to each instruction. This patch is originally authored by @yui-knk for showing which column a NoMethodError occurred. https://github.com/ruby/ruby/compare/master...yui-knk:feature/node_id Co-Authored-By: Yuichiro Kaneko <yui-knk@ruby-lang.org>
* Remove unused rb_ast_parse_array declarationS.H2021-03-201-1/+0
|
* Unfreeze string-literal-only interpolated string-literalNobuyoshi Nakada2020-09-301-3/+9
| | | | [Feature #17104]
* Hoisted out functions for no name rest argument symbolNobuyoshi Nakada2020-07-081-8/+19
|
* Constified NODE pointer in ASTNodeDataNobuyoshi Nakada2020-07-081-7/+7
|
* Added `NODE_SPECIAL_EXCESSIVE_COMMA` info to `ARGS` of ↵manga_osyo2020-07-081-1/+3
| | | | `RubyVM::AbstractSyntaxTree`.
* Add operator info to `OP_ASGN2` of `RubyVM::AbstractSyntaxTree`.manga_osyo2020-07-061-1/+2
|
* node_children: do not goto into a branch卜部昌平2020-06-291-29/+2
| | | | | | Was this an autogenerated function? This tendency of avoiding empty branches are no longer preserved (see for instance NODE_IVAR). Let's just delete those unnecessary jumps into branches.
* Introduce find pattern [Feature #16828]Kazuki Tsujimoto2020-06-141-0/+13
|
* Separate builtin initialization callsNobuyoshi Nakada2019-12-291-2/+0
|
* decouple internal.h headers卜部昌平2019-12-261-3/+7
| | | | | | | | | | | | | | | | | | Saves comitters' daily life by avoid #include-ing everything from internal.h to make each file do so instead. This would significantly speed up incremental builds. We take the following inclusion order in this changeset: 1. "ruby/config.h", where _GNU_SOURCE is defined (must be the very first thing among everything). 2. RUBY_EXTCONF_H if any. 3. Standard C headers, sorted alphabetically. 4. Other system headers, maybe guarded by #ifdef 5. Everything else, sorted alphabetically. Exceptions are those win32-related headers, which tend not be self- containing (headers have inclusion order dependencies).
* Revert "Method reference operator"Nobuyoshi Nakada2019-11-121-3/+0
| | | | | This reverts commit 67c574736912003c377218153f9d3b9c0c96a17b. [Feature #16275]
* Renamed `load_*.inc` as `*.rbinc` to utilize a suffix ruleNobuyoshi Nakada2019-11-081-1/+1
|
* use builtin for RubyVM::AbstractSyntaxTree.Koichi Sasada2019-11-081-127/+15
| | | | | Define RubyVM::AbstractSyntaxTree in ast.rb with __builtin functions.
* Rename NODE_ARRAY to NODE_LIST to reflect its actual use casesYusuke Endoh2019-09-071-3/+3
| | | | | | | | | | and NODE_ZARRAY to NODE_ZLIST. NODE_ARRAY is used not only by an Array literal, but also the contents of Hash literals, method call arguments, dynamic string literals, etc. In addition, the structure of NODE_ARRAY is a linked list, not an array. This is very confusing, so I believe `NODE_LIST` is a better name.
* Make pattern matching support **nil syntaxKazuki Tsujimoto2019-09-011-1/+4
|
* Make RubyVM::AbstractSyntaxTree handle **nil syntaxJeremy Evans2019-08-301-2/+2
| | | | | Use false instead of nil for the keyword and keyword rest values in that case.
* Make it as clear as possible that RubyVM is MRI-specific and only exists on ↵Benoit Daloze2019-08-191-0/+5
| | | | | | | | | | | | | MRI (#2113) [ci skip] * Make it clear as possible that RubyVM is MRI-specific and only exists on MRI * See [Bug #15743]. * Use "CRuby VM" instead of "Ruby VM" for clarity. * Use YARV rather than "CRuby VM" for documenting RubyVM::InstructionSequence * Avoid introducing a new "CRuby VM" term in documentation
* Support memsize of ASTNobuyoshi Nakada2019-07-231-1/+8
|
* ast.c: update inspect results in the documentsNobuyoshi Nakada2019-05-221-4/+4
|
* Distinguish pre-condition and post-condition loopsNobuyoshi Nakada2019-05-181-1/+2
|
* Avoid usage of the dummy empty BEGIN nodektsj2019-04-201-1/+3
| | | | | | Use NODE_SPECIAL_NO_NAME_REST instead. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67629 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Add symbol to the result of `RubyVM::AbstractSyntaxTree#children`.yui-knk2019-04-201-3/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add symbol to the result to make pattern match easily. For example: (1) NODE_MASGN * NODE_SPECIAL_NO_NAME_REST ``` $ ./miniruby -e 'p RubyVM::AbstractSyntaxTree.parse("a, * = b").children[-1].children' [#<RubyVM::AbstractSyntaxTree::Node:VCALL@1:7-1:8>, #<RubyVM::AbstractSyntaxTree::Node:ARRAY@1:0-1:1>, :NODE_SPECIAL_NO_NAME_REST] ``` (2) NODE_POSTARG * NODE_SPECIAL_NO_NAME_REST ``` $ ./miniruby -e 'p RubyVM::AbstractSyntaxTree.parse("a, *, _ = b").children[-1].children[-1].children' [:NODE_SPECIAL_NO_NAME_REST, #<RubyVM::AbstractSyntaxTree::Node:ARRAY@1:6-1:7>] ``` (3) NODE_LASGN * NODE_SPECIAL_REQUIRED_KEYWORD ``` $ ./miniruby -e 'p RubyVM::AbstractSyntaxTree.parse("def a(k:) end").children[-1].children[-1].children[1].children[7].children[0].children' [:k, :NODE_SPECIAL_REQUIRED_KEYWORD] ``` git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67627 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Introduce pattern matching [EXPERIMENTAL]ktsj2019-04-171-0/+20
| | | | | | [ruby-core:87945] [Feature #14912] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67586 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* [DOC] fix markups [ci skip]nobu2019-03-221-5/+5
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67337 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ast.c: fix missing head part in dynamic literalnobu2019-01-141-1/+3
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66819 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
View Lorry Controller Status