| Commit message (Collapse) | Author | Age | Files | Lines |
... | |
|
|
|
|
|
| |
Renamed the def function for regions to reflect that it is for regions. Took
the isInstance out of both functions. All lexical regions are instances (in the
ragel sense) and all regular language definitions are not.
|
|
|
|
|
|
|
| |
The scanner code was derivied from ragel, where the same map of names to graphs
is used for regular language defintions and scanners. Some of the regular
lanuage defintiions are instantiations, meaning they create states. Starting to
retire this by creating a separate map for regular language defs (rlMap).
|
| |
|
|
|
|
|
|
|
| |
Added a shell script test harness that executes all the lm tests in the current
directory. Doesn't require any makefile generation, TESTS file, etc. The shell
script be easily programmed to add extra steps during testing, such as pre/post
execution, linking, etc. Just the better way to go.
|
|
|
|
|
|
| |
codegen.cc for writing the colm program
compiler.cc for the main compiler logic
synthesis.cc for the bytecode program generation
|
|
|
|
|
| |
The exports.cc file is for writing the C++ interface. Write for generic code
writing. Currently has only the main file.
|
|
|
|
| |
Merged parsedata.cc and analysis.cc and renamed it colm.cc
|
| |
|
|
|
|
|
| |
Allocate scanners for included files on the heap. Consistent with the main
line.
|
|
|
|
|
| |
Allocating the primary processing objects in the mainline and calling them,
previously had scanners allocating parser and parsers allocating parse data.
|
| |
|
| |
|
|
|
|
|
|
| |
PDA construction and execution is complicated too much by the automatic
insertion of collect ignore tokens when the collect-ignore property is set.
Instead put the collect ignores into grammars, patterns and replacements.
|
| |
|
|
|
|
|
| |
Suppress code generation of types that are duplicates into the ignore/token/ci
regions. Removed some print statements used for debugging.
|
|
|
|
|
|
|
| |
Now possible to parse patterns that have collect-ignores. Sometimes you need
them present in the input stream when you pass over the production. Other times
you don't when you pass over the nonterminal. Built skipping of them into the
backtracker.
|
|
|
|
|
|
|
|
|
| |
Can say that a production should collect ignores from a region. There is a
collect ignore region created, but the states from the ignore-version of the
region is used. When the scanner fails to produce a token from the
collect-ignore region, the collect-ignore token is generated and accepted by
the fsm. Need to take it out of the data tree on reductions and put it into an
ignore list. Reverse this during unparsing.
|
| |
|
|
|
|
|
|
| |
Added the keyword 'ni', which can go ahead of or before a token pattern
(literal or usual), which means no-ignore. Sets the noPreIgnore and
noPostIgnore bits in the token, which affect the ignore scanning and attaching.
|
| |
|
|
|
|
|
|
| |
The right ignore attaching failed to take into account that the accume ignore
list is in reverse order. Need to take the tail that is for right ignore, not
the head.
|
| |
|
|
|
|
|
| |
Exercises the attaching of tokens to the side that the ignore definitions came
from.
|
|
|
|
|
|
|
|
|
|
|
|
| |
Every region now also has a duplicate scanning region that is only for tokens.
The duplicate ignores and tokens generate the original tokens through a
TokenDef ignore mechanism.
Can turn off post ignore parsing and pre-igore parsing on a token-by-token
basis. Probably want to move it into the productions and specify it there.
Currently don't have a specification mechanism.
If an ignore is a post-token ignore it is not right-attached.
|
|
|
|
|
| |
The text_notrim() functions retrieve the text of a token without automatically
trimming off whitespace.
|
|
|
|
|
|
|
|
| |
The print implemenation now takes a trim flag. The colm print function now sets
this flag by default. This is a change to the colm language back to 0.5
semantics. The $ conversion uses this flag too (also 0.5 semantics), in the
previous commit it issued a tree trim operation. The % operation gives a string
conversion without triming.
|
| |
|
|
|
|
|
|
| |
No longer need to trim trees before converting to strings because the string
conversion does it automatically. To convert to a string without trimming the %
operator is used. This may change.
|
|
|
|
|
| |
The repeat2 test is the current doc gen program from ragel. There is a custom
language with a traversal that calls prints (no real transformation).
|
| |
|
|
|
|
|
| |
The $ operation automatically adds a TRIM. The '%' opertion was added, which is
the original $ conversion without the trim.
|
|
|
|
| |
Eliminated final clone of the push ignores, this one was in the trim operation.
|
|
|
|
| |
Clone elimination on calls pushIgnore.
|
|
|
|
|
|
| |
Removed sp from the pushIgnore functions. Need to use it in contexts where it
is not currently available. Actually not needed because we can directly access
refs when trees are moved around during the push.
|
| |
|
|
|
|
| |
The ignore node handling code has been frequently cloned. Cleaning that up.
|
|
|
|
|
| |
Eliminated the IgnoreTree struct because it no longer contains any extensions
of Tree. Just using Tree.
|
|
|
|
|
| |
This field was the only field that extends the basic tree. Can now eliminate
the structure and just use Tree.
|
|
|
|
|
| |
Whitespace is shifting. Most of the updates involve triming whitespace where it
was previously trimmed automatically.
|
|
|
|
|
| |
Added a missing downref in the detach of right ignores. This was lost when
converting to the no-dupe of ignores.
|
|
|
|
|
| |
Explictly suppress leading and trailing ignores using the TRIM operation.
Adjusted output for the no-dupignores too. Whitespace shifts around.
|
|
|
|
|
|
| |
Need to put the visitType on the stack if we are to reference it past a
recursive call. Implement the suppress left when going backwards to reverse the
list. Implement suppress-right on the pass forward for printing.
|
|
|
|
|
| |
Don't need to adjust the refcounts in TREE_TRIM. The trim function will split
the tree, making it safe to write to. Just pop, split, modify, push back.
|
|
|
|
|
|
|
|
|
| |
The trim operation wraps the ignores with nodes that have the suppress-left and
suppress-right flags set in the ignore list struct. These are now added to the
list of ignores to ouput once a terminal is hit, but are not printed since they
have no data. They are there just for their flags. Implemented the
suppress-left and suppress-right operations in a walk of the ignores to output
by altering the list.
|
| |
|
|
|
|
| |
These tests are helpful when testing the no-kf-dupign branch.
|
|
|
|
|
|
|
|
|
|
|
|
| |
Trying out an elimination of the kid flags and the duplicate ignore tokens. The
kid flags waste a lot of space. There is also a bug WRT iterators. Some kids in
the ref chain are actually on the stack only the tree is safe to edit because
the stack variables are not full kids.
The duplicate ignores causes a complicated implementation in the print
function. It may actually be unneeded now that we have follow ignores. We can
also provide some control over where the ignore tokens go, to the right of the
left token or to the left of the right token.
|
| |
|
|
|
|
| |
Final adjustments related to shifting ignore token placements.
|
|
|
|
|
| |
Removed the token matching the empty string. Adjusted the output for shifting
placement of ignore tokens.
|