| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
| |
Fix spelling on various files pertaining to core Perl.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The main exceptions being dist/, ext/, and Configure related
files, which will be updated in a subsequent commit. Files in the cpan/
directory are also omitted as they are not owned by the core.
'#define' has seven characters, so following it with a \t makes it look
like '#define ' when it is not, which then frustrates attempts to find
where a given define is. If you *know* then you do a
git grep -P 'define\s+WHATEVER'
but if don't or you forget, you can get very confused trying to find
where a given define is located. This fixes all such cases so they
actually are 'define WHATEVER' instead.
If this patch is getting in your way with blame analysis then view it
with the -w option to blame.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
This helps with HPUX builds where we need to ensure everything
is aligned the same (on 32 bit boundaries). It also strongly
encourages everything to use the accessor macros and not access
the members directly.
By using a union for the variadic fields we make it more obvious
that some regops use the field in different ways. This patch
also converts all the arg unions into a standardized union with
standardized member names.
|
| |
|
|
|
|
|
| |
It is confusing to have two different members, at different struct
offsets and with different sizes, with the same name. So rename them
so they have different names that include their size so it is obvious
what is going on.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
These two defines are related to each other, and even though
VOLATILE_REF is not explicitly used in regexec.c which would require
it being placed in regcomp.h, it is implicitly, and RE_PESSIMISTIC_PARENS
*is* used in regexec.c. So put them both in regcomp.h and document them
together. This adds copious documentation for what they both are for.
RE_PESSIMISTIC_PARENS is effectively a "build option" (although intended
for debugging regex engine bugs only). VOLATILE_REF is the name of a
flag which is used to mark REF nodes as requiring special backtracking
support in regexec.c
|
| |
|
|
|
|
|
| |
Anonymous unions/structs are a C11 feature (previously a GNU extension)
and not available in C90 or C99.
Fixes #20932.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This eliminates the regnode_2L data structure, and merges it with the older
regnode_2 data structure. At the same time it makes each "arg" property of the
various regnode types that have one be consistently structured as an anonymous
union like this:
union {
U32 arg1u;
I32 arg2i;
struct {
U16 arg1a;
U16 arg1b;
};
};
We then expose four macros for accessing each slot: ARG1u() ARG1i() and
ARG1a() and ARG1b(). Code then explicitly designates which they want. The old
logic used ARG() to access an U32 arg1, and ARG1() to access an I32 arg1,
which was confusing to say the least. The regnode_2L structure had a U32 arg1,
and I32 arg2, and the regnode_2 data strucutre had two I32 args. With the new
set of macros we use the regnode_2 for both, and use the appropriate macros to
show whether we want to signed or unsigned values.
This also renames the regnode_4 to regnode_3. The 3 stands for "three 32-bit
args". However as each slot can also store two U16s, a regnode_3 can hold up
to 6 U16s, or as 3 I32's, or a combination. For instance the CURLY style nodes
use regnode_3 to store 4 values, ARG1i() for min count, ARG2i() for max count
and ARG3a() and ARG3b() for parens before and inside the quantifier.
It also changes the functions reganode() to reg1node() and changes reg2Lanode()
to reg2node(). The 2L thing was just confusing.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This way we can do the required paren restoration only when it is in use. When
we match a REF type node which is potentially a reference to an unclosed paren
we push the match context information, currently for "everything", but in a
future patch we can teach it to be more efficient by adding a new parameter to
the REF regop to track which parens it should save.
This converts the backtracking changes from the previous commit, so that it is
run only when specifically enabled via the define RE_PESSIMISTIC_PARENS which
is by default 0. We don't make the new fields in the struct conditional as the
stack frames are large and our changes don't make any real difference and it
keeps things simpler to not have conditional members, especially since some of
the structures have to line up with each other.
If enabling RE_PESSIMISTIC_PARENS fixes a backtracking bug then it means
something is sensitive to us not necessarily restoring the parens properly on
failure. We make some assumptions that the paren state after a failing state
will be corrected by a future successful state, or that the state of the
parens is irrelevant as we will fail anyway. This can be made not true by
EVAL, backrefs, and potentially some other scenarios. Thus I have left this
inefficient logic in place but guarded by the flag.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
In /((a)(b)|(a))+/ we should not end up with $2 and $4 being set at
the same time. When a branch fails it should reset any capture buffers
that might be touched by its branch.
We change BRANCH and BRANCHJ to store the number of parens before the
branch, and the number of parens after the branch was completed. When
a BRANCH operation fails, we clear the buffers it contains before we
continue on.
It is a bit more complex than it should be because we have BRANCHJ
and BRANCH. (One of these days we should merge them together.)
This is also made somewhat more complex because TRIE nodes are actually
branches, and may need to track capture buffers also, at two levels.
The overall TRIE op, and for jump tries especially where we emulate
the behavior of branches. So we have to do the same clearing logic if
a trie branch fails as well.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This was originally a patch which made somewhat drastic changes to how
we represent capture buffers, which Dave M and I and are still
discussing offline and which has a larger impact than is acceptable to
address at the current time. As such I have reverted the controversial
parts of this patch for now, while keeping most of it intact even if in
some cases the changes are unused except for debugging purposes.
This patch still contains valuable changes, for instance teaching CURLYX
and CURLYM about how many parens there are before the curly[1] (which
will be useful in follow up patches even if stricly speaking they are
not directly used yet), tests and other cleanups. Also this patch is
sufficiently large that reverting it out would have a large effect on
the patches that were made on top of it.
Thus keeping most of this patch while eliminating the controversial
parts of it for now seemed the best approach, especially as some of the
changes it introduces and the follow up patches based on it are very
useful in cleaning up the structures we use to represent regops.
[1] Curly is the regexp internals term for quantifiers, named after
x{min,max} "curly brace" quantifiers.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This adds (*{ ... }) and (**{ ... }) as equivalents to (?{ ... }) and
(??{ ... }). The only difference being that the star variants are
"optimisitic" and are defined to never disable optimisations. This is
especially relevant now that use of (?{ ... }) prevents important
optimisations anywhere in the pattern, instead of the older and inconsistent
rules where it only affected the parts that contained the EVAL.
It is also very useful for injecting debugging style expressions to the
pattern to understand what the regex engine is actually doing. The older
style (?{ ... }) variants would change the regex engines behavior, meaning
this was not as effective a tool as it could have been.
Similarly it is now possible to test that a given regex optimisation
works correctly using (*{ ... }), which was not possible with (?{ ... }).
|
| |
|
|
|
|
|
|
| |
This allows us to resolve a test inconsistency between CURLYX and CURLY
and CURLYM, which have different maximums. We use I32 and not U32 because
the existing count logic uses -1 internally and using an I32 for the min/max
prevents warnings about comparing signed and unsigned values when the
count is compared against the min or max.
|
| |
|
|
| |
They are unused these days.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
In GH 20435 many typos in our C code were corrected. However, this pull
request was not applied to blead and developed merge conflicts. I
extracted diffs for the individual modified files and applied them with
'git apply', excepting four files where patch conflicts were reported.
Those files were:
handy.h
locale.c
regcomp.c
toke.c
We can handle these in a subsequent commit. Also, had to run these two
programs to keep 'make test_porting' happy:
$ ./perl -Ilib regen/uconfig_h.pl
$ ./perl -Ilib regen/regcomp.pl regnodes.h
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This splits a bunch of the subcomponents of the regex engine into
smaller files.
regcomp_debug.c
regcomp_internal.h
regcomp_invlist.c
regcomp_study.c
regcomp_trie.c
The only real change besides to the build machine to achieve the split
is to also adds some new defines which can be used in embed.fnc to control
exports without having to enumerate /every/ regex engine file. For
instance all of regcomp*.c defines PERL_IN_REGCOMP_ANY, and this is used
in embed.fnc to manage exports.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Having internal tabs causes confusion in diffs and reviews. In the
following patch I will move a lot of code around, creating new files
and they will all be whitespace clean: no trailing whitespace,
tabs expanded to the next tabstop properly, and no trailing empty
lines at the bottom of the file.
This patch prepares for that split, and future splits and changes to
the regex engine by precleaning the main regex engine files with the
same rules.
It should show no changes under '-w'.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
We used the ARG() macro to access the parno data for the OPEN
and CLOSE regops. This made it difficult to find what needed to
change when the type and size or location of this data in the
node was modified. Replacing this access with a specific macro
makes the code more legible and future proof.
This was actually backported from finding everything that broke
by changing the regnode type for OPEN and CLOSE to 2L and moving
the paren parameter to the 2L slot. We might do something like this
in the future and separating the PARNO() macros from their
implementation will make it easier.
|
| | |
|
| |
|
|
| |
CID 353002
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
This replaces PL_regnode_arg_len, PL_regnode_arg_len_varies,
PL_regnode_off_by_arg and PL_regnode_kind with a single PL_regnode_info
array, which is an array of struct regnode_meta, which contains the same
data but as a struct. Since PL_regnode_name is only used in debugging
builds of the regex engine we keep it separate. If we add more debug
properties it might be good to create a PL_regnode_debug_info[] to hold
that data instead.
This means when we add new properties we do not need to modify any
secondary sources to add new properites, just the struct definition
and regen/regcomp.pl
|
| | |
|
| |
|
|
| |
REGNODE_ARG_LEN_VARIES()
|
| | |
|
| | |
|
| |
|
|
| |
The code confusing uses type and kind as synonyms. Lets end that bad habit
|
| |
|
|
|
| |
This rewrites one comment to include more explanation of the difference
between Perl_regnext() and REGNODE_AFTER().
|
| |
|
|
|
|
|
|
|
|
| |
This adds REGNODE_AFTER_varies() which is used when the called *knows*
that the current regnode is variable length. We then use it to handle
EXACTish style nodes as determined by PL_regnode_arg_len_varies.
As part of this patch Perl_regnext() Perl_regnode_after() and
Perl_check_regnode_after() are moved to reginline.h, which is loaded via
regcomp.c only when we are compiling the regex engine.
|
| |
|
|
|
| |
Now that REGNODE_AFTER() can handle all cases it makes sense
to remove the dynamic() suffix.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It is really easy to get confused about the difference between
NEXTOPER() and regnext() of a regnode. The two concepts are related,
similar, but importantly distinct. NEXTOPER() is also defined in such a
way that it is easy to abuse and misunderstand and encourages producing
code that is fragile to larger change, effectively "baking in"
assumptions to the code that are difficult to discover by searching.
Changing the type and storage requirements of a regnode may break things
in subtle and hard to debug ways.
An example of how NEXTOPER() is problematic is that this:
NEXTOPER(NEXTOPER(branch)) does not mean "find the second node after the
branch node", it means "jump forward by a regnode which happens to be
two regnodes large". In other words NEXTOPER is just a fancy way of
writing "node+1".
This patch replaces NEXTOPER() with three new macros:
REGNODE_AFTER_dynamic(node)
REGNODE_AFTER_opcode(node,op)
REGNODE_AFTER_type(node,tregnode_OPNAME)
The first is the most generic case, it jumps forward by the size of the
node, and determines that size by consulting OP(node). The second is
where you have already extracted OP(node), and the third is where you
know the actual structure that you want to jump forward by. Every
regnode type has a corresponding type, which is known at compile time,
so using the third will produce the most efficient code. However in many
cases the code operates on one of several types, whose size may be the
same now, but may change in the future, in which case one of the other
forms is preferred. The run time logic in regexec.c should probably
only use the REGNODE_AFTER_type() interface.
Note that there is also a REGNODE_BEFORE() which replaces PREVOPER(),
which is used in a specific piece of legacy logic but should not be
used otherwise. It is not safe to go backwards from an arbitrary node,
we simply have no way to know how large the previous node is and thus
where it starts.
This patch includes some logic that validates assumptions during DEBUG
mode which should catch errors from resizing regnodes.
After this patch changing the size of an existing regnode should be
relatively safe and errors related to sizing should trigger assertion
fails.
This patch includes changes to perlreguts.pod to explain this stuff
better.
|
| |
|
|
|
|
|
|
|
| |
In a follow up patch we will use this data from regexec.c which
currently cannot see the variable.
This changes a comment in regen/mk_invlists.pl which necessitated
rebuilding several files related to unicode. Only the hashes associated
with mk_invlists.pl were changed.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We cannot safely store 64 bit aligned data in a regnode structure due
to the implicit 32 bit alignment the base structure forces on the
data. Thanks to Tony Cook for the suggestion on how to cleanly support
variable sized pointers without alignment issues.
I am pretty sure we should not be storing pointers in the regexp program
like this. In most cases where we need an SV attached to a regnode
structure we store it in the 'data' array which part of the regexp
structure, and then store an index to that item in the regnode. This
allows the use of a smaller member for the index instead.
This was identified by running "make test_reonly" under the ubsan build:
./Configure -d -Doptimize=-g -Dusedevel -DDEBUGGING \
-Accflags='-fsanitize=address -fsanitize=undefined \
-ggdb3' -Aldflags='-Wl,--no-as-needed -lasan -lubsan' \
-Dcc=ccache\ gcc -Dld=gcc
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It turns out that any character class whose UTF-8 representation is two
bytes long, and where all elements share the same first byte can be
represented by a compact, fast regnode designed for the purpose.
This commit adds that regnode, ANYOFHbbm. ANYOFHb already exists for
classes where all elements have the same first byte, and this just
changes the two-byte ones to use a bitmap instead of an inversion list.
The advantages of this are that no conversion to code point is required
(the continuation byte is just looked up in the bitmap) and no inversion
list is needed. The inversion list would occupy more space, from 4 to
34 extra 64-bit words, plus an AV and SV, depending on what elements the
class matches.
Many characters in the Latin, Greek, Cyrillic, Greek, Hebrew, Arabic,
and several other (lesser-known) scripts are of this form.
It would be possible to extend this technique to larger bitmaps, but
this commit is a start.
|
| |
|
|
|
|
| |
This introduces a new macro and converts to use it so that bitmaps other
than the traditional ones in ANYOF nodes may be defined in a common
manner.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A bracketed character class in a pattern is generally represented by
some form of ANYOF node, with matches of characters in the Latin1 range
handled by a bitmap, and an inversion list for higher code point
matches. But some patterns only have low matches, and some only high,
and some match everything that is high.
This commit refactors a little so that the distinction between nothing
high matches vs everything high matches is done through the same
technique. Previously one was indicated by a flag, and the other by a
special value in the node's structure. Now there are two special
values, and the flag is freed up for a potential future use. In the
past the meaning of the flags has had to be overloaded go accommodate
all the needs. freeing of a flag means
This all allows for some slight simplicfications.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
In ANYOF nodes (generated for qr/[]/), there is a bitmap component, and
possibly a non-bitmap component. It turns out that a single flag can be
used to indicate the existence of the latter. When looked at this way,
the name of the flag becomes simpler, and incorporates the meaning of
another bit, which was previously shared with yet another meaning. Thus
that other meaning can become an unshared bit.
This allows for some simplification, and being able to handle the
uncommon Turkish locale with fewer main-line conditionals being executed
at runtime.
|
| |
|
|
| |
This is a small thing, but might as well use the mnemonic.
|
| | |
|
| |
|
|
|
|
|
|
|
| |
These long names are designed to remind the coder that they have
multiple meanings. But move the reminder text to the end, as it
obscures the purposes.
And some have two halves for the separate meanings; change the names so
the halves are split by two underscores to visually emphasize this.
|
| |
|
|
|
|
|
| |
This reverts commit d62feba66bf43f35d092bb026694f927e9f94d38.
As explained in its commit message. It adds some comments to point out
that the commit exists, for the curious.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Several of the POSIXA classes are a single range on ASCII platforms, and
[:digit:] is a single range on both ASCII and EBCDIC. This regnode was
designed to replace the POSIXA regnode for such classes to get a bit of
performance by not needing to do an array lookup. Instead it encodes
some bits in the flags field that with shifting and masking get the
right values for the single range's bounds for any such node.
However, performance tests conducted by Sergey Aleynikov showed this was
actually slower than what it intended to replace. Rather than
completely drop this work, I'm adding it to blead, and immediately
reverting it, so that should parts of it ever become useful, it would be
available.
A few tests fail; those are skipped for the purposes of this commit so
that it doesn't interfere with bisecting.
The code also isn't completely commented.
One could add a regnode for each posix class it was decided should have
the expected performance boost. But regnodes are a finite resource, and
the boost is probably not large enough to justify doing so.
|
| |
|
|
|
|
|
| |
C reserves symbols beginning with underscores for its own use. This
commit moves the underscore so it is trailing, which is legal. The
symbols changed here are many of the ones in handy.h that have
significant uses outside it.
|
| |
|
|
| |
Easier to read
|
| |
|
|
|
|
|
|
| |
This names a flag bit whose meaning depends on context. The previous
name contained both meanings, and was simply too long to be
comprehensible. This commit splits it into two names, with the suffix
'_shared'. It's not necessary to know precisely what the other meaning
is when reading the code, only that it has another meaning.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add funcs to parse empty constructs which boil down to OPFAIL or NOTHING,
and then consistently use them while parsing lookahead and lookbehind.
This allows us to eliminate duplicated code and optimize the various cases
which boil down to these constructs.
Eg. (?!) and (?<!) are equivalent to OPFAIL ops,
and (?=) and (?<=) are equivalent to NOTHING ops.
This patch does not deal with the (* ... ) forms, that will come in a
follow up.
One advantage of this optimization is it does not need to set the various
special flags related to lookaround as they aren't really lookaround ops.
As a bonus this also improves the error messages from incomplete patterns,
and add tests for various error messages.
Note the functions are given awkward but shortish names so they are not
always forced to be line broken. I would prefer to name them something
more descriptive, but that would make their use harder to read by
forcing line breaks. So I chose to use an abbreviation.
|
| |
|
|
|
|
|
|
|
|
| |
This changes the name_list_idx attribute from I32 to a U32 as it will
never be negative, and as of a963d6d5acabdd8c7 a 0 can be safely used
to represent "no value" for items in the 'data' array.
I noticed this while cleaning up the offsets debug logic and updating
the perlreguts documentation, so I figured I might as well clean it up
at the same time.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This code was added by Mark Jason Dominus to aid a regex debugger
he wrote for ActiveState. The basic premise is that every opcode
in a regex can be attributed back to a contiguous sequence of characters
that make up the pattern. This assumption has not been true ever since
the "jump" TRIE optimizations were added to the engine.
I spoke to MJD many years ago about whether it was ok to remove this
from the regex engine and he said he had no objections.
An example of a pattern that cannot be handled correctly by this logic is
/(?: a x+ | b y+ | c z+ )/x
where the
(?:a ... | b ... | c ...)
parts will now be handled by the TRIE logic and not by the BRANCH/EXACT
opcodes that it would have been in the past. The offset debug output
cannot handle this type of transformation, and produce nonsense output
that mention opcodes that have been optimized away from the final program.
The regex compiler is complicated enough without having to maintain this
logic. There are essentially no tests for it, and the few tests that do
cover it do so as a byproduct of testing other things. Despite the offsets
logic only being used in debug supporting it does have a cost to non-debug
logic as various internal routines include parameters related to it that
are otherwise unused.
Note this output is only usable or visible by enabling special flags
in re.pm, there is no formal API to access it short of parsing the
output of the debug mode of the regex engine, which has changed multiple
time over the past years.
|
