| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| | |
|
| |
|
|
|
|
|
|
|
|
|
| |
This just detabifies to get rid of the mixed tab/space indentation.
Applying consistent indentation and dealing with other tabs are another issue.
Done with `expand -i`.
* vutil.* left alone, it's part of version.
* Left regen managed files alone for now.
|
| | |
|
| |
|
|
|
|
| |
For some reason I was storing the counts of sub signature parameters and
optional parameters as signed ints. Since these can never be negative,
change them to UV instead.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
RT #132760
A recent commit (v5.27.7-212-g894f226) moved subroutine attributes back
before the subroutine's signature: e.g.
sub foo :prototype($$) ($a, $b) { ... } # 5.18 and 5.28 +
sub foo ($a, $b) :prototype($$) { ... } # 5.20 .. 5.26
This change means that any code still using an attribute following the
signature is going to trigger a syntax error. However, the error, followed
by error recovery and further warnings and errors, is very unfriendly and
gives no indication of the root cause. This commit introduces a new error,
"Subroutine attributes must come before the signature".
For example, List::Lazy, the subject of the ticket, failed to compile
tests, with output like:
Array found where operator expected at blib/lib/List/Lazy.pm line 43,
near "$$@)" (Missing operator before @)?)
"my" variable $step masks earlier declaration in same statement at
blib/lib/List/Lazy.pm line 44.
syntax error at blib/lib/List/Lazy.pm line 36, near ") :"
Global symbol "$generator" requires explicit package name (did you
forget to declare "my $generator"?) at blib/lib/List/Lazy.pm line 38.
Global symbol "$state" requires explicit package name (did you forget
to declare "my $state"?) at blib/lib/List/Lazy.pm line 39.
Global symbol "$min" requires explicit package name (did you forget to
declare "my $min"?) at blib/lib/List/Lazy.pm line 43.
Global symbol "$max" requires explicit package name (did you forget to
declare "my $max"?) at blib/lib/List/Lazy.pm line 43.
Global symbol "$step" requires explicit package name (did you forget
to declare "my $step"?) at blib/lib/List/Lazy.pm line 43.
Invalid separator character '{' in attribute list at
blib/lib/List/Lazy.pm line 44, near "$step : sub "
Global symbol "$step" requires explicit package name (did you forget
to declare "my $step"?) at blib/lib/List/Lazy.pm line 44.
But following this commit, it now just outputs:
Subroutine attributes must come before the signature at
blib/lib/List/Lazy.pm line 36.
Compilation failed in require at t/append.t line 5.
BEGIN failed--compilation aborted at t/append.t line 5.
It works by:
1) adding a boolean flag (sig_seen) to the parser state to indicate that a
signature has been parsed;
2) at the end of parsing a signature, PL_expect is set to XATTRBLOCK
rather than XBLOCK.
Then if something looking like one or more attributes is encountered
by the lexer immediately afterwards, it scans it as if it were an
attribute, but then if sig_seen is true, it croaks.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
We've had a few reports of segmentation faults and other misbehaviour
when sub-parsing, such as within interpolated expressions, fails.
This change aborts compilation if anything complex enough to not be
parsed by the lexer is compiled in a sub-parse *and* an error
occurs within the sub-parse.
An earlier version of this patch failed on simpler expressions,
which caused many test failures, which this version doesn't (which may
just mean we need more tests...)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previous commits have tightened up the checking of UTF-8 for
well-formedness in the input program or string eval. This is done in
lex_next_chunk and lex_start. But it doesn't handle the case of
use utf8; foo
because 'foo' is checked while UTF-8 is still off. This solves that
problem by noticing when utf8 is turned on, and then rechecking at the
next opportunity.
See thread beginning at
http://nntp.perl.org/group/perl.perl5.porters/242916
This fixes [perl #130675]. A test will be added in a future commit
This catches some errors earlier than they used to be and aborts. so
some tests in the suite had to be split into multiple parts.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
This reverts v5.25.7-60-gb2c9b6e and adds a test.
In that previous commit of mine, for efficiency I changed it so that it
checked and extended the parser stack only after every reduce rather than
every shift, but when it did check, it extended it by at least 15 slots to
allow for all the elements of the longest possible rule to be shifted.
Turns out this was bad reasoning. The following type of code can shift
indefinitely without ever reducing:
[{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{
|
| |
|
|
|
|
| |
Rather than checking before each individual shift whether the parse stack
needs extending, only check once per rule, making sure there's enough
space to shift all the items for the longest possible rule
|
| |
|
|
| |
Makes testing whether the parser stack needs extending cheaper
|
| | |
|
| |
|
|
| |
This is meant to be a more correct version of d64e121b07.
|
| |
|
|
|
|
|
| |
This reverts commit d64e121b07bda895f7f3a5d0e449fc948986e2f1.
It contained stupid blunders. See
<20160803202545.2999.qmail@lists-nntp.develooper.com>.
|
| |
|
|
|
|
|
|
|
| |
During the course of parsing end exection, these values get stored
as ints and UVs, then used as SSize_t.
Standardise on IVs instead. Technically they can never be negative, but
their final use is as indices into AVs, which is SSize_t, so it's
easier to standardise on a signed value throughout.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Currently the signature of a sub (i.e. the '($a, $b = 1)' bit) is parsed
in toke.c using a roll-your-own mini-parser. This commit makes
the signature be part of the general grammar in perly.y instead.
In theory it should still generate the same optree as before, except
that an OP_STUB is no longer appended to each signature optree: it's
unnecessary, and I assume that was a hangover from early development of
the original signature code.
Error messages have changed somewhat: the generic 'Parse error' has
changed to the generic 'syntax error', with the addition of ', near "xyz"'
now appended to each message.
Also, some specific error messages have been added; for example
(@a=1) now says that slurpy params can't have a default vale, rather than
just giving 'Parse error'.
It introduces a new lexer expect state, XSIGVAR, since otherwise when
the lexer saw something like '($, ...)' it would see the identifier
'$,' rather than the tokens '$' and ','.
Since it no longer uses parse_termexpr(), it is no longer subject to the
bug (#123010) associated with that; so sub f($x = print, $y) {}
is no longer mis-interpreted as sub f($x = print($_, $y)) {}
|
| |
|
|
|
| |
Follow-up to 25e092d6d, which left some gaping holes. I committed it
too soon.
|
| |
|
|
|
|
| |
We still have a 16-bit hole, which I have noted. There’s no point in
moving the two U8s near the end of the struct (lex_fakeeof and lex_flags)
to fill that hole, as we would just have a bigger hole at the end.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
sublex_info is never validly copied or set* all at once and no pointer
is ever taken to it. It seems to be left over from the time when
PL_sublex_info was a global variable. (Indeed, the struct is still
defined in perl.h, an odd place for something used only by parser.h.)
It will be easier to eliminate alignment holes in the parser struct if
we just empty it out.
* The one instance of sublex_info being copied, in
sv.c:Perl_parser_dup, ended up potentially sharing an SV between
threads, which is a no-no. I say potentially, because I can’t see how
it could be non-null during thread cloning, which would have to happen
between sublex_start and sublex_push.
|
| |
|
|
|
|
|
| |
In commit 8f0d8652cd, I mentioned that PL_parser->lex_expect is
still assigned to by Data::Alias. That is no longer the case as
of Data::Alias 1.19. So there is no need to keep this member
around.
|
| |
|
|
|
| |
As mentioned in the commit message for a2867f88735d4, this can
only contain numbers from 0 to 5.
|
| |
|
|
|
|
| |
We will need to store characters > 255 in here.
Also, cast accordingly in toke.c.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
An empty cpan/.dir-locals.el stops Emacs using the core defaults for
code imported from CPAN.
Committer's work:
To keep t/porting/cmp_version.t and t/porting/utils.t happy, $VERSION needed
to be incremented in many files, including throughout dist/PathTools.
perldelta entry for module updates.
Add two Emacs control files to MANIFEST; re-sort MANIFEST.
For: RT #124119.
|
| |
|
|
|
|
| |
This is to avoid a compiler warning at toke.c:1912 (the assertion in
S_force_next). The only values this member can contain are 0 to 5, so
this change is safe. (I’ll probably change it to U8 after 5.22.)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
8635e3c2 (5.21.6) changed the COP sequence numbers for nested blocks,
such that most BEGIN blocks (incl. ‘use’ statements) and sub declara-
tions end up in the right place. However, it had the side effect of
causing declarations at the end of the enclosing scope to fall out of
it and appear below.
This commit fixes that by adding an extra nulled COP to the end of the
enclosing scope if that scope ends with a sub, so the final declara-
tion gets deparsed before it.
The frequency of sub declarations at the end of the enclosing scope is
sufficiently low (I’m guessing a bit here) that this slight increase
in run-time memory usage is probably acceptable.
I had to change B::Deparse to deparse nulled COPs the same way it does
live COPs, which means we get more extraneous semicolons than before.
I hope to fix that in a forthcoming commit. I also ran into a B bug,
in that null ops are not presented to Perl code with the right op
class (see the blessing in the patch). I plan to fix that in a separ-
ate commit, too.
|
| |
|
|
|
|
|
| |
There is at least one CPAN module (Data::Alias) that assigns to this.
Removing it won’t shrink the parser struct because of alignment, so
it doesn’t gain us anything. Just leave it for now. We can remove
it later if we have to.
|
| |
|
|
|
|
| |
MAD = Misc Attribute Decoration; unmaintained attempt at preserving
the Perl parse tree more faithfully so that automatic conversion to
Perl 6 would have been easier.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This turned out to be tricky. Normally @ at the beginning of the
interpolated code signals to the lexer to emit ‘join($",’ immediately.
With "$_->@*" we would have to retract the $ _ -> tokens upon encoun-
tering @*, which we obviously cannot do.
Waiting until we reach the end of the interpolated text before emit-
ting anything could not work either, as it may contain BEGIN blocks
that affect the way part of the interpolated code is parsed.
So what we do is introduce an egregious or clever hack, depending on
how you look at it.
Normally, the lexer turns "@foo" into:
stringify ( join ( $ " , @ foo ) )
(The " is a WORD token, representing a variable name.)
"$_" becomes:
stringify ( $ _ )
We can turn "$_->@*" into:
stringify ( $ _ -> @ * POSTJOIN )
Where POSTJOIN is a new lexer token with special handling that creates
a join op just the way join($", ...) does.
To make "foo$_->@*bar" work as well, we have to make POSTJOIN have
precedence just below ->, so that
stringify ( "foo" . $ _ -> @ * POSTJOIN . "bar" )
(what the parser sees) is equivalent to:
stringify ( "foo" . ( $ _ -> @ * POSTJOIN ) . "bar" )
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Commit 2179133 inadvertently stopped the PERL5DB env var from being
truncated just before the first line break. (I’m considering that
a bug fix.)
The result is that #!perl -d:foo will throw the line numbers off by
one, as will line breaks in PERL5DB:
$ PERL5DB='sub DB::DB{}'$'\n\n\n''' ./perl -dle 'warn "ok"'
ok at -e line 4.
#!perl -d:foo has thrown off line numbers since f0e67a1d291 in 5.12.
This commit fixes both, by storing the line number of #! -d or the
number 0 for -d on the command line in the new PL_parser->preambling
member, which now overrides any number in PL_curcop.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously, the line number was localised in lexing scopes. herelines
had to be in the lex_shared struct so that inner lexing scopes
could peek into values belonging to outer lexing scopes and set the
herelines value belonging to the same scope that a here-doc body was
extracted from. (herelines records how much extra to increase the
line number at the next line ending, to jump over a here-doc.)
In commit ffdb8b167e, I changed things so that lexing scopes no longer
localised the line number, except for here-docs, and the line number
was incremented within the inner lexing scope, instead of during the
initial scan for the terminator. That meant the herelines value had
to be copied into the inner lexing scope.
For nested here-docs, the inner here-doc’s body is always inside the
outer here-doc, so no peeking into outer scopes is necessary.
Hence, there is no longer any reason for herelines to be inside the
lex_shared struct. We can put it directly inside the parser struct.
Here-docs will localise it. Other quote-like constructs will not (and
can avoid the copy.)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
‘Ambiguous use of * resolved as operator *’: This message can occur in
cases where there is no multiplication operator, so what it is saying
is completely wrong.
When the lexer parses a bareword, it looks at the previous character
and warns if it happens to match /[*%&]/, so foo**bar and foo&&bar
result in this warning, as does print $%foo.
The purpose of the code is to catch *bar *bar or &black &sheep.
To avoid false positives, when emitting one of the three operators
* % & the lexer can record that fact, so when it sees a bareword pre-
ceded by one of those three characters, instead of guessing that the
infix operator was used, it will *know*.
The test cases added also trigger ‘Bareword found where operator
expected’. I don’t know whether that should change, but at least the
current behaviour is tested, so we will know when it does change.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When re-parsing a pattern for run-time (?{}) code blocks,
we end up with the EVAL_RE_REPARSING flag set in PL_in_eval.
Currently we clear this flag as soon as scan_str() returns, to ensure that
it's not set if we happen to parse further patterns (e.g. within the
(?{ ... }) code itself.
However, a soon-to-be-applied bugfix requires us to know the reparsing
state beyond this point. To solve this, we add a new boolean flag to the
parser struct, which is set from PL_in_eval in S_sublex_push() (with the
old value being saved). This allows us to have the flag around for the
entire pattern string parsing phase, without it affecting nested pattern
compilation.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
yylex must emit exactly one token each time it is called. Some-
times yylex needs to parse several tokens at once. That’s what
the various force functions are for. But that is also what
PL_pending_ident is for.
The various force_next, force_word, force_ident, etc., functions keep
a stack of tokens (PL_nextval/PL_nexttype) that yylex will check imme-
diately when called.
PL_pending_ident is used to track a single identifier that yylex will
hand off to S_pending_ident to handle.
S_pending_ident is the only piece of code for resolving an identi-
fier that could be lexical but could also be a package variable.
force_ident assumes it is looking for a package variable.
force_* takes precedence over PL_pending_ident.
All this means that, if an identifier needs to be looked up in the pad
on the next yylex invocation, it has to use PL_pending_ident, and the
force_* functions cannot be used at the same time.
Not realising that, when I made ‘our sub foo’ store the sub in the
pad I also made ‘our sub foo ($)’ into a syntax error, because it
was being parsed as ‘our sub ($) foo’ (the prototype being ‘forced’);
i.e., the pending tokens were being pulled out of the ‘queue’ in the
wrong order. (I put queue in quotes, because one queue and one unre-
lated buffer together don’t exactly count as ‘a queue’.)
Changing PL_pending_ident to have precedence over the force stack
breaks ext/XS-APItest/t/swaptwostmts.t, because the statement-parsing
interface does not localise PL_pending_ident. It could be changed to
do that, but I don’t think it is the right solution.
Having two separate pending token mechanisms makes things need-
lessly fragile.
This commit eliminates the PL_pending_ident mechanism and
modifies S_pending_ident (renaming it in the process to
S_force_ident_maybe_lex) to work with the force mechanism. I was
going to merge it with force_ident, but the two make incompatible
assumptions that just complicate the code if merged. S_pending_ident
needs the sigil in the same string buffer, to pass to the pad inter-
face. force_ident needs to be able to work without a sigil present.
So now we only have one queue for pending tokens and the order is more
predictable.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commit fixes here-docs in single-line re-evals in files (as
opposed to evals) and here-docs in single-line quote-like operators
inside re-evals.
In both cases, the here-doc parser has to look into an outer
lexing scope to find the here-doc body. And in both cases it
was stomping on PL_linestr (the current line buffer) while
PL_sublex_info.re_eval_start was pointing to an offset in that buffer.
(re_eval_start is used to construct the string to include in the
regexp’s stringification once the lexer reaches the end of the
re-eval.)
Fixing this entails moving re_eval_start and re_eval_str to
PL_parser->lex_shared, making the pre-localised values visible.
This is so that the code that peeks into an outer linestr buffer to
steal the here-doc body can set up re_eval_str in the right scope.
(re_eval_str is used to store the re-eval text when the here-
oc parser has no choice but to modify linestr; see also commit
db4442662555874019.)
It also entails making the stream-based parser (i.e., that reads from
an input stream) leave PL_linestr alone, instead of clobbering it and
then reconstructing part of it afterwards.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Unfortunately, PL_parser->linestr and PL_parser->bufptr are both
part of the API, so we can’t just move them to PL_parser->lex_shared.
Instead, we have to copy them in sublex_push, to make them visible to
inner lexing scopes.
This allows the SvIVX(PL_linestr) and SvNVX(PL_linestr) hack to
be removed.
It should also speed things up slightly. We are already allocating
PL_parser->lex_shared in sublex_push, so there should be no need to
upgrade PL_linestr to SvNVX as well.
I was pleasantly surprised to see how the here-doc code seemed to
shrink all by itself when modified to account.
PL_sublex_info.super_bufptr is also superseded by the addition of
->ls_bufptr to the LEXSHARED struct. Its old values when localised
were not visible, being stashed away on the savestack, so it was
harder to use.
|
| |
|
|
|
| |
It took me a while to figure this out, so here it is for
future readers.
|
| |
|
|
|
|
|
|
| |
PL_parser->herelines needs to be visible to inner lexing scopes, which
also need to have their own copy of it, so that the here-doc parser
can modify the right herelines variable corresponding to the
PL_linestr from which it is stealing its body. (A subsequent commit
will take take of that.)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The line numbers for operators after a here-doc marker on the same
line were off by the length of the here-doc.
This is because the here-doc parser would artificially increase the
line number as it went, because it was stealing lines out of the
input stream.
Instead, we can record the number of lines in the here-doc, and add it
to the line number the next time we need to increment it.
This also fixes the line numbers after s//<<END/e to the end of the
file, which were off because the line number adjusted by the <<END was
localised to the s///.
Since herelines is visible to inner lexing scopes, the outer lexing
scope can see changes made by the inner one.
The lack of localisation does cause problems with line numbers inside
quote-like operators (but they were off by one already), which will be
addressed in subsequent commits.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
For re-evals, this is something that broke recently, post-5.16 (the
jumbo fix). For other interpolating constructs, this has never
worked, as far as I can tell.
The lexer was losing track of PL_lex_state (aka PL_parser->lex_state)
when parsing formats. Usually, the state alternates between
LEX_FORMLINE (a picture line) and LEX_NORMAL (an argument line), but
the LEX_NORMAL should actually be whatever the state was before the
format started.
This commit adds a new parser member to track the ‘normal’ state when
parsing a format.
It also tweaks S_scan_formline to handle multi-line buffers outside of
string eval (such as happens in interpolating constructs).
That bufend assignment that is removed as a result is not necessary as
of a0d0e21ea6ea (perl 5.000). That very commit added a bufend assign-
ment after the sv_gets (later filter_gets; later lex_next_chunk) fur-
ther down in the loop in scan_formline.
|
| | |
|
| | |
|
| |
|
|
|
| |
This updates the editor hints in our files for Emacs and vim to request
that tabs be inserted as spaces.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously it would leave the file handle open if it was (equal to) stdin,
on the assumption that this must have been because no script name was
supplied on the interpreter command line, so the interpreter was defaulting
to reading the script from standard input.
However, if the program has closed STDIN, then the next file handle opened
(for any reason) will have file descriptor 0. So in this situation, the
handle that require opened to read the module would be mistaken for the above
situation and left open. Effectively, this leaked a file handle.
This is now fixed, by explicitly tracking from parser creation time whether
it should keep the file handle open, and only setting this flag when
defaulting to reading the main program from standard input. This resolves
RT #37033.
|
| |
|
|
|
|
|
|
|
|
| |
lex_flags holds 4 flag bits, with multiple flag bits manipulated together
at times, so they can't be split out into individual bitfields. This change
permits the C compiler to generate simpler code, reducing toke.o by about
400 bytes on this platform, but doesn't change the size of the structure.
lex_flags was added in commit 802a15e9c01d1a0b in August 2011, so is not in
any stable release.
|
| |
|
|
|
|
|
| |
Sync copyright dates with actual changes according to git history.
[Plus run regen_perly.h to update the SHA-256 checksums, and
regen/regcharclass.pl to update regcharclass.h]
|
| | |
|
| |
|
|
|
|
|
| |
Perl_lex_start copies the string passed to it unconditionally.
Sometimes pp_entereval makes a copy before passing the string
to lex_start. So in those cases we can pass a flag to avoid a
redundant copy.
|
| |
|
|
|
|
|
| |
By combining two booleans with the flags field, we save some space.
By making it a 16-bit instead of 32-bit field (only two flag bits
are currently used), we also avoid alignment holes (I hope; I’m
not very good at this).
|
| |
|
|
|
|
| |
When a filter is added, the current buffer is hung on the end of
the filters array, and a new substring of it becomes the current
buffer.
|
| |
|
|
|
| |
Put LEX_IGNORE_UTF8_HINTS near the only other constant passed
to lex_start
|
| |
|
|
|
| |
(modified by the committer only to apply when the unicode_eval
feature is enabled)
|
