summaryrefslogtreecommitdiff
path: root/parser.h
Commit message (Collapse)AuthorAgeFilesLines
* Fix our sub with protoFather Chrysostomos2012-09-151-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | yylex must emit exactly one token each time it is called. Some- times yylex needs to parse several tokens at once. That’s what the various force functions are for. But that is also what PL_pending_ident is for. The various force_next, force_word, force_ident, etc., functions keep a stack of tokens (PL_nextval/PL_nexttype) that yylex will check imme- diately when called. PL_pending_ident is used to track a single identifier that yylex will hand off to S_pending_ident to handle. S_pending_ident is the only piece of code for resolving an identi- fier that could be lexical but could also be a package variable. force_ident assumes it is looking for a package variable. force_* takes precedence over PL_pending_ident. All this means that, if an identifier needs to be looked up in the pad on the next yylex invocation, it has to use PL_pending_ident, and the force_* functions cannot be used at the same time. Not realising that, when I made ‘our sub foo’ store the sub in the pad I also made ‘our sub foo ($)’ into a syntax error, because it was being parsed as ‘our sub ($) foo’ (the prototype being ‘forced’); i.e., the pending tokens were being pulled out of the ‘queue’ in the wrong order. (I put queue in quotes, because one queue and one unre- lated buffer together don’t exactly count as ‘a queue’.) Changing PL_pending_ident to have precedence over the force stack breaks ext/XS-APItest/t/swaptwostmts.t, because the statement-parsing interface does not localise PL_pending_ident. It could be changed to do that, but I don’t think it is the right solution. Having two separate pending token mechanisms makes things need- lessly fragile. This commit eliminates the PL_pending_ident mechanism and modifies S_pending_ident (renaming it in the process to S_force_ident_maybe_lex) to work with the force mechanism. I was going to merge it with force_ident, but the two make incompatible assumptions that just complicate the code if merged. S_pending_ident needs the sigil in the same string buffer, to pass to the pad inter- face. force_ident needs to be able to work without a sigil present. So now we only have one queue for pending tokens and the order is more predictable.
* Finish fixing here-docs in re-evalsFather Chrysostomos2012-08-301-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | This commit fixes here-docs in single-line re-evals in files (as opposed to evals) and here-docs in single-line quote-like operators inside re-evals. In both cases, the here-doc parser has to look into an outer lexing scope to find the here-doc body. And in both cases it was stomping on PL_linestr (the current line buffer) while PL_sublex_info.re_eval_start was pointing to an offset in that buffer. (re_eval_start is used to construct the string to include in the regexp’s stringification once the lexer reaches the end of the re-eval.) Fixing this entails moving re_eval_start and re_eval_str to PL_parser->lex_shared, making the pre-localised values visible. This is so that the code that peeks into an outer linestr buffer to steal the here-doc body can set up re_eval_str in the right scope. (re_eval_str is used to store the re-eval text when the here- oc parser has no choice but to modify linestr; see also commit db4442662555874019.) It also entails making the stream-based parser (i.e., that reads from an input stream) leave PL_linestr alone, instead of clobbering it and then reconstructing part of it afterwards.
* Use PL_parser->lex_shared instead of Sv[IN]VX(PL_linestr)Father Chrysostomos2012-08-281-0/+2
| | | | | | | | | | | | | | | | | | | | | | Unfortunately, PL_parser->linestr and PL_parser->bufptr are both part of the API, so we can’t just move them to PL_parser->lex_shared. Instead, we have to copy them in sublex_push, to make them visible to inner lexing scopes. This allows the SvIVX(PL_linestr) and SvNVX(PL_linestr) hack to be removed. It should also speed things up slightly. We are already allocating PL_parser->lex_shared in sublex_push, so there should be no need to upgrade PL_linestr to SvNVX as well. I was pleasantly surprised to see how the here-doc code seemed to shrink all by itself when modified to account. PL_sublex_info.super_bufptr is also superseded by the addition of ->ls_bufptr to the LEXSHARED struct. Its old values when localised were not visible, being stashed away on the savestack, so it was harder to use.
* parser.h: Document copline with more detailFather Chrysostomos2012-08-271-1/+5
| | | | | It took me a while to figure this out, so here it is for future readers.
* Add PL_parser->lex_shared struct; move herelines into itFather Chrysostomos2012-08-271-1/+8
| | | | | | | | PL_parser->herelines needs to be visible to inner lexing scopes, which also need to have their own copy of it, so that the here-doc parser can modify the right herelines variable corresponding to the PL_linestr from which it is stealing its body. (A subsequent commit will take take of that.)
* [perl #114070] Fix lines nums after <<fooFather Chrysostomos2012-08-271-0/+1
| | | | | | | | | | | | | | | | | | | | | | | The line numbers for operators after a here-doc marker on the same line were off by the length of the here-doc. This is because the here-doc parser would artificially increase the line number as it went, because it was stealing lines out of the input stream. Instead, we can record the number of lines in the here-doc, and add it to the line number the next time we need to increment it. This also fixes the line numbers after s//<<END/e to the end of the file, which were off because the line number adjusted by the <<END was localised to the s///. Since herelines is visible to inner lexing scopes, the outer lexing scope can see changes made by the inner one. The lack of localisation does cause problems with line numbers inside quote-like operators (but they were off by one already), which will be addressed in subsequent commits.
* [perl #114040] Parse formats in interpolating constructsFather Chrysostomos2012-08-061-0/+1
| | | | | | | | | | | | | | | | | | | | | | | For re-evals, this is something that broke recently, post-5.16 (the jumbo fix). For other interpolating constructs, this has never worked, as far as I can tell. The lexer was losing track of PL_lex_state (aka PL_parser->lex_state) when parsing formats. Usually, the state alternates between LEX_FORMLINE (a picture line) and LEX_NORMAL (an argument line), but the LEX_NORMAL should actually be whatever the state was before the format started. This commit adds a new parser member to track the ‘normal’ state when parsing a format. It also tweaks S_scan_formline to handle multi-line buffers outside of string eval (such as happens in interpolating constructs). That bufend assignment that is removed as a result is not necessary as of a0d0e21ea6ea (perl 5.000). That very commit added a bufend assign- ment after the sv_gets (later filter_gets; later lex_next_chunk) fur- ther down in the loop in scan_formline.
* parser.h: Add comments explaining *bufptrFather Chrysostomos2012-07-031-3/+5
|
* parser.h: Correct comment explaining last_lop_opFather Chrysostomos2012-07-031-1/+1
|
* update the editor hints for spaces, not tabsRicardo Signes2012-05-291-2/+2
| | | | | This updates the editor hints in our files for Emacs and vim to request that tabs be inserted as spaces.
* The parser should always close the file handle that it opened.Nicholas Clark2012-02-271-1/+2
| | | | | | | | | | | | | | | | | Previously it would leave the file handle open if it was (equal to) stdin, on the assumption that this must have been because no script name was supplied on the interpreter command line, so the interpreter was defaulting to reading the script from standard input. However, if the program has closed STDIN, then the next file handle opened (for any reason) will have file descriptor 0. So in this situation, the handle that require opened to read the module would be mistaken for the above situation and left open. Effectively, this leaked a file handle. This is now fixed, by explicitly tracking from parser creation time whether it should keep the file handle open, and only setting this flag when defaulting to reading the main program from standard input. This resolves RT #37033.
* In struct yy_parser, change lex_flags to a U8, from part of a bitfield.Nicholas Clark2012-02-271-1/+1
| | | | | | | | | | lex_flags holds 4 flag bits, with multiple flag bits manipulated together at times, so they can't be split out into individual bitfields. This change permits the C compiler to generate simpler code, reducing toke.o by about 400 bytes on this platform, but doesn't change the size of the structure. lex_flags was added in commit 802a15e9c01d1a0b in August 2011, so is not in any stable release.
* Bump several file copyright datesSteffen Schwigon2012-01-191-1/+1
| | | | | | | Sync copyright dates with actual changes according to git history. [Plus run regen_perly.h to update the SHA-256 checksums, and regen/regcharclass.pl to update regcharclass.h]
* Fix a 5-year-old typo in a commentRodolfo Carvalho2011-11-201-1/+1
|
* Avoid redundant copies in string evalsFather Chrysostomos2011-11-061-1/+3
| | | | | | | Perl_lex_start copies the string passed to it unconditionally. Sometimes pp_entereval makes a copy before passing the string to lex_start. So in those cases we can pass a flag to avoid a redundant copy.
* Merge some parser fieldsFather Chrysostomos2011-11-061-3/+3
| | | | | | | By combining two booleans with the flags field, we save some space. By making it a 16-bit instead of 32-bit field (only two flag bits are currently used), we also avoid alignment holes (I hope; I’m not very good at this).
* Make source filters work in evalbytesFather Chrysostomos2011-11-061-1/+5
| | | | | | When a filter is added, the current buffer is hung on the end of the filters array, and a new substring of it becomes the current buffer.
* parser.h: Rearrange constantsFather Chrysostomos2011-11-061-1/+2
| | | | | Put LEX_IGNORE_UTF8_HINTS near the only other constant passed to lex_start
* eval STRING UTF8 cleanup.Brian Fraser2011-11-061-0/+2
| | | | | (modified by the committer only to apply when the unicode_eval feature is enabled)
* [perl #87064] eval no longer shares filtersFather Chrysostomos2011-04-031-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Before this commit: commit f07ec6dd59215a56bc1159449a9631be7a02a94d Author: Zefram <zefram@fysh.org> Date: Wed Oct 13 19:05:19 2010 +0100 remove filter inheritance option from lex_start The only uses of lex_start that had the new_filter parameter false, to make the new lexer context share source filters with the previous lexer context, were uses with rsfp null, which therefore never invoked source filters. Inheriting source filters from a logically unrelated file seems like a silly idea anyway. string evals could inherit the same source filter space as the cur- rently compiling code. Despite what the quoted commit message says, sharing source filters allows filters to be inherited in both direc- tions: A source filter created when the eval is being compiled also applies to the file with which it is sharing its space. There are at least 20 CPAN distributions relying on this behaviour (or, rather, what could be considered a Test::More bug). So this com- mit restores the source-filter-sharing capability. It does not change the current API or make public the API for sharing source filters, as this is supposed to be a temporary stop-gap measure for 5.14.
* recursive-descent expression parsingZefram2010-12-111-2/+19
| | | | | | New API functions parse_fullexpr(), parse_listexpr(), parse_termexpr(), and parse_arithexpr(), to parse an expression at various precedence levels.
* function to parse isolated labelZefram2010-10-251-0/+3
| | | | | | | New API function parse_label() parses a label, separate from statements. If a label has not already been lexed and queued up, it does not use yylex(), but parses the label itself at the character level, to avoid unwanted lexing past an absent optional label.
* replace PL_doextract with better kinds of variableZefram2010-10-211-0/+1
| | | | | | | | PL_doextract had two unrelated jobs, neither best served by an interpreter global variable. The first was to track the -x command-line switch. That is replaced with a local variable in S_parse_body(). The second was to track whether the lexer is in the middle of a =pod section. That is replaced with an element in PL_parser.
* Store the PL_compcv instead of the the PL_comppad in parser stack, and make ↵Gerard Goossen2009-12-161-1/+1
| | | | it reference counted. Properly solves [perl #66094]
* lexer APIZefram2009-11-151-0/+4
| | | | | | | | | Attached is a patch that adds a public API for the lowest layers of lexing. This is meant to provide a solid foundation for the parsing that Devel::Declare and similar modules do, and it complements the pluggable keyword mechanism. The API consists of some existing variables combined with some new functions, all marked as experimental (which making them public certainly is).
* Add editor blocks to some header files.Marcus Holland-Moritz2008-01-011-1/+9
| | | p4raw-id: //depot/perl@32793
* Re-order struct yy_parser to save space on most systems.Nicholas Clark2007-07-161-3/+2
| | | | | Re-order struct yy_stack_frame to save space on LP64 systems. p4raw-id: //depot/perl@31618
* change #31615 added the new field in the wrong placeDave Mitchell2007-07-161-1/+1
| | | | | p4raw-link: @31615 on //depot/perl: 503de4705ff6537018ae94e9179e16636748b2a6 p4raw-id: //depot/perl@31616
* [perl #43425] local $[: fix scoping during parser error handling.Dave Mitchell2007-07-161-0/+1
| | | | | | | | | Change 22306# inadvertently made 'local $[' statement-scoped rather than block-scoped; so revert that change and add a different fix. The problem was to ensure that the savestack got popped correctly while popping errored tokens. We how record the current value of PL_savestack_ix with each pushed parser state. p4raw-id: //depot/perl@31615
* move PL_error_count into the PL_parser structDave Mitchell2007-05-211-0/+1
| | | p4raw-id: //depot/perl@31255
* move PL_multi_end into the PL_parser structDave Mitchell2007-05-211-0/+2
| | | p4raw-id: //depot/perl@31254
* move PL_tokenbuf into the PL_parser structDave Mitchell2007-05-211-0/+1
| | | p4raw-id: //depot/perl@31252
* move PL_in_my and PL_in_my_stash into the PL_parser structDave Mitchell2007-05-121-1/+4
| | | p4raw-id: //depot/perl@31203
* save old PL_curcop value in parser structDave Mitchell2007-05-121-0/+2
| | | p4raw-id: //depot/perl@31201
* move PL_rsfp_filters into the parser structDave Mitchell2007-05-121-0/+1
| | | p4raw-id: //depot/perl@31200
* move PL_rsfp into the PL_parser structDave Mitchell2007-05-111-0/+1
| | | | | and simplify its creation and destruction p4raw-id: //depot/perl@31199
* move PL_lex_state into the PL_parser structDave Mitchell2007-05-051-0/+1
| | | p4raw-id: //depot/perl@31154
* migrate more variables to PL_parser struct:Dave Mitchell2007-05-051-0/+9
| | | | | PL_nexttoke PL_curforce PL_nextval PL_nexttype p4raw-id: //depot/perl@31148
* move some more variables into the PL_parser struct:Dave Mitchell2007-05-041-0/+8
| | | | | | | PL_bufptr PL_oldbufptr PL_oldoldbufptr PL_linestart PL_bufend PL_last_uni PL_last_lop PL_last_lop_op p4raw-id: //depot/perl@31147
* move PL_expect and PL_copline into the PL_parser structureDave Mitchell2007-05-041-0/+2
| | | p4raw-id: //depot/perl@31134
* move PL_linestr from the interpreter struct to the parser structDave Mitchell2007-04-241-0/+1
| | | p4raw-id: //depot/perl@31058
* Several members of struct yy_parser can go on a diet. Some I32s wereNicholas Clark2007-04-211-6/+6
| | | | | actually only holding chars. p4raw-id: //depot/perl@31015
* Rearrange members of structures to reduce memory size on someNicholas Clark2007-03-311-2/+2
| | | | | | platforms. On LP64 structs stackinfo, refcounted_he, and magic shrink by 8 bytes, struct yy_parser by 16. p4raw-id: //depot/perl@30817
* Update copyright years to include 2007. (Plus a couple of 2006s andNicholas Clark2007-01-021-1/+1
| | | | | earlier we missed in av.h and hv.h) p4raw-id: //depot/perl@29670
* move easy lexer state variables into PL_parserDave Mitchell2007-01-021-0/+41
| | | | | (where "easy" == "only appear in toke.c") p4raw-id: //depot/perl@29655
* split parser initialisation from parser executionDave Mitchell2007-01-011-0/+2
| | | p4raw-id: //depot/perl@29652
* split parser stack from parser objectDave Mitchell2007-01-011-2/+3
| | | p4raw-id: //depot/perl@29649
* move parser state into new parser object, PL_parserDave Mitchell2006-12-181-0/+34
p4raw-id: //depot/perl@29570
View Lorry Controller Status