Don’t let format arguments ‘leak out’ of formline

When parsing formats, the lexer invents tokens to feed to the parser.

So when the lexer dissects this:

format =
@<<<< @>>>>
$foo, $bar, $baz
.

The parser actually sees this (the parser knows that = . is like { }):

format =
; formline "@<<<< @>>>>\n", $foo, $bar, $baz;
.

The lexer makes no effort to make sure that the argument line is con-
tained within formline’s arguments.  To make

{ do_stuff; $foo, bar }

work, the lexer supplies a ‘do’ before the block, if it is
inside a format.

This means that

$a, $b; $c, $d

feeds ($a, $b) to formline, wheras

{ $a, $b; $c, $d }

feeds ($c, $d) to formline.  It also has various other
strange effects:

This script prints "# 0" as I would expect:

print "# ";
format =
@
(0 and die)
.
write

This one, locking parentheses, dies because ‘and’ has low precedence:

print "# ";
format =
@
0 and die
.
write

This does not work:

my $day = "Wed";
format =
@<<<<<<<<<<
({qw[ Sun 0 Mon 1 Tue 2 Wed 3 Thu 4 Fri 5 Sat 6 ]}->{$day})
.
write

You have to do this:

my $day = "Wed";
format =
@<<<<<<<<<<
({my %d = qw[ Sun 0 Mon 1 Tue 2 Wed 3 Thu 4 Fri 5 Sat 6 ]; \%d}->{$day})
.
write

which is very strange and shouldn’t even be valid syntax.

This does not work, because ‘no’ is not allowed in an expression:

use strict;
$::foo = "bar"
format =
@<<<<<<<<<<<
no strict; $foo
.
write;

Putting a block around it makes it work.  Putting a semicolon before
‘no’ stop it from being a syntax error, but it silently does the
wrong thing.

I thought I could fix all these by putting an implicit do { ... }
around the argument line and removing the special-casing for an open-
ing brace, allowing anonymous hashrefs to work in formats, such
that this:

format =
@<<<< @>>>>
$foo, $bar, $baz
.

would turn into this:

format =
; formline "@<<<< @>>>>\n", do { $foo, $bar, $baz; };
.

But that will lead to madness like this ‘working’:

format =
@
}+do{
.

It would also stop lexicals declared in one format line from being
visible in another.

So instead this commit starts being honest with the parser.  We still
have some ‘invented’ tokens, to indicate the start and end of a format
line, but now it is the parser itself that understands a sequence of
format lines, instead of being fed generated code.

So the example above is now presented to the parser like this:

format = ; FORMRBRACK
"@<<<< @>>>>\n" FORMLBRACK $foo, $bar, $baz ; FORMRBRACK
; .

Note about the semicolons:  The parser expects to see a semicolon at
the end of each statement.  So the lexer has to supply one before
FORMRBRACK.  The final dot goes through the same code that handles
closing braces, which generates a semicolon for the same reason.  It’s
easier to make the parser expect a semicolon before the final dot than
to change the } code in the lexer.  We use the } code for . because it
handles the internal variables that keep track of how many nested lev-
els there, what kind, etc.

The extra ;FORMRBRACK after the = is there also to keep the lexer sim-
ple (ahem).  When a newline is encountered during ‘normal’ (as opposed
to format picture) parsing inside a format, that’s when the semicolon
and FORMRBRACK are emitted.  (There was already a semicolon there
before this commit.  I have just added FORMRBRACK in the same spot.)

1 files changed, 49 insertions, 45 deletions

diff --git a/perly.h b/perly.h
index 29c317c3c4..daa777be2f 100644
--- a/perly.h
+++ b/perly.h
@@ -103,28 +103,30 @@
      MYSUB = 312,
      REQUIRE = 313,
      COLONATTR = 314,
-     PREC_LOW = 315,
-     DOROP = 316,
-     OROP = 317,
-     ANDOP = 318,
-     NOTOP = 319,
-     ASSIGNOP = 320,
-     DORDOR = 321,
-     OROR = 322,
-     ANDAND = 323,
-     BITOROP = 324,
-     BITANDOP = 325,
-     SHIFTOP = 326,
-     MATCHOP = 327,
-     REFGEN = 328,
-     UMINUS = 329,
-     POWOP = 330,
-     POSTDEC = 331,
-     POSTINC = 332,
-     PREDEC = 333,
-     PREINC = 334,
-     ARROW = 335,
-     PEG = 336
+     FORMLBRACK = 315,
+     FORMRBRACK = 316,
+     PREC_LOW = 317,
+     DOROP = 318,
+     OROP = 319,
+     ANDOP = 320,
+     NOTOP = 321,
+     ASSIGNOP = 322,
+     DORDOR = 323,
+     OROR = 324,
+     ANDAND = 325,
+     BITOROP = 326,
+     BITANDOP = 327,
+     SHIFTOP = 328,
+     MATCHOP = 329,
+     REFGEN = 330,
+     UMINUS = 331,
+     POWOP = 332,
+     POSTDEC = 333,
+     POSTINC = 334,
+     PREDEC = 335,
+     PREINC = 336,
+     ARROW = 337,
+     PEG = 338
    };
 #endif
 /* Tokens.  */
@@ -185,28 +187,30 @@
 #define MYSUB 312
 #define REQUIRE 313
 #define COLONATTR 314
-#define PREC_LOW 315
-#define DOROP 316
-#define OROP 317
-#define ANDOP 318
-#define NOTOP 319
-#define ASSIGNOP 320
-#define DORDOR 321
-#define OROR 322
-#define ANDAND 323
-#define BITOROP 324
-#define BITANDOP 325
-#define SHIFTOP 326
-#define MATCHOP 327
-#define REFGEN 328
-#define UMINUS 329
-#define POWOP 330
-#define POSTDEC 331
-#define POSTINC 332
-#define PREDEC 333
-#define PREINC 334
-#define ARROW 335
-#define PEG 336
+#define FORMLBRACK 315
+#define FORMRBRACK 316
+#define PREC_LOW 317
+#define DOROP 318
+#define OROP 319
+#define ANDOP 320
+#define NOTOP 321
+#define ASSIGNOP 322
+#define DORDOR 323
+#define OROR 324
+#define ANDAND 325
+#define BITOROP 326
+#define BITANDOP 327
+#define SHIFTOP 328
+#define MATCHOP 329
+#define REFGEN 330
+#define UMINUS 331
+#define POWOP 332
+#define POSTDEC 333
+#define POSTINC 334
+#define PREDEC 335
+#define PREINC 336
+#define ARROW 337
+#define PEG 338
 
 
 
@@ -264,6 +268,6 @@ typedef union YYSTYPE
 
 
 /* Generated from:
- * 50649beb21bb272a5dafa43d5fbe244206c6a99e0e67b39b3123a22a35702873 perly.y
+ * 580d1bac82838fb0ae1e092010e776fb247f0d4cb68834a166507fae86707700 perly.y
  * 4a3ced51fe6585f6ec59e65e1ffd74bc976c498b0e6992eaab337219fee72e51 regen_perly.pl
  * ex: set ro: */