Don’t let format arguments ‘leak out’ of formline

Don’t let format arguments ‘leak out’ of formline

When parsing formats, the lexer invents tokens to feed to the parser.

So when the lexer dissects this:

format =
@<<<< @>>>>
$foo, $bar, $baz
.

The parser actually sees this (the parser knows that = . is like { }):

format =
; formline "@<<<< @>>>>\n", $foo, $bar, $baz;
.

The lexer makes no effort to make sure that the argument line is con-
tained within formline’s arguments.  To make

{ do_stuff; $foo, bar }

work, the lexer supplies a ‘do’ before the block, if it is
inside a format.

This means that

$a, $b; $c, $d

feeds ($a, $b) to formline, wheras

{ $a, $b; $c, $d }

feeds ($c, $d) to formline.  It also has various other
strange effects:

This script prints "# 0" as I would expect:

print "# ";
format =
@
(0 and die)
.
write

This one, locking parentheses, dies because ‘and’ has low precedence:

print "# ";
format =
@
0 and die
.
write

This does not work:

my $day = "Wed";
format =
@<<<<<<<<<<
({qw[ Sun 0 Mon 1 Tue 2 Wed 3 Thu 4 Fri 5 Sat 6 ]}->{$day})
.
write

You have to do this:

my $day = "Wed";
format =
@<<<<<<<<<<
({my %d = qw[ Sun 0 Mon 1 Tue 2 Wed 3 Thu 4 Fri 5 Sat 6 ]; \%d}->{$day})
.
write

which is very strange and shouldn’t even be valid syntax.

This does not work, because ‘no’ is not allowed in an expression:

use strict;
$::foo = "bar"
format =
@<<<<<<<<<<<
no strict; $foo
.
write;

Putting a block around it makes it work.  Putting a semicolon before
‘no’ stop it from being a syntax error, but it silently does the
wrong thing.

I thought I could fix all these by putting an implicit do { ... }
around the argument line and removing the special-casing for an open-
ing brace, allowing anonymous hashrefs to work in formats, such
that this:

format =
@<<<< @>>>>
$foo, $bar, $baz
.

would turn into this:

format =
; formline "@<<<< @>>>>\n", do { $foo, $bar, $baz; };
.

But that will lead to madness like this ‘working’:

format =
@
}+do{
.

It would also stop lexicals declared in one format line from being
visible in another.

So instead this commit starts being honest with the parser.  We still
have some ‘invented’ tokens, to indicate the start and end of a format
line, but now it is the parser itself that understands a sequence of
format lines, instead of being fed generated code.

So the example above is now presented to the parser like this:

format = ; FORMRBRACK
"@<<<< @>>>>\n" FORMLBRACK $foo, $bar, $baz ; FORMRBRACK
; .

Note about the semicolons:  The parser expects to see a semicolon at
the end of each statement.  So the lexer has to supply one before
FORMRBRACK.  The final dot goes through the same code that handles
closing braces, which generates a semicolon for the same reason.  It’s
easier to make the parser expect a semicolon before the final dot than
to change the } code in the lexer.  We use the } code for . because it
handles the internal variables that keep track of how many nested lev-
els there, what kind, etc.

The extra ;FORMRBRACK after the = is there also to keep the lexer sim-
ple (ahem).  When a newline is encountered during ‘normal’ (as opposed
to format picture) parsing inside a format, that’s when the semicolon
and FORMRBRACK are emitted.  (There was already a semicolon there
before this commit.  I have just added FORMRBRACK in the same spot.)