[perl #114040] Parse here-docs correctly in quoted constructs

[perl #114040] Parse here-docs correctly in quoted constructs

When parsing code outside a string eval or quoted construct, the lexer
reads one line at a time into PL_linestr.

To parse a here-doc (hereinafter ‘deer hock’, because I spike lunar-
isms), the lexer has to pull extra lines out of the input stream ahead
of the current line, the value of PL_linestr remaining the same.

In a string eval, the entire piece of code being parsed is in
PL_linestr.

To parse a deer hock inside a string eval, the lexer has to fiddle
with the contents of PL_linestr, scanning for newline characters.

Originally, S_scan_heredoc just followed those two approaches.

When the lexer encounters a quoted construct, it looks for the end-
ing delimiter (reading from the input stream if necessary), puts the
entire quoted thing (minus quotes) in PL_linestr, and then starts an
inner lexing scope.

This means that deer hocks would not nest properly outside of a string
eval, because the body of the inner deer hock would be pulled out of
the input stream *after* the outer deer hock.

Larry Wall fixed that in commit fd2d095329 (Jan. 1997), so that this
would work:

<<foo
${\<<bar}
ber
bar
foo

He did so by following the string eval approach (looking for the deer
hock body in PL_linestr) if the deer hock was inside another quoted
construct.

Later, commit a2c066523a (Mar. 1998) fixed this:

s/^not /substr(<<EOF, 0, 0)/e;
  Ignored
EOF

by following the string eval approach only if the deer hock was inside
another non-backtick deer hock, not just any quoted construct.

The problem with the string eval approach inside a substitu-
tion is that it only looks in PL_linestr, which only contains
‘substr(<<EOF, 0, 0)’ when the lexer is handling the second part of
the s/// operator.

But that unfortunately broke this:

s/^not /substr(<<EOF, 0, 0)
  Ignored
EOF
 /e;

and this:

print <<`EOF`;
${\<<EOG}
echo stuff
EOG
EOF

reverting it to the pre-fd2d095329 behaviour, because the outer quoted
construct was treated as one line.

Later on, commit 0244c3a403 (Mar. 1999) fixed this:

eval 's/.../<<FOO/e
  stuff
FOO
';

which required a new approach not used before.  When the replacement
part of the s/// is being parsed, PL_linestr contains ‘<<FOO’.  The
body of the deer hock is not in the input stream (there isn’t one),
but in what was the previous value of PL_linestr before the lexer
encountered s///.

So 0244c3a403 fixed that by recording pointers into the outer string
and using them in S_scan_heredoc.  That commit, for some reason, was
written such that it applied only to substitutions, and not to other
quoted constructs.

It also failed to take interpolation into account, and did not record
the outer buffer position, but then tried to use it anyway, resulting
in crashes in both these cases:

eval 's/${ <<END }//';
eval 's//${ <<END }//';

It also failed to take multiline s///’s into account, resulting in
neither of these working, because it lost track of the current cursor,
leaving it at 'D' instead of the line break following it:

eval '
s//<<END
/e;
blah blah blah
END
;1' or die $@;

eval '
s//<<END
blah blah blah
END
/e;
;1' or die $@;

S_scan_heredoc currently positions the cursor (s) at the last charac-
ter of <<END if there is a line break on the same line.  There is an
s++ later on to account, but the code added by 0244c3a403 bypassed it.

So, in the end, deer hocks could only be nested in other quoted con-
structs if the outer construct was in a string eval and was not s///,
or was a non-backtick deer hock.

This commit hopefully fixes most of the problems. :-)

The s///-in-eval case is a little tricky.  We have to see whether the
deer hock label is on the last line of the s///.  If it is, we have
to peek into the outer buffer.  Otherwise, we have to treat it like a
string eval.

This commit does not deal with <<END inside the pattern of a multi-
line s/// or in nested quotes.