Max Maischein [Thu, 3 Oct 2013 16:48:27 +0000 (18:48 +0200)]
Elide use of `grep`
The new approach uses more memory as it reads the whole
11 MB CPAN packages file into memory. Also, it makes
less use of parallel multiprocessing now available on
many machines.
Max Maischein [Sat, 5 Oct 2013 18:01:44 +0000 (20:01 +0200)]
Run correct make
Use Config.pm to determine correct kind of `make` tool
For Windows, the make process is supposed to get kicked off
in ./Win32.
We now run `make test-prep` before trying the module self-tests
Max Maischein [Sat, 5 Oct 2013 17:59:11 +0000 (19:59 +0200)]
Elide use of `chmod`
We simplify life here and try to set Porting/Makefile.PL
always to mode 755 (u=rwx , ao=rx) instead of being
more precise.
Max Maischein [Thu, 3 Oct 2013 16:38:36 +0000 (18:38 +0200)]
Elide use of `ls`, `find` and `touch`
File::Find and code copied from ExtUtils::Command do the same.
This patch makes the code to remove the exec bit a little less
efficient by not restricting itself to files that now have
the exec bit set, but instead looking at all files.
This change also uses $ENV{TEMP} in preference to
/tmp as a temp directory for caching the CPAN index.
Max Maischein [Thu, 3 Oct 2013 16:17:39 +0000 (18:17 +0200)]
Elide use of `rm`
We remove the reliance on an external 'rm' tool in
favour of using File::Path::remove_tree.
We also eliminate some dir-changing by using
more absolute directory names.
Max Maischein [Tue, 21 May 2013 17:11:22 +0000 (19:11 +0200)]
First try with HTTP::Tiny, fallback on `wget`
This tries downloads first with the built-in HTTP::Tiny
before it falls back to the external `wget` tool. Arguably,
the reliance on `wget` could be eliminated to reduce the
amount of code clutter.
Ricardo Signes [Sat, 5 Oct 2013 19:26:40 +0000 (15:26 -0400)]
correct documentation of postfix glob deref
thanks, mst
Ricardo Signes [Sat, 5 Oct 2013 18:58:46 +0000 (14:58 -0400)]
perldelta: experimental postfix dereference syntax
Ricardo Signes [Sat, 5 Oct 2013 18:41:27 +0000 (14:41 -0400)]
Merge branch 'postderef' into blead
Ricardo Signes [Sat, 5 Oct 2013 18:41:04 +0000 (14:41 -0400)]
tweak postfix deref docs, update for glob deref
Ricardo Signes [Wed, 2 Oct 2013 16:08:49 +0000 (12:08 -0400)]
new warnings category, so bump warnings.pm
Ricardo Signes [Fri, 27 Sep 2013 02:44:00 +0000 (22:44 -0400)]
preliminary postfix dereference docs
This commit adds an overview of the feature to perlref and a pointer
to the section in perlref to perlop's documentation of the arrow.
If/when this feature becomes non-experimental, the documentation
should be merged upward into Using References.
This documentation was written against a previous state of the
branch. Is should be fact-checked before any merge.
Father Chrysostomos [Mon, 16 Sep 2013 19:49:12 +0000 (12:49 -0700)]
Fix a bug with ->@ inside "@{...}"
When encountering ->@[ or ->@{, we should only record that we are
going to need a POSTJOIN at the top level of interpolation, not
inside any brackets. Otherwise the ->@[ can interfere with an outer
"@{...}", causing syntax errors.
Father Chrysostomos [Mon, 16 Sep 2013 16:57:13 +0000 (09:57 -0700)]
Make postderef experimental
Father Chrysostomos [Mon, 16 Sep 2013 16:44:34 +0000 (09:44 -0700)]
Allow ->@ ->$ interpolation under postderef_qq feature
This turned out to be tricky. Normally @ at the beginning of the
interpolated code signals to the lexer to emit ‘join($",’ immediately.
With "$_->@*" we would have to retract the $ _ -> tokens upon encoun-
tering @*, which we obviously cannot do.
Waiting until we reach the end of the interpolated text before emit-
ting anything could not work either, as it may contain BEGIN blocks
that affect the way part of the interpolated code is parsed.
So what we do is introduce an egregious or clever hack, depending on
how you look at it.
Normally, the lexer turns "@foo" into:
stringify ( join ( $ " , @ foo ) )
(The " is a WORD token, representing a variable name.)
"$_" becomes:
stringify ( $ _ )
We can turn "$_->@*" into:
stringify ( $ _ -> @ * POSTJOIN )
Where POSTJOIN is a new lexer token with special handling that creates
a join op just the way join($", ...) does.
To make "foo$_->@*bar" work as well, we have to make POSTJOIN have
precedence just below ->, so that
stringify ( "foo" . $ _ -> @ * POSTJOIN . "bar" )
(what the parser sees) is equivalent to:
stringify ( "foo" . ( $ _ -> @ * POSTJOIN ) . "bar" )
Father Chrysostomos [Sun, 15 Sep 2013 23:31:19 +0000 (16:31 -0700)]
Add postderef_qq feature feature
Father Chrysostomos [Sun, 15 Sep 2013 23:29:21 +0000 (16:29 -0700)]
Hide postderef behind the feature feature
Father Chrysostomos [Sun, 15 Sep 2013 23:22:41 +0000 (16:22 -0700)]
toke.c:postderef: rmv obsolete comment
Father Chrysostomos [Sun, 15 Sep 2013 23:21:54 +0000 (16:21 -0700)]
Increase $feature::VERSION to 1.34
Father Chrysostomos [Sun, 15 Sep 2013 23:20:58 +0000 (16:20 -0700)]
Add postderef feature feature
Father Chrysostomos [Sun, 15 Sep 2013 23:07:11 +0000 (16:07 -0700)]
->%{ ->%[
Father Chrysostomos [Wed, 14 Aug 2013 01:12:58 +0000 (18:12 -0700)]
Fix up postfixderef.t
Father Chrysostomos [Tue, 13 Aug 2013 21:22:20 +0000 (14:22 -0700)]
Add postfixderef.t to MANIFEST
David Nicol [Tue, 13 Aug 2013 21:22:04 +0000 (14:22 -0700)]
Tests for postfix dereference
Father Chrysostomos [Fri, 5 Jul 2013 08:10:08 +0000 (01:10 -0700)]
Postfix dereference syntax
$_->$* means $$_ (and compiled down to the same op tree)
$_->@* means @$_ ( ditto ditto blah blah blah )
$_->%* means %$_ (...)
$_->&* means &$_
$_->** means *$_
$_->@[...] means @$_[...]
$_->@{...} means @$_{...}
$_->*{...} means *$_{...}
$_->@* is not always equivalent to @$_, particularly in contexts like
@foo[0], which cannot be written foo->@*[0]. (Just omit the asterisk
and it works.)
James E Keenan [Sat, 5 Oct 2013 13:00:38 +0000 (15:00 +0200)]
Remove pod/perlembed.pod in wake of RT #120093 corrections.
Provide author's preferred email address.
Chris 'BinGOs' Williams [Sat, 5 Oct 2013 10:47:14 +0000 (11:47 +0100)]
Update Unicode-Normalize to CPAN version 1.17
[DELTA]
1.17 Sat Oct 5 11:36:43 2013
- assertion using unpack
Karl Williamson [Sat, 5 Oct 2013 03:15:01 +0000 (21:15 -0600)]
mktables: Verify input files' version
Since Unicode 3.2, all Unicode database source files (except
unfortunately, the most important one, UCD.txt), have their first lines
be identifying information, their name and version number. This commit
checks that the version is the expected one. This should prevent the
database from being out-of-sync. Perl changes the names of some files
so that they are distinct on DOS filesystems, so we can't easily check
that the name in the file is the same as the name of the file.
Karl Williamson [Sat, 5 Oct 2013 03:05:09 +0000 (21:05 -0600)]
Bump Unicode version
In commit
a9c9e371c40cf388593577cf577494e91793f62a, I forgot to update
the Unicode version in the file that states it.
Gideon Israel Dsouza [Thu, 3 Oct 2013 17:03:14 +0000 (22:33 +0530)]
Fixed 27 issues in perlembed.pod reported by podcheck
Fixed mostly lines exceeding 80 characters by decreasing leading
space in verbatim lines to one space. Also fixed long code lines
with a slash and line break.
Add Gideon Israel Dsouza to list of Perl authors.
For: RT #120093
Ricardo Signes [Fri, 4 Oct 2013 14:42:49 +0000 (10:42 -0400)]
perlfunc: replace splice example with simpler one
based on suggestions from Nicholas Clark and David Golden on
perl5-porters
Nicholas Clark [Fri, 4 Oct 2013 13:51:13 +0000 (15:51 +0200)]
Remove unused variable 'mark' from pp_last.
mark is unused since commit
0c0c317c3b754aee,
"[perl #3112] Stop last from returning values"
Karl Williamson [Tue, 1 Oct 2013 15:26:40 +0000 (09:26 -0600)]
Upgrade to Unicode 6.3
Karl Williamson [Fri, 4 Oct 2013 04:37:51 +0000 (22:37 -0600)]
mktables: Fix logic with binary vs enum properties
The code was confused about what certain variables signified, and raises
erroneous warnings at other times. These bugs did not show up until
compiling Unicode 6.3.
Karl Williamson [Fri, 4 Oct 2013 04:06:29 +0000 (22:06 -0600)]
mktables: Do some table-driven code generation
The Unicode Character Database consists of many files in various
different formats. mktables has a single routine that processes the
most common format type. Files with different formats are run through
filters to transform them into this format, so that almost all end up
being handles by this common function.
This commit adds a way of specifying the format for one of the other
format types, and then automatically generating the code to do the
transformation. This doesn't work if the file has lines that have
special cases, such as if there is a known typo in it; the current
scheme can be used for those.
Unfortunately, all but one of the candidate files in Unicode 6.2 aren't
suitable for this table-driven approach. But a second one is coming in
6.3, and I anticipate more in the future, as Unicode has tightened their
quality control significantly in recent releases.
Karl Williamson [Fri, 4 Oct 2013 03:52:14 +0000 (21:52 -0600)]
perluniprops: Add correct ignored files docs
Unicode furnishes various files that Perl ignores. perluniprops lists
these, with a brief reason of what they are for and why they aren't used
by Perl. Two files weren't listed, and one had a typo in the name and
an inadequate description.
Karl Williamson [Tue, 1 Oct 2013 04:41:51 +0000 (22:41 -0600)]
lib/unicore/README.perl: Update
This changes this to conform to changes in Unicode 6.2
Steve Hay [Thu, 3 Oct 2013 12:11:26 +0000 (13:11 +0100)]
Porting/corelist.pl - Fix breakage since commit
c676c8385f
Steve Hay [Thu, 3 Oct 2013 08:04:57 +0000 (09:04 +0100)]
Porting/Maintainers.pl - No need to list IGNORABLES as EXCLUDED
Steve Hay [Thu, 3 Oct 2013 07:58:10 +0000 (08:58 +0100)]
Porting/Maintainers.pl - Consistent quotes
Steve Hay [Thu, 3 Oct 2013 07:56:56 +0000 (08:56 +0100)]
Porting/Maintainers.pl - Use PAUSE user IDs
Since we now encourage the use of the DISTRIBUTION field to find the PAUSE
user ID of the maintainer rather than explicitly listing most maintainers,
it makes sense to also use PAUSE user IDs where possible for the few
MAINTAINERS that are still given explicitly.
Steve Hay [Wed, 2 Oct 2013 17:16:33 +0000 (18:16 +0100)]
Porting/Maintainers.pl - Remove most MAINTAINER fields
Remove the MAINTAINER field if it was 'p5p' (or 'blead', which was the case for
Unicode::UCD, presumably in error) or if there is a DISTRIBUTION field. Only a
few MAINTAINER fields are now left, for modules with non-'p5p' maintainers but
no CPAN distribution. For modules now having no MAINTAINER field, the maintainer
is understood to be either the Perl 5 Porters if there is no DISTRIBUTION field
or the UPSTREAM field is 'blead', or else the CPAN author whose PAUSE user ID
forms the first part of the DISTRIBUTION value.
Set up a legacy MAINTAINER field for backwards compatibility with existing
functionality in Porting/Maintainers, Porting/corecpan.pl and
Porting/corelist.pl.
The latter program now omits modules such as Attribute::Handlers from %upstream
and %bug_tracker in CoreList.pm because the MAINTAINER is now understood to be
'p5p' since the UPSTREAM field is 'blead'. Previously it was included because a
non-'p5p' MAINTAINER was given, but that was really at odds with UPSTREAM being
'blead'.
Ricardo Signes [Thu, 3 Oct 2013 01:11:35 +0000 (21:11 -0400)]
perlexperiment: add rt.perl.org link for lexsubs
David Golden [Wed, 2 Oct 2013 18:03:29 +0000 (14:03 -0400)]
perlfunc.pod: clarify return value of -X [perl #120077]
Craig A. Berry [Wed, 2 Oct 2013 02:43:13 +0000 (21:43 -0500)]
Fix ext/File-Glob/t/threads.t filenames for VMS.
As noted in
http://www.nntp.perl.org/group/perl.perl5.porters/2013/09/msg208134.html
zero-length extensions always get a trailing dot on VMS, and the
easiest workaround is to always use an explicit extension.
Chris 'BinGOs' Williams [Tue, 1 Oct 2013 12:15:39 +0000 (13:15 +0100)]
Regen META files for CPAN::Meta update
Chris 'BinGOs' Williams [Tue, 1 Oct 2013 12:14:11 +0000 (13:14 +0100)]
Update CPAN-Meta to CPAN version 2.132661
[DELTA]
2.132661 2013-09-23 13:27:46 America/New_York
[FIXED]
- updated Makefile.PL logic to support PERL_NO_HIGHLANDER
[PREREQS]
- Dropped ExtUtils::MakeMaker configure_requires dependency
to 6.17
2.132660 2013-09-23 06:04:04 America/New_York
[FIXED]
- Installation on Perls < 5.12 will uninstall older versions installed
due to being bundled with ExtUtils::MakeMaker
Chris 'BinGOs' Williams [Tue, 1 Oct 2013 12:12:33 +0000 (13:12 +0100)]
Update CPAN-Meta-Requirements to CPAN version 2.125
[DELTA]
2.125 2013-09-23 14:51:14 America/New_York
[FIXED]
- updated Makefile.PL logic to support PERL_NO_HIGHLANDER
2.124 2013-09-23 06:40:07 America/New_York
[FIXED]
- Better fix than 2.123. On old perls, rather than install into
'core', we continue to install into the proper 'site' library,
but force UNINST=1 when necessary to remove stale copies
from ExtUtils::MakeMaker
Chris 'BinGOs' Williams [Tue, 1 Oct 2013 12:10:22 +0000 (13:10 +0100)]
Update Parse-CPAN-Meta to CPAN version 1.4409
[DELTA]
1.4409 2013-09-23 13:17:18 America/New_York
[FIXED]
- updated Makefile.PL logic to support PERL_NO_HIGHLANDER
1.4408 2013-09-23 11:06:35 America/New_York
[FIXED]
- Installation on Perls < 5.12 will uninstall older versions installed
due to being bundled with ExtUtils::MakeMaker
Chris 'BinGOs' Williams [Tue, 1 Oct 2013 12:08:33 +0000 (13:08 +0100)]
Update CPAN-Meta-YAML to CPAN version 0.010
[DELTA]
0.010 2013-09-23 13:11:20 America/New_York
- Generated from ETHER/YAML-Tiny-1.55.tar.gz
- Fix broken test when copying from YAML-Tiny
- updated Makefile.PL logic to support PERL_NO_HIGHLANDER
0.009 2013-09-23 10:16:19 America/New_York
- Generated from ETHER/YAML-Tiny-1.55.tar.gz
- Makefile.PL will use UNINST=1 on old perls that might have
an old version incorrectly installed into the core library path
Chris 'BinGOs' Williams [Tue, 1 Oct 2013 12:06:43 +0000 (13:06 +0100)]
Update Getopt-Long to CPAN version 2.42
[DELTA]
Changes in version 2.42
-----------------------
* Fix bug https://rt.cpan.org/Ticket/Display.html?id=88707
Karl Williamson [Mon, 30 Sep 2013 20:13:57 +0000 (14:13 -0600)]
regcomp.c: Fix up confusing comment
This was discussed as part of [perl #120041].
Karl Williamson [Mon, 30 Sep 2013 20:06:23 +0000 (14:06 -0600)]
PATCH: [perl #120041] regcomp.c missing parens and broken STCLASS
This was caused by a problem in commit
43a64b8b430604bd4d76fd256a5babdccaf0ab2b
which was masked by problems in commit
a0dd42312a1f26356d2fdf49656e45b77c2cefb5.
The first commit deleted a line it shouldn't have. I thought that a
variable wasn't modified before reaching that line, but it turns out it
was because another variable is set to point to its address, and was
modified inside a subroutine call. The other two lines deleted in the
first commit are ok to have deleted.
The second commit masked the problems of the first by omitting some
necessary grouping parentheses, which caused things to not work as
intended, and we didn't have a good test case in our suite to find this
problem. (Thanks to Lukas Mai for spotting the problem and submitting a
test case.)
Karl Williamson [Mon, 30 Sep 2013 15:57:55 +0000 (09:57 -0600)]
Revert "Optimise if/unless wrt OP_AND/OP_OR/OP_DOR. Also optimise OP_OR/OP_DOR chains."
This reverts commit
15a4d87479c14a0808c36a762bcd182890b84815
at the submitter's request.
Karl Williamson [Sat, 28 Sep 2013 18:06:35 +0000 (12:06 -0600)]
utf8.c: Silence Win32 compiler warnings
The Win32 compiler doesn't realize that the values in these places can
be a max of 255. Other compilers don't warn.
Karl Williamson [Sat, 28 Sep 2013 18:06:07 +0000 (12:06 -0600)]
utf8.h: White-space only
I believe this makes the macro easier to read
Ricardo Signes [Mon, 30 Sep 2013 02:14:55 +0000 (22:14 -0400)]
standardize perlre cross-refs to operator-based flags
Ricardo Signes [Mon, 30 Sep 2013 01:51:30 +0000 (21:51 -0400)]
reword the description of what the /m flag does
Ricardo Signes [Mon, 30 Sep 2013 01:47:28 +0000 (21:47 -0400)]
slightly clarify the meaning of $ in regex
This patch was suggested in #116773.
Chris 'BinGOs' Williams [Sun, 29 Sep 2013 09:35:13 +0000 (10:35 +0100)]
Update Perl-OSType to CPAN version 1.006
[DELTA]
1.006 2013-09-25 13:57:38 America/New_York
[FIXES]
- Compile test could hang on Windows
[PREREQS]
- Dropped configure_requires for ExtUtils::MakeMaker to 6.17
Chris 'BinGOs' Williams [Sun, 29 Sep 2013 09:32:41 +0000 (10:32 +0100)]
Update File-Temp to CPAN version 0.2302
[DELTA]
0.2302 2013-09-26 09:45:35 America/New_York
* Drop minimum Perl version back to 5.6 (erroneously bumped by dzil)
* Do not inherit from Exporter (requires Exporter 5.57) (thanks to
Olivier Mengué)
* 'use base ...' => 'use parent ...' as parent is lighter (thanks to
Olivier Mengué)
Chris 'BinGOs' Williams [Sun, 29 Sep 2013 09:25:33 +0000 (10:25 +0100)]
Update HTTP-Tiny to CPAN version 0.036
[DELTA]
0.036 2013-09-25 12:10:06 America/New_York
[FIXED]
- Compile test could hang on Windows
[PREREQS]
- Dropped configure_requires for ExtUtils::MakeMaker to 6.17
[META]
- Updated support files
Chris 'BinGOs' Williams [Sun, 29 Sep 2013 09:23:43 +0000 (10:23 +0100)]
Update File-Fetch to CPAN version 0.44
James E Keenan [Fri, 27 Sep 2013 23:53:07 +0000 (01:53 +0200)]
Increment version number for threads.
For: RT #120033
Jerry D. Hedden [Fri, 27 Sep 2013 18:36:15 +0000 (14:36 -0400)]
Upgrade to threads 1.88
Daniel Dragan [Thu, 26 Sep 2013 08:39:55 +0000 (04:39 -0400)]
better pod for threads::_handle
First time I tried to get the OS handle I wrote
unpack('P[Q || L /*PICK ONE*/]', $thread->_handle())
it crashed because _handle returns a number, not a packed string. unpack
wants a packed string. Mention the pointer is numeric and not to ever pass
the retval to unpack('P[.
Brian Fraser [Fri, 27 Sep 2013 17:30:32 +0000 (14:30 -0300)]
File::Glob::VERSION++
Brian Fraser [Sat, 21 Sep 2013 06:19:52 +0000 (03:19 -0300)]
File::Glob: Dup glob state in CLONE()
This solves [perl #119897] and [perl #117823], and restores the
behavior of glob() in conjunction with threads of 5.14 and older.
Since 5.16, code that used glob() inside a thread had been
unintentionally sharing state between threads, which lead to things
like this crashing and failing assertions:
./perl -Ilib -Mthreads -e 'scalar glob("*"); threads->create(sub { glob("*") })->join();'
Andy Dougherty [Wed, 25 Sep 2013 12:35:58 +0000 (08:35 -0400)]
Use a more reliable check for BSDish systems.
Look for BSD defined in <sys/param.h>, rather than keeping a manual list
of BSD derivatives.
Karl Williamson [Tue, 24 Sep 2013 18:49:56 +0000 (12:49 -0600)]
regcomp.c: Improve -Dr output
Sometimes sequences like [\w\s] were output as separate classes
[\w][\s]. And sometimes, an empty [] was output. This fixes those and
eliminates all but one call to a macro, which is hence now just in-lined.
Karl Williamson [Tue, 24 Sep 2013 18:27:22 +0000 (12:27 -0600)]
regcomp.c: Don't exceed array bounds
The upper limit of this loop was wrong. It should be the number of
Posix classes that have space stored for them. This caused a valgrind
error, but it's only when outputting a regular expression under, for
example, -Dr
Karl Williamson [Tue, 24 Sep 2013 18:11:36 +0000 (12:11 -0600)]
perldelta for regex optimizer handling above-Latin1
cdd87c1d4df41f9a54cccff996fa64d291adcee8
Karl Williamson [Tue, 24 Sep 2013 16:32:37 +0000 (10:32 -0600)]
regcomp.c: Move bit to different data structure
Commit
899d20b99829f8ecdc14e1351b533bc62a354dea was used to free up a
bit in a flags field that had run out of bits at the time. Further work
has made that unnecessary, and this commit moves it back to the flags
field, which even after this commit has a spare bit (which is intended
to be used in a future commit).
Doing so makes this bit "just one of the guys", so can be operated on
en-masse with the others. This allows a little code to be removed, and
the knowledge of this flag mostly confined to lower level subroutines.
Karl Williamson [Tue, 24 Sep 2013 16:25:45 +0000 (10:25 -0600)]
regcomp.c: Remove unnecessary flag settings
I've looked at the code around each of these flag sets, and the flag
should already be set when these are called, so they are redundant.
Karl Williamson [Tue, 24 Sep 2013 03:40:05 +0000 (21:40 -0600)]
Make 'OR's logically 'or'
The regex optimizer in several OR operations cleared the synthetic start
class (SSC) node from matching an empty string. This doesn't seem
right. The end result of an OR should be the same size or larger than
either of its inputs. I talked it over with Yves Orton, and we decided
that even though we don't have test cases that demonstrate problems with
the current behavior, we should do the logical thing. This change
should not cause code failures, as it only increases the things that the
SSC can match, perhaps these are false positives.
Karl Williamson [Mon, 23 Sep 2013 05:12:20 +0000 (23:12 -0600)]
regcomp.c: White-space, comments only
This moves the static functions introduced a few commits ago to more
logical places in the file, and wraps some long lines to 79 columns, and
a few nits in comments
Karl Williamson [Mon, 23 Sep 2013 04:56:20 +0000 (22:56 -0600)]
regcomp.c: Remove unused parameter in static function
This parameter is no longer used, since a few commits ago in this
series.
Karl Williamson [Mon, 23 Sep 2013 04:46:10 +0000 (22:46 -0600)]
Add some tests for the regex optimizer
We don't have the infrastructure to test the regex optimizer, and I'm
not sure how to do it properly, without tying the tests to particular
optimizations. What I did, however, was to go through the recently
changed optimizer code and write tests to exercise every branch, as far
as I could tell.
Karl Williamson [Mon, 23 Sep 2013 04:36:57 +0000 (22:36 -0600)]
regcomp.c: Tighten optimizer for /li matches
The synthetic start class (ssc) generated by the regex optimizer
frequently has case-sensitive matching enabled, even if nowhere in the
pattern is there a /i. This commit causes any pattern that doesn't have
/i to not have its ssc contain a /i.
Karl Williamson [Mon, 23 Sep 2013 03:36:29 +0000 (21:36 -0600)]
Teach regex optimizer to handle above-Latin1
Until this commit, the regular expression optimizer has essentially
punted on above-Latin1 code points. Under some circumstances, they
would be taken into account, more or less, but often, the generated
synthetic start class would end up matching all above-Latin1 code
points. With the advent of inversion lists, it becomes feasible to
actually fully handle such code points, as inversion lists are a
convenient way to express arbitrary lists of code points and take their
union, intersection, etc. This commit changes the optimizer to use
inversion lists for operating on the code points the synthetic start
class can match.
I don't much understand the overall operation of the optimizer. I'm
told that previous porters found that perturbing it caused unexpected
behaviors. I had promised to get this change in 5.18, but didn't. I'm
trying to get it in early enough into the 5.20 preliminary series that
any problems will surface before 5.20 ships.
This commit doesn't change the macro level logic, but does significantly
change various micro level things. Thus the 'and' and 'or' subroutines
have been rewritten to use inversion lists. I'm pretty confident that
they do what their names suggest. I re-derived the equations for what
these operations should do, getting the same results in some cases, but
extending others where the previous code mostly punted. The derivations
are given in comments in the respective routines.
Some of the code is greatly simplified, as it no longer has to treat
above-Latin1 specially.
It is now feasible for /i matching of above-Latin1 code points to know
explicitly the folds that should be in the synthetic start class. But
more prepatory work needs to be done before putting that into place.
...
Karl Williamson [Mon, 23 Sep 2013 02:43:02 +0000 (20:43 -0600)]
regcomp.c: Add some static functions
This commit adds some functions that are currently unused, but will be
used in a future commit. This commit is essentially to make the
differences smaller in that commit, as 'diff' is getting confused and
not outputting the logical differences. The functions are added in a
block at the beginning of the file to avoid the 'diff' issues. A later
white-space only commit will move them to more appropriate positions.
Karl Williamson [Tue, 10 Sep 2013 02:33:48 +0000 (20:33 -0600)]
regcomp.c: Use macro accessor uniformly
These instances were using the structure field directly; everywhere else
uses a macro that hides the field's location in the structure. This
converts to use the macro everywhere.
Karl Williamson [Sun, 15 Sep 2013 01:03:39 +0000 (19:03 -0600)]
regcomp.c: Optimize e.g. /[\w\W]/l into dot
This is an unlikely scenario for someone to include a Posix class and
its complement in the same bracketed character class, but looking for
this and optimizing it away helps the algorithm coming in a future
commit to look at the synthetic start class.
This commit only does this for /l matching. For all other matching, if
we know at compile time what the posix classes match, this optimization
is already done.
Karl Williamson [Fri, 6 Sep 2013 04:40:54 +0000 (22:40 -0600)]
Enlarge dummy regex pass1 compilation node
In pass 1 of compiling regular expressions, the needed size is
calculated. There is space allocated for a scratch node that can be
used for the things that the real one will hold in pass 2. It is valid
only while working on the current node, and gets overwritten in the next
node.
Until this commit, this scratch space was sized only for the smallest
node type, meaning that larger types could not use it for scratch. Now
it is sized to be the largest non EXACTish node.
We could make it an array of 256 + overhead bytes instead to be able to
hold the EXACTish nodes, but I don't see a need for that now.
Karl Williamson [Thu, 15 Aug 2013 21:27:08 +0000 (15:27 -0600)]
regcomp.c: Use STR_WITH_LEN to avoid bookkeeping
By changing the order of the parameters to the static function
S_add_data, we can call it with STR_WITH_LEN and avoid a human having to
count characters.
Karl Williamson [Thu, 15 Aug 2013 21:07:44 +0000 (15:07 -0600)]
Rename regex flag bit for clarity
ANYOF_UNICODE_ALL doesn't mean every Unicode code point. It means those
above the Latin1 range. Rename it, while retaining the old one for back
compat.
Karl Williamson [Thu, 15 Aug 2013 20:55:16 +0000 (14:55 -0600)]
regcomp.c: Better DEBUGGING builds error detection
The code had a default: catch-all in the switch statement, but the
comments indicated that it was uncertain what all was being caught.
This changes this to panic only in DEBUGGING builds so that we can find
out if there are indeed other possibilities that we haven't handled, and
which could use better handling than the default, match everything.
The two known possibilities are given separate case: statements in
preparation for handling them differently.
Karl Williamson [Thu, 15 Aug 2013 20:49:37 +0000 (14:49 -0600)]
regcomp.c: Change some static parameters to const
I found I needed const in a future commit.
Karl Williamson [Thu, 15 Aug 2013 20:27:53 +0000 (14:27 -0600)]
Retain an inversion list's mortality in its replacement
A couple of inversion list handling functions end up sometimes creating
a new inversion list, replacing the old one instead of modifying it.
This commit causes the replacement list to have the same mortality of
the old one. That is, mortality is now preserved across these
operations.
Karl Williamson [Thu, 15 Aug 2013 20:04:43 +0000 (14:04 -0600)]
perl.c: Clean up some SV*s at termination
These were omitted from cleaning up when PERL_DESTRUCT_LEVEL is non-zero
Karl Williamson [Thu, 15 Aug 2013 17:19:02 +0000 (11:19 -0600)]
regcomp.c: Add parameter to static function
This parameter will be used in future commits. This commit is really
only to make the difference listing smaller in those, by committing
separately just the book-keeping parts. This parameter requires also
passing the aTHX_ thread parameter
Karl Williamson [Thu, 15 Aug 2013 16:59:01 +0000 (10:59 -0600)]
Remove PL_ASCII; use existing array slots for it
PL_ASCII contains an inversion list to match the ASCII-range code
points. It is unusable outside the core regular expression code because
all the functions that manipulate inversion lists are defined only
within a few core files. Therefore no outside code should be depending
on it.
It turns out that there are arrays of similar inversion lists, and these
all have slots which should have this inversion list in them. This
commit fills them, instead of using PL_ASCII.
Karl Williamson [Thu, 15 Aug 2013 16:51:24 +0000 (10:51 -0600)]
regcomp.c: Typos in comments; Fix another comment
The non-typo fix is the result of allowing a parameter to the function
be NULL, and not updating the comments to reflect that.
Karl Williamson [Thu, 15 Aug 2013 16:39:14 +0000 (10:39 -0600)]
regcomp.c: Fix syntax error in #ifdef'd out code
This line is currently not compiled, but would fail if the #ifdef is
changed.
Karl Williamson [Thu, 15 Aug 2013 16:36:29 +0000 (10:36 -0600)]
perl.h: Don't pollute global namespace
These structures are used internally in the regular expression files,
and are declared here only because of #include ordering issues. Wrap
them in an #ifdef so only visible to the correct files.
Karl Williamson [Thu, 15 Aug 2013 03:13:52 +0000 (21:13 -0600)]
Make typedef fully typedef
The regcomp.c struct RExC_state_t has not been usable fully as a
typedef, requiring the 'struct' at times. This has caused me, and I
presume others, wasted time when we forget to use it under those
circumstances when it should be used, but it's never been a big enough
issue to cause me to spend tuits on it. But, working on something else,
I finally came to the realization of what the problem is. It is because
proto.h is #included before regcomp.h is, and so functions that are
declared in proto.h that have something that is a RExC_state_t as a
parameter don't know that it is a typedef because that is defined in
regcomp.h. A way around this is already used for other similar
structures, and that is to declare them in perl.h which is always read
in before proto.h, leaving the definitions to regcomp.h. Thus proto.h
knows enough to compile.
The structure was already declared in perl.h; just not typedef'd.
Otherwise proto.h would not know about it at all. This patch moves two
regcomp.c related declarations in perl.h to the same section as the
others, and changes the one for RExC_state_t to be a typedef. All the
'struct' uses are removed.
Karl Williamson [Wed, 14 Aug 2013 17:39:38 +0000 (11:39 -0600)]
regcomp.h: Create new typedef synonym for clarity
This commit finishes (at least for now) removing some of the overloading
of the term class. A 'regnode_charclass_class' node contains space for
storing the posix classes it matches that are never defined until the
moment of matching because they are subject to the current run-time
locale. This commit creates a typedef 'regnode_charclass_posixl'
synonym that doesn't re-use the term 'class' for two different purposes.
Karl Williamson [Fri, 9 Aug 2013 18:21:53 +0000 (12:21 -0600)]
regcomp.h: Parenthesize macro formal parameter
Not doing so can cause problems, so it is standard procedure to
parenthesize all parameters within a macro definition.
Karl Williamson [Fri, 9 Aug 2013 17:51:09 +0000 (11:51 -0600)]
regcomp.h: Add better named synonyms
This continues the process started two commits ago of removing some of
the overloading of the term 'class'.
In this case, this commit adds some #defines referring to the portions
of the regnode associated with bracketed character classes, the ANYOF
node. Specifically those portions that deal with the Posix character
classes, like \w and [:punct:] under /l (locale) matching are renamed
substituting POSIXL for CLASS. POSIXL is already used for POSIX-related
things under /l. I remember being terribly confused when I started
reading this code about this. One had a class within a class. This
should clarify things somewhat.
The old names are retained in case files outside the core #include and
use it (there are a few such in cpan).
Karl Williamson [Sun, 15 Sep 2013 00:57:26 +0000 (18:57 -0600)]
regcomp.c: Clarify comment
This continues the process of removing some overloading of the word
'class', by changing this comment to use 'bracketed class', and
re-wrapping