If C<use locale> is in effect, the case map is taken from the current
locale for code points less than 255, and from Unicode rules for larger
-code points. See L<perllocale>.
+code points. However, matches that would cross the Unicode
+rules/non-Unicode rules boundary (ords 255/256) will not succeed. See
+L<perllocale>.
+
+There are a number of Unicode characters that match multiple characters
+under C</i>. For example, C<LATIN SMALL LIGATURE FI>
+should match the sequence C<fi>. Perl is not
+currently able to do this when the multiple characters are in the pattern and
+are split between groupings, or when one or more are quantified. Thus
+
+ "\N{LATIN SMALL LIGATURE FI}" =~ /fi/i; # Matches
+ "\N{LATIN SMALL LIGATURE FI}" =~ /[fi][fi]/i; # Doesn't match!
+ "\N{LATIN SMALL LIGATURE FI}" =~ /fi*/i; # Doesn't match!
+
+ # The below doesn't match, and it isn't clear what $1 and $2 would
+ # be even if it did!!
+ "\N{LATIN SMALL LIGATURE FI}" =~ /(f)(i)/i; # Doesn't match!
+
+Also, this matching doesn't fully conform to the current Unicode
+recommendations, which ask that the matching be made upon the NFD
+(Normalization Form Decomposed) of the text. However, Unicode is
+in the process of reconsidering and revising their recommendations.
=item x
X</x>
Extend your pattern's legibility by permitting whitespace and comments.
+Details in L</"/x">
=item p
X</p> X<regex, preserve> X<regexp, preserve>
X</a> X</d> X</l> X</u>
These modifiers, new in 5.14, affect which character-set semantics
-(Unicode, ASCII, etc.) are used, as described below.
+(Unicode, ASCII, etc.) are used, as described below in
+L</Character set modifiers>.
=back
These are usually written as "the C</x> modifier", even though the delimiter
in question might not really be a slash. The modifiers C</imsxadlup>
may also be embedded within the regular expression itself using
-the C<(?...)> construct.
+the C<(?...)> construct, see L</Extended Patterns> below.
The C</x>, C</l>, C</u>, C</a> and C</d> modifiers need a little more
explanation.
+=head3 /x
+
C</x> tells
the regular expression parser to ignore most whitespace that is neither
backslashed nor within a character class. You can use this to break up
L<perluniprops/Properties accessible through \p{} and \P{}>.
X</x>
-C</l> means to use a locale (see L<perllocale>) when pattern matching.
-The locale used will be the one in effect at the time of execution of
-the pattern match. This may not be the same as the compilation-time
-locale, and can differ from one match to another if there is an
-intervening call of the
+=head3 Character set modifiers
+
+C</d>, C</u>, C</a>, and C</l>, available starting in 5.14, are called
+the character set modifiers; they affect the character set semantics
+used for the regular expression.
+
+At any given time, exactly one of these modifiers is in effect. Once
+compiled, the behavior doesn't change regardless of what rules are in
+effect when the regular expression is executed. And if a regular
+expression is interpolated into a larger one, the original's rules
+continue to apply to it, and only it.
+
+=head4 /l
+
+means to use the current locale's rules (see L<perllocale>) when pattern
+matching. For example, C<\w> will match the "word" characters of that
+locale, and C<"/i"> case-insensitive matching will match according to
+the locale's case folding rules. The locale used will be the one in
+effect at the time of execution of the pattern match. This may not be
+the same as the compilation-time locale, and can differ from one match
+to another if there is an intervening call of the
L<setlocale() function|perllocale/The setlocale function>.
-This modifier is automatically set if the regular expression is compiled
-within the scope of a C<"use locale"> pragma.
-Perl only allows single-byte locales. This means that code points above
-255 are treated as Unicode no matter what locale is in effect.
-Under Unicode rules, there are a few case-insensitive matches that cross the
-255/256 boundary. These are disallowed. For example,
-0xFF does not caselessly match the character at 0x178, LATIN CAPITAL
-LETTER Y WITH DIAERESIS, because 0xFF may not be LATIN SMALL LETTER Y
-in the current locale, and Perl has no way of knowing if that character
-even exists in the locale, much less what code point it is.
+
+Perl only supports single-byte locales. This means that code points
+above 255 are treated as Unicode no matter what locale is in effect.
+Under Unicode rules, there are a few case-insensitive matches that cross
+the 255/256 boundary. These are disallowed under C</l>. For example,
+0xFF does not caselessly match the character at 0x178, C<LATIN CAPITAL
+LETTER Y WITH DIAERESIS>, because 0xFF may not be C<LATIN SMALL LETTER Y
+WITH DIAERESIS> in the current locale, and Perl has no way of knowing if
+that character even exists in the locale, much less what code point it
+is.
+
+This modifier may be specified to be the default by C<use locale>, but
+see L</Which character set modifier is in effect?>.
X</l>
-C</u> means to use Unicode semantics when pattern matching. It is
-automatically set if the regular expression is encoded in utf8 internally,
-or is compiled within the scope of a
-L<C<"use feature 'unicode_strings">|feature> pragma (and isn't also in
-the scope of the L<C<"use locale">|locale> or the L<C<"use bytes">|bytes>
-pragma). On ASCII platforms, the code points between 128 and 255 take on their
+=head4 /u
+
+means to use Unicode rules when pattern matching. On ASCII platforms,
+this means that the code points between 128 and 255 take on their
Latin-1 (ISO-8859-1) meanings (which are the same as Unicode's), whereas
in strict ASCII their meanings are undefined. Thus the platform
-effectively becomes a Unicode platform. The ASCII characters remain as
-ASCII characters (since ASCII is a subset of Latin-1 and Unicode). For
-example, when this option is not on, on a non-utf8 string, C<"\w">
-matches precisely C<[A-Za-z0-9_]>. When the option is on, it matches
-not just those, but all the Latin-1 word characters (such as an "n" with
-a tilde). On EBCDIC platforms, which already are equivalent to Latin-1,
-this modifier changes behavior only when the C<"/i"> modifier is also
-specified, and affects only two characters, giving them full Unicode
-semantics: the C<MICRO SIGN> will match the Greek capital and
-small letters C<MU>; otherwise not; and the C<LATIN CAPITAL LETTER SHARP
-S> will match any of C<SS>, C<Ss>, C<sS>, and C<ss>, otherwise not.
-(This last case is buggy, however.)
+effectively becomes a Unicode platform, hence, for example, C<\w> will
+match any of the more than 100_000 word characters in Unicode.
+
+Unlike most locales, which are specific to a language and country pair,
+Unicode classifies all the characters that are letters I<somewhere> as
+C<\w>. For example, your locale might not think that C<LATIN SMALL
+LETTER ETH> is a letter (unless you happen to speak Icelandic), but
+Unicode does. Similarly, all the characters that are decimal digits
+somewhere in the world will match C<\d>; this is hundreds, not 10,
+possible matches. And some of those digits look like some of the 10
+ASCII digits, but mean a different number, so a human could easily think
+a number is a different quantity than it really is. For example,
+C<BENGALI DIGIT FOUR> (U+09EA) looks very much like an
+C<ASCII DIGIT EIGHT> (U+0038). And, C<\d+>, may match strings of digits
+that are a mixture from different writing systems, creating a security
+issue. L<Unicode::UCD/num()> can be used to sort this out.
+
+Also, case-insensitive matching works on the full set of Unicode
+characters. The C<KELVIN SIGN>, for example matches the letters "k" and
+"K"; and C<LATIN SMALL LIGATURE FF> matches the sequence "ff", which,
+if you're not prepared, might make it look like a hexadecimal constant,
+presenting another potential security issue. See
+L<http://unicode.org/reports/tr36> for a detailed discussion of Unicode
+security issues.
+
+On EBCDIC platforms, which already are equivalent to Latin-1 (at least
+the ones that Perl handles), this modifier changes behavior only when
+the C<"/i"> modifier is also specified, and it turns out it affects only
+two characters, giving them full Unicode semantics: the C<MICRO SIGN>
+will match the Greek capital and small letters C<MU>; otherwise not; and
+the C<LATIN CAPITAL LETTER SHARP S> will match any of C<SS>, C<Ss>,
+C<sS>, and C<ss>, otherwise not.
+
+This modifier may be specified to be the default by C<use feature
+'unicode_strings>, but see
+L</Which character set modifier is in effect?>.
X</u>
-C</a> is the same as C</u>, except that C<\d>, C<\s>, C<\w>, and the
+=head4 /a
+
+is the same as C</u>, except that C<\d>, C<\s>, C<\w>, and the
Posix character classes are restricted to matching in the ASCII range
only. That is, with this modifier, C<\d> always means precisely the
digits C<"0"> to C<"9">; C<\s> means the five characters C<[ \f\n\r\t]>;
C<\w> means the 63 characters C<[A-Za-z0-9_]>; and likewise, all the
Posix classes such as C<[[:print:]]> match only the appropriate
-ASCII-range characters. As you would expect, this modifier causes, for
-example, C<\D> to mean the same thing as C<[^0-9]>; in fact, all
-non-ASCII characters match C<\D>, C<\S>, and C<\W>. C<\b> still means
-to match at the boundary between C<\w> and C<\W>, using the C<"a">
-definitions of them (similarly for C<\B>). Otherwise, C<"a"> behaves
-like the C<"u"> modifier, in that case-insensitive matching uses Unicode
-semantics; for example, "k" will match the Unicode C<\N{KELVIN SIGN}>
-under C</i> matching, and code points in the Latin1 range, above ASCII
-will have Unicode semantics when it comes to case-insensitive matching.
-But writing two in "a"'s in a row will increase its effect, causing the
-Kelvin sign and all other non-ASCII characters not to match any ASCII
-character under C</i> matching.
+ASCII-range characters.
+
+This modifier is useful for people who only incidentally use Unicode.
+With it, one can write C<\d> with confidence that it will only match
+ASCII characters, and should the need arise to match beyond ASCII, you
+can use C<\p{Digit}>, or C<\p{Word}> for C<\w>. There are similar
+C<\p{...}> constructs that can match white space and Posix classes
+beyond ASCII. See L<perlrecharclass>.
+
+As you would expect, this modifier causes, for example, C<\D> to mean
+the same thing as C<[^0-9]>; in fact, all non-ASCII characters match
+C<\D>, C<\S>, and C<\W>. C<\b> still means to match at the boundary
+between C<\w> and C<\W>, using the C</a> definitions of them (similarly
+for C<\B>).
+
+Otherwise, C</a> behaves like the C</u> modifier, in that
+case-insensitive matching uses Unicode semantics; for example, "k" will
+match the Unicode C<\N{KELVIN SIGN}> under C</i> matching, and code
+points in the Latin1 range, above ASCII will have Unicode rules when it
+comes to case-insensitive matching.
+
+To forbid ASCII/non-ASCII matches (like "k" with C<\N{KELVIN SIGN}>),
+specify the "a" twice, for example C</aai> or C</aia>
+
+To reiterate, this modifier provides protection for applications that
+don't wish to be exposed to all of Unicode. Specifying it twice
+gives added protection.
+
+This modifier may be specified to be the default by C<use re '/a'>
+or C<use re '/aa'>, but see
+L</Which character set modifier is in effect?>.
X</a>
+X</aa>
+
+=head4 /d
+
+This modifier means to use the "Default" native rules of the platform
+except when there is cause to use Unicode rules instead, as follows:
+
+=over 4
+
+=item 1
+
+the target string is encoded in UTF-8; or
+
+=item 2
+
+the pattern is encoded in UTF-8; or
+
+=item 3
+
+the pattern explicitly mentions a code point that is above 255 (say by
+C<\x{100}>); or
+
+=item 4
-C</d> means to use the traditional Perl pattern-matching behavior.
-This is dualistic (hence the name C</d>, which also could stand for
-"depends"). When this is in effect, Perl matches according to the
-platform's native character set rules unless there is something that
-indicates to use Unicode rules. If either the target string or the
-pattern itself is encoded in UTF-8, Unicode rules are used. Also, if
-the pattern contains Unicode-only features, such as code points above
-255, C<\p()> Unicode properties or C<\N{}> Unicode names, Unicode rules
-will be used. It is automatically selected by default if the regular
-expression is compiled neither within the scope of a C<"use locale">
-pragma nor a <C<"use feature 'unicode_strings"> pragma.
-This behavior causes a number of glitches, see
-L<perlunicode/The "Unicode Bug">.
-X</d>
+the pattern uses a Unicode name (C<\N{...}>); or
+
+=item 5
+
+the pattern uses a Unicode property (C<\p{...}>)
+
+=back
+
+Another mnemonic for this modifier is "Depends", as the rules actually
+used depend on various things, and as a result you can get unexpected
+results. See L<perlunicode/The "Unicode Bug">.
+
+On ASCII platforms, the native rules are ASCII, and on EBCDIC platforms
+(at least the ones that Perl handles), they are Latin-1.
+
+Here are some examples of how that works on an ASCII platform:
+
+ $str = "\xDF"; # $str is not in UTF-8 format.
+ $str =~ /^\w/; # No match, as $str isn't in UTF-8 format.
+ $str .= "\x{0e0b}"; # Now $str is in UTF-8 format.
+ $str =~ /^\w/; # Match! $str is now in UTF-8 format.
+ chop $str;
+ $str =~ /^\w/; # Still a match! $str remains in UTF-8 format.
+
+=head4 Which character set modifier is in effect?
+
+Which of these modifiers is in effect at any given point in a regular
+expression depends on a fairly complex set of interactions. As
+explained below in L</Extended Patterns> it is possible to explicitly
+specify modifiers that apply only to portions of a regular expression.
+The innermost always has priority over any outer ones, and one applying
+to the whole expression has priority over any default settings that are
+described in the next few paragraphs.
+
+The C<L<use re 'E<sol>foo'|re/'E<sol>flags' mode">> pragma can be used to set
+default modifiers (including these) for regular expressions compiled
+within its scope. This pragma has precedence over the other pragmas
+that change the defaults, as listed below.
+
+Otherwise, C<L<use locale|perllocale>> sets the default modifier to C</l>;
+and C<L<use feature 'unicode_strings|feature>> or
+C<L<use 5.012|perlfunc/use VERSION>> (or higher) set the default to
+C</u> when not in the same scope as either C<L<use locale|perllocale>>
+or C<L<use bytes|bytes>> .
+
+If none of the above apply, for backwards compatibility reasons, the
+C</d> modifier is the one in effect by default. As this can lead to
+unexpected results, it is best to specify which other rule set should be
+used.
+
+=head4 Character set modifier behavior prior to Perl 5.14
+
+Prior to 5.14, there were no explicit modifiers, but C</l> was implied
+for regexes compiled within the scope of C<use locale>, and C</d> was
+implied otherwise. However, interpolating a regex into a larger regex
+would ignore the original compilation in favor of whatever was in effect
+at the time of the second compilation. There were a number of
+inconsistencies (bugs) with the C</d> modifier, where Unicode rules
+would be used when inappropriate, and vice versa. C<\p{}> did not imply
+Unicode rules, and neither did all occurrences of C<\N{}>, until 5.12.
=head2 Regular Expressions
constant.
The C<\I<digit>> notation also works in certain circumstances outside
-the pattern. See L</Warning on \1 Instead of $1> below for details.)
+the pattern. See L</Warning on \1 Instead of $1> below for details.
Examples:
Note that the C<a>, C<d>, C<l>, C<p>, and C<u> modifiers are special in
that they can only be enabled, not disabled, and the C<a>, C<d>, C<l>, and
C<u> modifiers are mutually exclusive: specifying one de-specifies the
-others, and a maximum of one may appear in the construct. Thus, for
+others, and a maximum of one (or two C<a>'s) may appear in the
+construct. Thus, for
example, C<(?-p)> will warn when compiled under C<use warnings>;
C<(?-d:...)> and C<(?dl:...)> are fatal errors.
=head1 BUGS
-There are numerous problems with case-insensitive matching of characters
-outside the ASCII range, especially with those whose folds are multiple
-characters, such as ligatures like C<LATIN SMALL LIGATURE FF>.
-
-In a bracketed character class with case-insensitive matching, ranges only work
-for ASCII characters. For example,
-C<m/[\N{CYRILLIC CAPITAL LETTER A}-\N{CYRILLIC CAPITAL LETTER YA}]/i>
-doesn't match all the Russian upper and lower case letters.
-
Many regular expression constructs don't work on EBCDIC platforms.
+There are a number of issues with regard to case-insensitive matching
+in Unicode rules. See C<i> under L</Modifiers> above.
+
This document varies from difficult to understand to completely
and utterly opaque. The wandering prose riddled with jargon is
hard to fathom in several places.
projects / platform / upstream / perl.git / commitdiff