2 Copyright 2006-2007 John Maddock.
3 Distributed under the Boost Software License, Version 1.0.
4 (See accompanying file LICENSE_1_0.txt or copy at
5 http://www.boost.org/LICENSE_1_0.txt).
9 [section:perl_syntax Perl Regular Expression Syntax]
13 The Perl regular expression syntax is based on that used by the
14 programming language Perl . Perl regular expressions are the
15 default behavior in Boost.Regex or you can pass the flag =perl= to the
16 [basic_regex] constructor, for example:
18 // e1 is a case sensitive Perl regular expression:
19 // since Perl is the default option there's no need to explicitly specify the syntax used here:
20 boost::regex e1(my_expression);
21 // e2 a case insensitive Perl regular expression:
22 boost::regex e2(my_expression, boost::regex::perl|boost::regex::icase);
24 [h3 Perl Regular Expression Syntax]
26 In Perl regular expressions, all characters match themselves except for the
27 following special characters:
29 [pre .\[{}()\\\*+?|^$]
33 The single character '.' when used outside of a character set will match
34 any single character except:
36 * The NULL character when the [link boost_regex.ref.match_flag_type flag
37 =match_not_dot_null=] is passed to the matching algorithms.
38 * The newline character when the [link boost_regex.ref.match_flag_type
39 flag =match_not_dot_newline=] is passed to
40 the matching algorithms.
44 A '^' character shall match the start of a line.
46 A '$' character shall match the end of a line.
48 [h4 Marked sub-expressions]
50 A section beginning =(= and ending =)= acts as a marked sub-expression.
51 Whatever matched the sub-expression is split out in a separate field by
52 the matching algorithms. Marked sub-expressions can also repeated, or
53 referred to by a back-reference.
55 [h4 Non-marking grouping]
57 A marked sub-expression is useful to lexically group part of a regular
58 expression, but has the side-effect of spitting out an extra field in
59 the result. As an alternative you can lexically group part of a
60 regular expression, without generating a marked sub-expression by using
61 =(?:= and =)= , for example =(?:ab)+= will repeat =ab= without splitting
62 out any separate sub-expressions.
66 Any atom (a single character, a marked sub-expression, or a character class)
67 can be repeated with the =*=, =+=, =?=, and ={}= operators.
69 The =*= operator will match the preceding atom zero or more times,
70 for example the expression =a*b= will match any of the following:
76 The =+= operator will match the preceding atom one or more times, for
77 example the expression =a+b= will match any of the following:
86 The =?= operator will match the preceding atom zero or one times, for
87 example the expression ca?b will match any of the following:
96 An atom can also be repeated with a bounded repeat:
98 =a{n}= Matches 'a' repeated exactly n times.
100 =a{n,}= Matches 'a' repeated n or more times.
102 =a{n, m}= Matches 'a' repeated between n and m times inclusive.
108 Will match either of:
118 It is an error to use a repeat operator, if the preceding construct can not
119 be repeated, for example:
123 Will raise an error, as there is nothing for the =*= operator to be applied to.
125 [h4 Non greedy repeats]
127 The normal repeat operators are "greedy", that is to say they will consume as
128 much input as possible. There are non-greedy versions available that will
129 consume as little input as possible while still producing a match.
131 =*?= Matches the previous atom zero or more times, while consuming as little
134 =+?= Matches the previous atom one or more times, while consuming as
135 little input as possible.
137 =??= Matches the previous atom zero or one times, while consuming
138 as little input as possible.
140 ={n,}?= Matches the previous atom n or more times, while consuming as
141 little input as possible.
143 ={n,m}?= Matches the previous atom between n and m times, while
144 consuming as little input as possible.
146 [h4 Possessive repeats]
148 By default when a repeated pattern does not match then the engine will backtrack until
149 a match is found. However, this behaviour can sometime be undesireable so there are
150 also "possessive" repeats: these match as much as possible and do not then allow
151 backtracking if the rest of the expression fails to match.
153 =*+= Matches the previous atom zero or more times, while giving nothing back.
155 =++= Matches the previous atom one or more times, while giving nothing back.
157 =?+= Matches the previous atom zero or one times, while giving nothing back.
159 ={n,}+= Matches the previous atom n or more times, while giving nothing back.
161 ={n,m}+= Matches the previous atom between n and m times, while giving nothing back.
165 An escape character followed by a digit /n/, where /n/ is in the range 1-9,
166 matches the same string that was matched by sub-expression /n/. For example
171 Will match the string:
179 You can also use the \g escape for the same function, for example:
183 [[=\g1=][Match whatever matched sub-expression 1]]
184 [[=\g{1}=][Match whatever matched sub-expression 1: this form allows for safer
185 parsing of the expression in cases like =\g{1}2= or for indexes higher than 9 as in =\g{1234}=]]
186 [[=\g-1=][Match whatever matched the last opened sub-expression]]
187 [[=\g{-2}=][Match whatever matched the last but one opened sub-expression]]
188 [[=\g{one}=][Match whatever matched the sub-expression named "one"]]
191 Finally the \k escape can be used to refer to named subexpressions, for example [^\k<two>] will match
192 whatever matched the subexpression named "two".
196 The =|= operator will match either of its arguments, so for example:
197 =abc|def= will match either "abc" or "def".
199 Parenthesis can be used to group alternations, for example: =ab(d|ef)=
200 will match either of "abd" or "abef".
202 Empty alternatives are not allowed (these are almost always a mistake), but
203 if you really want an empty alternative use =(?:)= as a placeholder, for example:
205 =|abc= is not a valid expression, but
207 =(?:)|abc= is and is equivalent, also the expression:
209 =(?:abc)??= has exactly the same effect.
213 A character set is a bracket-expression starting with =[= and ending with =]=,
214 it defines a set of characters, and matches any single character that is a
217 A bracket expression may contain any combination of the following:
219 [h5 Single characters]
221 For example =[abc]=, will match any of the characters 'a', 'b', or 'c'.
223 [h5 Character ranges]
225 For example =[a-c]= will match any single character in the range 'a' to 'c'.
226 By default, for Perl regular expressions, a character x is within the
227 range y to z, if the code point of the character lies within the codepoints of
228 the endpoints of the range. Alternatively, if you set the
229 [link boost_regex.ref.syntax_option_type.syntax_option_type_perl =collate= flag]
230 when constructing the regular expression, then ranges are locale sensitive.
234 If the bracket-expression begins with the ^ character, then it matches the
235 complement of the characters it contains, for example =[^a-c]= matches
236 any character that is not in the range =a-c=.
238 [h5 Character classes]
240 An expression of the form [^\[\[:name:\]\]] matches the named character class
241 "name", for example [^\[\[:lower:\]\]] matches any lower case character.
242 See [link boost_regex.syntax.character_classes character class names].
244 [h5 Collating Elements]
246 An expression of the form [^\[\[.col.\]\]] matches the collating element /col/.
247 A collating element is any single character, or any sequence of characters
248 that collates as a single unit. Collating elements may also be used
249 as the end point of a range, for example: [^\[\[.ae.\]-c\]] matches the
250 character sequence "ae", plus any single character in the range "ae"-c,
251 assuming that "ae" is treated as a single collating element in the current locale.
253 As an extension, a collating element may also be specified via it's
254 [link boost_regex.syntax.collating_names symbolic name], for example:
258 matches a =\0= character.
260 [h5 Equivalence classes]
262 An expression of the form [^\[\[\=col\=\]\]], matches any character or collating element
263 whose primary sort key is the same as that for collating element /col/, as with
264 collating elements the name /col/ may be a
265 [link boost_regex.syntax.collating_names symbolic name]. A primary sort key is
266 one that ignores case, accentation, or locale-specific tailorings; so for
267 example `[[=a=]]` matches any of the characters:
268 a, '''À''', '''Á''', '''Â''',
269 '''Ã''', '''Ä''', '''Å''', A, '''à''', '''á''',
270 '''â''', '''ã''', '''ä''' and '''å'''.
271 Unfortunately implementation of this is reliant on the platform's collation
272 and localisation support; this feature can not be relied upon to work portably
273 across all platforms, or even all locales on one platform.
275 [h5 Escaped Characters]
277 All the escape sequences that match a single character, or a single character
278 class are permitted within a character class definition. For example
279 `[\[\]]` would match either of `[` or `]` while `[\W\d]` would match any character
280 that is either a "digit", /or/ is /not/ a "word" character.
284 All of the above can be combined in one character set declaration, for example:
285 [^\[\[:digit:\]a-c\[.NUL.\]\]].
289 Any special character preceded by an escape shall match itself.
291 The following escape sequences are all synonyms for single characters:
294 [[Escape][Character]]
302 [[=\b=][=\b= (but only inside a character class declaration).]]
303 [[=\cX=][An ASCII escape sequence - the character whose code point is X % 32]]
304 [[=\xdd=][A hexadecimal escape sequence - matches the single character whose
305 code point is 0xdd.]]
306 [[=\x{dddd}=][A hexadecimal escape sequence - matches the single character whose
307 code point is 0xdddd.]]
308 [[=\0ddd=][An octal escape sequence - matches the single character whose
309 code point is 0ddd.]]
310 [[=\N{name}=][Matches the single character which has the
311 [link boost_regex.syntax.collating_names symbolic name] /name/.
312 For example =\N{newline}= matches the single character \\n.]]
315 [h5 "Single character" character classes:]
317 Any escaped character /x/, if /x/ is the name of a character class shall
318 match any character that is a member of that class, and any
319 escaped character /X/, if /x/ is the name of a character class, shall
320 match any character not in that class.
322 The following are supported by default:
325 [[Escape sequence][Equivalent to]]
326 [[`\d`][`[[:digit:]]`]]
327 [[`\l`][`[[:lower:]]`]]
328 [[`\s`][`[[:space:]]`]]
329 [[`\u`][`[[:upper:]]`]]
330 [[`\w`][`[[:word:]]`]]
331 [[`\h`][Horizontal whitespace]]
332 [[`\v`][Vertical whitespace]]
333 [[`\D`][`[^[:digit:]]`]]
334 [[`\L`][`[^[:lower:]]`]]
335 [[`\S`][`[^[:space:]]`]]
336 [[`\U`][`[^[:upper:]]`]]
337 [[`\W`][`[^[:word:]]`]]
338 [[`\H`][Not Horizontal whitespace]]
339 [[`\V`][Not Vertical whitespace]]
342 [h5 Character Properties]
344 The character property names in the following table are all equivalent
345 to the [link boost_regex.syntax.character_classes names used in character classes].
348 [[Form][Description][Equivalent character set form]]
349 [[`\pX`][Matches any character that has the property X.][`[[:X:]]`]]
350 [[`\p{Name}`][Matches any character that has the property Name.][`[[:Name:]]`]]
351 [[`\PX`][Matches any character that does not have the property X.][`[^[:X:]]`]]
352 [[`\P{Name}`][Matches any character that does not have the property Name.][`[^[:Name:]]`]]
355 For example =\pd= matches any "digit" character, as does =\p{digit}=.
359 The following escape sequences match the boundaries of words:
361 =\<= Matches the start of a word.
363 =\>= Matches the end of a word.
365 =\b= Matches a word boundary (the start or end of a word).
367 =\B= Matches only when not at a word boundary.
369 [h5 Buffer boundaries]
371 The following match only at buffer boundaries: a "buffer" in this
372 context is the whole of the input text that is being matched against
373 (note that ^ and $ may match embedded newlines within the text).
375 \\\` Matches at the start of a buffer only.
377 \\' Matches at the end of a buffer only.
379 \\A Matches at the start of a buffer only (the same as =\\\`=).
381 \\z Matches at the end of a buffer only (the same as =\\'=).
383 \\Z Matches a zero-width assertion consisting of an optional sequence of newlines at the end of a buffer:
384 equivalent to the regular expression [^(?=\\v*\\z)]. Note that this is subtly different from Perl which
385 behaves as if matching [^(?=\\n?\\z)].
387 [h5 Continuation Escape]
389 The sequence =\G= matches only at the end of the last match found, or at
390 the start of the text being matched if no previous match was found.
391 This escape useful if you're iterating over the matches contained within a
392 text, and you want each subsequence match to start where the last one ended.
396 The escape sequence =\Q= begins a "quoted sequence": all the subsequent characters
397 are treated as literals, until either the end of the regular expression or \\E
398 is found. For example the expression: =\Q\*+\Ea+= would match either of:
405 =\C= Matches a single code point: in Boost regex this has exactly the
406 same effect as a "." operator.
407 =\X= Matches a combining character sequence: that is any non-combining
408 character followed by a sequence of zero or more combining characters.
410 [h5 Matching Line Endings]
412 The escape sequence =\R= matches any line ending character sequence, specifically it is identical to
413 the expression [^(?>\x0D\x0A?|\[\x0A-\x0C\x85\x{2028}\x{2029}\])].
415 [h5 Keeping back some text]
417 =\K= Resets the start location of $0 to the current text position: in other words everything to the
418 left of \K is "kept back" and does not form part of the regular expression match. $` is updated
421 For example =foo\Kbar= matched against the text "foobar" would return the match "bar" for $0 and "foo"
422 for $`. This can be used to simulate variable width lookbehind assertions.
424 [h5 Any other escape]
426 Any other escape sequence matches the character that is escaped, for example
427 \\@ matches a literal '@'.
429 [h4 Perl Extended Patterns]
431 Perl-specific extensions to the regular expression syntax all start with =(?=.
433 [h5 Named Subexpressions]
435 You can create a named subexpression using:
439 Which can be then be refered to by the name /NAME/. Alternatively you can delimit the name
444 These named subexpressions can be refered to in a backreference using either [^\g{NAME}] or [^\k<NAME>]
445 and can also be refered to by name in a [perl_format] format string for search and replace operations, or in the
446 [match_results] member functions.
450 =(?# ... )= is treated as a comment, it's contents are ignored.
454 =(?imsx-imsx ... )= alters which of the perl modifiers are in effect within
455 the pattern, changes take effect from the point that the block is first seen
456 and extend to any enclosing =)=. Letters before a '-' turn that perl
457 modifier on, letters afterward, turn it off.
459 =(?imsx-imsx:pattern)= applies the specified modifiers to pattern only.
461 [h5 Non-marking groups]
463 =(?:pattern)= lexically groups pattern, without generating an additional
468 =(?|pattern)= resets the subexpression count at the start of each "|" alternative within /pattern/.
470 The sub-expression count following this construct is that of whichever branch had the largest number of
471 sub-expressions. This construct is useful when you want to capture one of a number of alternative matches
472 in a single sub-expression index.
474 In the following example the index of each sub-expression is shown below the expression:
477 # before ---------------branch-reset----------- after
478 / ( a ) (?| x ( y ) z | (p (q) r) | (t) u (v) ) ( z ) /x
484 [^(?=pattern)] consumes zero characters, only if pattern matches.
486 =(?!pattern)= consumes zero characters, only if pattern does not match.
488 Lookahead is typically used to create the logical AND of two regular
489 expressions, for example if a password must contain a lower case letter,
490 an upper case letter, a punctuation symbol, and be at least 6 characters long,
493 (?=.*[[:lower:]])(?=.*[[:upper:]])(?=.*[[:punct:]]).{6,}
495 could be used to validate the password.
499 [^(?<=pattern)] consumes zero characters, only if pattern could be matched
500 against the characters preceding the current position (pattern must be
503 =(?<!pattern)= consumes zero characters, only if pattern could not be
504 matched against the characters preceding the current position (pattern must
507 [h5 Independent sub-expressions]
509 =(?>pattern)= /pattern/ is matched independently of the surrounding patterns,
510 the expression will never backtrack into /pattern/. Independent sub-expressions
511 are typically used to improve performance; only the best possible match
512 for pattern will be considered, if this doesn't allow the expression as a
513 whole to match then no match is found at all.
515 [h5 Recursive Expressions]
517 [^(?['N]) (?-['N]) (?+['N]) (?R) (?0) (?&NAME)]
519 =(?R)= and =(?0)= recurse to the start of the entire pattern.
521 [^(?['N])] executes sub-expression /N/ recursively, for example =(?2)= will recurse to sub-expression 2.
523 [^(?-['N])] and [^(?+['N])] are relative recursions, so for example =(?-1)= recurses to the last sub-expression to be declared,
524 and =(?+1)= recurses to the next sub-expression to be declared.
526 [^(?&NAME)] recurses to named sub-expression ['NAME].
528 [h5 Conditional Expressions]
530 =(?(condition)yes-pattern|no-pattern)= attempts to match /yes-pattern/ if
531 the /condition/ is true, otherwise attempts to match /no-pattern/.
533 =(?(condition)yes-pattern)= attempts to match /yes-pattern/ if the /condition/
534 is true, otherwise matches the NULL string.
536 /condition/ may be either: a forward lookahead assert, the index of
537 a marked sub-expression (the condition becomes true if the sub-expression
538 has been matched), or an index of a recursion (the condition become true if we are executing
539 directly inside the specified recursion).
541 Here is a summary of the possible predicates:
543 * [^(?(?\=assert)yes-pattern|no-pattern)] Executes /yes-pattern/ if the forward look-ahead assert matches, otherwise
544 executes /no-pattern/.
545 * =(?(?!assert)yes-pattern|no-pattern)= Executes /yes-pattern/ if the forward look-ahead assert does not match, otherwise
546 executes /no-pattern/.
547 * =(?(['N])yes-pattern|no-pattern)= Executes /yes-pattern/ if subexpression /N/ has been matched, otherwise
548 executes /no-pattern/.
549 * =(?(<['name]>)yes-pattern|no-pattern)= Executes /yes-pattern/ if named subexpression /name/ has been matched, otherwise
550 executes /no-pattern/.
551 * =(?('['name]')yes-pattern|no-pattern)= Executes /yes-pattern/ if named subexpression /name/ has been matched, otherwise
552 executes /no-pattern/.
553 * =(?(R)yes-pattern|no-pattern)= Executes /yes-pattern/ if we are executing inside a recursion, otherwise
554 executes /no-pattern/.
555 * [^(?(R['N])yes-pattern|no-pattern)] Executes /yes-pattern/ if we are executing inside a recursion to sub-expression /N/, otherwise
556 executes /no-pattern/.
557 * [^(?(R&['name])yes-pattern|no-pattern)] Executes /yes-pattern/ if we are executing inside a recursion to named sub-expression /name/, otherwise
558 executes /no-pattern/.
559 * [^(?(DEFINE)never-exectuted-pattern)] Defines a block of code that is never executed and matches no characters:
560 this is usually used to define one or more named sub-expressions which are refered to from elsewhere in the pattern.
562 [h4 Operator precedence]
564 The order of precedence for of operators is as follows:
566 # Collation-related bracket symbols `[==] [::] [..]`
567 # Escaped characters =\=
568 # Character set (bracket expression) `[]`
570 # Single-character-ERE duplication =* + ? {m,n}=
575 [h3 What gets matched]
577 If you view the regular expression as a directed (possibly cyclic)
578 graph, then the best match found is the first match found by a
579 depth-first-search performed on that graph, while matching the input text.
583 The best match found is the
584 [link boost_regex.syntax.leftmost_longest_rule leftmost match],
585 with individual elements matched as follows;
588 [[Construct][What gets matched]]
589 [[=AtomA AtomB=][Locates the best match for /AtomA/ that has a following match for /AtomB/.]]
590 [[=Expression1 | Expression2=][If /Expresion1/ can be matched then returns that match,
591 otherwise attempts to match /Expression2/.]]
592 [[=S{N}=][Matches /S/ repeated exactly N times.]]
593 [[=S{N,M}=][Matches S repeated between N and M times, and as many times as possible.]]
594 [[=S{N,M}?=][Matches S repeated between N and M times, and as few times as possible.]]
595 [[=S?, S*, S+=][The same as =S{0,1}=, =S{0,UINT_MAX}=, =S{1,UINT_MAX}= respectively.]]
596 [[=S??, S*?, S+?=][The same as =S{0,1}?=, =S{0,UINT_MAX}?=, =S{1,UINT_MAX}?= respectively.]]
597 [[=(?>S)=][Matches the best match for /S/, and only that.]]
598 [[[^(?=S), (?<=S)]][Matches only the best match for /S/ (this is only
599 visible if there are capturing parenthesis within /S/).]]
600 [[=(?!S), (?<!S)=][Considers only whether a match for S exists or not.]]
601 [[=(?(condition)yes-pattern | no-pattern)=][If condition is true, then
602 only yes-pattern is considered, otherwise only no-pattern is considered.]]
607 The [link boost_regex.ref.syntax_option_type.syntax_option_type_perl options =normal=,
608 =ECMAScript=, =JavaScript= and =JScript=] are all synonyms for
613 There are a [link boost_regex.ref.syntax_option_type.syntax_option_type_perl
614 variety of flags] that may be combined with the =perl= option when
615 constructing the regular expression, in particular note that the
616 =newline_alt= option alters the syntax, while the =collate=, =nosubs= and
617 =icase= options modify how the case and locale sensitivity are to be applied.
619 [h3 Pattern Modifiers]
621 The perl =smix= modifiers can either be applied using a =(?smix-smix)=
622 prefix to the regular expression, or with one of the
623 [link boost_regex.ref.syntax_option_type.syntax_option_type_perl regex-compile time
624 flags =no_mod_m=, =mod_x=, =mod_s=, and =no_mod_s=].
628 [@http://perldoc.perl.org/perlre.html Perl 5.8].