diff options
| author | Scott MacVicar <scottmac@php.net> | 2009-11-03 12:15:03 +0000 |
|---|---|---|
| committer | Scott MacVicar <scottmac@php.net> | 2009-11-03 12:15:03 +0000 |
| commit | f03b175f7cb5b1ffc0b4f59d83836beb3794ebfd (patch) | |
| tree | 30ce6bf2f30beb84bbbc70d257a2666d032e0398 /ext/pcre/pcrelib/doc/pcre.txt | |
| parent | 26e3082abca953a2add5ae02b6fd3f6f301e00ec (diff) | |
| download | php-git-f03b175f7cb5b1ffc0b4f59d83836beb3794ebfd.tar.gz | |
Update PCRE to 8.00
Diffstat (limited to 'ext/pcre/pcrelib/doc/pcre.txt')
| -rw-r--r-- | ext/pcre/pcrelib/doc/pcre.txt | 2888 |
1 files changed, 1641 insertions, 1247 deletions
diff --git a/ext/pcre/pcrelib/doc/pcre.txt b/ext/pcre/pcrelib/doc/pcre.txt index 9a2ce31598..2ccc7bb4b6 100644 --- a/ext/pcre/pcrelib/doc/pcre.txt +++ b/ext/pcre/pcrelib/doc/pcre.txt @@ -2,8 +2,9 @@ This file contains a concatenation of the PCRE man pages, converted to plain text format for ease of searching with a text editor, or for use on systems that do not have a man page processor. The small individual files that give -synopses of each function in the library have not been included. There are -separate text files for the pcregrep and pcretest commands. +synopses of each function in the library have not been included. Neither has +the pcredemo program. There are separate text files for the pcregrep and +pcretest commands. ----------------------------------------------------------------------------- @@ -18,23 +19,23 @@ INTRODUCTION The PCRE library is a set of functions that implement regular expres- sion pattern matching using the same syntax and semantics as Perl, with - just a few differences. Certain features that appeared in Python and - PCRE before they appeared in Perl are also available using the Python - syntax. There is also some support for certain .NET and Oniguruma syn- - tax items, and there is an option for requesting some minor changes - that give better JavaScript compatibility. - - The current implementation of PCRE (release 7.x) corresponds approxi- - mately with Perl 5.10, including support for UTF-8 encoded strings and - Unicode general category properties. However, UTF-8 and Unicode support - has to be explicitly enabled; it is not the default. The Unicode tables - correspond to Unicode release 5.1. + just a few differences. Some features that appeared in Python and PCRE + before they appeared in Perl are also available using the Python syn- + tax, there is some support for one or two .NET and Oniguruma syntax + items, and there is an option for requesting some minor changes that + give better JavaScript compatibility. + + The current implementation of PCRE corresponds approximately with Perl + 5.10, including support for UTF-8 encoded strings and Unicode general + category properties. However, UTF-8 and Unicode support has to be + explicitly enabled; it is not the default. The Unicode tables corre- + spond to Unicode release 5.1. In addition to the Perl-compatible matching function, PCRE contains an - alternative matching function that matches the same compiled patterns - in a different way. In certain circumstances, the alternative function - has some advantages. For a discussion of the two matching algorithms, - see the pcrematching page. + alternative function that matches the same compiled patterns in a dif- + ferent way. In certain circumstances, the alternative function has some + advantages. For a discussion of the two matching algorithms, see the + pcrematching page. PCRE is written in C and released as a C library. A number of people have written wrappers and interfaces of various kinds. In particular, @@ -54,8 +55,8 @@ INTRODUCTION library is built. The pcre_config() function makes it possible for a client to discover which features are available. The features them- selves are described in the pcrebuild page. Documentation about build- - ing PCRE for various operating systems can be found in the README file - in the source distribution. + ing PCRE for various operating systems can be found in the README and + NON-UNIX-USE files in the source distribution. The library contains a number of undocumented internal functions and data tables that are used by more than one of the exported external @@ -71,8 +72,9 @@ USER DOCUMENTATION The user documentation for PCRE comprises a number of different sec- tions. In the "man" format, each of these is a separate "man page". In the HTML format, each is a separate page, linked from the index page. - In the plain text format, all the sections are concatenated, for ease - of searching. The sections are as follows: + In the plain text format, all the sections, except the pcredemo sec- + tion, are concatenated, for ease of searching. The sections are as fol- + lows: pcre this document pcre-config show PCRE installation configuration information @@ -81,34 +83,35 @@ USER DOCUMENTATION pcrecallout details of the callout feature pcrecompat discussion of Perl compatibility pcrecpp details of the C++ wrapper + pcredemo a demonstration C program that uses PCRE pcregrep description of the pcregrep command pcrematching discussion of the two matching algorithms pcrepartial details of the partial matching facility pcrepattern syntax and semantics of supported regular expressions - pcresyntax quick syntax reference pcreperform discussion of performance issues pcreposix the POSIX-compatible C API pcreprecompile details of saving and re-using precompiled patterns - pcresample discussion of the sample program + pcresample discussion of the pcredemo program pcrestack discussion of stack usage + pcresyntax quick syntax reference pcretest description of the pcretest testing command - In addition, in the "man" and HTML formats, there is a short page for + In addition, in the "man" and HTML formats, there is a short page for each C library function, listing its arguments and results. LIMITATIONS - There are some size limitations in PCRE but it is hoped that they will + There are some size limitations in PCRE but it is hoped that they will never in practice be relevant. - The maximum length of a compiled pattern is 65539 (sic) bytes if PCRE + The maximum length of a compiled pattern is 65539 (sic) bytes if PCRE is compiled with the default internal linkage size of 2. If you want to - process regular expressions that are truly enormous, you can compile - PCRE with an internal linkage size of 3 or 4 (see the README file in - the source distribution and the pcrebuild documentation for details). - In these cases the limit is substantially larger. However, the speed + process regular expressions that are truly enormous, you can compile + PCRE with an internal linkage size of 3 or 4 (see the README file in + the source distribution and the pcrebuild documentation for details). + In these cases the limit is substantially larger. However, the speed of execution is slower. All values in repeating quantifiers must be less than 65536. @@ -119,131 +122,131 @@ LIMITATIONS The maximum length of name for a named subpattern is 32 characters, and the maximum number of named subpatterns is 10000. - The maximum length of a subject string is the largest positive number - that an integer variable can hold. However, when using the traditional + The maximum length of a subject string is the largest positive number + that an integer variable can hold. However, when using the traditional matching function, PCRE uses recursion to handle subpatterns and indef- - inite repetition. This means that the available stack space may limit + inite repetition. This means that the available stack space may limit the size of a subject string that can be processed by certain patterns. For a discussion of stack issues, see the pcrestack documentation. UTF-8 AND UNICODE PROPERTY SUPPORT - From release 3.3, PCRE has had some support for character strings - encoded in the UTF-8 format. For release 4.0 this was greatly extended - to cover most common requirements, and in release 5.0 additional sup- + From release 3.3, PCRE has had some support for character strings + encoded in the UTF-8 format. For release 4.0 this was greatly extended + to cover most common requirements, and in release 5.0 additional sup- port for Unicode general category properties was added. - In order process UTF-8 strings, you must build PCRE to include UTF-8 - support in the code, and, in addition, you must call pcre_compile() - with the PCRE_UTF8 option flag, or the pattern must start with the - sequence (*UTF8). When either of these is the case, both the pattern - and any subject strings that are matched against it are treated as - UTF-8 strings instead of just strings of bytes. + In order process UTF-8 strings, you must build PCRE to include UTF-8 + support in the code, and, in addition, you must call pcre_compile() + with the PCRE_UTF8 option flag, or the pattern must start with the + sequence (*UTF8). When either of these is the case, both the pattern + and any subject strings that are matched against it are treated as + UTF-8 strings instead of strings of 1-byte characters. - If you compile PCRE with UTF-8 support, but do not use it at run time, - the library will be a bit bigger, but the additional run time overhead + If you compile PCRE with UTF-8 support, but do not use it at run time, + the library will be a bit bigger, but the additional run time overhead is limited to testing the PCRE_UTF8 flag occasionally, so should not be very big. If PCRE is built with Unicode character property support (which implies - UTF-8 support), the escape sequences \p{..}, \P{..}, and \X are sup- + UTF-8 support), the escape sequences \p{..}, \P{..}, and \X are sup- ported. The available properties that can be tested are limited to the - general category properties such as Lu for an upper case letter or Nd - for a decimal number, the Unicode script names such as Arabic or Han, - and the derived properties Any and L&. A full list is given in the + general category properties such as Lu for an upper case letter or Nd + for a decimal number, the Unicode script names such as Arabic or Han, + and the derived properties Any and L&. A full list is given in the pcrepattern documentation. Only the short names for properties are sup- - ported. For example, \p{L} matches a letter. Its Perl synonym, \p{Let- - ter}, is not supported. Furthermore, in Perl, many properties may - optionally be prefixed by "Is", for compatibility with Perl 5.6. PCRE + ported. For example, \p{L} matches a letter. Its Perl synonym, \p{Let- + ter}, is not supported. Furthermore, in Perl, many properties may + optionally be prefixed by "Is", for compatibility with Perl 5.6. PCRE does not support this. Validity of UTF-8 strings - When you set the PCRE_UTF8 flag, the strings passed as patterns and + When you set the PCRE_UTF8 flag, the strings passed as patterns and subjects are (by default) checked for validity on entry to the relevant - functions. From release 7.3 of PCRE, the check is according the rules - of RFC 3629, which are themselves derived from the Unicode specifica- - tion. Earlier releases of PCRE followed the rules of RFC 2279, which - allows the full range of 31-bit values (0 to 0x7FFFFFFF). The current + functions. From release 7.3 of PCRE, the check is according the rules + of RFC 3629, which are themselves derived from the Unicode specifica- + tion. Earlier releases of PCRE followed the rules of RFC 2279, which + allows the full range of 31-bit values (0 to 0x7FFFFFFF). The current check allows only values in the range U+0 to U+10FFFF, excluding U+D800 to U+DFFF. - The excluded code points are the "Low Surrogate Area" of Unicode, of - which the Unicode Standard says this: "The Low Surrogate Area does not - contain any character assignments, consequently no character code + The excluded code points are the "Low Surrogate Area" of Unicode, of + which the Unicode Standard says this: "The Low Surrogate Area does not + contain any character assignments, consequently no character code charts or namelists are provided for this area. Surrogates are reserved - for use with UTF-16 and then must be used in pairs." The code points - that are encoded by UTF-16 pairs are available as independent code - points in the UTF-8 encoding. (In other words, the whole surrogate + for use with UTF-16 and then must be used in pairs." The code points + that are encoded by UTF-16 pairs are available as independent code + points in the UTF-8 encoding. (In other words, the whole surrogate thing is a fudge for UTF-16 which unfortunately messes up UTF-8.) - If an invalid UTF-8 string is passed to PCRE, an error return + If an invalid UTF-8 string is passed to PCRE, an error return (PCRE_ERROR_BADUTF8) is given. In some situations, you may already know that your strings are valid, and therefore want to skip these checks in order to improve performance. If you set the PCRE_NO_UTF8_CHECK flag at - compile time or at run time, PCRE assumes that the pattern or subject - it is given (respectively) contains only valid UTF-8 codes. In this + compile time or at run time, PCRE assumes that the pattern or subject + it is given (respectively) contains only valid UTF-8 codes. In this case, it does not diagnose an invalid UTF-8 string. - If you pass an invalid UTF-8 string when PCRE_NO_UTF8_CHECK is set, - what happens depends on why the string is invalid. If the string con- + If you pass an invalid UTF-8 string when PCRE_NO_UTF8_CHECK is set, + what happens depends on why the string is invalid. If the string con- forms to the "old" definition of UTF-8 (RFC 2279), it is processed as a - string of characters in the range 0 to 0x7FFFFFFF. In other words, + string of characters in the range 0 to 0x7FFFFFFF. In other words, apart from the initial validity test, PCRE (when in UTF-8 mode) handles - strings according to the more liberal rules of RFC 2279. However, if - the string does not even conform to RFC 2279, the result is undefined. + strings according to the more liberal rules of RFC 2279. However, if + the string does not even conform to RFC 2279, the result is undefined. Your program may crash. - If you want to process strings of values in the full range 0 to - 0x7FFFFFFF, encoded in a UTF-8-like manner as per the old RFC, you can + If you want to process strings of values in the full range 0 to + 0x7FFFFFFF, encoded in a UTF-8-like manner as per the old RFC, you can set PCRE_NO_UTF8_CHECK to bypass the more restrictive test. However, in this situation, you will have to apply your own validity check. General comments about UTF-8 mode - 1. An unbraced hexadecimal escape sequence (such as \xb3) matches a + 1. An unbraced hexadecimal escape sequence (such as \xb3) matches a two-byte UTF-8 character if the value is greater than 127. - 2. Octal numbers up to \777 are recognized, and match two-byte UTF-8 + 2. Octal numbers up to \777 are recognized, and match two-byte UTF-8 characters for values greater than \177. - 3. Repeat quantifiers apply to complete UTF-8 characters, not to indi- + 3. Repeat quantifiers apply to complete UTF-8 characters, not to indi- vidual bytes, for example: \x{100}{3}. - 4. The dot metacharacter matches one UTF-8 character instead of a sin- + 4. The dot metacharacter matches one UTF-8 character instead of a sin- gle byte. - 5. The escape sequence \C can be used to match a single byte in UTF-8 - mode, but its use can lead to some strange effects. This facility is + 5. The escape sequence \C can be used to match a single byte in UTF-8 + mode, but its use can lead to some strange effects. This facility is not available in the alternative matching function, pcre_dfa_exec(). - 6. The character escapes \b, \B, \d, \D, \s, \S, \w, and \W correctly - test characters of any code value, but the characters that PCRE recog- - nizes as digits, spaces, or word characters remain the same set as + 6. The character escapes \b, \B, \d, \D, \s, \S, \w, and \W correctly + test characters of any code value, but the characters that PCRE recog- + nizes as digits, spaces, or word characters remain the same set as before, all with values less than 256. This remains true even when PCRE - includes Unicode property support, because to do otherwise would slow - down PCRE in many common cases. If you really want to test for a wider - sense of, say, "digit", you must use Unicode property tests such as - \p{Nd}. Note that this also applies to \b, because it is defined in + includes Unicode property support, because to do otherwise would slow + down PCRE in many common cases. If you really want to test for a wider + sense of, say, "digit", you must use Unicode property tests such as + \p{Nd}. Note that this also applies to \b, because it is defined in terms of \w and \W. - 7. Similarly, characters that match the POSIX named character classes + 7. Similarly, characters that match the POSIX named character classes are all low-valued characters. - 8. However, the Perl 5.10 horizontal and vertical whitespace matching + 8. However, the Perl 5.10 horizontal and vertical whitespace matching escapes (\h, \H, \v, and \V) do match all the appropriate Unicode char- acters. - 9. Case-insensitive matching applies only to characters whose values - are less than 128, unless PCRE is built with Unicode property support. - Even when Unicode property support is available, PCRE still uses its - own character tables when checking the case of low-valued characters, - so as not to degrade performance. The Unicode property information is + 9. Case-insensitive matching applies only to characters whose values + are less than 128, unless PCRE is built with Unicode property support. + Even when Unicode property support is available, PCRE still uses its + own character tables when checking the case of low-valued characters, + so as not to degrade performance. The Unicode property information is used only for characters with higher values. Even when Unicode property support is available, PCRE supports case-insensitive matching only when - there is a one-to-one mapping between a letter's cases. There are a - small number of many-to-one mappings in Unicode; these are not sup- + there is a one-to-one mapping between a letter's cases. There are a + small number of many-to-one mappings in Unicode; these are not sup- ported by PCRE. @@ -253,14 +256,14 @@ AUTHOR University Computing Service Cambridge CB2 3QH, England. - Putting an actual email address here seems to have been a spam magnet, - so I've taken it away. If you want to email me, use my two initials, + Putting an actual email address here seems to have been a spam magnet, + so I've taken it away. If you want to email me, use my two initials, followed by the two digits 10, at the domain cam.ac.uk. REVISION - Last updated: 11 April 2009 + Last updated: 28 September 2009 Copyright (c) 1997-2009 University of Cambridge. ------------------------------------------------------------------------------ @@ -279,20 +282,25 @@ PCRE BUILD-TIME OPTIONS script, where the optional features are selected or deselected by pro- viding options to configure before running the make command. However, the same options can be selected in both Unix-like and non-Unix-like - environments using the GUI facility of CMakeSetup if you are using - CMake instead of configure to build PCRE. + environments using the GUI facility of cmake-gui if you are using CMake + instead of configure to build PCRE. + + There is a lot more information about building PCRE in non-Unix-like + environments in the file called NON_UNIX_USE, which is part of the PCRE + distribution. You should consult this file as well as the README file + if you are building in a non-Unix-like environment. The complete list of options for configure (which includes the standard - ones such as the selection of the installation directory) can be + ones such as the selection of the installation directory) can be obtained by running ./configure --help - The following sections include descriptions of options whose names + The following sections include descriptions of options whose names begin with --enable or --disable. These settings specify changes to the - defaults for the configure command. Because of the way that configure - works, --enable and --disable always come in pairs, so the complemen- - tary option always exists as well, but as it specifies the default, it + defaults for the configure command. Because of the way that configure + works, --enable and --disable always come in pairs, so the complemen- + tary option always exists as well, but as it specifies the default, it is not described. @@ -313,46 +321,46 @@ UTF-8 SUPPORT --enable-utf8 - to the configure command. Of itself, this does not make PCRE treat - strings as UTF-8. As well as compiling PCRE with this option, you also - have have to set the PCRE_UTF8 option when you call the pcre_compile() - function. + to the configure command. Of itself, this does not make PCRE treat + strings as UTF-8. As well as compiling PCRE with this option, you also + have have to set the PCRE_UTF8 option when you call the pcre_compile() + or pcre_compile2() functions. - If you set --enable-utf8 when compiling in an EBCDIC environment, PCRE + If you set --enable-utf8 when compiling in an EBCDIC environment, PCRE expects its input to be either ASCII or UTF-8 (depending on the runtime - option). It is not possible to support both EBCDIC and UTF-8 codes in - the same version of the library. Consequently, --enable-utf8 and + option). It is not possible to support both EBCDIC and UTF-8 codes in + the same version of the library. Consequently, --enable-utf8 and --enable-ebcdic are mutually exclusive. UNICODE CHARACTER PROPERTY SUPPORT - UTF-8 support allows PCRE to process character values greater than 255 - in the strings that it handles. On its own, however, it does not pro- + UTF-8 support allows PCRE to process character values greater than 255 + in the strings that it handles. On its own, however, it does not pro- vide any facilities for accessing the properties of such characters. If - you want to be able to use the pattern escapes \P, \p, and \X, which + you want to be able to use the pattern escapes \P, \p, and \X, which refer to Unicode character properties, you must add --enable-unicode-properties - to the configure command. This implies UTF-8 support, even if you have + to the configure command. This implies UTF-8 support, even if you have not explicitly requested it. - Including Unicode property support adds around 30K of tables to the - PCRE library. Only the general category properties such as Lu and Nd + Including Unicode property support adds around 30K of tables to the + PCRE library. Only the general category properties such as Lu and Nd are supported. Details are given in the pcrepattern documentation. CODE VALUE OF NEWLINE - By default, PCRE interprets the linefeed (LF) character as indicating - the end of a line. This is the normal newline character on Unix-like - systems. You can compile PCRE to use carriage return (CR) instead, by + By default, PCRE interprets the linefeed (LF) character as indicating + the end of a line. This is the normal newline character on Unix-like + systems. You can compile PCRE to use carriage return (CR) instead, by adding --enable-newline-is-cr - to the configure command. There is also a --enable-newline-is-lf + to the configure command. There is also a --enable-newline-is-lf option, which explicitly specifies linefeed as the newline character. Alternatively, you can specify that line endings are to be indicated by @@ -364,35 +372,35 @@ CODE VALUE OF NEWLINE --enable-newline-is-anycrlf - which causes PCRE to recognize any of the three sequences CR, LF, or + which causes PCRE to recognize any of the three sequences CR, LF, or CRLF as indicating a line ending. Finally, a fifth option, specified by --enable-newline-is-any causes PCRE to recognize any Unicode newline sequence. - Whatever line ending convention is selected when PCRE is built can be - overridden when the library functions are called. At build time it is + Whatever line ending convention is selected when PCRE is built can be + overridden when the library functions are called. At build time it is conventional to use the standard for your operating system. WHAT \R MATCHES - By default, the sequence \R in a pattern matches any Unicode newline - sequence, whatever has been selected as the line ending sequence. If + By default, the sequence \R in a pattern matches any Unicode newline + sequence, whatever has been selected as the line ending sequence. If you specify --enable-bsr-anycrlf - the default is changed so that \R matches only CR, LF, or CRLF. What- - ever is selected when PCRE is built can be overridden when the library + the default is changed so that \R matches only CR, LF, or CRLF. What- + ever is selected when PCRE is built can be overridden when the library functions are called. BUILDING SHARED AND STATIC LIBRARIES - The PCRE building process uses libtool to build both shared and static - Unix libraries by default. You can suppress one of these by adding one + The PCRE building process uses libtool to build both shared and static + Unix libraries by default. You can suppress one of these by adding one of --disable-shared @@ -404,9 +412,9 @@ BUILDING SHARED AND STATIC LIBRARIES POSIX MALLOC USAGE When PCRE is called through the POSIX interface (see the pcreposix doc- - umentation), additional working storage is required for holding the - pointers to capturing substrings, because PCRE requires three integers - per substring, whereas the POSIX interface provides only two. If the + umentation), additional working storage is required for holding the + pointers to capturing substrings, because PCRE requires three integers + per substring, whereas the POSIX interface provides only two. If the number of expected substrings is small, the wrapper function uses space on the stack, because this is faster than using malloc() for each call. The default threshold above which the stack is no longer used is 10; it @@ -419,50 +427,49 @@ POSIX MALLOC USAGE HANDLING VERY LARGE PATTERNS - Within a compiled pattern, offset values are used to point from one - part to another (for example, from an opening parenthesis to an alter- - nation metacharacter). By default, two-byte values are used for these - offsets, leading to a maximum size for a compiled pattern of around - 64K. This is sufficient to handle all but the most gigantic patterns. - Nevertheless, some people do want to process enormous patterns, so it - is possible to compile PCRE to use three-byte or four-byte offsets by - adding a setting such as + Within a compiled pattern, offset values are used to point from one + part to another (for example, from an opening parenthesis to an alter- + nation metacharacter). By default, two-byte values are used for these + offsets, leading to a maximum size for a compiled pattern of around + 64K. This is sufficient to handle all but the most gigantic patterns. + Nevertheless, some people do want to process truyl enormous patterns, + so it is possible to compile PCRE to use three-byte or four-byte off- + sets by adding a setting such as --with-link-size=3 - to the configure command. The value given must be 2, 3, or 4. Using - longer offsets slows down the operation of PCRE because it has to load + to the configure command. The value given must be 2, 3, or 4. Using + longer offsets slows down the operation of PCRE because it has to load additional bytes when handling them. AVOIDING EXCESSIVE STACK USAGE When matching with the pcre_exec() function, PCRE implements backtrack- - ing by making recursive calls to an internal function called match(). - In environments where the size of the stack is limited, this can se- - verely limit PCRE's operation. (The Unix environment does not usually + ing by making recursive calls to an internal function called match(). + In environments where the size of the stack is limited, this can se- + verely limit PCRE's operation. (The Unix environment does not usually suffer from this problem, but it may sometimes be necessary to increase - the maximum stack size. There is a discussion in the pcrestack docu- - mentation.) An alternative approach to recursion that uses memory from - the heap to remember data, instead of using recursive function calls, - has been implemented to work round the problem of limited stack size. + the maximum stack size. There is a discussion in the pcrestack docu- + mentation.) An alternative approach to recursion that uses memory from + the heap to remember data, instead of using recursive function calls, + has been implemented to work round the problem of limited stack size. If you want to build a version of PCRE that works this way, add --disable-stack-for-recursion - to the configure command. With this configuration, PCRE will use the - pcre_stack_malloc and pcre_stack_free variables to call memory manage- - ment functions. By default these point to malloc() and free(), but you - can replace the pointers so that your own functions are used. + to the configure command. With this configuration, PCRE will use the + pcre_stack_malloc and pcre_stack_free variables to call memory manage- + ment functions. By default these point to malloc() and free(), but you + can replace the pointers so that your own functions are used instead. - Separate functions are provided rather than using pcre_malloc and - pcre_free because the usage is very predictable: the block sizes - requested are always the same, and the blocks are always freed in - reverse order. A calling program might be able to implement optimized - functions that perform better than malloc() and free(). PCRE runs + Separate functions are provided rather than using pcre_malloc and + pcre_free because the usage is very predictable: the block sizes + requested are always the same, and the blocks are always freed in + reverse order. A calling program might be able to implement optimized + functions that perform better than malloc() and free(). PCRE runs noticeably more slowly when built in this way. This option affects only - the pcre_exec() function; it is not relevant for the the - pcre_dfa_exec() function. + the pcre_exec() function; it is not relevant for pcre_dfa_exec(). LIMITING PCRE RESOURCE USAGE @@ -551,7 +558,7 @@ PCRETEST OPTION FOR LIBREADLINE SUPPORT to the configure command, pcretest is linked with the libreadline library, and when its input is from a terminal, it reads it using the readline() function. This provides line-editing and history facilities. - Note that libreadline is GPL-licenced, so if you distribute a binary of + Note that libreadline is GPL-licensed, so if you distribute a binary of pcretest linked in this way, there may be licensing issues. Setting this option causes the -lreadline option to be added to the @@ -587,7 +594,7 @@ AUTHOR REVISION - Last updated: 17 March 2009 + Last updated: 29 September 2009 Copyright (c) 1997-2009 University of Cambridge. ------------------------------------------------------------------------------ @@ -675,13 +682,19 @@ THE ALTERNATIVE MATCHING ALGORITHM though it is not implemented as a traditional finite state machine (it keeps multiple states active simultaneously). + Although the general principle of this matching algorithm is that it + scans the subject string only once, without backtracking, there is one + exception: when a lookaround assertion is encountered, the characters + following or preceding the current point have to be independently + inspected. + The scan continues until either the end of the subject is reached, or there are no more unterminated paths. At this point, terminated paths represent the different matching possibilities (if there are none, the match has failed). Thus, if there is more than one possible match, this algorithm finds all of them, and in particular, it finds the long- - est. In PCRE, there is an option to stop the algorithm after the first - match (which is necessarily the shortest) has been found. + est. There is an option to stop the algorithm after the first match + (which is necessarily the shortest) is found. Note that all the matches that are found start at the same point in the subject. If the pattern @@ -751,24 +764,19 @@ ADVANTAGES OF THE ALTERNATIVE ALGORITHM more than one match using the standard algorithm, you have to do kludgy things with callouts. - 2. There is much better support for partial matching. The restrictions - on the content of the pattern that apply when using the standard algo- - rithm for partial matching do not apply to the alternative algorithm. - For non-anchored patterns, the starting position of a partial match is - available. - - 3. Because the alternative algorithm scans the subject string just + 2. Because the alternative algorithm scans the subject string just once, and never needs to backtrack, it is possible to pass very long subject strings to the matching function in several pieces, checking - for partial matching each time. + for partial matching each time. The pcrepartial documentation gives + details of partial matching. DISADVANTAGES OF THE ALTERNATIVE ALGORITHM The alternative algorithm suffers from a number of disadvantages: - 1. It is substantially slower than the standard algorithm. This is - partly because it has to search for all possible matches, but is also + 1. It is substantially slower than the standard algorithm. This is + partly because it has to search for all possible matches, but is also because it is less susceptible to optimization. 2. Capturing parentheses and back references are not supported. @@ -786,8 +794,8 @@ AUTHOR REVISION - Last updated: 19 April 2008 - Copyright (c) 1997-2008 University of Cambridge. + Last updated: 29 September 2009 + Copyright (c) 1997-2009 University of Cambridge. ------------------------------------------------------------------------------ @@ -898,16 +906,18 @@ PCRE API OVERVIEW pcre_exec() are used for compiling and matching regular expressions in a Perl-compatible manner. A sample program that demonstrates the sim- plest way of using them is provided in the file called pcredemo.c in - the source distribution. The pcresample documentation describes how to - compile and run it. + the PCRE source distribution. A listing of this program is given in the + pcredemo documentation, and the pcresample documentation describes how + to compile and run it. A second matching function, pcre_dfa_exec(), which is not Perl-compati- - ble, is also provided. This uses a different algorithm for the match- - ing. The alternative algorithm finds all possible matches (at a given - point in the subject), and scans the subject just once. However, this - algorithm does not return captured substrings. A description of the two - matching algorithms and their advantages and disadvantages is given in - the pcrematching documentation. + ble, is also provided. This uses a different algorithm for the match- + ing. The alternative algorithm finds all possible matches (at a given + point in the subject), and scans the subject just once (unless there + are lookbehind assertions). However, this algorithm does not return + captured substrings. A description of the two matching algorithms and + their advantages and disadvantages is given in the pcrematching docu- + mentation. In addition to the main compiling and matching functions, there are convenience functions for extracting captured substrings from a subject @@ -1117,7 +1127,9 @@ COMPILING A PATTERN Either of the functions pcre_compile() or pcre_compile2() can be called to compile a pattern into an internal form. The only difference between the two interfaces is that pcre_compile2() has an additional argument, - errorcodeptr, via which a numerical error code can be returned. + errorcodeptr, via which a numerical error code can be returned. To + avoid too much repetition, we refer just to pcre_compile() below, but + the information applies equally to pcre_compile2(). The pattern is a C string terminated by a binary zero, and is passed in the pattern argument. A pointer to a single block of memory that is @@ -1135,22 +1147,24 @@ COMPILING A PATTERN The options argument contains various bit settings that affect the com- pilation. It should be zero if no options are required. The available options are described below. Some of them (in particular, those that - are compatible with Perl, but also some others) can also be set and + are compatible with Perl, but some others as well) can also be set and unset from within the pattern (see the detailed description in the pcrepattern documentation). For those options that can be different in different parts of the pattern, the contents of the options argument - specifies their initial settings at the start of compilation and execu- - tion. The PCRE_ANCHORED and PCRE_NEWLINE_xxx options can be set at the - time of matching as well as at compile time. + specifies their settings at the start of compilation and execution. The + PCRE_ANCHORED, PCRE_BSR_xxx, and PCRE_NEWLINE_xxx options can be set at + the time of matching as well as at compile time. If errptr is NULL, pcre_compile() returns NULL immediately. Otherwise, if compilation of a pattern fails, pcre_compile() returns NULL, and sets the variable pointed to by errptr to point to a textual error mes- sage. This is a static string that is part of the library. You must not - try to free it. The offset from the start of the pattern to the charac- - ter where the error was discovered is placed in the variable pointed to - by erroffset, which must not be NULL. If it is, an immediate error is - given. + try to free it. The byte offset from the start of the pattern to the + character that was being processed when the error was discovered is + placed in the variable pointed to by erroffset, which must not be NULL. + If it is, an immediate error is given. Some errors are not detected + until checks are carried out when the whole pattern has been scanned; + in this case the offset is set to the end of the pattern. If pcre_compile2() is used instead of pcre_compile(), and the error- codeptr argument is not NULL, a non-zero error code number is returned @@ -1480,14 +1494,14 @@ STUDYING A PATTERN the results of the study. The returned value from pcre_study() can be passed directly to - pcre_exec(). However, a pcre_extra block also contains other fields - that can be set by the caller before the block is passed; these are - described below in the section on matching a pattern. + pcre_exec() or pcre_dfa_exec(). However, a pcre_extra block also con- + tains other fields that can be set by the caller before the block is + passed; these are described below in the section on matching a pattern. - If studying the pattern does not produce any additional information + If studying the pattern does not produce any useful information, pcre_study() returns NULL. In that circumstance, if the calling program - wants to pass any of the other fields to pcre_exec(), it must set up - its own pcre_extra block. + wants to pass any of the other fields to pcre_exec() or + pcre_dfa_exec(), it must set up its own pcre_extra block. The second argument of pcre_study() contains option bits. At present, no options are defined, and this argument should always be zero. @@ -1507,63 +1521,72 @@ STUDYING A PATTERN 0, /* no options exist */ &error); /* set to NULL or points to a message */ - At present, studying a pattern is useful only for non-anchored patterns - that do not have a single fixed starting character. A bitmap of possi- - ble starting bytes is created. + Studying a pattern does two things: first, a lower bound for the length + of subject string that is needed to match the pattern is computed. This + does not mean that there are any strings of that length that match, but + it does guarantee that no shorter strings match. The value is used by + pcre_exec() and pcre_dfa_exec() to avoid wasting time by trying to + match strings that are shorter than the lower bound. You can find out + the value in a calling program via the pcre_fullinfo() function. + + Studying a pattern is also useful for non-anchored patterns that do not + have a single fixed starting character. A bitmap of possible starting + bytes is created. This speeds up finding a position in the subject at + which to start matching. LOCALE SUPPORT - PCRE handles caseless matching, and determines whether characters are - letters, digits, or whatever, by reference to a set of tables, indexed - by character value. When running in UTF-8 mode, this applies only to - characters with codes less than 128. Higher-valued codes never match - escapes such as \w or \d, but can be tested with \p if PCRE is built - with Unicode character property support. The use of locales with Uni- - code is discouraged. If you are handling characters with codes greater - than 128, you should either use UTF-8 and Unicode, or use locales, but + PCRE handles caseless matching, and determines whether characters are + letters, digits, or whatever, by reference to a set of tables, indexed + by character value. When running in UTF-8 mode, this applies only to + characters with codes less than 128. Higher-valued codes never match + escapes such as \w or \d, but can be tested with \p if PCRE is built + with Unicode character property support. The use of locales with Uni- + code is discouraged. If you are handling characters with codes greater + than 128, you should either use UTF-8 and Unicode, or use locales, but not try to mix the two. - PCRE contains an internal set of tables that are used when the final - argument of pcre_compile() is NULL. These are sufficient for many + PCRE contains an internal set of tables that are used when the final + argument of pcre_compile() is NULL. These are sufficient for many applications. Normally, the internal tables recognize only ASCII char- acters. However, when PCRE is built, it is possible to cause the inter- nal tables to be rebuilt in the default "C" locale of the local system, which may cause them to be different. - The internal tables can always be overridden by tables supplied by the + The internal tables can always be overridden by tables supplied by the application that calls PCRE. These may be created in a different locale - from the default. As more and more applications change to using Uni- + from the default. As more and more applications change to using Uni- code, the need for this locale support is expected to die away. - External tables are built by calling the pcre_maketables() function, - which has no arguments, in the relevant locale. The result can then be - passed to pcre_compile() or pcre_exec() as often as necessary. For - example, to build and use tables that are appropriate for the French - locale (where accented characters with values greater than 128 are + External tables are built by calling the pcre_maketables() function, + which has no arguments, in the relevant locale. The result can then be + passed to pcre_compile() or pcre_exec() as often as necessary. For + example, to build and use tables that are appropriate for the French + locale (where accented characters with values greater than 128 are treated as letters), the following code could be used: setlocale(LC_CTYPE, "fr_FR"); tables = pcre_maketables(); re = pcre_compile(..., tables); - The locale name "fr_FR" is used on Linux and other Unix-like systems; + The locale name "fr_FR" is used on Linux and other Unix-like systems; if you are using Windows, the name for the French locale is "french". - When pcre_maketables() runs, the tables are built in memory that is - obtained via pcre_malloc. It is the caller's responsibility to ensure - that the memory containing the tables remains available for as long as + When pcre_maketables() runs, the tables are built in memory that is + obtained via pcre_malloc. It is the caller's responsibility to ensure + that the memory containing the tables remains available for as long as it is needed. The pointer that is passed to pcre_compile() is saved with the compiled - pattern, and the same tables are used via this pointer by pcre_study() + pattern, and the same tables are used via this pointer by pcre_study() and normally also by pcre_exec(). Thus, by default, for any single pat- tern, compilation, studying and matching all happen in the same locale, but different patterns can be compiled in different locales. - It is possible to pass a table pointer or NULL (indicating the use of - the internal tables) to pcre_exec(). Although not intended for this - purpose, this facility could be used to match a pattern in a different + It is possible to pass a table pointer or NULL (indicating the use of + the internal tables) to pcre_exec(). Although not intended for this + purpose, this facility could be used to match a pattern in a different locale from the one in which it was compiled. Passing table pointers at run time is discussed below in the section on matching a pattern. @@ -1573,15 +1596,15 @@ INFORMATION ABOUT A PATTERN int pcre_fullinfo(const pcre *code, const pcre_extra *extra, int what, void *where); - The pcre_fullinfo() function returns information about a compiled pat- + The pcre_fullinfo() function returns information about a compiled pat- tern. It replaces the obsolete pcre_info() function, which is neverthe- less retained for backwards compability (and is documented below). - The first argument for pcre_fullinfo() is a pointer to the compiled - pattern. The second argument is the result of pcre_study(), or NULL if - the pattern was not studied. The third argument specifies which piece - of information is required, and the fourth argument is a pointer to a - variable to receive the data. The yield of the function is zero for + The first argument for pcre_fullinfo() is a pointer to the compiled + pattern. The second argument is the result of pcre_study(), or NULL if + the pattern was not studied. The third argument specifies which piece + of information is required, and the fourth argument is a pointer to a + variable to receive the data. The yield of the function is zero for success, or one of the following negative numbers: PCRE_ERROR_NULL the argument code was NULL @@ -1589,9 +1612,9 @@ INFORMATION ABOUT A PATTERN PCRE_ERROR_BADMAGIC the "magic number" was not found PCRE_ERROR_BADOPTION the value of what was invalid - The "magic number" is placed at the start of each compiled pattern as - an simple check against passing an arbitrary memory pointer. Here is a - typical call of pcre_fullinfo(), to obtain the length of the compiled + The "magic number" is placed at the start of each compiled pattern as + an simple check against passing an arbitrary memory pointer. Here is a + typical call of pcre_fullinfo(), to obtain the length of the compiled pattern: int rc; @@ -1602,111 +1625,131 @@ INFORMATION ABOUT A PATTERN PCRE_INFO_SIZE, /* what is required */ &length); /* where to put the data */ - The possible values for the third argument are defined in pcre.h, and + The possible values for the third argument are defined in pcre.h, and are as follows: PCRE_INFO_BACKREFMAX - Return the number of the highest back reference in the pattern. The - fourth argument should point to an int variable. Zero is returned if + Return the number of the highest back reference in the pattern. The + fourth argument should point to an int variable. Zero is returned if there are no back references. PCRE_INFO_CAPTURECOUNT - Return the number of capturing subpatterns in the pattern. The fourth + Return the number of capturing subpatterns in the pattern. The fourth argument should point to an int variable. PCRE_INFO_DEFAULT_TABLES - Return a pointer to the internal default character tables within PCRE. - The fourth argument should point to an unsigned char * variable. This + Return a pointer to the internal default character tables within PCRE. + The fourth argument should point to an unsigned char * variable. This information call is provided for internal use by the pcre_study() func- - tion. External callers can cause PCRE to use its internal tables by + tion. External callers can cause PCRE to use its internal tables by passing a NULL table pointer. PCRE_INFO_FIRSTBYTE - Return information about the first byte of any matched string, for a - non-anchored pattern. The fourth argument should point to an int vari- - able. (This option used to be called PCRE_INFO_FIRSTCHAR; the old name + Return information about the first byte of any matched string, for a + non-anchored pattern. The fourth argument should point to an int vari- + able. (This option used to be called PCRE_INFO_FIRSTCHAR; the old name is still recognized for backwards compatibility.) - If there is a fixed first byte, for example, from a pattern such as + If there is a fixed first byte, for example, from a pattern such as (cat|cow|coyote), its value is returned. Otherwise, if either - (a) the pattern was compiled with the PCRE_MULTILINE option, and every + (a) the pattern was compiled with the PCRE_MULTILINE option, and every branch starts with "^", or (b) every branch of the pattern starts with ".*" and PCRE_DOTALL is not set (if it were set, the pattern would be anchored), - -1 is returned, indicating that the pattern matches only at the start - of a subject string or after any newline within the string. Otherwise + -1 is returned, indicating that the pattern matches only at the start + of a subject string or after any newline within the string. Otherwise -2 is returned. For anchored patterns, -2 is returned. PCRE_INFO_FIRSTTABLE - If the pattern was studied, and this resulted in the construction of a + If the pattern was studied, and this resulted in the construction of a 256-bit table indicating a fixed set of bytes for the first byte in any - matching string, a pointer to the table is returned. Otherwise NULL is - returned. The fourth argument should point to an unsigned char * vari- + matching string, a pointer to the table is returned. Otherwise NULL is + returned. The fourth argument should point to an unsigned char * vari- able. PCRE_INFO_HASCRORLF - Return 1 if the pattern contains any explicit matches for CR or LF - characters, otherwise 0. The fourth argument should point to an int - variable. An explicit match is either a literal CR or LF character, or + Return 1 if the pattern contains any explicit matches for CR or LF + characters, otherwise 0. The fourth argument should point to an int + variable. An explicit match is either a literal CR or LF character, or \r or \n. PCRE_INFO_JCHANGED - Return 1 if the (?J) or (?-J) option setting is used in the pattern, - otherwise 0. The fourth argument should point to an int variable. (?J) + Return 1 if the (?J) or (?-J) option setting is used in the pattern, + otherwise 0. The fourth argument should point to an int variable. (?J) and (?-J) set and unset the local PCRE_DUPNAMES option, respectively. PCRE_INFO_LASTLITERAL - Return the value of the rightmost literal byte that must exist in any - matched string, other than at its start, if such a byte has been + Return the value of the rightmost literal byte that must exist in any + matched string, other than at its start, if such a byte has been recorded. The fourth argument should point to an int variable. If there - is no such byte, -1 is returned. For anchored patterns, a last literal - byte is recorded only if it follows something of variable length. For + is no such byte, -1 is returned. For anchored patterns, a last literal + byte is recorded only if it follows something of variable length. For example, for the pattern /^a\d+z\d+/ the returned value is "z", but for /^a\dz\d/ the returned value is -1. + PCRE_INFO_MINLENGTH + + If the pattern was studied and a minimum length for matching subject + strings was computed, its value is returned. Otherwise the returned + value is -1. The value is a number of characters, not bytes (this may + be relevant in UTF-8 mode). The fourth argument should point to an int + variable. A non-negative value is a lower bound to the length of any + matching string. There may not be any strings of that length that do + actually match, but every string that does match is at least that long. + PCRE_INFO_NAMECOUNT PCRE_INFO_NAMEENTRYSIZE PCRE_INFO_NAMETABLE - PCRE supports the use of named as well as numbered capturing parenthe- - ses. The names are just an additional way of identifying the parenthe- + PCRE supports the use of named as well as numbered capturing parenthe- + ses. The names are just an additional way of identifying the parenthe- ses, which still acquire numbers. Several convenience functions such as - pcre_get_named_substring() are provided for extracting captured sub- - strings by name. It is also possible to extract the data directly, by - first converting the name to a number in order to access the correct + pcre_get_named_substring() are provided for extracting captured sub- + strings by name. It is also possible to extract the data directly, by + first converting the name to a number in order to access the correct pointers in the output vector (described with pcre_exec() below). To do - the conversion, you need to use the name-to-number map, which is + the conversion, you need to use the name-to-number map, which is described by these three values. The map consists of a number of fixed-size entries. PCRE_INFO_NAMECOUNT gives the number of entries, and PCRE_INFO_NAMEENTRYSIZE gives the size - of each entry; both of these return an int value. The entry size - depends on the length of the longest name. PCRE_INFO_NAMETABLE returns - a pointer to the first entry of the table (a pointer to char). The + of each entry; both of these return an int value. The entry size + depends on the length of the longest name. PCRE_INFO_NAMETABLE returns + a pointer to the first entry of the table (a pointer to char). The first two bytes of each entry are the number of the capturing parenthe- - sis, most significant byte first. The rest of the entry is the corre- - sponding name, zero terminated. The names are in alphabetical order. - When PCRE_DUPNAMES is set, duplicate names are in order of their paren- - theses numbers. For example, consider the following pattern (assume - PCRE_EXTENDED is set, so white space - including newlines - is - ignored): + sis, most significant byte first. The rest of the entry is the corre- + sponding name, zero terminated. + + The names are in alphabetical order. Duplicate names may appear if (?| + is used to create multiple groups with the same number, as described in + the section on duplicate subpattern numbers in the pcrepattern page. + Duplicate names for subpatterns with different numbers are permitted + only if PCRE_DUPNAMES is set. In all cases of duplicate names, they + appear in the table in the order in which they were found in the pat- + tern. In the absence of (?| this is the order of increasing number; + when (?| is used this is not necessarily the case because later subpat- + terns may have lower numbers. + + As a simple example of the name/number table, consider the following + pattern (assume PCRE_EXTENDED is set, so white space - including new- + lines - is ignored): (?<date> (?<year>(\d\d)?\d\d) - (?<month>\d\d) - (?<day>\d\d) ) - There are four named subpatterns, so the table has four entries, and - each entry in the table is eight bytes long. The table is as follows, + There are four named subpatterns, so the table has four entries, and + each entry in the table is eight bytes long. The table is as follows, with non-printing bytes shows in hexadecimal, and undefined bytes shown as ??: @@ -1715,29 +1758,31 @@ INFORMATION ABOUT A PATTERN 00 04 m o n t h 00 00 02 y e a r 00 ?? - When writing code to extract data from named subpatterns using the - name-to-number map, remember that the length of the entries is likely + When writing code to extract data from named subpatterns using the + name-to-number map, remember that the length of the entries is likely to be different for each compiled pattern. PCRE_INFO_OKPARTIAL - Return 1 if the pattern can be used for partial matching, otherwise 0. - The fourth argument should point to an int variable. The pcrepartial - documentation lists the restrictions that apply to patterns when par- - tial matching is used. + Return 1 if the pattern can be used for partial matching with + pcre_exec(), otherwise 0. The fourth argument should point to an int + variable. From release 8.00, this always returns 1, because the + restrictions that previously applied to partial matching have been + lifted. The pcrepartial documentation gives details of partial match- + ing. PCRE_INFO_OPTIONS - Return a copy of the options with which the pattern was compiled. The - fourth argument should point to an unsigned long int variable. These + Return a copy of the options with which the pattern was compiled. The + fourth argument should point to an unsigned long int variable. These option bits are those specified in the call to pcre_compile(), modified by any top-level option settings at the start of the pattern itself. In - other words, they are the options that will be in force when matching - starts. For example, if the pattern /(?im)abc(?-i)d/ is compiled with - the PCRE_EXTENDED option, the result is PCRE_CASELESS, PCRE_MULTILINE, + other words, they are the options that will be in force when matching + starts. For example, if the pattern /(?im)abc(?-i)d/ is compiled with + the PCRE_EXTENDED option, the result is PCRE_CASELESS, PCRE_MULTILINE, and PCRE_EXTENDED. - A pattern is automatically anchored by PCRE if all of its top-level + A pattern is automatically anchored by PCRE if all of its top-level alternatives begin with one of the following: ^ unless PCRE_MULTILINE is set @@ -1751,7 +1796,7 @@ INFORMATION ABOUT A PATTERN PCRE_INFO_SIZE - Return the size of the compiled pattern, that is, the value that was + Return the size of the compiled pattern, that is, the value that was passed as the argument to pcre_malloc() when PCRE was getting memory in which to place the compiled data. The fourth argument should point to a size_t variable. @@ -1759,9 +1804,10 @@ INFORMATION ABOUT A PATTERN PCRE_INFO_STUDYSIZE Return the size of the data block pointed to by the study_data field in - a pcre_extra block. That is, it is the value that was passed to + a pcre_extra block. That is, it is the value that was passed to pcre_malloc() when PCRE was getting memory into which to place the data - created by pcre_study(). The fourth argument should point to a size_t + created by pcre_study(). If pcre_extra is NULL, or there is no study + data, zero is returned. The fourth argument should point to a size_t variable. @@ -1817,7 +1863,7 @@ MATCHING A PATTERN: THE TRADITIONAL FUNCTION The function pcre_exec() is called to match a subject string against a compiled pattern, which is passed in the code argument. If the pattern - has been studied, the result of the study should be passed in the extra + was studied, the result of the study should be passed in the extra argument. This function is the main matching facility of the library, and it operates in a Perl-like manner. For specialist use there is also an alternative matching function, which is described below in the sec- @@ -1876,8 +1922,8 @@ MATCHING A PATTERN: THE TRADITIONAL FUNCTION The match_limit field provides a means of preventing PCRE from using up a vast amount of resources when running patterns that are not going to match, but which have a very large number of possibilities in their - search trees. The classic example is the use of nested unlimited - repeats. + search trees. The classic example is a pattern that uses nested unlim- + ited repeats. Internally, PCRE uses a function called match() which it calls repeat- edly (sometimes recursively). The limit set by match_limit is imposed @@ -1910,8 +1956,8 @@ MATCHING A PATTERN: THE TRADITIONAL FUNCTION PCRE_EXTRA_MATCH_LIMIT_RECURSION is set in the flags field. If the limit is exceeded, pcre_exec() returns PCRE_ERROR_RECURSIONLIMIT. - The pcre_callout field is used in conjunction with the "callout" fea- - ture, which is described in the pcrecallout documentation. + The callout_data field is used in conjunction with the "callout" fea- + ture, and is described in the pcrecallout documentation. The tables field is used to pass a character tables pointer to pcre_exec(); this overrides the value that is stored with the compiled @@ -1928,22 +1974,23 @@ MATCHING A PATTERN: THE TRADITIONAL FUNCTION The unused bits of the options argument for pcre_exec() must be zero. The only bits that may be set are PCRE_ANCHORED, PCRE_NEWLINE_xxx, - PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, PCRE_NO_START_OPTIMIZE, - PCRE_NO_UTF8_CHECK and PCRE_PARTIAL. + PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, PCRE_NOTEMPTY_ATSTART, + PCRE_NO_START_OPTIMIZE, PCRE_NO_UTF8_CHECK, PCRE_PARTIAL_SOFT, and + PCRE_PARTIAL_HARD. PCRE_ANCHORED - The PCRE_ANCHORED option limits pcre_exec() to matching at the first - matching position. If a pattern was compiled with PCRE_ANCHORED, or - turned out to be anchored by virtue of its contents, it cannot be made + The PCRE_ANCHORED option limits pcre_exec() to matching at the first + matching position. If a pattern was compiled with PCRE_ANCHORED, or + turned out to be anchored by virtue of its contents, it cannot be made unachored at matching time. PCRE_BSR_ANYCRLF PCRE_BSR_UNICODE These options (which are mutually exclusive) control what the \R escape - sequence matches. The choice is either to match only CR, LF, or CRLF, - or to match any Unicode newline sequence. These options override the + sequence matches. The choice is either to match only CR, LF, or CRLF, + or to match any Unicode newline sequence. These options override the choice that was made or defaulted when the pattern was compiled. PCRE_NEWLINE_CR @@ -1952,76 +1999,83 @@ MATCHING A PATTERN: THE TRADITIONAL FUNCTION PCRE_NEWLINE_ANYCRLF PCRE_NEWLINE_ANY - These options override the newline definition that was chosen or - defaulted when the pattern was compiled. For details, see the descrip- - tion of pcre_compile() above. During matching, the newline choice - affects the behaviour of the dot, circumflex, and dollar metacharac- - ters. It may also alter the way the match position is advanced after a + These options override the newline definition that was chosen or + defaulted when the pattern was compiled. For details, see the descrip- + tion of pcre_compile() above. During matching, the newline choice + affects the behaviour of the dot, circumflex, and dollar metacharac- + ters. It may also alter the way the match position is advanced after a match failure for an unanchored pattern. - When PCRE_NEWLINE_CRLF, PCRE_NEWLINE_ANYCRLF, or PCRE_NEWLINE_ANY is - set, and a match attempt for an unanchored pattern fails when the cur- - rent position is at a CRLF sequence, and the pattern contains no - explicit matches for CR or LF characters, the match position is + When PCRE_NEWLINE_CRLF, PCRE_NEWLINE_ANYCRLF, or PCRE_NEWLINE_ANY is + set, and a match attempt for an unanchored pattern fails when the cur- + rent position is at a CRLF sequence, and the pattern contains no + explicit matches for CR or LF characters, the match position is advanced by two characters instead of one, in other words, to after the CRLF. The above rule is a compromise that makes the most common cases work as - expected. For example, if the pattern is .+A (and the PCRE_DOTALL + expected. For example, if the pattern is .+A (and the PCRE_DOTALL option is not set), it does not match the string "\r\nA" because, after - failing at the start, it skips both the CR and the LF before retrying. - However, the pattern [\r\n]A does match that string, because it con- + failing at the start, it skips both the CR and the LF before retrying. + However, the pattern [\r\n]A does match that string, because it con- tains an explicit CR or LF reference, and so advances only by one char- acter after the first failure. An explicit match for CR of LF is either a literal appearance of one of - those characters, or one of the \r or \n escape sequences. Implicit - matches such as [^X] do not count, nor does \s (which includes CR and + those characters, or one of the \r or \n escape sequences. Implicit + matches such as [^X] do not count, nor does \s (which includes CR and LF in the characters that it matches). - Notwithstanding the above, anomalous effects may still occur when CRLF + Notwithstanding the above, anomalous effects may still occur when CRLF is a valid newline sequence and explicit \r or \n escapes appear in the pattern. PCRE_NOTBOL This option specifies that first character of the subject string is not - the beginning of a line, so the circumflex metacharacter should not - match before it. Setting this without PCRE_MULTILINE (at compile time) - causes circumflex never to match. This option affects only the behav- + the beginning of a line, so the circumflex metacharacter should not + match before it. Setting this without PCRE_MULTILINE (at compile time) + causes circumflex never to match. This option affects only the behav- iour of the circumflex metacharacter. It does not affect \A. PCRE_NOTEOL This option specifies that the end of the subject string is not the end - of a line, so the dollar metacharacter should not match it nor (except - in multiline mode) a newline immediately before it. Setting this with- + of a line, so the dollar metacharacter should not match it nor (except + in multiline mode) a newline immediately before it. Setting this with- out PCRE_MULTILINE (at compile time) causes dollar never to match. This - option affects only the behaviour of the dollar metacharacter. It does + option affects only the behaviour of the dollar metacharacter. It does not affect \Z or \z. PCRE_NOTEMPTY An empty string is not considered to be a valid match if this option is - set. If there are alternatives in the pattern, they are tried. If all - the alternatives match the empty string, the entire match fails. For + set. If there are alternatives in the pattern, they are tried. If all + the alternatives match the empty string, the entire match fails. For example, if the pattern a?b? - is applied to a string not beginning with "a" or "b", it matches the - empty string at the start of the subject. With PCRE_NOTEMPTY set, this + is applied to a string not beginning with "a" or "b", it matches an + empty string at the start of the subject. With PCRE_NOTEMPTY set, this match is not valid, so PCRE searches further into the string for occur- rences of "a" or "b". - Perl has no direct equivalent of PCRE_NOTEMPTY, but it does make a spe- - cial case of a pattern match of the empty string within its split() - function, and when using the /g modifier. It is possible to emulate - Perl's behaviour after matching a null string by first trying the match - again at the same offset with PCRE_NOTEMPTY and PCRE_ANCHORED, and then - if that fails by advancing the starting offset (see below) and trying - an ordinary match again. There is some code that demonstrates how to do - this in the pcredemo.c sample program. + PCRE_NOTEMPTY_ATSTART + + This is like PCRE_NOTEMPTY, except that an empty string match that is + not at the start of the subject is permitted. If the pattern is + anchored, such a match can occur only if the pattern contains \K. + + Perl has no direct equivalent of PCRE_NOTEMPTY or + PCRE_NOTEMPTY_ATSTART, but it does make a special case of a pattern + match of the empty string within its split() function, and when using + the /g modifier. It is possible to emulate Perl's behaviour after + matching a null string by first trying the match again at the same off- + set with PCRE_NOTEMPTY_ATSTART and PCRE_ANCHORED, and then if that + fails, by advancing the starting offset (see below) and trying an ordi- + nary match again. There is some code that demonstrates how to do this + in the pcredemo sample program. PCRE_NO_START_OPTIMIZE @@ -2056,128 +2110,132 @@ MATCHING A PATTERN: THE TRADITIONAL FUNCTION value of startoffset that does not point to the start of a UTF-8 char- acter, is undefined. Your program may crash. - PCRE_PARTIAL - - This option turns on the partial matching feature. If the subject - string fails to match the pattern, but at some point during the match- - ing process the end of the subject was reached (that is, the subject - partially matches the pattern and the failure to match occurred only - because there were not enough subject characters), pcre_exec() returns - PCRE_ERROR_PARTIAL instead of PCRE_ERROR_NOMATCH. When PCRE_PARTIAL is - used, there are restrictions on what may appear in the pattern. These - are discussed in the pcrepartial documentation. + PCRE_PARTIAL_HARD + PCRE_PARTIAL_SOFT + + These options turn on the partial matching feature. For backwards com- + patibility, PCRE_PARTIAL is a synonym for PCRE_PARTIAL_SOFT. A partial + match occurs if the end of the subject string is reached successfully, + but there are not enough subject characters to complete the match. If + this happens when PCRE_PARTIAL_HARD is set, pcre_exec() immediately + returns PCRE_ERROR_PARTIAL. Otherwise, if PCRE_PARTIAL_SOFT is set, + matching continues by testing any other alternatives. Only if they all + fail is PCRE_ERROR_PARTIAL returned (instead of PCRE_ERROR_NOMATCH). + The portion of the string that was inspected when the partial match was + found is set as the first matching string. There is a more detailed + discussion in the pcrepartial documentation. The string to be matched by pcre_exec() - The subject string is passed to pcre_exec() as a pointer in subject, a + The subject string is passed to pcre_exec() as a pointer in subject, a length (in bytes) in length, and a starting byte offset in startoffset. In UTF-8 mode, the byte offset must point to the start of a UTF-8 char- - acter. Unlike the pattern string, the subject may contain binary zero - bytes. When the starting offset is zero, the search for a match starts - at the beginning of the subject, and this is by far the most common + acter. Unlike the pattern string, the subject may contain binary zero + bytes. When the starting offset is zero, the search for a match starts + at the beginning of the subject, and this is by far the most common case. - A non-zero starting offset is useful when searching for another match - in the same subject by calling pcre_exec() again after a previous suc- - cess. Setting startoffset differs from just passing over a shortened - string and setting PCRE_NOTBOL in the case of a pattern that begins + A non-zero starting offset is useful when searching for another match + in the same subject by calling pcre_exec() again after a previous suc- + cess. Setting startoffset differs from just passing over a shortened + string and setting PCRE_NOTBOL in the case of a pattern that begins with any kind of lookbehind. For example, consider the pattern \Biss\B - which finds occurrences of "iss" in the middle of words. (\B matches - only if the current position in the subject is not a word boundary.) - When applied to the string "Mississipi" the first call to pcre_exec() - finds the first occurrence. If pcre_exec() is called again with just - the remainder of the subject, namely "issipi", it does not match, + which finds occurrences of "iss" in the middle of words. (\B matches + only if the current position in the subject is not a word boundary.) + When applied to the string "Mississipi" the first call to pcre_exec() + finds the first occurrence. If pcre_exec() is called again with just + the remainder of the subject, namely "issipi", it does not match, because \B is always false at the start of the subject, which is deemed - to be a word boundary. However, if pcre_exec() is passed the entire + to be a word boundary. However, if pcre_exec() is passed the entire string again, but with startoffset set to 4, it finds the second occur- - rence of "iss" because it is able to look behind the starting point to + rence of "iss" because it is able to look behind the starting point to discover that it is preceded by a letter. - If a non-zero starting offset is passed when the pattern is anchored, + If a non-zero starting offset is passed when the pattern is anchored, one attempt to match at the given offset is made. This can only succeed - if the pattern does not require the match to be at the start of the + if the pattern does not require the match to be at the start of the subject. How pcre_exec() returns captured substrings - In general, a pattern matches a certain portion of the subject, and in - addition, further substrings from the subject may be picked out by - parts of the pattern. Following the usage in Jeffrey Friedl's book, - this is called "capturing" in what follows, and the phrase "capturing - subpattern" is used for a fragment of a pattern that picks out a sub- - string. PCRE supports several other kinds of parenthesized subpattern + In general, a pattern matches a certain portion of the subject, and in + addition, further substrings from the subject may be picked out by + parts of the pattern. Following the usage in Jeffrey Friedl's book, + this is called "capturing" in what follows, and the phrase "capturing + subpattern" is used for a fragment of a pattern that picks out a sub- + string. PCRE supports several other kinds of parenthesized subpattern that do not cause substrings to be captured. Captured substrings are returned to the caller via a vector of integers - whose address is passed in ovector. The number of elements in the vec- - tor is passed in ovecsize, which must be a non-negative number. Note: + whose address is passed in ovector. The number of elements in the vec- + tor is passed in ovecsize, which must be a non-negative number. Note: this argument is NOT the size of ovector in bytes. - The first two-thirds of the vector is used to pass back captured sub- - strings, each substring using a pair of integers. The remaining third - of the vector is used as workspace by pcre_exec() while matching cap- - turing subpatterns, and is not available for passing back information. - The number passed in ovecsize should always be a multiple of three. If + The first two-thirds of the vector is used to pass back captured sub- + strings, each substring using a pair of integers. The remaining third + of the vector is used as workspace by pcre_exec() while matching cap- + turing subpatterns, and is not available for passing back information. + The number passed in ovecsize should always be a multiple of three. If it is not, it is rounded down. - When a match is successful, information about captured substrings is - returned in pairs of integers, starting at the beginning of ovector, - and continuing up to two-thirds of its length at the most. The first - element of each pair is set to the byte offset of the first character - in a substring, and the second is set to the byte offset of the first - character after the end of a substring. Note: these values are always + When a match is successful, information about captured substrings is + returned in pairs of integers, starting at the beginning of ovector, + and continuing up to two-thirds of its length at the most. The first + element of each pair is set to the byte offset of the first character + in a substring, and the second is set to the byte offset of the first + character after the end of a substring. Note: these values are always byte offsets, even in UTF-8 mode. They are not character counts. - The first pair of integers, ovector[0] and ovector[1], identify the - portion of the subject string matched by the entire pattern. The next - pair is used for the first capturing subpattern, and so on. The value + The first pair of integers, ovector[0] and ovector[1], identify the + portion of the subject string matched by the entire pattern. The next + pair is used for the first capturing subpattern, and so on. The value returned by pcre_exec() is one more than the highest numbered pair that - has been set. For example, if two substrings have been captured, the - returned value is 3. If there are no capturing subpatterns, the return + has been set. For example, if two substrings have been captured, the + returned value is 3. If there are no capturing subpatterns, the return value from a successful match is 1, indicating that just the first pair of offsets has been set. If a capturing subpattern is matched repeatedly, it is the last portion of the string that it matched that is returned. - If the vector is too small to hold all the captured substring offsets, + If the vector is too small to hold all the captured substring offsets, it is used as far as possible (up to two-thirds of its length), and the - function returns a value of zero. If the substring offsets are not of - interest, pcre_exec() may be called with ovector passed as NULL and - ovecsize as zero. However, if the pattern contains back references and - the ovector is not big enough to remember the related substrings, PCRE - has to get additional memory for use during matching. Thus it is usu- + function returns a value of zero. If the substring offsets are not of + interest, pcre_exec() may be called with ovector passed as NULL and + ovecsize as zero. However, if the pattern contains back references and + the ovector is not big enough to remember the related substrings, PCRE + has to get additional memory for use during matching. Thus it is usu- ally advisable to supply an ovector. - The pcre_info() function can be used to find out how many capturing - subpatterns there are in a compiled pattern. The smallest size for - ovector that will allow for n captured substrings, in addition to the + The pcre_fullinfo() function can be used to find out how many capturing + subpatterns there are in a compiled pattern. The smallest size for + ovector that will allow for n captured substrings, in addition to the offsets of the substring matched by the whole pattern, is (n+1)*3. - It is possible for capturing subpattern number n+1 to match some part + It is possible for capturing subpattern number n+1 to match some part of the subject when subpattern n has not been used at all. For example, - if the string "abc" is matched against the pattern (a|(z))(bc) the + if the string "abc" is matched against the pattern (a|(z))(bc) the return from the function is 4, and subpatterns 1 and 3 are matched, but - 2 is not. When this happens, both values in the offset pairs corre- + 2 is not. When this happens, both values in the offset pairs corre- sponding to unused subpatterns are set to -1. - Offset values that correspond to unused subpatterns at the end of the - expression are also set to -1. For example, if the string "abc" is - matched against the pattern (abc)(x(yz)?)? subpatterns 2 and 3 are not - matched. The return from the function is 2, because the highest used + Offset values that correspond to unused subpatterns at the end of the + expression are also set to -1. For example, if the string "abc" is + matched against the pattern (abc)(x(yz)?)? subpatterns 2 and 3 are not + matched. The return from the function is 2, because the highest used capturing subpattern number is 1. However, you can refer to the offsets - for the second and third capturing subpatterns if you wish (assuming + for the second and third capturing subpatterns if you wish (assuming the vector is large enough, of course). - Some convenience functions are provided for extracting the captured + Some convenience functions are provided for extracting the captured substrings as separate strings. These are described below. Error return values from pcre_exec() - If pcre_exec() fails, it returns a negative number. The following are + If pcre_exec() fails, it returns a negative number. The following are defined in the header file: PCRE_ERROR_NOMATCH (-1) @@ -2186,7 +2244,7 @@ MATCHING A PATTERN: THE TRADITIONAL FUNCTION PCRE_ERROR_NULL (-2) - Either code or subject was passed as NULL, or ovector was NULL and + Either code or subject was passed as NULL, or ovector was NULL and ovecsize was not zero. PCRE_ERROR_BADOPTION (-3) @@ -2195,65 +2253,66 @@ MATCHING A PATTERN: THE TRADITIONAL FUNCTION PCRE_ERROR_BADMAGIC (-4) - PCRE stores a 4-byte "magic number" at the start of the compiled code, + PCRE stores a 4-byte "magic number" at the start of the compiled code, to catch the case when it is passed a junk pointer and to detect when a pattern that was compiled in an environment of one endianness is run in - an environment with the other endianness. This is the error that PCRE + an environment with the other endianness. This is the error that PCRE gives when the magic number is not present. PCRE_ERROR_UNKNOWN_OPCODE (-5) While running the pattern match, an unknown item was encountered in the - compiled pattern. This error could be caused by a bug in PCRE or by + compiled pattern. This error could be caused by a bug in PCRE or by overwriting of the compiled pattern. PCRE_ERROR_NOMEMORY (-6) - If a pattern contains back references, but the ovector that is passed + If a pattern contains back references, but the ovector that is passed to pcre_exec() is not big enough to remember the referenced substrings, - PCRE gets a block of memory at the start of matching to use for this - purpose. If the call via pcre_malloc() fails, this error is given. The + PCRE gets a block of memory at the start of matching to use for this + purpose. If the call via pcre_malloc() fails, this error is given. The memory is automatically freed at the end of matching. PCRE_ERROR_NOSUBSTRING (-7) - This error is used by the pcre_copy_substring(), pcre_get_substring(), + This error is used by the pcre_copy_substring(), pcre_get_substring(), and pcre_get_substring_list() functions (see below). It is never returned by pcre_exec(). PCRE_ERROR_MATCHLIMIT (-8) - The backtracking limit, as specified by the match_limit field in a - pcre_extra structure (or defaulted) was reached. See the description + The backtracking limit, as specified by the match_limit field in a + pcre_extra structure (or defaulted) was reached. See the description above. PCRE_ERROR_CALLOUT (-9) This error is never generated by pcre_exec() itself. It is provided for - use by callout functions that want to yield a distinctive error code. + use by callout functions that want to yield a distinctive error code. See the pcrecallout documentation for details. PCRE_ERROR_BADUTF8 (-10) - A string that contains an invalid UTF-8 byte sequence was passed as a + A string that contains an invalid UTF-8 byte sequence was passed as a subject. PCRE_ERROR_BADUTF8_OFFSET (-11) The UTF-8 byte sequence that was passed as a subject was valid, but the - value of startoffset did not point to the beginning of a UTF-8 charac- + value of startoffset did not point to the beginning of a UTF-8 charac- ter. PCRE_ERROR_PARTIAL (-12) - The subject string did not match, but it did match partially. See the + The subject string did not match, but it did match partially. See the pcrepartial documentation for details of partial matching. PCRE_ERROR_BADPARTIAL (-13) - The PCRE_PARTIAL option was used with a compiled pattern containing - items that are not supported for partial matching. See the pcrepartial - documentation for details of partial matching. + This code is no longer in use. It was formerly returned when the + PCRE_PARTIAL option was used with a compiled pattern containing items + that were not supported for partial matching. From release 8.00 + onwards, there are no restrictions on partial matching. PCRE_ERROR_INTERNAL (-14) @@ -2412,10 +2471,13 @@ EXTRACTING CAPTURED SUBSTRINGS BY NAME ate. NOTE: If PCRE_DUPNAMES is set and there are duplicate names, the behaviour may not be what you want (see the next section). - Warning: If the pattern uses the "(?|" feature to set up multiple sub- - patterns with the same number, you cannot use names to distinguish - them, because names are not included in the compiled code. The matching - process uses only numbers. + Warning: If the pattern uses the (?| feature to set up multiple subpat- + terns with the same number, as described in the section on duplicate + subpattern numbers in the pcrepattern page, you cannot use names to + distinguish the different subpatterns, because names are not included + in the compiled code. The matching process uses only numbers. For this + reason, the use of different names for subpatterns of the same number + causes an error at compile time. DUPLICATE SUBPATTERN NAMES @@ -2423,47 +2485,51 @@ DUPLICATE SUBPATTERN NAMES int pcre_get_stringtable_entries(const pcre *code, const char *name, char **first, char **last); - When a pattern is compiled with the PCRE_DUPNAMES option, names for - subpatterns are not required to be unique. Normally, patterns with - duplicate names are such that in any one match, only one of the named - subpatterns participates. An example is shown in the pcrepattern docu- - mentation. + When a pattern is compiled with the PCRE_DUPNAMES option, names for + subpatterns are not required to be unique. (Duplicate names are always + allowed for subpatterns with the same number, created by using the (?| + feature. Indeed, if such subpatterns are named, they are required to + use the same names.) + + Normally, patterns with duplicate names are such that in any one match, + only one of the named subpatterns participates. An example is shown in + the pcrepattern documentation. - When duplicates are present, pcre_copy_named_substring() and - pcre_get_named_substring() return the first substring corresponding to - the given name that is set. If none are set, PCRE_ERROR_NOSUBSTRING - (-7) is returned; no data is returned. The pcre_get_stringnumber() - function returns one of the numbers that are associated with the name, + When duplicates are present, pcre_copy_named_substring() and + pcre_get_named_substring() return the first substring corresponding to + the given name that is set. If none are set, PCRE_ERROR_NOSUBSTRING + (-7) is returned; no data is returned. The pcre_get_stringnumber() + function returns one of the numbers that are associated with the name, but it is not defined which it is. - If you want to get full details of all captured substrings for a given - name, you must use the pcre_get_stringtable_entries() function. The + If you want to get full details of all captured substrings for a given + name, you must use the pcre_get_stringtable_entries() function. The first argument is the compiled pattern, and the second is the name. The - third and fourth are pointers to variables which are updated by the + third and fourth are pointers to variables which are updated by the function. After it has run, they point to the first and last entries in - the name-to-number table for the given name. The function itself - returns the length of each entry, or PCRE_ERROR_NOSUBSTRING (-7) if - there are none. The format of the table is described above in the sec- - tion entitled Information about a pattern. Given all the relevant - entries for the name, you can extract each of their numbers, and hence + the name-to-number table for the given name. The function itself + returns the length of each entry, or PCRE_ERROR_NOSUBSTRING (-7) if + there are none. The format of the table is described above in the sec- + tion entitled Information about a pattern. Given all the relevant + entries for the name, you can extract each of their numbers, and hence the captured data, if any. FINDING ALL POSSIBLE MATCHES - The traditional matching function uses a similar algorithm to Perl, + The traditional matching function uses a similar algorithm to Perl, which stops when it finds the first match, starting at a given point in - the subject. If you want to find all possible matches, or the longest - possible match, consider using the alternative matching function (see - below) instead. If you cannot use the alternative function, but still - need to find all possible matches, you can kludge it up by making use + the subject. If you want to find all possible matches, or the longest + possible match, consider using the alternative matching function (see + below) instead. If you cannot use the alternative function, but still + need to find all possible matches, you can kludge it up by making use of the callout facility, which is described in the pcrecallout documen- tation. What you have to do is to insert a callout right at the end of the pat- - tern. When your callout function is called, extract and save the cur- - rent matched substring. Then return 1, which forces pcre_exec() to - backtrack and try other alternatives. Ultimately, when it runs out of + tern. When your callout function is called, extract and save the cur- + rent matched substring. Then return 1, which forces pcre_exec() to + backtrack and try other alternatives. Ultimately, when it runs out of matches, pcre_exec() will yield PCRE_ERROR_NOMATCH. @@ -2474,14 +2540,15 @@ MATCHING A PATTERN: THE ALTERNATIVE FUNCTION int options, int *ovector, int ovecsize, int *workspace, int wscount); - The function pcre_dfa_exec() is called to match a subject string - against a compiled pattern, using a matching algorithm that scans the - subject string just once, and does not backtrack. This has different - characteristics to the normal algorithm, and is not compatible with - Perl. Some of the features of PCRE patterns are not supported. Never- - theless, there are times when this kind of matching can be useful. For - a discussion of the two matching algorithms, see the pcrematching docu- - mentation. + The function pcre_dfa_exec() is called to match a subject string + against a compiled pattern, using a matching algorithm that scans the + subject string just once, and does not backtrack. This has different + characteristics to the normal algorithm, and is not compatible with + Perl. Some of the features of PCRE patterns are not supported. Never- + theless, there are times when this kind of matching can be useful. For + a discussion of the two matching algorithms, and a list of features + that pcre_dfa_exec() does not support, see the pcrematching documenta- + tion. The arguments for the pcre_dfa_exec() function are the same as for pcre_exec(), plus two extras. The ovector argument is used in a differ- @@ -2516,38 +2583,43 @@ MATCHING A PATTERN: THE ALTERNATIVE FUNCTION The unused bits of the options argument for pcre_dfa_exec() must be zero. The only bits that may be set are PCRE_ANCHORED, PCRE_NEW- - LINE_xxx, PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, PCRE_NO_UTF8_CHECK, - PCRE_PARTIAL, PCRE_DFA_SHORTEST, and PCRE_DFA_RESTART. All but the last - three of these are the same as for pcre_exec(), so their description is - not repeated here. - - PCRE_PARTIAL - - This has the same general effect as it does for pcre_exec(), but the - details are slightly different. When PCRE_PARTIAL is set for - pcre_dfa_exec(), the return code PCRE_ERROR_NOMATCH is converted into - PCRE_ERROR_PARTIAL if the end of the subject is reached, there have - been no complete matches, but there is still at least one matching pos- - sibility. The portion of the string that provided the partial match is - set as the first matching string. + LINE_xxx, PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, + PCRE_NOTEMPTY_ATSTART, PCRE_NO_UTF8_CHECK, PCRE_PARTIAL_HARD, PCRE_PAR- + TIAL_SOFT, PCRE_DFA_SHORTEST, and PCRE_DFA_RESTART. All but the last + four of these are exactly the same as for pcre_exec(), so their + description is not repeated here. + + PCRE_PARTIAL_HARD + PCRE_PARTIAL_SOFT + + These have the same general effect as they do for pcre_exec(), but the + details are slightly different. When PCRE_PARTIAL_HARD is set for + pcre_dfa_exec(), it returns PCRE_ERROR_PARTIAL if the end of the sub- + ject is reached and there is still at least one matching possibility + that requires additional characters. This happens even if some complete + matches have also been found. When PCRE_PARTIAL_SOFT is set, the return + code PCRE_ERROR_NOMATCH is converted into PCRE_ERROR_PARTIAL if the end + of the subject is reached, there have been no complete matches, but + there is still at least one matching possibility. The portion of the + string that was inspected when the longest partial match was found is + set as the first matching string in both cases. PCRE_DFA_SHORTEST - Setting the PCRE_DFA_SHORTEST option causes the matching algorithm to + Setting the PCRE_DFA_SHORTEST option causes the matching algorithm to stop as soon as it has found one match. Because of the way the alterna- - tive algorithm works, this is necessarily the shortest possible match + tive algorithm works, this is necessarily the shortest possible match at the first possible matching point in the subject string. PCRE_DFA_RESTART - When pcre_dfa_exec() is called with the PCRE_PARTIAL option, and - returns a partial match, it is possible to call it again, with addi- - tional subject characters, and have it continue with the same match. - The PCRE_DFA_RESTART option requests this action; when it is set, the - workspace and wscount options must reference the same vector as before - because data about the match so far is left in them after a partial - match. There is more discussion of this facility in the pcrepartial - documentation. + When pcre_dfa_exec() returns a partial match, it is possible to call it + again, with additional subject characters, and have it continue with + the same match. The PCRE_DFA_RESTART option requests this action; when + it is set, the workspace and wscount options must reference the same + vector as before because data about the match so far is left in them + after a partial match. There is more discussion of this facility in the + pcrepartial documentation. Successful returns from pcre_dfa_exec() @@ -2636,7 +2708,7 @@ AUTHOR REVISION - Last updated: 11 April 2009 + Last updated: 03 October 2009 Copyright (c) 1997-2009 University of Cambridge. ------------------------------------------------------------------------------ @@ -2666,10 +2738,10 @@ PCRE CALLOUTS (?C1)abc(?C2)def - If the PCRE_AUTO_CALLOUT option bit is set when pcre_compile() is - called, PCRE automatically inserts callouts, all with number 255, - before each item in the pattern. For example, if PCRE_AUTO_CALLOUT is - used with the pattern + If the PCRE_AUTO_CALLOUT option bit is set when pcre_compile() or + pcre_compile2() is called, PCRE automatically inserts callouts, all + with number 255, before each item in the pattern. For example, if + PCRE_AUTO_CALLOUT is used with the pattern A(\d{2}|--) @@ -2698,18 +2770,23 @@ MISSING CALLOUTS ever start, and the callout is never reached. However, with "abyd", though the result is still no match, the callout is obeyed. - You can disable these optimizations by passing the PCRE_NO_START_OPTI- - MIZE option to pcre_exec() or pcre_dfa_exec(). This slows down the - matching process, but does ensure that callouts such as the example + If the pattern is studied, PCRE knows the minimum length of a matching + string, and will immediately give a "no match" return without actually + running a match if the subject is not long enough, or, for unanchored + patterns, if it has been scanned far enough. + + You can disable these optimizations by passing the PCRE_NO_START_OPTI- + MIZE option to pcre_exec() or pcre_dfa_exec(). This slows down the + matching process, but does ensure that callouts such as the example above are obeyed. THE CALLOUT INTERFACE - During matching, when PCRE reaches a callout point, the external func- - tion defined by pcre_callout is called (if it is set). This applies to - both the pcre_exec() and the pcre_dfa_exec() matching functions. The - only argument to the callout function is a pointer to a pcre_callout + During matching, when PCRE reaches a callout point, the external func- + tion defined by pcre_callout is called (if it is set). This applies to + both the pcre_exec() and the pcre_dfa_exec() matching functions. The + only argument to the callout function is a pointer to a pcre_callout block. This structure contains the following fields: int version; @@ -2725,81 +2802,81 @@ THE CALLOUT INTERFACE int pattern_position; int next_item_length; - The version field is an integer containing the version number of the - block format. The initial version was 0; the current version is 1. The - version number will change again in future if additional fields are + The version field is an integer containing the version number of the + block format. The initial version was 0; the current version is 1. The + version number will change again in future if additional fields are added, but the intention is never to remove any of the existing fields. - The callout_number field contains the number of the callout, as com- - piled into the pattern (that is, the number after ?C for manual call- + The callout_number field contains the number of the callout, as com- + piled into the pattern (that is, the number after ?C for manual call- outs, and 255 for automatically generated callouts). - The offset_vector field is a pointer to the vector of offsets that was - passed by the caller to pcre_exec() or pcre_dfa_exec(). When - pcre_exec() is used, the contents can be inspected in order to extract - substrings that have been matched so far, in the same way as for - extracting substrings after a match has completed. For pcre_dfa_exec() + The offset_vector field is a pointer to the vector of offsets that was + passed by the caller to pcre_exec() or pcre_dfa_exec(). When + pcre_exec() is used, the contents can be inspected in order to extract + substrings that have been matched so far, in the same way as for + extracting substrings after a match has completed. For pcre_dfa_exec() this field is not useful. The subject and subject_length fields contain copies of the values that were passed to pcre_exec(). - The start_match field normally contains the offset within the subject - at which the current match attempt started. However, if the escape - sequence \K has been encountered, this value is changed to reflect the - modified starting point. If the pattern is not anchored, the callout + The start_match field normally contains the offset within the subject + at which the current match attempt started. However, if the escape + sequence \K has been encountered, this value is changed to reflect the + modified starting point. If the pattern is not anchored, the callout function may be called several times from the same point in the pattern for different starting points in the subject. - The current_position field contains the offset within the subject of + The current_position field contains the offset within the subject of the current match pointer. - When the pcre_exec() function is used, the capture_top field contains - one more than the number of the highest numbered captured substring so - far. If no substrings have been captured, the value of capture_top is - one. This is always the case when pcre_dfa_exec() is used, because it + When the pcre_exec() function is used, the capture_top field contains + one more than the number of the highest numbered captured substring so + far. If no substrings have been captured, the value of capture_top is + one. This is always the case when pcre_dfa_exec() is used, because it does not support captured substrings. - The capture_last field contains the number of the most recently cap- - tured substring. If no substrings have been captured, its value is -1. + The capture_last field contains the number of the most recently cap- + tured substring. If no substrings have been captured, its value is -1. This is always the case when pcre_dfa_exec() is used. - The callout_data field contains a value that is passed to pcre_exec() - or pcre_dfa_exec() specifically so that it can be passed back in call- - outs. It is passed in the pcre_callout field of the pcre_extra data - structure. If no such data was passed, the value of callout_data in a - pcre_callout block is NULL. There is a description of the pcre_extra + The callout_data field contains a value that is passed to pcre_exec() + or pcre_dfa_exec() specifically so that it can be passed back in call- + outs. It is passed in the pcre_callout field of the pcre_extra data + structure. If no such data was passed, the value of callout_data in a + pcre_callout block is NULL. There is a description of the pcre_extra structure in the pcreapi documentation. - The pattern_position field is present from version 1 of the pcre_call- + The pattern_position field is present from version 1 of the pcre_call- out structure. It contains the offset to the next item to be matched in the pattern string. - The next_item_length field is present from version 1 of the pcre_call- + The next_item_length field is present from version 1 of the pcre_call- out structure. It contains the length of the next item to be matched in - the pattern string. When the callout immediately precedes an alterna- - tion bar, a closing parenthesis, or the end of the pattern, the length - is zero. When the callout precedes an opening parenthesis, the length + the pattern string. When the callout immediately precedes an alterna- + tion bar, a closing parenthesis, or the end of the pattern, the length + is zero. When the callout precedes an opening parenthesis, the length is that of the entire subpattern. - The pattern_position and next_item_length fields are intended to help - in distinguishing between different automatic callouts, which all have + The pattern_position and next_item_length fields are intended to help + in distinguishing between different automatic callouts, which all have the same callout number. However, they are set for all callouts. RETURN VALUES - The external callout function returns an integer to PCRE. If the value - is zero, matching proceeds as normal. If the value is greater than - zero, matching fails at the current point, but the testing of other + The external callout function returns an integer to PCRE. If the value + is zero, matching proceeds as normal. If the value is greater than + zero, matching fails at the current point, but the testing of other matching possibilities goes ahead, just as if a lookahead assertion had - failed. If the value is less than zero, the match is abandoned, and - pcre_exec() (or pcre_dfa_exec()) returns the negative value. + failed. If the value is less than zero, the match is abandoned, and + pcre_exec() or pcre_dfa_exec() returns the negative value. - Negative values should normally be chosen from the set of + Negative values should normally be chosen from the set of PCRE_ERROR_xxx values. In particular, PCRE_ERROR_NOMATCH forces a stan- - dard "no match" failure. The error number PCRE_ERROR_CALLOUT is - reserved for use by callout functions; it will never be used by PCRE + dard "no match" failure. The error number PCRE_ERROR_CALLOUT is + reserved for use by callout functions; it will never be used by PCRE itself. @@ -2812,7 +2889,7 @@ AUTHOR REVISION - Last updated: 15 March 2009 + Last updated: 29 September 2009 Copyright (c) 1997-2009 University of Cambridge. ------------------------------------------------------------------------------ @@ -2827,46 +2904,49 @@ NAME DIFFERENCES BETWEEN PCRE AND PERL This document describes the differences in the ways that PCRE and Perl - handle regular expressions. The differences described here are mainly - with respect to Perl 5.8, though PCRE versions 7.0 and later contain - some features that are expected to be in the forthcoming Perl 5.10. + handle regular expressions. The differences described here are with + respect to Perl 5.10. - 1. PCRE has only a subset of Perl's UTF-8 and Unicode support. Details - of what it does have are given in the section on UTF-8 support in the + 1. PCRE has only a subset of Perl's UTF-8 and Unicode support. Details + of what it does have are given in the section on UTF-8 support in the main pcre page. 2. PCRE does not allow repeat quantifiers on lookahead assertions. Perl - permits them, but they do not mean what you might think. For example, + permits them, but they do not mean what you might think. For example, (?!a){3} does not assert that the next three characters are not "a". It just asserts that the next character is not "a" three times. - 3. Capturing subpatterns that occur inside negative lookahead asser- - tions are counted, but their entries in the offsets vector are never - set. Perl sets its numerical variables from any such patterns that are + 3. Capturing subpatterns that occur inside negative lookahead asser- + tions are counted, but their entries in the offsets vector are never + set. Perl sets its numerical variables from any such patterns that are matched before the assertion fails to match something (thereby succeed- - ing), but only if the negative lookahead assertion contains just one + ing), but only if the negative lookahead assertion contains just one branch. - 4. Though binary zero characters are supported in the subject string, + 4. Though binary zero characters are supported in the subject string, they are not allowed in a pattern string because it is passed as a nor- mal C string, terminated by zero. The escape sequence \0 can be used in the pattern to represent a binary zero. - 5. The following Perl escape sequences are not supported: \l, \u, \L, + 5. The following Perl escape sequences are not supported: \l, \u, \L, \U, and \N. In fact these are implemented by Perl's general string-han- - dling and are not part of its pattern matching engine. If any of these + dling and are not part of its pattern matching engine. If any of these are encountered by PCRE, an error is generated. - 6. The Perl escape sequences \p, \P, and \X are supported only if PCRE - is built with Unicode character property support. The properties that - can be tested with \p and \P are limited to the general category prop- - erties such as Lu and Nd, script names such as Greek or Han, and the - derived properties Any and L&. + 6. The Perl escape sequences \p, \P, and \X are supported only if PCRE + is built with Unicode character property support. The properties that + can be tested with \p and \P are limited to the general category prop- + erties such as Lu and Nd, script names such as Greek or Han, and the + derived properties Any and L&. PCRE does support the Cs (surrogate) + property, which Perl does not; the Perl documentation says "Because + Perl hides the need for the user to understand the internal representa- + tion of Unicode characters, there is no need to implement the somewhat + messy concept of surrogates." 7. PCRE does support the \Q...\E escape for quoting substrings. Charac- - ters in between are treated as literals. This is slightly different - from Perl in that $ and @ are also handled as literals inside the - quotes. In Perl, they cause variable interpolation (but of course PCRE + ters in between are treated as literals. This is slightly different + from Perl in that $ and @ are also handled as literals inside the + quotes. In Perl, they cause variable interpolation (but of course PCRE does not have variables). Note the following examples: Pattern PCRE matches Perl matches @@ -2876,55 +2956,68 @@ DIFFERENCES BETWEEN PCRE AND PERL \Qabc\$xyz\E abc\$xyz abc\$xyz \Qabc\E\$\Qxyz\E abc$xyz abc$xyz - The \Q...\E sequence is recognized both inside and outside character + The \Q...\E sequence is recognized both inside and outside character classes. 8. Fairly obviously, PCRE does not support the (?{code}) and (??{code}) - constructions. However, there is support for recursive patterns. This - is not available in Perl 5.8, but will be in Perl 5.10. Also, the PCRE - "callout" feature allows an external function to be called during pat- + constructions. However, there is support for recursive patterns. This + is not available in Perl 5.8, but it is in Perl 5.10. Also, the PCRE + "callout" feature allows an external function to be called during pat- tern matching. See the pcrecallout documentation for details. - 9. Subpatterns that are called recursively or as "subroutines" are - always treated as atomic groups in PCRE. This is like Python, but - unlike Perl. + 9. Subpatterns that are called recursively or as "subroutines" are + always treated as atomic groups in PCRE. This is like Python, but + unlike Perl. There is a discussion of an example that explains this in + more detail in the section on recursion differences from Perl in the + pcrepattern page. - 10. There are some differences that are concerned with the settings of - captured strings when part of a pattern is repeated. For example, - matching "aba" against the pattern /^(a(b)?)+$/ in Perl leaves $2 + 10. There are some differences that are concerned with the settings of + captured strings when part of a pattern is repeated. For example, + matching "aba" against the pattern /^(a(b)?)+$/ in Perl leaves $2 unset, but in PCRE it is set to "b". 11. PCRE does support Perl 5.10's backtracking verbs (*ACCEPT), - (*FAIL), (*F), (*COMMIT), (*PRUNE), (*SKIP), and (*THEN), but only in - the forms without an argument. PCRE does not support (*MARK). If - (*ACCEPT) is within capturing parentheses, PCRE does not set that cap- - ture group; this is different to Perl. - - 12. PCRE provides some extensions to the Perl regular expression facil- - ities. Perl 5.10 will include new features that are not in earlier - versions, some of which (such as named parentheses) have been in PCRE - for some time. This list is with respect to Perl 5.10: - - (a) Although lookbehind assertions must match fixed length strings, - each alternative branch of a lookbehind assertion can match a different - length of string. Perl requires them all to have the same length. + (*FAIL), (*F), (*COMMIT), (*PRUNE), (*SKIP), and (*THEN), but only in + the forms without an argument. PCRE does not support (*MARK). + + 12. PCRE's handling of duplicate subpattern numbers and duplicate sub- + pattern names is not as general as Perl's. This is a consequence of the + fact the PCRE works internally just with numbers, using an external ta- + ble to translate between numbers and names. In particular, a pattern + such as (?|(?<a>A)|(?<b)B), where the two capturing parentheses have + the same number but different names, is not supported, and causes an + error at compile time. If it were allowed, it would not be possible to + distinguish which parentheses matched, because both names map to cap- + turing subpattern number 1. To avoid this confusing situation, an error + is given at compile time. + + 13. PCRE provides some extensions to the Perl regular expression facil- + ities. Perl 5.10 includes new features that are not in earlier ver- + sions of Perl, some of which (such as named parentheses) have been in + PCRE for some time. This list is with respect to Perl 5.10: + + (a) Although lookbehind assertions in PCRE must match fixed length + strings, each alternative branch of a lookbehind assertion can match a + different length of string. Perl requires them all to have the same + length. - (b) If PCRE_DOLLAR_ENDONLY is set and PCRE_MULTILINE is not set, the $ + (b) If PCRE_DOLLAR_ENDONLY is set and PCRE_MULTILINE is not set, the $ meta-character matches only at the very end of the string. (c) If PCRE_EXTRA is set, a backslash followed by a letter with no spe- cial meaning is faulted. Otherwise, like Perl, the backslash is quietly ignored. (Perl can be made to issue a warning.) - (d) If PCRE_UNGREEDY is set, the greediness of the repetition quanti- + (d) If PCRE_UNGREEDY is set, the greediness of the repetition quanti- fiers is inverted, that is, by default they are not greedy, but if fol- lowed by a question mark they are. (e) PCRE_ANCHORED can be used at matching time to force a pattern to be tried only at the first matching position in the subject string. - (f) The PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, and PCRE_NO_AUTO_CAP- - TURE options for pcre_exec() have no Perl equivalents. + (f) The PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, PCRE_NOTEMPTY_ATSTART, + and PCRE_NO_AUTO_CAPTURE options for pcre_exec() have no Perl equiva- + lents. (g) The \R escape sequence can be restricted to match only CR, LF, or CRLF by the PCRE_BSR_ANYCRLF option. @@ -2953,8 +3046,8 @@ AUTHOR REVISION - Last updated: 11 September 2007 - Copyright (c) 1997-2007 University of Cambridge. + Last updated: 04 October 2009 + Copyright (c) 1997-2009 University of Cambridge. ------------------------------------------------------------------------------ @@ -2984,9 +3077,9 @@ PCRE REGULAR EXPRESSION DETAILS The original operation of PCRE was on strings of one-byte characters. However, there is now also support for UTF-8 character strings. To use - this, you must build PCRE to include UTF-8 support, and then call - pcre_compile() with the PCRE_UTF8 option. There is also a special - sequence that can be given at the start of a pattern: + this, PCRE must be built to include UTF-8 support, and you must call + pcre_compile() or pcre_compile2() with the PCRE_UTF8 option. There is + also a special sequence that can be given at the start of a pattern: (*UTF8) @@ -3024,9 +3117,9 @@ NEWLINE CONVENTIONS (*ANYCRLF) any of the three above (*ANY) all Unicode newline sequences - These override the default and the options given to pcre_compile(). For - example, on a Unix system where LF is the default newline sequence, the - pattern + These override the default and the options given to pcre_compile() or + pcre_compile2(). For example, on a Unix system where LF is the default + newline sequence, the pattern (*CR)a.b @@ -3143,7 +3236,7 @@ BACKSLASH acters in patterns in a visible manner. There is no restriction on the appearance of non-printing characters, apart from the binary zero that terminates a pattern, but when a pattern is being prepared by text - editing, it is usually easier to use one of the following escape + editing, it is often easier to use one of the following escape sequences than the binary character it represents: \a alarm, that is, the BEL character (hex 07) @@ -3355,13 +3448,13 @@ BACKSLASH (*BSR_ANYCRLF) CR, LF, or CRLF only (*BSR_UNICODE) any Unicode newline sequence - These override the default and the options given to pcre_compile(), but - they can be overridden by options given to pcre_exec(). Note that these - special settings, which are not Perl-compatible, are recognized only at - the very start of a pattern, and that they must be in upper case. If - more than one of them is present, the last one is used. They can be - combined with a change of newline convention, for example, a pattern - can start with: + These override the default and the options given to pcre_compile() or + pcre_compile2(), but they can be overridden by options given to + pcre_exec() or pcre_dfa_exec(). Note that these special settings, which + are not Perl-compatible, are recognized only at the very start of a + pattern, and that they must be in upper case. If more than one of them + is present, the last one is used. They can be combined with a change of + newline convention, for example, a pattern can start with: (*ANY)(*BSR_ANYCRLF) @@ -3472,9 +3565,9 @@ BACKSLASH U+D800 to U+DFFF. Such characters are not valid in UTF-8 strings (see RFC 3629) and so cannot be tested by PCRE, unless UTF-8 validity check- ing has been turned off (see the discussion of PCRE_NO_UTF8_CHECK in - the pcreapi page). + the pcreapi page). Perl does not support the Cs property. - The long synonyms for these properties that Perl supports (such as + The long synonyms for property names that Perl supports (such as \p{Letter}) are not supported by PCRE, nor is it permitted to prefix any of these properties with "Is". @@ -3544,34 +3637,37 @@ BACKSLASH A word boundary is a position in the subject string where the current character and the previous character do not both match \w or \W (i.e. one matches \w and the other matches \W), or the start or end of the - string if the first or last character matches \w, respectively. + string if the first or last character matches \w, respectively. Neither + PCRE nor Perl has a separte "start of word" or "end of word" metase- + quence. However, whatever follows \b normally determines which it is. + For example, the fragment \ba matches "a" at the start of a word. - The \A, \Z, and \z assertions differ from the traditional circumflex + The \A, \Z, and \z assertions differ from the traditional circumflex and dollar (described in the next section) in that they only ever match - at the very start and end of the subject string, whatever options are - set. Thus, they are independent of multiline mode. These three asser- + at the very start and end of the subject string, whatever options are + set. Thus, they are independent of multiline mode. These three asser- tions are not affected by the PCRE_NOTBOL or PCRE_NOTEOL options, which - affect only the behaviour of the circumflex and dollar metacharacters. - However, if the startoffset argument of pcre_exec() is non-zero, indi- + affect only the behaviour of the circumflex and dollar metacharacters. + However, if the startoffset argument of pcre_exec() is non-zero, indi- cating that matching is to start at a point other than the beginning of - the subject, \A can never match. The difference between \Z and \z is + the subject, \A can never match. The difference between \Z and \z is that \Z matches before a newline at the end of the string as well as at the very end, whereas \z matches only at the end. - The \G assertion is true only when the current matching position is at - the start point of the match, as specified by the startoffset argument - of pcre_exec(). It differs from \A when the value of startoffset is - non-zero. By calling pcre_exec() multiple times with appropriate argu- + The \G assertion is true only when the current matching position is at + the start point of the match, as specified by the startoffset argument + of pcre_exec(). It differs from \A when the value of startoffset is + non-zero. By calling pcre_exec() multiple times with appropriate argu- ments, you can mimic Perl's /g option, and it is in this kind of imple- mentation where \G can be useful. - Note, however, that PCRE's interpretation of \G, as the start of the + Note, however, that PCRE's interpretation of \G, as the start of the current match, is subtly different from Perl's, which defines it as the - end of the previous match. In Perl, these can be different when the - previously matched string was empty. Because PCRE does just one match + end of the previous match. In Perl, these can be different when the + previously matched string was empty. Because PCRE does just one match at a time, it cannot reproduce this behaviour. - If all the alternatives of a pattern begin with \G, the expression is + If all the alternatives of a pattern begin with \G, the expression is anchored to the starting match position, and the "anchored" flag is set in the compiled regular expression. @@ -3579,90 +3675,90 @@ BACKSLASH CIRCUMFLEX AND DOLLAR Outside a character class, in the default matching mode, the circumflex - character is an assertion that is true only if the current matching - point is at the start of the subject string. If the startoffset argu- - ment of pcre_exec() is non-zero, circumflex can never match if the - PCRE_MULTILINE option is unset. Inside a character class, circumflex + character is an assertion that is true only if the current matching + point is at the start of the subject string. If the startoffset argu- + ment of pcre_exec() is non-zero, circumflex can never match if the + PCRE_MULTILINE option is unset. Inside a character class, circumflex has an entirely different meaning (see below). - Circumflex need not be the first character of the pattern if a number - of alternatives are involved, but it should be the first thing in each - alternative in which it appears if the pattern is ever to match that - branch. If all possible alternatives start with a circumflex, that is, - if the pattern is constrained to match only at the start of the sub- - ject, it is said to be an "anchored" pattern. (There are also other + Circumflex need not be the first character of the pattern if a number + of alternatives are involved, but it should be the first thing in each + alternative in which it appears if the pattern is ever to match that + branch. If all possible alternatives start with a circumflex, that is, + if the pattern is constrained to match only at the start of the sub- + ject, it is said to be an "anchored" pattern. (There are also other constructs that can cause a pattern to be anchored.) - A dollar character is an assertion that is true only if the current - matching point is at the end of the subject string, or immediately + A dollar character is an assertion that is true only if the current + matching point is at the end of the subject string, or immediately before a newline at the end of the string (by default). Dollar need not - be the last character of the pattern if a number of alternatives are - involved, but it should be the last item in any branch in which it + be the last character of the pattern if a number of alternatives are + involved, but it should be the last item in any branch in which it appears. Dollar has no special meaning in a character class. - The meaning of dollar can be changed so that it matches only at the - very end of the string, by setting the PCRE_DOLLAR_ENDONLY option at + The meaning of dollar can be changed so that it matches only at the + very end of the string, by setting the PCRE_DOLLAR_ENDONLY option at compile time. This does not affect the \Z assertion. The meanings of the circumflex and dollar characters are changed if the - PCRE_MULTILINE option is set. When this is the case, a circumflex - matches immediately after internal newlines as well as at the start of - the subject string. It does not match after a newline that ends the - string. A dollar matches before any newlines in the string, as well as - at the very end, when PCRE_MULTILINE is set. When newline is specified - as the two-character sequence CRLF, isolated CR and LF characters do + PCRE_MULTILINE option is set. When this is the case, a circumflex + matches immediately after internal newlines as well as at the start of + the subject string. It does not match after a newline that ends the + string. A dollar matches before any newlines in the string, as well as + at the very end, when PCRE_MULTILINE is set. When newline is specified + as the two-character sequence CRLF, isolated CR and LF characters do not indicate newlines. - For example, the pattern /^abc$/ matches the subject string "def\nabc" - (where \n represents a newline) in multiline mode, but not otherwise. - Consequently, patterns that are anchored in single line mode because - all branches start with ^ are not anchored in multiline mode, and a - match for circumflex is possible when the startoffset argument of - pcre_exec() is non-zero. The PCRE_DOLLAR_ENDONLY option is ignored if + For example, the pattern /^abc$/ matches the subject string "def\nabc" + (where \n represents a newline) in multiline mode, but not otherwise. + Consequently, patterns that are anchored in single line mode because + all branches start with ^ are not anchored in multiline mode, and a + match for circumflex is possible when the startoffset argument of + pcre_exec() is non-zero. The PCRE_DOLLAR_ENDONLY option is ignored if PCRE_MULTILINE is set. - Note that the sequences \A, \Z, and \z can be used to match the start - and end of the subject in both modes, and if all branches of a pattern - start with \A it is always anchored, whether or not PCRE_MULTILINE is + Note that the sequences \A, \Z, and \z can be used to match the start + and end of the subject in both modes, and if all branches of a pattern + start with \A it is always anchored, whether or not PCRE_MULTILINE is set. FULL STOP (PERIOD, DOT) Outside a character class, a dot in the pattern matches any one charac- - ter in the subject string except (by default) a character that signi- - fies the end of a line. In UTF-8 mode, the matched character may be + ter in the subject string except (by default) a character that signi- + fies the end of a line. In UTF-8 mode, the matched character may be more than one byte long. - When a line ending is defined as a single character, dot never matches - that character; when the two-character sequence CRLF is used, dot does - not match CR if it is immediately followed by LF, but otherwise it - matches all characters (including isolated CRs and LFs). When any Uni- - code line endings are being recognized, dot does not match CR or LF or + When a line ending is defined as a single character, dot never matches + that character; when the two-character sequence CRLF is used, dot does + not match CR if it is immediately followed by LF, but otherwise it + matches all characters (including isolated CRs and LFs). When any Uni- + code line endings are being recognized, dot does not match CR or LF or any of the other line ending characters. - The behaviour of dot with regard to newlines can be changed. If the - PCRE_DOTALL option is set, a dot matches any one character, without + The behaviour of dot with regard to newlines can be changed. If the + PCRE_DOTALL option is set, a dot matches any one character, without exception. If the two-character sequence CRLF is present in the subject string, it takes two dots to match it. - The handling of dot is entirely independent of the handling of circum- - flex and dollar, the only relationship being that they both involve + The handling of dot is entirely independent of the handling of circum- + flex and dollar, the only relationship being that they both involve newlines. Dot has no special meaning in a character class. MATCHING A SINGLE BYTE Outside a character class, the escape sequence \C matches any one byte, - both in and out of UTF-8 mode. Unlike a dot, it always matches any - line-ending characters. The feature is provided in Perl in order to - match individual bytes in UTF-8 mode. Because it breaks up UTF-8 char- - acters into individual bytes, what remains in the string may be a mal- - formed UTF-8 string. For this reason, the \C escape sequence is best + both in and out of UTF-8 mode. Unlike a dot, it always matches any + line-ending characters. The feature is provided in Perl in order to + match individual bytes in UTF-8 mode. Because it breaks up UTF-8 char- + acters into individual bytes, what remains in the string may be a mal- + formed UTF-8 string. For this reason, the \C escape sequence is best avoided. - PCRE does not allow \C to appear in lookbehind assertions (described - below), because in UTF-8 mode this would make it impossible to calcu- + PCRE does not allow \C to appear in lookbehind assertions (described + below), because in UTF-8 mode this would make it impossible to calcu- late the length of the lookbehind. @@ -3670,97 +3766,99 @@ SQUARE BRACKETS AND CHARACTER CLASSES An opening square bracket introduces a character class, terminated by a closing square bracket. A closing square bracket on its own is not spe- - cial. If a closing square bracket is required as a member of the class, - it should be the first data character in the class (after an initial - circumflex, if present) or escaped with a backslash. - - A character class matches a single character in the subject. In UTF-8 - mode, the character may occupy more than one byte. A matched character + cial by default. However, if the PCRE_JAVASCRIPT_COMPAT option is set, + a lone closing square bracket causes a compile-time error. If a closing + square bracket is required as a member of the class, it should be the + first data character in the class (after an initial circumflex, if + present) or escaped with a backslash. + + A character class matches a single character in the subject. In UTF-8 + mode, the character may be more than one byte long. A matched character must be in the set of characters defined by the class, unless the first - character in the class definition is a circumflex, in which case the - subject character must not be in the set defined by the class. If a - circumflex is actually required as a member of the class, ensure it is + character in the class definition is a circumflex, in which case the + subject character must not be in the set defined by the class. If a + circumflex is actually required as a member of the class, ensure it is not the first character, or escape it with a backslash. - For example, the character class [aeiou] matches any lower case vowel, - while [^aeiou] matches any character that is not a lower case vowel. + For example, the character class [aeiou] matches any lower case vowel, + while [^aeiou] matches any character that is not a lower case vowel. Note that a circumflex is just a convenient notation for specifying the - characters that are in the class by enumerating those that are not. A - class that starts with a circumflex is not an assertion: it still con- - sumes a character from the subject string, and therefore it fails if + characters that are in the class by enumerating those that are not. A + class that starts with a circumflex is not an assertion; it still con- + sumes a character from the subject string, and therefore it fails if the current pointer is at the end of the string. - In UTF-8 mode, characters with values greater than 255 can be included - in a class as a literal string of bytes, or by using the \x{ escaping + In UTF-8 mode, characters with values greater than 255 can be included + in a class as a literal string of bytes, or by using the \x{ escaping mechanism. - When caseless matching is set, any letters in a class represent both - their upper case and lower case versions, so for example, a caseless - [aeiou] matches "A" as well as "a", and a caseless [^aeiou] does not - match "A", whereas a caseful version would. In UTF-8 mode, PCRE always - understands the concept of case for characters whose values are less - than 128, so caseless matching is always possible. For characters with - higher values, the concept of case is supported if PCRE is compiled - with Unicode property support, but not otherwise. If you want to use - caseless matching for characters 128 and above, you must ensure that - PCRE is compiled with Unicode property support as well as with UTF-8 - support. - - Characters that might indicate line breaks are never treated in any - special way when matching character classes, whatever line-ending - sequence is in use, and whatever setting of the PCRE_DOTALL and + When caseless matching is set, any letters in a class represent both + their upper case and lower case versions, so for example, a caseless + [aeiou] matches "A" as well as "a", and a caseless [^aeiou] does not + match "A", whereas a caseful version would. In UTF-8 mode, PCRE always + understands the concept of case for characters whose values are less + than 128, so caseless matching is always possible. For characters with + higher values, the concept of case is supported if PCRE is compiled + with Unicode property support, but not otherwise. If you want to use + caseless matching in UTF8-mode for characters 128 and above, you must + ensure that PCRE is compiled with Unicode property support as well as + with UTF-8 support. + + Characters that might indicate line breaks are never treated in any + special way when matching character classes, whatever line-ending + sequence is in use, and whatever setting of the PCRE_DOTALL and PCRE_MULTILINE options is used. A class such as [^a] always matches one of these characters. - The minus (hyphen) character can be used to specify a range of charac- - ters in a character class. For example, [d-m] matches any letter - between d and m, inclusive. If a minus character is required in a - class, it must be escaped with a backslash or appear in a position - where it cannot be interpreted as indicating a range, typically as the + The minus (hyphen) character can be used to specify a range of charac- + ters in a character class. For example, [d-m] matches any letter + between d and m, inclusive. If a minus character is required in a + class, it must be escaped with a backslash or appear in a position + where it cannot be interpreted as indicating a range, typically as the first or last character in the class. It is not possible to have the literal character "]" as the end charac- - ter of a range. A pattern such as [W-]46] is interpreted as a class of - two characters ("W" and "-") followed by a literal string "46]", so it - would match "W46]" or "-46]". However, if the "]" is escaped with a - backslash it is interpreted as the end of range, so [W-\]46] is inter- - preted as a class containing a range followed by two other characters. - The octal or hexadecimal representation of "]" can also be used to end + ter of a range. A pattern such as [W-]46] is interpreted as a class of + two characters ("W" and "-") followed by a literal string "46]", so it + would match "W46]" or "-46]". However, if the "]" is escaped with a + backslash it is interpreted as the end of range, so [W-\]46] is inter- + preted as a class containing a range followed by two other characters. + The octal or hexadecimal representation of "]" can also be used to end a range. - Ranges operate in the collating sequence of character values. They can - also be used for characters specified numerically, for example - [\000-\037]. In UTF-8 mode, ranges can include characters whose values + Ranges operate in the collating sequence of character values. They can + also be used for characters specified numerically, for example + [\000-\037]. In UTF-8 mode, ranges can include characters whose values are greater than 255, for example [\x{100}-\x{2ff}]. If a range that includes letters is used when caseless matching is set, it matches the letters in either case. For example, [W-c] is equivalent - to [][\\^_`wxyzabc], matched caselessly, and in non-UTF-8 mode, if - character tables for a French locale are in use, [\xc8-\xcb] matches - accented E characters in both cases. In UTF-8 mode, PCRE supports the - concept of case for characters with values greater than 128 only when + to [][\\^_`wxyzabc], matched caselessly, and in non-UTF-8 mode, if + character tables for a French locale are in use, [\xc8-\xcb] matches + accented E characters in both cases. In UTF-8 mode, PCRE supports the + concept of case for characters with values greater than 128 only when it is compiled with Unicode property support. - The character types \d, \D, \p, \P, \s, \S, \w, and \W may also appear - in a character class, and add the characters that they match to the + The character types \d, \D, \p, \P, \s, \S, \w, and \W may also appear + in a character class, and add the characters that they match to the class. For example, [\dABCDEF] matches any hexadecimal digit. A circum- - flex can conveniently be used with the upper case character types to - specify a more restricted set of characters than the matching lower - case type. For example, the class [^\W_] matches any letter or digit, + flex can conveniently be used with the upper case character types to + specify a more restricted set of characters than the matching lower + case type. For example, the class [^\W_] matches any letter or digit, but not underscore. - The only metacharacters that are recognized in character classes are - backslash, hyphen (only where it can be interpreted as specifying a - range), circumflex (only at the start), opening square bracket (only - when it can be interpreted as introducing a POSIX class name - see the - next section), and the terminating closing square bracket. However, + The only metacharacters that are recognized in character classes are + backslash, hyphen (only where it can be interpreted as specifying a + range), circumflex (only at the start), opening square bracket (only + when it can be interpreted as introducing a POSIX class name - see the + next section), and the terminating closing square bracket. However, escaping other non-alphanumeric characters does no harm. POSIX CHARACTER CLASSES Perl supports the POSIX notation for character classes. This uses names - enclosed by [: and :] within the enclosing square brackets. PCRE also + enclosed by [: and :] within the enclosing square brackets. PCRE also supports this notation. For example, [01[:alpha:]%] @@ -3783,18 +3881,18 @@ POSIX CHARACTER CLASSES word "word" characters (same as \w) xdigit hexadecimal digits - The "space" characters are HT (9), LF (10), VT (11), FF (12), CR (13), - and space (32). Notice that this list includes the VT character (code + The "space" characters are HT (9), LF (10), VT (11), FF (12), CR (13), + and space (32). Notice that this list includes the VT character (code 11). This makes "space" different to \s, which does not include VT (for Perl compatibility). - The name "word" is a Perl extension, and "blank" is a GNU extension - from Perl 5.8. Another Perl extension is negation, which is indicated + The name "word" is a Perl extension, and "blank" is a GNU extension + from Perl 5.8. Another Perl extension is negation, which is indicated by a ^ character after the colon. For example, [12[:^digit:]] - matches "1", "2", or any non-digit. PCRE (and Perl) also recognize the + matches "1", "2", or any non-digit. PCRE (and Perl) also recognize the POSIX syntax [.ch.] and [=ch=] where "ch" is a "collating element", but these are not supported, and an error is given if they are encountered. @@ -3804,24 +3902,24 @@ POSIX CHARACTER CLASSES VERTICAL BAR - Vertical bar characters are used to separate alternative patterns. For + Vertical bar characters are used to separate alternative patterns. For example, the pattern gilbert|sullivan - matches either "gilbert" or "sullivan". Any number of alternatives may - appear, and an empty alternative is permitted (matching the empty + matches either "gilbert" or "sullivan". Any number of alternatives may + appear, and an empty alternative is permitted (matching the empty string). The matching process tries each alternative in turn, from left - to right, and the first one that succeeds is used. If the alternatives - are within a subpattern (defined below), "succeeds" means matching the + to right, and the first one that succeeds is used. If the alternatives + are within a subpattern (defined below), "succeeds" means matching the rest of the main pattern as well as the alternative in the subpattern. INTERNAL OPTION SETTING - The settings of the PCRE_CASELESS, PCRE_MULTILINE, PCRE_DOTALL, and - PCRE_EXTENDED options (which are Perl-compatible) can be changed from - within the pattern by a sequence of Perl option letters enclosed + The settings of the PCRE_CASELESS, PCRE_MULTILINE, PCRE_DOTALL, and + PCRE_EXTENDED options (which are Perl-compatible) can be changed from + within the pattern by a sequence of Perl option letters enclosed between "(?" and ")". The option letters are i for PCRE_CASELESS @@ -3831,46 +3929,46 @@ INTERNAL OPTION SETTING For example, (?im) sets caseless, multiline matching. It is also possi- ble to unset these options by preceding the letter with a hyphen, and a - combined setting and unsetting such as (?im-sx), which sets PCRE_CASE- - LESS and PCRE_MULTILINE while unsetting PCRE_DOTALL and PCRE_EXTENDED, - is also permitted. If a letter appears both before and after the + combined setting and unsetting such as (?im-sx), which sets PCRE_CASE- + LESS and PCRE_MULTILINE while unsetting PCRE_DOTALL and PCRE_EXTENDED, + is also permitted. If a letter appears both before and after the hyphen, the option is unset. - The PCRE-specific options PCRE_DUPNAMES, PCRE_UNGREEDY, and PCRE_EXTRA - can be changed in the same way as the Perl-compatible options by using + The PCRE-specific options PCRE_DUPNAMES, PCRE_UNGREEDY, and PCRE_EXTRA + can be changed in the same way as the Perl-compatible options by using the characters J, U and X respectively. - When one of these option changes occurs at top level (that is, not - inside subpattern parentheses), the change applies to the remainder of + When one of these option changes occurs at top level (that is, not + inside subpattern parentheses), the change applies to the remainder of the pattern that follows. If the change is placed right at the start of a pattern, PCRE extracts it into the global options (and it will there- fore show up in data extracted by the pcre_fullinfo() function). - An option change within a subpattern (see below for a description of + An option change within a subpattern (see below for a description of subpatterns) affects only that part of the current pattern that follows it, so (a(?i)b)c matches abc and aBc and no other strings (assuming PCRE_CASELESS is not - used). By this means, options can be made to have different settings - in different parts of the pattern. Any changes made in one alternative - do carry on into subsequent branches within the same subpattern. For + used). By this means, options can be made to have different settings + in different parts of the pattern. Any changes made in one alternative + do carry on into subsequent branches within the same subpattern. For example, (a(?i)b|c) - matches "ab", "aB", "c", and "C", even though when matching "C" the - first branch is abandoned before the option setting. This is because - the effects of option settings happen at compile time. There would be + matches "ab", "aB", "c", and "C", even though when matching "C" the + first branch is abandoned before the option setting. This is because + the effects of option settings happen at compile time. There would be some very weird behaviour otherwise. - Note: There are other PCRE-specific options that can be set by the - application when the compile or match functions are called. In some + Note: There are other PCRE-specific options that can be set by the + application when the compile or match functions are called. In some cases the pattern can contain special leading sequences such as (*CRLF) - to override what the application has set or what has been defaulted. - Details are given in the section entitled "Newline sequences" above. - There is also the (*UTF8) leading sequence that can be used to set + to override what the application has set or what has been defaulted. + Details are given in the section entitled "Newline sequences" above. + There is also the (*UTF8) leading sequence that can be used to set UTF-8 mode; this is equivalent to setting the PCRE_UTF8 option. @@ -3883,18 +3981,18 @@ SUBPATTERNS cat(aract|erpillar|) - matches one of the words "cat", "cataract", or "caterpillar". Without - the parentheses, it would match "cataract", "erpillar" or an empty + matches one of the words "cat", "cataract", or "caterpillar". Without + the parentheses, it would match "cataract", "erpillar" or an empty string. - 2. It sets up the subpattern as a capturing subpattern. This means - that, when the whole pattern matches, that portion of the subject + 2. It sets up the subpattern as a capturing subpattern. This means + that, when the whole pattern matches, that portion of the subject string that matched the subpattern is passed back to the caller via the - ovector argument of pcre_exec(). Opening parentheses are counted from - left to right (starting from 1) to obtain numbers for the capturing + ovector argument of pcre_exec(). Opening parentheses are counted from + left to right (starting from 1) to obtain numbers for the capturing subpatterns. - For example, if the string "the red king" is matched against the pat- + For example, if the string "the red king" is matched against the pat- tern the ((red|white) (king|queen)) @@ -3902,12 +4000,12 @@ SUBPATTERNS the captured substrings are "red king", "red", and "king", and are num- bered 1, 2, and 3, respectively. - The fact that plain parentheses fulfil two functions is not always - helpful. There are often times when a grouping subpattern is required - without a capturing requirement. If an opening parenthesis is followed - by a question mark and a colon, the subpattern does not do any captur- - ing, and is not counted when computing the number of any subsequent - capturing subpatterns. For example, if the string "the white queen" is + The fact that plain parentheses fulfil two functions is not always + helpful. There are often times when a grouping subpattern is required + without a capturing requirement. If an opening parenthesis is followed + by a question mark and a colon, the subpattern does not do any captur- + ing, and is not counted when computing the number of any subsequent + capturing subpatterns. For example, if the string "the white queen" is matched against the pattern the ((?:red|white) (king|queen)) @@ -3915,46 +4013,59 @@ SUBPATTERNS the captured substrings are "white queen" and "queen", and are numbered 1 and 2. The maximum number of capturing subpatterns is 65535. - As a convenient shorthand, if any option settings are required at the - start of a non-capturing subpattern, the option letters may appear + As a convenient shorthand, if any option settings are required at the + start of a non-capturing subpattern, the option letters may appear between the "?" and the ":". Thus the two patterns (?i:saturday|sunday) (?:(?i)saturday|sunday) match exactly the same set of strings. Because alternative branches are - tried from left to right, and options are not reset until the end of - the subpattern is reached, an option setting in one branch does affect - subsequent branches, so the above patterns match "SUNDAY" as well as + tried from left to right, and options are not reset until the end of + the subpattern is reached, an option setting in one branch does affect + subsequent branches, so the above patterns match "SUNDAY" as well as "Saturday". DUPLICATE SUBPATTERN NUMBERS Perl 5.10 introduced a feature whereby each alternative in a subpattern - uses the same numbers for its capturing parentheses. Such a subpattern - starts with (?| and is itself a non-capturing subpattern. For example, + uses the same numbers for its capturing parentheses. Such a subpattern + starts with (?| and is itself a non-capturing subpattern. For example, consider this pattern: (?|(Sat)ur|(Sun))day - Because the two alternatives are inside a (?| group, both sets of cap- - turing parentheses are numbered one. Thus, when the pattern matches, - you can look at captured substring number one, whichever alternative - matched. This construct is useful when you want to capture part, but + Because the two alternatives are inside a (?| group, both sets of cap- + turing parentheses are numbered one. Thus, when the pattern matches, + you can look at captured substring number one, whichever alternative + matched. This construct is useful when you want to capture part, but not all, of one of a number of alternatives. Inside a (?| group, paren- - theses are numbered as usual, but the number is reset at the start of - each branch. The numbers of any capturing buffers that follow the sub- - pattern start after the highest number used in any branch. The follow- - ing example is taken from the Perl documentation. The numbers under- + theses are numbered as usual, but the number is reset at the start of + each branch. The numbers of any capturing buffers that follow the sub- + pattern start after the highest number used in any branch. The follow- + ing example is taken from the Perl documentation. The numbers under- neath show in which buffer the captured content will be stored. # before ---------------branch-reset----------- after / ( a ) (?| x ( y ) z | (p (q) r) | (t) u (v) ) ( z ) /x # 1 2 2 3 2 3 4 - A backreference or a recursive call to a numbered subpattern always - refers to the first one in the pattern with the given number. + A backreference to a numbered subpattern uses the most recent value + that is set for that number by any subpattern. The following pattern + matches "abcabc" or "defdef": + + /(?|(abc)|(def))\1/ + + In contrast, a recursive or "subroutine" call to a numbered subpattern + always refers to the first one in the pattern with the given number. + The following pattern matches "abcabc" or "defabc": + + /(?|(abc)|(def))(?1)/ + + If a condition test for a subpattern's having matched refers to a non- + unique number, the test is true if any of the subpatterns of that num- + ber have matched. An alternative approach to using this "branch reset" feature is to use duplicate named subpatterns, as described in the next section. @@ -3969,26 +4080,29 @@ NAMED SUBPATTERNS patterns. This feature was not added to Perl until release 5.10. Python had the feature earlier, and PCRE introduced it at release 4.0, using the Python syntax. PCRE now supports both the Perl and the Python syn- - tax. + tax. Perl allows identically numbered subpatterns to have different + names, but PCRE does not. - In PCRE, a subpattern can be named in one of three ways: (?<name>...) - or (?'name'...) as in Perl, or (?P<name>...) as in Python. References + In PCRE, a subpattern can be named in one of three ways: (?<name>...) + or (?'name'...) as in Perl, or (?P<name>...) as in Python. References to capturing parentheses from other parts of the pattern, such as back- - references, recursion, and conditions, can be made by name as well as + references, recursion, and conditions, can be made by name as well as by number. - Names consist of up to 32 alphanumeric characters and underscores. - Named capturing parentheses are still allocated numbers as well as - names, exactly as if the names were not present. The PCRE API provides + Names consist of up to 32 alphanumeric characters and underscores. + Named capturing parentheses are still allocated numbers as well as + names, exactly as if the names were not present. The PCRE API provides function calls for extracting the name-to-number translation table from a compiled pattern. There is also a convenience function for extracting a captured substring by name. - By default, a name must be unique within a pattern, but it is possible + By default, a name must be unique within a pattern, but it is possible to relax this constraint by setting the PCRE_DUPNAMES option at compile - time. This can be useful for patterns where only one instance of the - named parentheses can match. Suppose you want to match the name of a - weekday, either as a 3-letter abbreviation or as the full name, and in + time. (Duplicate names are also always permitted for subpatterns with + the same number, set up as described in the previous section.) Dupli- + cate names can be useful for patterns where only one instance of the + named parentheses can match. Suppose you want to match the name of a + weekday, either as a 3-letter abbreviation or as the full name, and in both cases you want to extract the abbreviation. This pattern (ignoring the line breaks) does the job: @@ -3998,26 +4112,38 @@ NAMED SUBPATTERNS (?<DN>Thu)(?:rsday)?| (?<DN>Sat)(?:urday)? - There are five capturing substrings, but only one is ever set after a + There are five capturing substrings, but only one is ever set after a match. (An alternative way of solving this problem is to use a "branch reset" subpattern, as described in the previous section.) - The convenience function for extracting the data by name returns the - substring for the first (and in this example, the only) subpattern of - that name that matched. This saves searching to find which numbered - subpattern it was. If you make a reference to a non-unique named sub- - pattern from elsewhere in the pattern, the one that corresponds to the - lowest number is used. For further details of the interfaces for han- - dling named subpatterns, see the pcreapi documentation. + The convenience function for extracting the data by name returns the + substring for the first (and in this example, the only) subpattern of + that name that matched. This saves searching to find which numbered + subpattern it was. + + If you make a backreference to a non-unique named subpattern from else- + where in the pattern, the one that corresponds to the first occurrence + of the name is used. In the absence of duplicate numbers (see the pre- + vious section) this is the one with the lowest number. If you use a + named reference in a condition test (see the section about conditions + below), either to check whether a subpattern has matched, or to check + for recursion, all subpatterns with the same name are tested. If the + condition is true for any one of them, the overall condition is true. + This is the same behaviour as testing by number. For further details of + the interfaces for handling named subpatterns, see the pcreapi documen- + tation. Warning: You cannot use different names to distinguish between two sub- - patterns with the same number (see the previous section) because PCRE - uses only the numbers when matching. + patterns with the same number because PCRE uses only the numbers when + matching. For this reason, an error is given at compile time if differ- + ent names are given to subpatterns with the same number. However, you + can give the same name to subpatterns with the same number, even when + PCRE_DUPNAMES is not set. REPETITION - Repetition is specified by quantifiers, which can follow any of the + Repetition is specified by quantifiers, which can follow any of the following items: a literal data character @@ -4029,18 +4155,19 @@ REPETITION a character class a back reference (see next section) a parenthesized subpattern (unless it is an assertion) + a recursive or "subroutine" call to a subpattern - The general repetition quantifier specifies a minimum and maximum num- - ber of permitted matches, by giving the two numbers in curly brackets - (braces), separated by a comma. The numbers must be less than 65536, + The general repetition quantifier specifies a minimum and maximum num- + ber of permitted matches, by giving the two numbers in curly brackets + (braces), separated by a comma. The numbers must be less than 65536, and the first must be less than or equal to the second. For example: z{2,4} - matches "zz", "zzz", or "zzzz". A closing brace on its own is not a - special character. If the second number is omitted, but the comma is - present, there is no upper limit; if the second number and the comma - are both omitted, the quantifier specifies an exact number of required + matches "zz", "zzz", or "zzzz". A closing brace on its own is not a + special character. If the second number is omitted, but the comma is + present, there is no upper limit; if the second number and the comma + are both omitted, the quantifier specifies an exact number of required matches. Thus [aeiou]{3,} @@ -4049,49 +4176,49 @@ REPETITION \d{8} - matches exactly 8 digits. An opening curly bracket that appears in a - position where a quantifier is not allowed, or one that does not match - the syntax of a quantifier, is taken as a literal character. For exam- + matches exactly 8 digits. An opening curly bracket that appears in a + position where a quantifier is not allowed, or one that does not match + the syntax of a quantifier, is taken as a literal character. For exam- ple, {,6} is not a quantifier, but a literal string of four characters. - In UTF-8 mode, quantifiers apply to UTF-8 characters rather than to + In UTF-8 mode, quantifiers apply to UTF-8 characters rather than to individual bytes. Thus, for example, \x{100}{2} matches two UTF-8 char- acters, each of which is represented by a two-byte sequence. Similarly, when Unicode property support is available, \X{3} matches three Unicode - extended sequences, each of which may be several bytes long (and they + extended sequences, each of which may be several bytes long (and they may be of different lengths). The quantifier {0} is permitted, causing the expression to behave as if the previous item and the quantifier were not present. This may be use- - ful for subpatterns that are referenced as subroutines from elsewhere + ful for subpatterns that are referenced as subroutines from elsewhere in the pattern. Items other than subpatterns that have a {0} quantifier are omitted from the compiled pattern. - For convenience, the three most common quantifiers have single-charac- + For convenience, the three most common quantifiers have single-charac- ter abbreviations: * is equivalent to {0,} + is equivalent to {1,} ? is equivalent to {0,1} - It is possible to construct infinite loops by following a subpattern + It is possible to construct infinite loops by following a subpattern that can match no characters with a quantifier that has no upper limit, for example: (a?)* Earlier versions of Perl and PCRE used to give an error at compile time - for such patterns. However, because there are cases where this can be - useful, such patterns are now accepted, but if any repetition of the - subpattern does in fact match no characters, the loop is forcibly bro- + for such patterns. However, because there are cases where this can be + useful, such patterns are now accepted, but if any repetition of the + subpattern does in fact match no characters, the loop is forcibly bro- ken. - By default, the quantifiers are "greedy", that is, they match as much - as possible (up to the maximum number of permitted times), without - causing the rest of the pattern to fail. The classic example of where + By default, the quantifiers are "greedy", that is, they match as much + as possible (up to the maximum number of permitted times), without + causing the rest of the pattern to fail. The classic example of where this gives problems is in trying to match comments in C programs. These - appear between /* and */ and within the comment, individual * and / - characters may appear. An attempt to match C comments by applying the + appear between /* and */ and within the comment, individual * and / + characters may appear. An attempt to match C comments by applying the pattern /\*.*\*/ @@ -4100,19 +4227,19 @@ REPETITION /* first comment */ not comment /* second comment */ - fails, because it matches the entire string owing to the greediness of + fails, because it matches the entire string owing to the greediness of the .* item. - However, if a quantifier is followed by a question mark, it ceases to + However, if a quantifier is followed by a question mark, it ceases to be greedy, and instead matches the minimum number of times possible, so the pattern /\*.*?\*/ - does the right thing with the C comments. The meaning of the various - quantifiers is not otherwise changed, just the preferred number of - matches. Do not confuse this use of question mark with its use as a - quantifier in its own right. Because it has two uses, it can sometimes + does the right thing with the C comments. The meaning of the various + quantifiers is not otherwise changed, just the preferred number of + matches. Do not confuse this use of question mark with its use as a + quantifier in its own right. Because it has two uses, it can sometimes appear doubled, as in \d??\d @@ -4120,36 +4247,36 @@ REPETITION which matches one digit by preference, but can match two if that is the only way the rest of the pattern matches. - If the PCRE_UNGREEDY option is set (an option that is not available in - Perl), the quantifiers are not greedy by default, but individual ones - can be made greedy by following them with a question mark. In other + If the PCRE_UNGREEDY option is set (an option that is not available in + Perl), the quantifiers are not greedy by default, but individual ones + can be made greedy by following them with a question mark. In other words, it inverts the default behaviour. - When a parenthesized subpattern is quantified with a minimum repeat - count that is greater than 1 or with a limited maximum, more memory is - required for the compiled pattern, in proportion to the size of the + When a parenthesized subpattern is quantified with a minimum repeat + count that is greater than 1 or with a limited maximum, more memory is + required for the compiled pattern, in proportion to the size of the minimum or maximum. If a pattern starts with .* or .{0,} and the PCRE_DOTALL option (equiv- - alent to Perl's /s) is set, thus allowing the dot to match newlines, - the pattern is implicitly anchored, because whatever follows will be - tried against every character position in the subject string, so there - is no point in retrying the overall match at any position after the - first. PCRE normally treats such a pattern as though it were preceded + alent to Perl's /s) is set, thus allowing the dot to match newlines, + the pattern is implicitly anchored, because whatever follows will be + tried against every character position in the subject string, so there + is no point in retrying the overall match at any position after the + first. PCRE normally treats such a pattern as though it were preceded by \A. - In cases where it is known that the subject string contains no new- - lines, it is worth setting PCRE_DOTALL in order to obtain this opti- + In cases where it is known that the subject string contains no new- + lines, it is worth setting PCRE_DOTALL in order to obtain this opti- mization, or alternatively using ^ to indicate anchoring explicitly. - However, there is one situation where the optimization cannot be used. - When .* is inside capturing parentheses that are the subject of a - backreference elsewhere in the pattern, a match at the start may fail + However, there is one situation where the optimization cannot be used. + When .* is inside capturing parentheses that are the subject of a + backreference elsewhere in the pattern, a match at the start may fail where a later one succeeds. Consider, for example: (.*)abc\1 - If the subject is "xyz123abc123" the match point is the fourth charac- + If the subject is "xyz123abc123" the match point is the fourth charac- ter. For this reason, such a pattern is not implicitly anchored. When a capturing subpattern is repeated, the value captured is the sub- @@ -4158,8 +4285,8 @@ REPETITION (tweedle[dume]{3}\s*)+ has matched "tweedledum tweedledee" the value of the captured substring - is "tweedledee". However, if there are nested capturing subpatterns, - the corresponding captured values may have been set in previous itera- + is "tweedledee". However, if there are nested capturing subpatterns, + the corresponding captured values may have been set in previous itera- tions. For example, after /(a|(b))+/ @@ -4169,53 +4296,53 @@ REPETITION ATOMIC GROUPING AND POSSESSIVE QUANTIFIERS - With both maximizing ("greedy") and minimizing ("ungreedy" or "lazy") - repetition, failure of what follows normally causes the repeated item - to be re-evaluated to see if a different number of repeats allows the - rest of the pattern to match. Sometimes it is useful to prevent this, - either to change the nature of the match, or to cause it fail earlier - than it otherwise might, when the author of the pattern knows there is + With both maximizing ("greedy") and minimizing ("ungreedy" or "lazy") + repetition, failure of what follows normally causes the repeated item + to be re-evaluated to see if a different number of repeats allows the + rest of the pattern to match. Sometimes it is useful to prevent this, + either to change the nature of the match, or to cause it fail earlier + than it otherwise might, when the author of the pattern knows there is no point in carrying on. - Consider, for example, the pattern \d+foo when applied to the subject + Consider, for example, the pattern \d+foo when applied to the subject line 123456bar After matching all 6 digits and then failing to match "foo", the normal - action of the matcher is to try again with only 5 digits matching the - \d+ item, and then with 4, and so on, before ultimately failing. - "Atomic grouping" (a term taken from Jeffrey Friedl's book) provides - the means for specifying that once a subpattern has matched, it is not + action of the matcher is to try again with only 5 digits matching the + \d+ item, and then with 4, and so on, before ultimately failing. + "Atomic grouping" (a term taken from Jeffrey Friedl's book) provides + the means for specifying that once a subpattern has matched, it is not to be re-evaluated in this way. - If we use atomic grouping for the previous example, the matcher gives - up immediately on failing to match "foo" the first time. The notation + If we use atomic grouping for the previous example, the matcher gives + up immediately on failing to match "foo" the first time. The notation is a kind of special parenthesis, starting with (?> as in this example: (?>\d+)foo - This kind of parenthesis "locks up" the part of the pattern it con- - tains once it has matched, and a failure further into the pattern is - prevented from backtracking into it. Backtracking past it to previous + This kind of parenthesis "locks up" the part of the pattern it con- + tains once it has matched, and a failure further into the pattern is + prevented from backtracking into it. Backtracking past it to previous items, however, works as normal. - An alternative description is that a subpattern of this type matches - the string of characters that an identical standalone pattern would + An alternative description is that a subpattern of this type matches + the string of characters that an identical standalone pattern would match, if anchored at the current point in the subject string. Atomic grouping subpatterns are not capturing subpatterns. Simple cases such as the above example can be thought of as a maximizing repeat that - must swallow everything it can. So, while both \d+ and \d+? are pre- - pared to adjust the number of digits they match in order to make the + must swallow everything it can. So, while both \d+ and \d+? are pre- + pared to adjust the number of digits they match in order to make the rest of the pattern match, (?>\d+) can only match an entire sequence of digits. - Atomic groups in general can of course contain arbitrarily complicated - subpatterns, and can be nested. However, when the subpattern for an + Atomic groups in general can of course contain arbitrarily complicated + subpatterns, and can be nested. However, when the subpattern for an atomic group is just a single repeated item, as in the example above, a - simpler notation, called a "possessive quantifier" can be used. This - consists of an additional + character following a quantifier. Using + simpler notation, called a "possessive quantifier" can be used. This + consists of an additional + character following a quantifier. Using this notation, the previous example can be rewritten as \d++foo @@ -4225,45 +4352,45 @@ ATOMIC GROUPING AND POSSESSIVE QUANTIFIERS (abc|xyz){2,3}+ - Possessive quantifiers are always greedy; the setting of the + Possessive quantifiers are always greedy; the setting of the PCRE_UNGREEDY option is ignored. They are a convenient notation for the - simpler forms of atomic group. However, there is no difference in the - meaning of a possessive quantifier and the equivalent atomic group, - though there may be a performance difference; possessive quantifiers + simpler forms of atomic group. However, there is no difference in the + meaning of a possessive quantifier and the equivalent atomic group, + though there may be a performance difference; possessive quantifiers should be slightly faster. - The possessive quantifier syntax is an extension to the Perl 5.8 syn- - tax. Jeffrey Friedl originated the idea (and the name) in the first + The possessive quantifier syntax is an extension to the Perl 5.8 syn- + tax. Jeffrey Friedl originated the idea (and the name) in the first edition of his book. Mike McCloskey liked it, so implemented it when he - built Sun's Java package, and PCRE copied it from there. It ultimately + built Sun's Java package, and PCRE copied it from there. It ultimately found its way into Perl at release 5.10. PCRE has an optimization that automatically "possessifies" certain sim- - ple pattern constructs. For example, the sequence A+B is treated as - A++B because there is no point in backtracking into a sequence of A's + ple pattern constructs. For example, the sequence A+B is treated as + A++B because there is no point in backtracking into a sequence of A's when B must follow. - When a pattern contains an unlimited repeat inside a subpattern that - can itself be repeated an unlimited number of times, the use of an - atomic group is the only way to avoid some failing matches taking a + When a pattern contains an unlimited repeat inside a subpattern that + can itself be repeated an unlimited number of times, the use of an + atomic group is the only way to avoid some failing matches taking a very long time indeed. The pattern (\D+|<\d+>)*[!?] - matches an unlimited number of substrings that either consist of non- - digits, or digits enclosed in <>, followed by either ! or ?. When it + matches an unlimited number of substrings that either consist of non- + digits, or digits enclosed in <>, followed by either ! or ?. When it matches, it runs quickly. However, if it is applied to aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa - it takes a long time before reporting failure. This is because the - string can be divided between the internal \D+ repeat and the external - * repeat in a large number of ways, and all have to be tried. (The - example uses [!?] rather than a single character at the end, because - both PCRE and Perl have an optimization that allows for fast failure - when a single character is used. They remember the last single charac- - ter that is required for a match, and fail early if it is not present - in the string.) If the pattern is changed so that it uses an atomic + it takes a long time before reporting failure. This is because the + string can be divided between the internal \D+ repeat and the external + * repeat in a large number of ways, and all have to be tried. (The + example uses [!?] rather than a single character at the end, because + both PCRE and Perl have an optimization that allows for fast failure + when a single character is used. They remember the last single charac- + ter that is required for a match, and fail early if it is not present + in the string.) If the pattern is changed so that it uses an atomic group, like this: ((?>\D+)|<\d+>)*[!?] @@ -4275,37 +4402,37 @@ BACK REFERENCES Outside a character class, a backslash followed by a digit greater than 0 (and possibly further digits) is a back reference to a capturing sub- - pattern earlier (that is, to its left) in the pattern, provided there + pattern earlier (that is, to its left) in the pattern, provided there have been that many previous capturing left parentheses. However, if the decimal number following the backslash is less than 10, - it is always taken as a back reference, and causes an error only if - there are not that many capturing left parentheses in the entire pat- - tern. In other words, the parentheses that are referenced need not be - to the left of the reference for numbers less than 10. A "forward back - reference" of this type can make sense when a repetition is involved - and the subpattern to the right has participated in an earlier itera- + it is always taken as a back reference, and causes an error only if + there are not that many capturing left parentheses in the entire pat- + tern. In other words, the parentheses that are referenced need not be + to the left of the reference for numbers less than 10. A "forward back + reference" of this type can make sense when a repetition is involved + and the subpattern to the right has participated in an earlier itera- tion. - It is not possible to have a numerical "forward back reference" to a - subpattern whose number is 10 or more using this syntax because a - sequence such as \50 is interpreted as a character defined in octal. + It is not possible to have a numerical "forward back reference" to a + subpattern whose number is 10 or more using this syntax because a + sequence such as \50 is interpreted as a character defined in octal. See the subsection entitled "Non-printing characters" above for further - details of the handling of digits following a backslash. There is no - such problem when named parentheses are used. A back reference to any + details of the handling of digits following a backslash. There is no + such problem when named parentheses are used. A back reference to any subpattern is possible using named parentheses (see below). - Another way of avoiding the ambiguity inherent in the use of digits + Another way of avoiding the ambiguity inherent in the use of digits following a backslash is to use the \g escape sequence, which is a fea- - ture introduced in Perl 5.10. This escape must be followed by an - unsigned number or a negative number, optionally enclosed in braces. + ture introduced in Perl 5.10. This escape must be followed by an + unsigned number or a negative number, optionally enclosed in braces. These examples are all identical: (ring), \1 (ring), \g1 (ring), \g{1} - An unsigned number specifies an absolute reference without the ambigu- + An unsigned number specifies an absolute reference without the ambigu- ity that is present in the older syntax. It is also useful when literal digits follow the reference. A negative number is a relative reference. Consider this example: @@ -4313,33 +4440,33 @@ BACK REFERENCES (abc(def)ghi)\g{-1} The sequence \g{-1} is a reference to the most recently started captur- - ing subpattern before \g, that is, is it equivalent to \2. Similarly, + ing subpattern before \g, that is, is it equivalent to \2. Similarly, \g{-2} would be equivalent to \1. The use of relative references can be - helpful in long patterns, and also in patterns that are created by + helpful in long patterns, and also in patterns that are created by joining together fragments that contain references within themselves. - A back reference matches whatever actually matched the capturing sub- - pattern in the current subject string, rather than anything matching + A back reference matches whatever actually matched the capturing sub- + pattern in the current subject string, rather than anything matching the subpattern itself (see "Subpatterns as subroutines" below for a way of doing that). So the pattern (sens|respons)e and \1ibility - matches "sense and sensibility" and "response and responsibility", but - not "sense and responsibility". If caseful matching is in force at the - time of the back reference, the case of letters is relevant. For exam- + matches "sense and sensibility" and "response and responsibility", but + not "sense and responsibility". If caseful matching is in force at the + time of the back reference, the case of letters is relevant. For exam- ple, ((?i)rah)\s+\1 - matches "rah rah" and "RAH RAH", but not "RAH rah", even though the + matches "rah rah" and "RAH RAH", but not "RAH rah", even though the original capturing subpattern is matched caselessly. - There are several different ways of writing back references to named - subpatterns. The .NET syntax \k{name} and the Perl syntax \k<name> or - \k'name' are supported, as is the Python syntax (?P=name). Perl 5.10's + There are several different ways of writing back references to named + subpatterns. The .NET syntax \k{name} and the Perl syntax \k<name> or + \k'name' are supported, as is the Python syntax (?P=name). Perl 5.10's unified back reference syntax, in which \g can be used for both numeric - and named references, is also supported. We could rewrite the above + and named references, is also supported. We could rewrite the above example in any of the following ways: (?<p1>(?i)rah)\s+\k<p1> @@ -4347,22 +4474,25 @@ BACK REFERENCES (?P<p1>(?i)rah)\s+(?P=p1) (?<p1>(?i)rah)\s+\g{p1} - A subpattern that is referenced by name may appear in the pattern + A subpattern that is referenced by name may appear in the pattern before or after the reference. - There may be more than one back reference to the same subpattern. If a - subpattern has not actually been used in a particular match, any back - references to it always fail. For example, the pattern + There may be more than one back reference to the same subpattern. If a + subpattern has not actually been used in a particular match, any back + references to it always fail by default. For example, the pattern (a|(bc))\2 - always fails if it starts to match "a" rather than "bc". Because there - may be many capturing parentheses in a pattern, all digits following - the backslash are taken as part of a potential back reference number. - If the pattern continues with a digit character, some delimiter must be - used to terminate the back reference. If the PCRE_EXTENDED option is - set, this can be whitespace. Otherwise an empty comment (see "Com- - ments" below) can be used. + always fails if it starts to match "a" rather than "bc". However, if + the PCRE_JAVASCRIPT_COMPAT option is set at compile time, a back refer- + ence to an unset value matches an empty string. + + Because there may be many capturing parentheses in a pattern, all dig- + its following a backslash are taken as part of a potential back refer- + ence number. If the pattern continues with a digit character, some + delimiter must be used to terminate the back reference. If the + PCRE_EXTENDED option is set, this can be whitespace. Otherwise, the \g{ + syntax or an empty comment (see "Comments" below) can be used. A back reference that occurs inside the parentheses to which it refers fails when the subpattern is first used, so, for example, (a\1) never @@ -4425,19 +4555,20 @@ ASSERTIONS If you want to force a matching failure at some point in a pattern, the most convenient way to do it is with (?!) because an empty string always matches, so an assertion that requires there not to be an empty - string must always fail. + string must always fail. The Perl 5.10 backtracking control verb + (*FAIL) or (*F) is essentially a synonym for (?!). Lookbehind assertions - Lookbehind assertions start with (?<= for positive assertions and (?<! + Lookbehind assertions start with (?<= for positive assertions and (?<! for negative assertions. For example, (?<!foo)bar - does find an occurrence of "bar" that is not preceded by "foo". The - contents of a lookbehind assertion are restricted such that all the + does find an occurrence of "bar" that is not preceded by "foo". The + contents of a lookbehind assertion are restricted such that all the strings it matches must have a fixed length. However, if there are sev- - eral top-level alternatives, they do not all have to have the same + eral top-level alternatives, they do not all have to have the same fixed length. Thus (?<=bullock|donkey) @@ -4446,23 +4577,22 @@ ASSERTIONS (?<!dogs?|cats?) - causes an error at compile time. Branches that match different length - strings are permitted only at the top level of a lookbehind assertion. - This is an extension compared with Perl (at least for 5.8), which - requires all branches to match the same length of string. An assertion - such as + causes an error at compile time. Branches that match different length + strings are permitted only at the top level of a lookbehind assertion. + This is an extension compared with Perl (5.8 and 5.10), which requires + all branches to match the same length of string. An assertion such as (?<=ab(c|de)) is not permitted, because its single top-level branch can match two - different lengths, but it is acceptable if rewritten to use two top- - level branches: + different lengths, but it is acceptable to PCRE if rewritten to use two + top-level branches: (?<=abc|abde) In some cases, the Perl 5.10 escape sequence \K (see above) can be used - instead of a lookbehind assertion; this is not restricted to a fixed- - length. + instead of a lookbehind assertion to get round the fixed-length + restriction. The implementation of lookbehind assertions is, for each alternative, to temporarily move the current position back by the fixed length and @@ -4474,9 +4604,13 @@ ASSERTIONS ble to calculate the length of the lookbehind. The \X and \R escapes, which can match different numbers of bytes, are also not permitted. + "Subroutine" calls (see below) such as (?2) or (?&X) are permitted in + lookbehinds, as long as the subpattern matches a fixed-length string. + Recursion, however, is not supported. + Possessive quantifiers can be used in conjunction with lookbehind - assertions to specify efficient matching at the end of the subject - string. Consider a simple pattern such as + assertions to specify efficient matching of fixed-length strings at the + end of subject strings. Consider a simple pattern such as abcd$ @@ -4539,9 +4673,9 @@ CONDITIONAL SUBPATTERNS It is possible to cause the matching process to obey a subpattern con- ditionally or to choose between two alternative subpatterns, depending - on the result of an assertion, or whether a previous capturing subpat- - tern matched or not. The two possible forms of conditional subpattern - are + on the result of an assertion, or whether a specific capturing subpat- + tern has already been matched. The two possible forms of conditional + subpattern are: (?(condition)yes-pattern) (?(condition)yes-pattern|no-pattern) @@ -4556,138 +4690,144 @@ CONDITIONAL SUBPATTERNS Checking for a used subpattern by number If the text between the parentheses consists of a sequence of digits, - the condition is true if the capturing subpattern of that number has - previously matched. An alternative notation is to precede the digits - with a plus or minus sign. In this case, the subpattern number is rela- - tive rather than absolute. The most recently opened parentheses can be - referenced by (?(-1), the next most recent by (?(-2), and so on. In - looping constructs it can also make sense to refer to subsequent groups - with constructs such as (?(+2). - - Consider the following pattern, which contains non-significant white + the condition is true if a capturing subpattern of that number has pre- + viously matched. If there is more than one capturing subpattern with + the same number (see the earlier section about duplicate subpattern + numbers), the condition is true if any of them have been set. An alter- + native notation is to precede the digits with a plus or minus sign. In + this case, the subpattern number is relative rather than absolute. The + most recently opened parentheses can be referenced by (?(-1), the next + most recent by (?(-2), and so on. In looping constructs it can also + make sense to refer to subsequent groups with constructs such as + (?(+2). + + Consider the following pattern, which contains non-significant white space to make it more readable (assume the PCRE_EXTENDED option) and to divide it into three parts for ease of discussion: ( \( )? [^()]+ (?(1) \) ) - The first part matches an optional opening parenthesis, and if that + The first part matches an optional opening parenthesis, and if that character is present, sets it as the first captured substring. The sec- - ond part matches one or more characters that are not parentheses. The + ond part matches one or more characters that are not parentheses. The third part is a conditional subpattern that tests whether the first set of parentheses matched or not. If they did, that is, if subject started with an opening parenthesis, the condition is true, and so the yes-pat- - tern is executed and a closing parenthesis is required. Otherwise, - since no-pattern is not present, the subpattern matches nothing. In - other words, this pattern matches a sequence of non-parentheses, + tern is executed and a closing parenthesis is required. Otherwise, + since no-pattern is not present, the subpattern matches nothing. In + other words, this pattern matches a sequence of non-parentheses, optionally enclosed in parentheses. - If you were embedding this pattern in a larger one, you could use a + If you were embedding this pattern in a larger one, you could use a relative reference: ...other stuff... ( \( )? [^()]+ (?(-1) \) ) ... - This makes the fragment independent of the parentheses in the larger + This makes the fragment independent of the parentheses in the larger pattern. Checking for a used subpattern by name - Perl uses the syntax (?(<name>)...) or (?('name')...) to test for a - used subpattern by name. For compatibility with earlier versions of - PCRE, which had this facility before Perl, the syntax (?(name)...) is - also recognized. However, there is a possible ambiguity with this syn- - tax, because subpattern names may consist entirely of digits. PCRE - looks first for a named subpattern; if it cannot find one and the name - consists entirely of digits, PCRE looks for a subpattern of that num- - ber, which must be greater than zero. Using subpattern names that con- + Perl uses the syntax (?(<name>)...) or (?('name')...) to test for a + used subpattern by name. For compatibility with earlier versions of + PCRE, which had this facility before Perl, the syntax (?(name)...) is + also recognized. However, there is a possible ambiguity with this syn- + tax, because subpattern names may consist entirely of digits. PCRE + looks first for a named subpattern; if it cannot find one and the name + consists entirely of digits, PCRE looks for a subpattern of that num- + ber, which must be greater than zero. Using subpattern names that con- sist entirely of digits is not recommended. Rewriting the above example to use a named subpattern gives this: (?<OPEN> \( )? [^()]+ (?(<OPEN>) \) ) + If the name used in a condition of this kind is a duplicate, the test + is applied to all subpatterns of the same name, and is true if any one + of them has matched. Checking for pattern recursion If the condition is the string (R), and there is no subpattern with the - name R, the condition is true if a recursive call to the whole pattern + name R, the condition is true if a recursive call to the whole pattern or any subpattern has been made. If digits or a name preceded by amper- sand follow the letter R, for example: (?(R3)...) or (?(R&name)...) - the condition is true if the most recent recursion is into the subpat- - tern whose number or name is given. This condition does not check the - entire recursion stack. + the condition is true if the most recent recursion is into a subpattern + whose number or name is given. This condition does not check the entire + recursion stack. If the name used in a condition of this kind is a + duplicate, the test is applied to all subpatterns of the same name, and + is true if any one of them is the most recent recursion. - At "top level", all these recursion test conditions are false. Recur- - sive patterns are described below. + At "top level", all these recursion test conditions are false. The + syntax for recursive patterns is described below. Defining subpatterns for use by reference only - If the condition is the string (DEFINE), and there is no subpattern - with the name DEFINE, the condition is always false. In this case, - there may be only one alternative in the subpattern. It is always - skipped if control reaches this point in the pattern; the idea of - DEFINE is that it can be used to define "subroutines" that can be ref- - erenced from elsewhere. (The use of "subroutines" is described below.) - For example, a pattern to match an IPv4 address could be written like + If the condition is the string (DEFINE), and there is no subpattern + with the name DEFINE, the condition is always false. In this case, + there may be only one alternative in the subpattern. It is always + skipped if control reaches this point in the pattern; the idea of + DEFINE is that it can be used to define "subroutines" that can be ref- + erenced from elsewhere. (The use of "subroutines" is described below.) + For example, a pattern to match an IPv4 address could be written like this (ignore whitespace and line breaks): (?(DEFINE) (?<byte> 2[0-4]\d | 25[0-5] | 1\d\d | [1-9]?\d) ) \b (?&byte) (\.(?&byte)){3} \b - The first part of the pattern is a DEFINE group inside which a another - group named "byte" is defined. This matches an individual component of - an IPv4 address (a number less than 256). When matching takes place, - this part of the pattern is skipped because DEFINE acts like a false - condition. - - The rest of the pattern uses references to the named group to match the - four dot-separated components of an IPv4 address, insisting on a word - boundary at each end. + The first part of the pattern is a DEFINE group inside which a another + group named "byte" is defined. This matches an individual component of + an IPv4 address (a number less than 256). When matching takes place, + this part of the pattern is skipped because DEFINE acts like a false + condition. The rest of the pattern uses references to the named group + to match the four dot-separated components of an IPv4 address, insist- + ing on a word boundary at each end. Assertion conditions - If the condition is not in any of the above formats, it must be an - assertion. This may be a positive or negative lookahead or lookbehind - assertion. Consider this pattern, again containing non-significant + If the condition is not in any of the above formats, it must be an + assertion. This may be a positive or negative lookahead or lookbehind + assertion. Consider this pattern, again containing non-significant white space, and with the two alternatives on the second line: (?(?=[^a-z]*[a-z]) \d{2}-[a-z]{3}-\d{2} | \d{2}-\d{2}-\d{2} ) - The condition is a positive lookahead assertion that matches an - optional sequence of non-letters followed by a letter. In other words, - it tests for the presence of at least one letter in the subject. If a - letter is found, the subject is matched against the first alternative; - otherwise it is matched against the second. This pattern matches - strings in one of the two forms dd-aaa-dd or dd-dd-dd, where aaa are + The condition is a positive lookahead assertion that matches an + optional sequence of non-letters followed by a letter. In other words, + it tests for the presence of at least one letter in the subject. If a + letter is found, the subject is matched against the first alternative; + otherwise it is matched against the second. This pattern matches + strings in one of the two forms dd-aaa-dd or dd-dd-dd, where aaa are letters and dd are digits. COMMENTS - The sequence (?# marks the start of a comment that continues up to the - next closing parenthesis. Nested parentheses are not permitted. The - characters that make up a comment play no part in the pattern matching + The sequence (?# marks the start of a comment that continues up to the + next closing parenthesis. Nested parentheses are not permitted. The + characters that make up a comment play no part in the pattern matching at all. - If the PCRE_EXTENDED option is set, an unescaped # character outside a - character class introduces a comment that continues to immediately + If the PCRE_EXTENDED option is set, an unescaped # character outside a + character class introduces a comment that continues to immediately after the next newline in the pattern. RECURSIVE PATTERNS - Consider the problem of matching a string in parentheses, allowing for - unlimited nested parentheses. Without the use of recursion, the best - that can be done is to use a pattern that matches up to some fixed - depth of nesting. It is not possible to handle an arbitrary nesting + Consider the problem of matching a string in parentheses, allowing for + unlimited nested parentheses. Without the use of recursion, the best + that can be done is to use a pattern that matches up to some fixed + depth of nesting. It is not possible to handle an arbitrary nesting depth. For some time, Perl has provided a facility that allows regular expres- - sions to recurse (amongst other things). It does this by interpolating - Perl code in the expression at run time, and the code can refer to the + sions to recurse (amongst other things). It does this by interpolating + Perl code in the expression at run time, and the code can refer to the expression itself. A Perl pattern using code interpolation to solve the parentheses problem can be created like this: @@ -4697,44 +4837,41 @@ RECURSIVE PATTERNS refers recursively to the pattern in which it appears. Obviously, PCRE cannot support the interpolation of Perl code. Instead, - it supports special syntax for recursion of the entire pattern, and - also for individual subpattern recursion. After its introduction in - PCRE and Python, this kind of recursion was introduced into Perl at - release 5.10. + it supports special syntax for recursion of the entire pattern, and + also for individual subpattern recursion. After its introduction in + PCRE and Python, this kind of recursion was subsequently introduced + into Perl at release 5.10. - A special item that consists of (? followed by a number greater than + A special item that consists of (? followed by a number greater than zero and a closing parenthesis is a recursive call of the subpattern of - the given number, provided that it occurs inside that subpattern. (If - not, it is a "subroutine" call, which is described in the next sec- - tion.) The special item (?R) or (?0) is a recursive call of the entire + the given number, provided that it occurs inside that subpattern. (If + not, it is a "subroutine" call, which is described in the next sec- + tion.) The special item (?R) or (?0) is a recursive call of the entire regular expression. - In PCRE (like Python, but unlike Perl), a recursive subpattern call is - always treated as an atomic group. That is, once it has matched some of - the subject string, it is never re-entered, even if it contains untried - alternatives and there is a subsequent matching failure. - This PCRE pattern solves the nested parentheses problem (assume the PCRE_EXTENDED option is set so that white space is ignored): - \( ( (?>[^()]+) | (?R) )* \) + \( ( [^()]++ | (?R) )* \) First it matches an opening parenthesis. Then it matches any number of substrings which can either be a sequence of non-parentheses, or a recursive match of the pattern itself (that is, a correctly parenthe- - sized substring). Finally there is a closing parenthesis. + sized substring). Finally there is a closing parenthesis. Note the use + of a possessive quantifier to avoid backtracking into sequences of non- + parentheses. If this were part of a larger pattern, you would not want to recurse the entire pattern, so instead you could use this: - ( \( ( (?>[^()]+) | (?1) )* \) ) + ( \( ( [^()]++ | (?1) )* \) ) We have put the pattern into parentheses, and caused the recursion to refer to them instead of the whole pattern. In a larger pattern, keeping track of parenthesis numbers can be - tricky. This is made easier by the use of relative references. (A Perl - 5.10 feature.) Instead of (?1) in the pattern above you can write + tricky. This is made easier by the use of relative references (a Perl + 5.10 feature). Instead of (?1) in the pattern above you can write (?-2) to refer to the second most recently opened parentheses preceding the recursion. In other words, a negative number counts capturing parentheses leftwards from the point at which it is encountered. @@ -4749,46 +4886,40 @@ RECURSIVE PATTERNS syntax for this is (?&name); PCRE's earlier syntax (?P>name) is also supported. We could rewrite the above example as follows: - (?<pn> \( ( (?>[^()]+) | (?&pn) )* \) ) + (?<pn> \( ( [^()]++ | (?&pn) )* \) ) If there is more than one subpattern with the same name, the earliest one is used. This particular example pattern that we have been looking at contains - nested unlimited repeats, and so the use of atomic grouping for match- - ing strings of non-parentheses is important when applying the pattern - to strings that do not match. For example, when this pattern is applied - to + nested unlimited repeats, and so the use of a possessive quantifier for + matching strings of non-parentheses is important when applying the pat- + tern to strings that do not match. For example, when this pattern is + applied to (aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa() - it yields "no match" quickly. However, if atomic grouping is not used, - the match runs for a very long time indeed because there are so many - different ways the + and * repeats can carve up the subject, and all - have to be tested before failure can be reported. + it yields "no match" quickly. However, if a possessive quantifier is + not used, the match runs for a very long time indeed because there are + so many different ways the + and * repeats can carve up the subject, + and all have to be tested before failure can be reported. - At the end of a match, the values set for any capturing subpatterns are - those from the outermost level of the recursion at which the subpattern - value is set. If you want to obtain intermediate values, a callout - function can be used (see below and the pcrecallout documentation). If - the pattern above is matched against + At the end of a match, the values of capturing parentheses are those + from the outermost level. If you want to obtain intermediate values, a + callout function can be used (see below and the pcrecallout documenta- + tion). If the pattern above is matched against (ab(cd)ef) - the value for the capturing parentheses is "ef", which is the last - value taken on at the top level. If additional parentheses are added, - giving - - \( ( ( (?>[^()]+) | (?R) )* ) \) - ^ ^ - ^ ^ + the value for the inner capturing parentheses (numbered 2) is "ef", + which is the last value taken on at the top level. If a capturing sub- + pattern is not matched at the top level, its final value is unset, even + if it is (temporarily) set at a deeper level. - the string they capture is "ab(cd)ef", the contents of the top level - parentheses. If there are more than 15 capturing parentheses in a pat- - tern, PCRE has to obtain extra memory to store data during a recursion, - which it does by using pcre_malloc, freeing it via pcre_free after- - wards. If no memory can be obtained, the match fails with the - PCRE_ERROR_NOMEMORY error. + If there are more than 15 capturing parentheses in a pattern, PCRE has + to obtain extra memory to store data during a recursion, which it does + by using pcre_malloc, freeing it via pcre_free afterwards. If no memory + can be obtained, the match fails with the PCRE_ERROR_NOMEMORY error. Do not confuse the (?R) item with the condition (R), which tests for recursion. Consider this pattern, which matches text in angle brack- @@ -4802,6 +4933,80 @@ RECURSIVE PATTERNS two different alternatives for the recursive and non-recursive cases. The (?R) item is the actual recursive call. + Recursion difference from Perl + + In PCRE (like Python, but unlike Perl), a recursive subpattern call is + always treated as an atomic group. That is, once it has matched some of + the subject string, it is never re-entered, even if it contains untried + alternatives and there is a subsequent matching failure. This can be + illustrated by the following pattern, which purports to match a palin- + dromic string that contains an odd number of characters (for example, + "a", "aba", "abcba", "abcdcba"): + + ^(.|(.)(?1)\2)$ + + The idea is that it either matches a single character, or two identical + characters surrounding a sub-palindrome. In Perl, this pattern works; + in PCRE it does not if the pattern is longer than three characters. + Consider the subject string "abcba": + + At the top level, the first character is matched, but as it is not at + the end of the string, the first alternative fails; the second alterna- + tive is taken and the recursion kicks in. The recursive call to subpat- + tern 1 successfully matches the next character ("b"). (Note that the + beginning and end of line tests are not part of the recursion). + + Back at the top level, the next character ("c") is compared with what + subpattern 2 matched, which was "a". This fails. Because the recursion + is treated as an atomic group, there are now no backtracking points, + and so the entire match fails. (Perl is able, at this point, to re- + enter the recursion and try the second alternative.) However, if the + pattern is written with the alternatives in the other order, things are + different: + + ^((.)(?1)\2|.)$ + + This time, the recursing alternative is tried first, and continues to + recurse until it runs out of characters, at which point the recursion + fails. But this time we do have another alternative to try at the + higher level. That is the big difference: in the previous case the + remaining alternative is at a deeper recursion level, which PCRE cannot + use. + + To change the pattern so that matches all palindromic strings, not just + those with an odd number of characters, it is tempting to change the + pattern to this: + + ^((.)(?1)\2|.?)$ + + Again, this works in Perl, but not in PCRE, and for the same reason. + When a deeper recursion has matched a single character, it cannot be + entered again in order to match an empty string. The solution is to + separate the two cases, and write out the odd and even cases as alter- + natives at the higher level: + + ^(?:((.)(?1)\2|)|((.)(?3)\4|.)) + + If you want to match typical palindromic phrases, the pattern has to + ignore all non-word characters, which can be done like this: + + ^\W*+(?:((.)\W*+(?1)\W*+\2|)|((.)\W*+(?3)\W*+\4|\W*+.\W*+))\W*+$ + + If run with the PCRE_CASELESS option, this pattern matches phrases such + as "A man, a plan, a canal: Panama!" and it works well in both PCRE and + Perl. Note the use of the possessive quantifier *+ to avoid backtrack- + ing into sequences of non-word characters. Without this, PCRE takes a + great deal longer (ten times or more) to match typical phrases, and + Perl takes so long that you think it has gone into a loop. + + WARNING: The palindrome-matching patterns above work only if the sub- + ject string does not start with a palindrome that is shorter than the + entire string. For example, although "abcba" is correctly matched, if + the subject is "ababa", PCRE finds the palindrome "aba" at the start, + then fails at top level because the end of the string does not follow. + Once again, it cannot jump back into the recursion to try other alter- + natives, so the entire match fails. + SUBPATTERNS AS SUBROUTINES @@ -4828,10 +5033,12 @@ SUBPATTERNS AS SUBROUTINES two strings. Another example is given in the discussion of DEFINE above. - Like recursive subpatterns, a "subroutine" call is always treated as an + Like recursive subpatterns, a subroutine call is always treated as an atomic group. That is, once it has matched some of the subject string, it is never re-entered, even if it contains untried alternatives and - there is a subsequent matching failure. + there is a subsequent matching failure. Any capturing parentheses that + are set during the subroutine call revert to their previous values + afterwards. When a subpattern is used as a subroutine, processing options such as case-independence are fixed when the subpattern is defined. They cannot @@ -4914,6 +5121,12 @@ BACKTRACKING CONTROL (*FAIL), which behaves like a failing negative assertion, they cause an error if encountered by pcre_dfa_exec(). + If any of these verbs are used in an assertion or subroutine subpattern + (including recursive subpatterns), their effect is confined to that + subpattern; it does not extend to the surrounding pattern. Note that + such subpatterns are processed as anchored at the point where they are + tested. + The new verbs make use of what was previously invalid syntax: an open- ing parenthesis followed by an asterisk. In Perl, they are generally of the form (*VERB:ARG) but PCRE does not support the use of arguments, so @@ -4928,14 +5141,14 @@ BACKTRACKING CONTROL This verb causes the match to end successfully, skipping the remainder of the pattern. When inside a recursion, only the innermost pattern is - ended immediately. PCRE differs from Perl in what happens if the - (*ACCEPT) is inside capturing parentheses. In Perl, the data so far is - captured: in PCRE no data is captured. For example: + ended immediately. If (*ACCEPT) is inside capturing parentheses, the + data so far is captured. (This feature was added to PCRE at release + 8.00.) For example: - A(A|B(*ACCEPT)|C)D + A((?:A|B(*ACCEPT)|C)D) - This matches "AB", "AAD", or "ACD", but when it matches "AB", no data - is captured. + This matches "AB", "AAD", or "ACD"; when it matches "AB", "B" is cap- + tured by the outer parentheses. (*FAIL) or (*F) @@ -4962,9 +5175,9 @@ BACKTRACKING CONTROL This verb causes the whole match to fail outright if the rest of the pattern does not match. Even if the pattern is unanchored, no further - attempts to find a match by advancing the start point take place. Once - (*COMMIT) has been passed, pcre_exec() is committed to finding a match - at the current starting point, or not at all. For example: + attempts to find a match by advancing the starting point take place. + Once (*COMMIT) has been passed, pcre_exec() is committed to finding a + match at the current starting point, or not at all. For example: a+(*COMMIT)b @@ -4996,7 +5209,7 @@ BACKTRACKING CONTROL If the subject is "aaaac...", after the first match attempt fails (starting at the first character in the string), the starting point skips on to start the next attempt at "c". Note that a possessive quan- - tifer does not have the same effect in this example; although it would + tifer does not have the same effect as this example; although it would suppress backtracking during the first match attempt, the second attempt would start at the second character instead of skipping on to "c". @@ -5019,7 +5232,7 @@ BACKTRACKING CONTROL SEE ALSO - pcreapi(3), pcrecallout(3), pcrematching(3), pcre(3). + pcreapi(3), pcrecallout(3), pcrematching(3), pcresyntax(3), pcre(3). AUTHOR @@ -5031,7 +5244,7 @@ AUTHOR REVISION - Last updated: 11 April 2009 + Last updated: 18 October 2009 Copyright (c) 1997-2009 University of Cambridge. ------------------------------------------------------------------------------ @@ -5412,77 +5625,182 @@ PARTIAL MATCHING IN PCRE If the application sees the user's keystrokes one by one, and can check that what has been typed so far is potentially valid, it is able to - raise an error as soon as a mistake is made, possibly beeping and not - reflecting the character that has been typed. This immediate feedback - is likely to be a better user interface than a check that is delayed - until the entire string has been entered. - - PCRE supports the concept of partial matching by means of the PCRE_PAR- - TIAL option, which can be set when calling pcre_exec() or - pcre_dfa_exec(). When this flag is set for pcre_exec(), the return code - PCRE_ERROR_NOMATCH is converted into PCRE_ERROR_PARTIAL if at any time - during the matching process the last part of the subject string matched - part of the pattern. Unfortunately, for non-anchored matching, it is - not possible to obtain the position of the start of the partial match. - No captured data is set when PCRE_ERROR_PARTIAL is returned. + raise an error as soon as a mistake is made, by beeping and not + reflecting the character that has been typed, for example. This immedi- + ate feedback is likely to be a better user interface than a check that + is delayed until the entire string has been entered. Partial matching + can also sometimes be useful when the subject string is very long and + is not all available at once. + + PCRE supports partial matching by means of the PCRE_PARTIAL_SOFT and + PCRE_PARTIAL_HARD options, which can be set when calling pcre_exec() or + pcre_dfa_exec(). For backwards compatibility, PCRE_PARTIAL is a synonym + for PCRE_PARTIAL_SOFT. The essential difference between the two options + is whether or not a partial match is preferred to an alternative com- + plete match, though the details differ between the two matching func- + tions. If both options are set, PCRE_PARTIAL_HARD takes precedence. + + Setting a partial matching option disables two of PCRE's optimizations. + PCRE remembers the last literal byte in a pattern, and abandons match- + ing immediately if such a byte is not present in the subject string. + This optimization cannot be used for a subject string that might match + only partially. If the pattern was studied, PCRE knows the minimum + length of a matching string, and does not bother to run the matching + function on shorter strings. This optimization is also disabled for + partial matching. + + +PARTIAL MATCHING USING pcre_exec() + + A partial match occurs during a call to pcre_exec() whenever the end of + the subject string is reached successfully, but matching cannot con- + tinue because more characters are needed. However, at least one charac- + ter must have been matched. (In other words, a partial match can never + be an empty string.) + + If PCRE_PARTIAL_SOFT is set, the partial match is remembered, but + matching continues as normal, and other alternatives in the pattern are + tried. If no complete match can be found, pcre_exec() returns + PCRE_ERROR_PARTIAL instead of PCRE_ERROR_NOMATCH. If there are at least + two slots in the offsets vector, the first of them is set to the offset + of the earliest character that was inspected when the partial match was + found. For convenience, the second offset points to the end of the + string so that a substring can easily be identified. + + For the majority of patterns, the first offset identifies the start of + the partially matched string. However, for patterns that contain look- + behind assertions, or \K, or begin with \b or \B, earlier characters + have been inspected while carrying out the match. For example: + + /(?<=abc)123/ + + This pattern matches "123", but only if it is preceded by "abc". If the + subject string is "xyzabc12", the offsets after a partial match are for + the substring "abc12", because all these characters are needed if + another match is tried with extra characters added. + + If there is more than one partial match, the first one that was found + provides the data that is returned. Consider this pattern: + + /123\w+X|dogY/ + + If this is matched against the subject string "abc123dog", both alter- + natives fail to match, but the end of the subject is reached during + matching, so PCRE_ERROR_PARTIAL is returned instead of + PCRE_ERROR_NOMATCH. The offsets are set to 3 and 9, identifying + "123dog" as the first partial match that was found. (In this example, + there are two partial matches, because "dog" on its own partially + matches the second alternative.) + + If PCRE_PARTIAL_HARD is set for pcre_exec(), it returns PCRE_ERROR_PAR- + TIAL as soon as a partial match is found, without continuing to search + for possible complete matches. The difference between the two options + can be illustrated by a pattern such as: + + /dog(sbody)?/ + + This matches either "dog" or "dogsbody", greedily (that is, it prefers + the longer string if possible). If it is matched against the string + "dog" with PCRE_PARTIAL_SOFT, it yields a complete match for "dog". + However, if PCRE_PARTIAL_HARD is set, the result is PCRE_ERROR_PARTIAL. + On the other hand, if the pattern is made ungreedy the result is dif- + ferent: + + /dog(sbody)??/ + + In this case the result is always a complete match because pcre_exec() + finds that first, and it never continues after finding a match. It + might be easier to follow this explanation by thinking of the two pat- + terns like this: + + /dog(sbody)?/ is the same as /dogsbody|dog/ + /dog(sbody)??/ is the same as /dog|dogsbody/ + + The second pattern will never match "dogsbody" when pcre_exec() is + used, because it will always find the shorter match first. + + +PARTIAL MATCHING USING pcre_dfa_exec() + + The pcre_dfa_exec() function moves along the subject string character + by character, without backtracking, searching for all possible matches + simultaneously. If the end of the subject is reached before the end of + the pattern, there is the possibility of a partial match, again pro- + vided that at least one character has matched. + + When PCRE_PARTIAL_SOFT is set, PCRE_ERROR_PARTIAL is returned only if + there have been no complete matches. Otherwise, the complete matches + are returned. However, if PCRE_PARTIAL_HARD is set, a partial match + takes precedence over any complete matches. The portion of the string + that was inspected when the longest partial match was found is set as + the first matching string, provided there are at least two slots in the + offsets vector. + + Because pcre_dfa_exec() always searches for all possible matches, and + there is no difference between greedy and ungreedy repetition, its be- + haviour is different from pcre_exec when PCRE_PARTIAL_HARD is set. Con- + sider the string "dog" matched against the ungreedy pattern shown + above: - When PCRE_PARTIAL is set for pcre_dfa_exec(), the return code - PCRE_ERROR_NOMATCH is converted into PCRE_ERROR_PARTIAL if the end of - the subject is reached, there have been no complete matches, but there - is still at least one matching possibility. The portion of the string - that provided the partial match is set as the first matching string. + /dog(sbody)??/ - Using PCRE_PARTIAL disables one of PCRE's optimizations. PCRE remembers - the last literal byte in a pattern, and abandons matching immediately - if such a byte is not present in the subject string. This optimization - cannot be used for a subject string that might match only partially. + Whereas pcre_exec() stops as soon as it finds the complete match for + "dog", pcre_dfa_exec() also finds the partial match for "dogsbody", and + so returns that when PCRE_PARTIAL_HARD is set. -RESTRICTED PATTERNS FOR PCRE_PARTIAL +PARTIAL MATCHING AND WORD BOUNDARIES - Because of the way certain internal optimizations are implemented in - the pcre_exec() function, the PCRE_PARTIAL option cannot be used with - all patterns. These restrictions do not apply when pcre_dfa_exec() is - used. For pcre_exec(), repeated single characters such as + If a pattern ends with one of sequences \b or \B, which test for word + boundaries, partial matching with PCRE_PARTIAL_SOFT can give counter- + intuitive results. Consider this pattern: - a{2,4} + /\bcat\b/ - and repeated single metasequences such as + This matches "cat", provided there is a word boundary at either end. If + the subject string is "the cat", the comparison of the final "t" with a + following character cannot take place, so a partial match is found. + However, pcre_exec() carries on with normal matching, which matches \b + at the end of the subject when the last character is a letter, thus + finding a complete match. The result, therefore, is not PCRE_ERROR_PAR- + TIAL. The same thing happens with pcre_dfa_exec(), because it also + finds the complete match. - \d+ + Using PCRE_PARTIAL_HARD in this case does yield PCRE_ERROR_PARTIAL, + because then the partial match takes precedence. - are not permitted if the maximum number of occurrences is greater than - one. Optional items such as \d? (where the maximum is one) are permit- - ted. Quantifiers with any values are permitted after parentheses, so - the invalid examples above can be coded thus: - (a){2,4} - (\d)+ +FORMERLY RESTRICTED PATTERNS - These constructions run more slowly, but for the kinds of application - that are envisaged for this facility, this is not felt to be a major - restriction. + For releases of PCRE prior to 8.00, because of the way certain internal + optimizations were implemented in the pcre_exec() function, the + PCRE_PARTIAL option (predecessor of PCRE_PARTIAL_SOFT) could not be + used with all patterns. From release 8.00 onwards, the restrictions no + longer apply, and partial matching with pcre_exec() can be requested + for any pattern. - If PCRE_PARTIAL is set for a pattern that does not conform to the - restrictions, pcre_exec() returns the error code PCRE_ERROR_BADPARTIAL - (-13). You can use the PCRE_INFO_OKPARTIAL call to pcre_fullinfo() to - find out if a compiled pattern can be used for partial matching. + Items that were formerly restricted were repeated single characters and + repeated metasequences. If PCRE_PARTIAL was set for a pattern that did + not conform to the restrictions, pcre_exec() returned the error code + PCRE_ERROR_BADPARTIAL (-13). This error code is no longer in use. The + PCRE_INFO_OKPARTIAL call to pcre_fullinfo() to find out if a compiled + pattern can be used for partial matching now always returns 1. EXAMPLE OF PARTIAL MATCHING USING PCRETEST If the escape sequence \P is present in a pcretest data line, the - PCRE_PARTIAL flag is used for the match. Here is a run of pcretest that - uses the date example quoted above: + PCRE_PARTIAL_SOFT option is used for the match. Here is a run of + pcretest that uses the date example quoted above: re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/ data> 25jun04\P 0: 25jun04 1: jun data> 25dec3\P - Partial match + Partial match: 23dec3 data> 3ju\P - Partial match + Partial match: 3ju data> 3juj\P No match data> j\P @@ -5490,36 +5808,23 @@ EXAMPLE OF PARTIAL MATCHING USING PCRETEST The first data string is matched completely, so pcretest shows the matched substrings. The remaining four strings do not match the com- - plete pattern, but the first two are partial matches. The same test, - using pcre_dfa_exec() matching (by means of the \D escape sequence), - produces the following output: + plete pattern, but the first two are partial matches. Similar output is + obtained when pcre_dfa_exec() is used. - re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/ - data> 25jun04\P\D - 0: 25jun04 - data> 23dec3\P\D - Partial match: 23dec3 - data> 3ju\P\D - Partial match: 3ju - data> 3juj\P\D - No match - data> j\P\D - No match - - Notice that in this case the portion of the string that was matched is - made available. + If the escape sequence \P is present more than once in a pcretest data + line, the PCRE_PARTIAL_HARD option is set for the match. MULTI-SEGMENT MATCHING WITH pcre_dfa_exec() When a partial match has been found using pcre_dfa_exec(), it is possi- - ble to continue the match by providing additional subject data and - calling pcre_dfa_exec() again with the same compiled regular expres- - sion, this time setting the PCRE_DFA_RESTART option. You must also pass - the same working space as before, because this is where details of the - previous partial match are stored. Here is an example using pcretest, - using the \R escape sequence to set the PCRE_DFA_RESTART option (\P and - \D are as above): + ble to continue the match by providing additional subject data and + calling pcre_dfa_exec() again with the same compiled regular expres- + sion, this time setting the PCRE_DFA_RESTART option. You must pass the + same working space as before, because this is where details of the pre- + vious partial match are stored. Here is an example using pcretest, + using the \R escape sequence to set the PCRE_DFA_RESTART option (\D + specifies the use of pcre_dfa_exec()): re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/ data> 23ja\P\D @@ -5527,38 +5832,77 @@ MULTI-SEGMENT MATCHING WITH pcre_dfa_exec() data> n05\R\D 0: n05 - The first call has "23ja" as the subject, and requests partial match- - ing; the second call has "n05" as the subject for the continued - (restarted) match. Notice that when the match is complete, only the - last part is shown; PCRE does not retain the previously partially- - matched string. It is up to the calling program to do that if it needs + The first call has "23ja" as the subject, and requests partial match- + ing; the second call has "n05" as the subject for the continued + (restarted) match. Notice that when the match is complete, only the + last part is shown; PCRE does not retain the previously partially- + matched string. It is up to the calling program to do that if it needs to. - You can set PCRE_PARTIAL with PCRE_DFA_RESTART to continue partial - matching over multiple segments. This facility can be used to pass very - long subject strings to pcre_dfa_exec(). However, some care is needed - for certain types of pattern. + You can set the PCRE_PARTIAL_SOFT or PCRE_PARTIAL_HARD options with + PCRE_DFA_RESTART to continue partial matching over multiple segments. + This facility can be used to pass very long subject strings to + pcre_dfa_exec(). - 1. If the pattern contains tests for the beginning or end of a line, - you need to pass the PCRE_NOTBOL or PCRE_NOTEOL options, as appropri- - ate, when the subject string for any call does not contain the begin- - ning or end of a line. - 2. If the pattern contains backward assertions (including \b or \B), - you need to arrange for some overlap in the subject strings to allow - for this. For example, you could pass the subject in chunks that are - 500 bytes long, but in a buffer of 700 bytes, with the starting offset - set to 200 and the previous 200 bytes at the start of the buffer. +MULTI-SEGMENT MATCHING WITH pcre_exec() + + From release 8.00, pcre_exec() can also be used to do multi-segment + matching. Unlike pcre_dfa_exec(), it is not possible to restart the + previous match with a new segment of data. Instead, new data must be + added to the previous subject string, and the entire match re-run, + starting from the point where the partial match occurred. Earlier data + can be discarded. Consider an unanchored pattern that matches dates: + + re> /\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d/ + data> The date is 23ja\P + Partial match: 23ja + + At this stage, an application could discard the text preceding "23ja", + add on text from the next segment, and call pcre_exec() again. Unlike + pcre_dfa_exec(), the entire matching string must always be available, + and the complete matching process occurs for each call, so more memory + and more processing time is needed. - 3. Matching a subject string that is split into multiple segments does - not always produce exactly the same result as matching over one single - long string. The difference arises when there are multiple matching - possibilities, because a partial match result is given only when there - are no completed matches in a call to pcre_dfa_exec(). This means that - as soon as the shortest match has been found, continuation to a new - subject segment is no longer possible. Consider this pcretest example: + Note: If the pattern contains lookbehind assertions, or \K, or starts + with \b or \B, the string that is returned for a partial match will + include characters that precede the partially matched string itself, + because these must be retained when adding on more characters for a + subsequent matching attempt. + + +ISSUES WITH MULTI-SEGMENT MATCHING + + Certain types of pattern may give problems with multi-segment matching, + whichever matching function is used. + + 1. If the pattern contains tests for the beginning or end of a line, + you need to pass the PCRE_NOTBOL or PCRE_NOTEOL options, as appropri- + ate, when the subject string for any call does not contain the begin- + ning or end of a line. + + 2. Lookbehind assertions at the start of a pattern are catered for in + the offsets that are returned for a partial match. However, in theory, + a lookbehind assertion later in the pattern could require even earlier + characters to be inspected, and it might not have been reached when a + partial match occurs. This is probably an extremely unlikely case; you + could guard against it to a certain extent by always including extra + characters at the start. + + 3. Matching a subject string that is split into multiple segments may + not always produce exactly the same result as matching over one single + long string, especially when PCRE_PARTIAL_SOFT is used. The section + "Partial Matching and Word Boundaries" above describes an issue that + arises if the pattern ends with \b or \B. Another kind of difference + may occur when there are multiple matching possibilities, because a + partial match result is given only when there are no completed matches. + This means that as soon as the shortest match has been found, continua- + tion to a new subject segment is no longer possible. Consider again + this pcretest example: re> /dog(sbody)?/ + data> dogsb\P + 0: dog data> do\P\D Partial match: do data> gsb\R\P\D @@ -5567,33 +5911,63 @@ MULTI-SEGMENT MATCHING WITH pcre_dfa_exec() 0: dogsbody 1: dog - The pattern matches the words "dog" or "dogsbody". When the subject is - presented in several parts ("do" and "gsb" being the first two) the - match stops when "dog" has been found, and it is not possible to con- - tinue. On the other hand, if "dogsbody" is presented as a single - string, both matches are found. + The first data line passes the string "dogsb" to pcre_exec(), setting + the PCRE_PARTIAL_SOFT option. Although the string is a partial match + for "dogsbody", the result is not PCRE_ERROR_PARTIAL, because the + shorter string "dog" is a complete match. Similarly, when the subject + is presented to pcre_dfa_exec() in several parts ("do" and "gsb" being + the first two) the match stops when "dog" has been found, and it is not + possible to continue. On the other hand, if "dogsbody" is presented as + a single string, pcre_dfa_exec() finds both matches. + + Because of these problems, it is probably best to use PCRE_PARTIAL_HARD + when matching multi-segment data. The example above then behaves dif- + ferently: + + re> /dog(sbody)?/ + data> dogsb\P\P + Partial match: dogsb + data> do\P\D + Partial match: do + data> gsb\R\P\P\D + Partial match: gsb - Because of this phenomenon, it does not usually make sense to end a - pattern that is going to be matched in this way with a variable repeat. 4. Patterns that contain alternatives at the top level which do not all - start with the same pattern item may not work as expected. For example, - consider this pattern: + start with the same pattern item may not work as expected when + PCRE_DFA_RESTART is used with pcre_dfa_exec(). For example, consider + this pattern: 1234|3789 - If the first part of the subject is "ABC123", a partial match of the - first alternative is found at offset 3. There is no partial match for + If the first part of the subject is "ABC123", a partial match of the + first alternative is found at offset 3. There is no partial match for the second alternative, because such a match does not start at the same - point in the subject string. Attempting to continue with the string - "789" does not yield a match because only those alternatives that match - at one point in the subject are remembered. The problem arises because - the start of the second alternative matches within the first alterna- - tive. There is no problem with anchored patterns or patterns such as: + point in the subject string. Attempting to continue with the string + "7890" does not yield a match because only those alternatives that + match at one point in the subject are remembered. The problem arises + because the start of the second alternative matches within the first + alternative. There is no problem with anchored patterns or patterns + such as: 1234|ABCD - where no string can be a partial match for both alternatives. + where no string can be a partial match for both alternatives. This is + not a problem if pcre_exec() is used, because the entire match has to + be rerun each time: + + re> /1234|3789/ + data> ABC123\P + Partial match: 123 + data> 1237890 + 0: 3789 + + Of course, instead of using PCRE_DFA_PARTIAL, the same technique of re- + running the entire match can also be used with pcre_dfa_exec(). Another + possibility is to work with two buffers. If a partial match at offset n + in the first buffer is followed by "no match" when PCRE_DFA_RESTART is + used on the second buffer, you can then try a new match starting at + offset n+1 in the first buffer. AUTHOR @@ -5605,8 +5979,8 @@ AUTHOR REVISION - Last updated: 04 June 2007 - Copyright (c) 1997-2007 University of Cambridge. + Last updated: 19 October 2009 + Copyright (c) 1997-2009 University of Cambridge. ------------------------------------------------------------------------------ @@ -5927,6 +6301,10 @@ DESCRIPTION easier to slot in PCRE as a replacement library. Other POSIX options are not even defined. + There are also some other options that are not defined by POSIX. These + have been added at the request of users who want to make use of certain + PCRE-specific features via the POSIX calling interface. + When PCRE is called via these functions, it is only the API that is POSIX-like in style. The syntax and semantics of the regular expres- sions themselves are still those of Perl, subject to the setting of @@ -5981,6 +6359,12 @@ COMPILING A PATTERN ing, the nmatch and pmatch arguments are ignored, and no captured strings are returned. + REG_UNGREEDY + + The PCRE_UNGREEDY option is set when the regular expression is passed + for compilation to the native function. Note that REG_UNGREEDY is not + part of the POSIX standard. + REG_UTF8 The PCRE_UTF8 option is set when the regular expression is passed for @@ -5993,7 +6377,7 @@ COMPILING A PATTERN semantics. In particular, the way it handles newline characters in the subject string is the Perl way, not the POSIX way. Note that setting PCRE_MULTILINE has only some of the effects specified for REG_NEWLINE. - It does not affect the way newlines are matched by . (they aren't) or + It does not affect the way newlines are matched by . (they are not) or by a negative class such as [^a] (they are). The yield of regcomp() is zero on success, and non-zero otherwise. The @@ -6001,6 +6385,10 @@ COMPILING A PATTERN is public: re_nsub contains the number of capturing subpatterns in the regular expression. Various error codes are defined in the header file. + NOTE: If the yield of regcomp() is non-zero, you must not attempt to + use the contents of the preg structure. If, for example, you pass it to + regexec(), the result is undefined and your program is likely to crash. + MATCHING NEWLINE CHARACTERS @@ -6076,36 +6464,39 @@ MATCHING A PATTERN matched strings is returned. The nmatch and pmatch arguments of regexec() are ignored. + If the value of nmatch is zero, or if the value pmatch is NULL, no data + about any matched strings is returned. + Otherwise,the portion of the string that was matched, and also any cap- tured substrings, are returned via the pmatch argument, which points to - an array of nmatch structures of type regmatch_t, containing the mem- - bers rm_so and rm_eo. These contain the offset to the first character - of each substring and the offset to the first character after the end - of each substring, respectively. The 0th element of the vector relates - to the entire portion of string that was matched; subsequent elements - relate to the capturing subpatterns of the regular expression. Unused + an array of nmatch structures of type regmatch_t, containing the mem- + bers rm_so and rm_eo. These contain the offset to the first character + of each substring and the offset to the first character after the end + of each substring, respectively. The 0th element of the vector relates + to the entire portion of string that was matched; subsequent elements + relate to the capturing subpatterns of the regular expression. Unused entries in the array have both structure members set to -1. - A successful match yields a zero return; various error codes are - defined in the header file, of which REG_NOMATCH is the "expected" + A successful match yields a zero return; various error codes are + defined in the header file, of which REG_NOMATCH is the "expected" failure code. ERROR MESSAGES The regerror() function maps a non-zero errorcode from either regcomp() - or regexec() to a printable message. If preg is not NULL, the error + or regexec() to a printable message. If preg is not NULL, the error should have arisen from the use of that structure. A message terminated - by a binary zero is placed in errbuf. The length of the message, - including the zero, is limited to errbuf_size. The yield of the func- + by a binary zero is placed in errbuf. The length of the message, + including the zero, is limited to errbuf_size. The yield of the func- tion is the size of buffer needed to hold the whole message. MEMORY USAGE - Compiling a regular expression causes memory to be allocated and asso- - ciated with the preg structure. The function regfree() frees all such - memory, after which preg may no longer be used as a compiled expres- + Compiling a regular expression causes memory to be allocated and asso- + ciated with the preg structure. The function regfree() frees all such + memory, after which preg may no longer be used as a compiled expres- sion. @@ -6118,7 +6509,7 @@ AUTHOR REVISION - Last updated: 11 March 2009 + Last updated: 02 September 2009 Copyright (c) 1997-2009 University of Cambridge. ------------------------------------------------------------------------------ @@ -6474,53 +6865,56 @@ NAME PCRE SAMPLE PROGRAM A simple, complete demonstration program, to get you started with using - PCRE, is supplied in the file pcredemo.c in the PCRE distribution. + PCRE, is supplied in the file pcredemo.c in the PCRE distribution. A + listing of this program is given in the pcredemo documentation. If you + do not have a copy of the PCRE distribution, you can save this listing + to re-create pcredemo.c. The program compiles the regular expression that is its first argument, - and matches it against the subject string in its second argument. No - PCRE options are set, and default character tables are used. If match- - ing succeeds, the program outputs the portion of the subject that + and matches it against the subject string in its second argument. No + PCRE options are set, and default character tables are used. If match- + ing succeeds, the program outputs the portion of the subject that matched, together with the contents of any captured substrings. If the -g option is given on the command line, the program then goes on to check for further matches of the same regular expression in the same - subject string. The logic is a little bit tricky because of the possi- - bility of matching an empty string. Comments in the code explain what + subject string. The logic is a little bit tricky because of the possi- + bility of matching an empty string. Comments in the code explain what is going on. - If PCRE is installed in the standard include and library directories - for your system, you should be able to compile the demonstration pro- - gram using this command: + If PCRE is installed in the standard include and library directories + for your operating system, you should be able to compile the demonstra- + tion program using this command: gcc -o pcredemo pcredemo.c -lpcre - If PCRE is installed elsewhere, you may need to add additional options - to the command line. For example, on a Unix-like system that has PCRE - installed in /usr/local, you can compile the demonstration program + If PCRE is installed elsewhere, you may need to add additional options + to the command line. For example, on a Unix-like system that has PCRE + installed in /usr/local, you can compile the demonstration program using a command like this: gcc -o pcredemo -I/usr/local/include pcredemo.c \ -L/usr/local/lib -lpcre - Once you have compiled the demonstration program, you can run simple + Once you have compiled the demonstration program, you can run simple tests like this: ./pcredemo 'cat|dog' 'the cat sat on the mat' ./pcredemo -g 'cat|dog' 'the dog sat on the cat' - Note that there is a much more comprehensive test program, called - pcretest, which supports many more facilities for testing regular + Note that there is a much more comprehensive test program, called + pcretest, which supports many more facilities for testing regular expressions and the PCRE library. The pcredemo program is provided as a simple coding example. - On some operating systems (e.g. Solaris), when PCRE is not installed in - the standard library directory, you may get an error like this when you - try to run pcredemo: + When you try to run pcredemo when PCRE is not installed in the standard + library directory, you may get an error like this on some operating + systems (e.g. Solaris): - ld.so.1: a.out: fatal: libpcre.so.0: open failed: No such file or + ld.so.1: a.out: fatal: libpcre.so.0: open failed: No such file or directory - This is caused by the way shared library support works on those sys- + This is caused by the way shared library support works on those sys- tems. You need to add -R/usr/local/lib @@ -6537,8 +6931,8 @@ AUTHOR REVISION - Last updated: 23 January 2008 - Copyright (c) 1997-2008 University of Cambridge. + Last updated: 30 September 2009 + Copyright (c) 1997-2009 University of Cambridge. ------------------------------------------------------------------------------ PCRESTACK(3) PCRESTACK(3) |