diff options
| author | nigel <nigel@2f5784b3-3f2a-0410-8824-cb99058d5e15> | 2007-02-24 21:40:45 +0000 |
|---|---|---|
| committer | nigel <nigel@2f5784b3-3f2a-0410-8824-cb99058d5e15> | 2007-02-24 21:40:45 +0000 |
| commit | 97cb05691b9cabed35f1a853c74d48c692aaabcf (patch) | |
| tree | cb7c68a44f0b79c6d90d9a18a7ec640c8435a5e7 /doc | |
| parent | 455fcc7e13a175722acfd2cca6ab99caa9606a22 (diff) | |
| download | pcre-97cb05691b9cabed35f1a853c74d48c692aaabcf.tar.gz | |
Load pcre-6.0 into code/trunk.
git-svn-id: svn://vcs.exim.org/pcre/code/trunk@77 2f5784b3-3f2a-0410-8824-cb99058d5e15
Diffstat (limited to 'doc')
44 files changed, 5038 insertions, 1385 deletions
diff --git a/doc/Tech.Notes b/doc/Tech.Notes index 18eb72b..322cc2d 100644 --- a/doc/Tech.Notes +++ b/doc/Tech.Notes @@ -32,18 +32,42 @@ terminology. OK, here's the real stuff ------------------------- -For the set of functions that forms PCRE (which are unrelated to those -mentioned above), I tried at first to invent an algorithm that used an amount -of store bounded by a multiple of the number of characters in the pattern, to -save on compiling time. However, because of the greater complexity in Perl -regular expressions, I couldn't do this. In any case, a first pass through the -pattern is needed, for a number of reasons. PCRE works by running a very -degenerate first pass to calculate a maximum store size, and then a second pass -to do the real compile - which may use a bit less than the predicted amount of -store. The idea is that this is going to turn out faster because the first pass -is degenerate and the second pass can just store stuff straight into the -vector. It does make the compiling functions bigger, of course, but they have -got quite big anyway to handle all the Perl stuff. +For the set of functions that form the "basic" PCRE library (which are +unrelated to those mentioned above), I tried at first to invent an algorithm +that used an amount of store bounded by a multiple of the number of characters +in the pattern, to save on compiling time. However, because of the greater +complexity in Perl regular expressions, I couldn't do this. In any case, a +first pass through the pattern is needed, for a number of reasons. PCRE works +by running a very degenerate first pass to calculate a maximum store size, and +then a second pass to do the real compile - which may use a bit less than the +predicted amount of store. The idea is that this is going to turn out faster +because the first pass is degenerate and the second pass can just store stuff +straight into the vector, which it knows is big enough. It does make the +compiling functions bigger, of course, but they have got quite big anyway to +handle all the Perl stuff. + +Traditional matching function +----------------------------- + +The "traditional", and original, matching function is called pcre_exec(), and +it implements an NFA algorithm, similar to the original Henry Spencer algorithm +and the way that Perl works. Not surprising, since it is intended to be as +compatible with Perl as possible. This is the function most users of PCRE will +use most of the time. + +Supplementary matching function +------------------------------- + +From PCRE 6.0, there is also a supplementary matching function called +pcre_dfa_exec(). This implements a DFA matching algorithm that searches +simultaneously for all possible matches that start at one point in the subject +string. (Going back to my roots: see Historical Note 1 above.) This function +intreprets the same compiled pattern data as pcre_exec(); however, not all the +facilities are available, and those that are don't always work in quite the +same way. See the user documentation for details. + +Format of compiled patterns +--------------------------- The compiled form of a pattern is a vector of bytes, containing items of variable length. The first byte in an item is an opcode, and the length of the @@ -223,14 +247,14 @@ number. This opcode is ignored while matching, but is fished out when handling the bracket itself. (They could have all been done like this, but I was making minimal changes.) -A bracket opcode is followed by two bytes which give the offset to the next -alternative OP_ALT or, if there aren't any branches, to the matching OP_KET -opcode. Each OP_ALT is followed by two bytes giving the offset to the next one, -or to the OP_KET opcode. +A bracket opcode is followed by LINK_SIZE bytes which give the offset to the +next alternative OP_ALT or, if there aren't any branches, to the matching +OP_KET opcode. Each OP_ALT is followed by LINK_SIZE bytes giving the offset to +the next one, or to the OP_KET opcode. OP_KET is used for subpatterns that do not repeat indefinitely, while OP_KETRMIN and OP_KETRMAX are used for indefinite repetitions, minimally or -maximally respectively. All three are followed by two bytes giving (as a +maximally respectively. All three are followed by LINK_SIZE bytes giving (as a positive number) the offset back to the matching OP_BRA opcode. If a subpattern is quantified such that it is permitted to match zero times, it @@ -312,4 +336,4 @@ at compile time, and so does not cause anything to be put into the compiled data. Philip Hazel -September 2004 +March 2005 diff --git a/doc/html/index.html b/doc/html/index.html index c0dbf59..32664ff 100644 --- a/doc/html/index.html +++ b/doc/html/index.html @@ -24,9 +24,15 @@ The HTML documentation for PCRE comprises the following pages: <tr><td><a href="pcrecompat.html">pcrecompat</a></td> <td> Compability with Perl</td></tr> +<tr><td><a href="pcrecpp.html">pcrecpp</a></td> + <td> The C++ wrapper for the PCRE library</td></tr> + <tr><td><a href="pcregrep.html">pcregrep</a></td> <td> The <b>pcregrep</b> command</td></tr> +<tr><td><a href="pcrematching.html">pcrematching</a></td> + <td> Discussion of the two matching algorithms</td></tr> + <tr><td><a href="pcrepartial.html">pcrepartial</a></td> <td> Using PCRE for partial matching</td></tr> @@ -59,6 +65,9 @@ in the library: <tr><td><a href="pcre_compile.html">pcre_compile</a></td> <td> Compile a regular expression</td></tr> +<tr><td><a href="pcre_compile2.html">pcre_compile2</a></td> + <td> Compile a regular expression (alternate interface)</td></tr> + <tr><td><a href="pcre_config.html">pcre_config</a></td> <td> Show build-time configuration options</td></tr> @@ -68,8 +77,13 @@ in the library: <tr><td><a href="pcre_copy_substring.html">pcre_copy_substring</a></td> <td> Extract numbered substring into given buffer</td></tr> +<tr><td><a href="pcre_dfa_exec.html">pcre_dfa_exec</a></td> + <td> Match a compiled pattern to a subject string + (DFA algorithm; <i>not</i> Perl compatible)</td></tr> + <tr><td><a href="pcre_exec.html">pcre_exec</a></td> - <td> Match a compiled pattern to a subject string</td></tr> + <td> Match a compiled pattern to a subject string + (Perl compatible)</td></tr> <tr><td><a href="pcre_free_substring.html">pcre_free_substring</a></td> <td> Free extracted substring</td></tr> diff --git a/doc/html/pcre.html b/doc/html/pcre.html index b1caf80..c8db6b0 100644 --- a/doc/html/pcre.html +++ b/doc/html/pcre.html @@ -23,16 +23,27 @@ man page, in case the conversion went wrong. <P> The PCRE library is a set of functions that implement regular expression pattern matching using the same syntax and semantics as Perl, with just a few -differences. The current implementation of PCRE (release 5.x) corresponds +differences. The current implementation of PCRE (release 6.x) corresponds approximately with Perl 5.8, including support for UTF-8 encoded strings and Unicode general category properties. However, this support has to be explicitly enabled; it is not the default. </P> <P> +In addition to the Perl-compatible matching function, PCRE also contains an +alternative matching function that matches the same compiled patterns in a +different way. In certain circumstances, the alternative function has some +advantages. For a discussion of the two matching algorithms, see the +<a href="pcrematching.html"><b>pcrematching</b></a> +page. +</P> +<P> PCRE is written in C and released as a C library. A number of people have -written wrappers and interfaces of various kinds. A C++ class is included in -these contributions, which can be found in the <i>Contrib</i> directory at the -primary FTP site, which is: +written wrappers and interfaces of various kinds. In particular, Google Inc. +have provided a comprehensive C++ wrapper. This is now included as part of the +PCRE distribution. The +<a href="pcrecpp.html"><b>pcrecpp</b></a> +page has details of this interface. Other people's contributions can be found +in the <i>Contrib</i> directory at the primary FTP site, which is: <a href="ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre">ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre</a> </P> <P> @@ -53,6 +64,12 @@ available. The features themselves are described in the page. Documentation about building PCRE for various operating systems can be found in the <b>README</b> file in the source distribution. </P> +<P> +The library contains a number of undocumented internal functions and data +tables that are used by more than one of the exported external functions, but +which are not intended for use by external callers. Their names all begin with +"_pcre_", which hopefully will not provoke any name clashes. +</P> <br><a name="SEC2" href="#TOC1">USER DOCUMENTATION</a><br> <P> The user documentation for PCRE comprises a number of different sections. In @@ -62,21 +79,23 @@ all the sections are concatenated, for ease of searching. The sections are as follows: <pre> pcre this document - pcreapi details of PCRE's native API + pcreapi details of PCRE's native C API pcrebuild options for building PCRE pcrecallout details of the callout feature pcrecompat discussion of Perl compatibility + pcrecpp details of the C++ wrapper pcregrep description of the <b>pcregrep</b> command + pcrematching discussion of the two matching algorithms pcrepartial details of the partial matching facility pcrepattern syntax and semantics of supported regular expressions pcreperform discussion of performance issues - pcreposix the POSIX-compatible API + pcreposix the POSIX-compatible C API pcreprecompile details of saving and re-using precompiled patterns pcresample discussion of the sample program pcretest description of the <b>pcretest</b> testing command </pre> In addition, in the "man" and HTML formats, there is a short page for each -library function, listing its arguments and results. +C library function, listing its arguments and results. </P> <br><a name="SEC3" href="#TOC1">LIMITATIONS</a><br> <P> @@ -104,9 +123,10 @@ subpatterns, assertions, and other types of subpattern, is 200. </P> <P> The maximum length of a subject string is the largest positive number that an -integer variable can hold. However, PCRE uses recursion to handle subpatterns -and indefinite repetition. This means that the available stack space may limit -the size of a subject string that can be processed by certain patterns. +integer variable can hold. However, when using the traditional matching +function, PCRE uses recursion to handle subpatterns and indefinite repetition. +This means that the available stack space may limit the size of a subject +string that can be processed by certain patterns. <a name="utf8support"></a></P> <br><a name="SEC4" href="#TOC1">UTF-8 AND UNICODE PROPERTY SUPPORT</a><br> <P> @@ -174,7 +194,8 @@ bytes, for example: \x{100}{3}. </P> <P> 6. The escape sequence \C can be used to match a single byte in UTF-8 mode, -but its use can lead to some strange effects. +but its use can lead to some strange effects. This facility is not available in +the alternative matching function, <b>pcre_dfa_exec()</b>. </P> <P> 7. The character escapes \b, \B, \d, \D, \s, \S, \w, and \W correctly @@ -199,16 +220,19 @@ values. </P> <br><a name="SEC5" href="#TOC1">AUTHOR</a><br> <P> -Philip Hazel <ph10@cam.ac.uk> +Philip Hazel <br> University Computing Service, <br> Cambridge CB2 3QG, England. +</P> +<P> +Putting an actual email address here seems to have been a spam magnet, so I've +taken it away. If you want to email me, use my initial and surname, separated +by a dot, at the domain ucs.cam.ac.uk. +Last updated: 07 March 2005 <br> -Phone: +44 1223 334714 -Last updated: 09 September 2004 -<br> -Copyright © 1997-2004 University of Cambridge. +Copyright © 1997-2005 University of Cambridge. <p> Return to the <a href="index.html">PCRE index page</a>. </p> diff --git a/doc/html/pcre_compile.html b/doc/html/pcre_compile.html index 0d417a1..7ed2c32 100644 --- a/doc/html/pcre_compile.html +++ b/doc/html/pcre_compile.html @@ -48,6 +48,7 @@ The option bits are: PCRE_EXTENDED Ignore whitespace and # comments PCRE_EXTRA PCRE extra features (not much use currently) + PCRE_FIRSTLINE Force matching to be before newline PCRE_MULTILINE ^ and $ match newlines within data PCRE_NO_AUTO_CAPTURE Disable numbered capturing paren- theses (named ones available) diff --git a/doc/html/pcre_compile2.html b/doc/html/pcre_compile2.html new file mode 100644 index 0000000..fc39929 --- /dev/null +++ b/doc/html/pcre_compile2.html @@ -0,0 +1,81 @@ +<html> +<head> +<title>pcre_compile2 specification</title> +</head> +<body bgcolor="#FFFFFF" text="#00005A" link="#0066FF" alink="#3399FF" vlink="#2222BB"> +<h1>pcre_compile2 man page</h1> +<p> +Return to the <a href="index.html">PCRE index page</a>. +</p> +<p> +This page is part of the PCRE HTML documentation. It was generated automatically +from the original man page. If there is any nonsense in it, please consult the +man page, in case the conversion went wrong. +<br> +<br><b> +SYNOPSIS +</b><br> +<P> +<b>#include <pcre.h></b> +</P> +<P> +<b>pcre *pcre_compile2(const char *<i>pattern</i>, int <i>options</i>,</b> +<b>int *<i>errorcodeptr</i>,</b> +<b>const char **<i>errptr</i>, int *<i>erroffset</i>,</b> +<b>const unsigned char *<i>tableptr</i>);</b> +</P> +<br><b> +DESCRIPTION +</b><br> +<P> +This function compiles a regular expression into an internal form. It is the +same as <b>pcre_compile()</b>, except for the addition of the <i>errorcodeptr</i> +argument. The arguments are: +</P> +<P> +<pre> + <i>pattern</i> A zero-terminated string containing the + regular expression to be compiled + <i>options</i> Zero or more option bits + <i>errorcodeptr</i> Where to put an error code + <i>errptr</i> Where to put an error message + <i>erroffset</i> Offset in pattern where error was found + <i>tableptr</i> Pointer to character tables, or NULL to + use the built-in default +</pre> +The option bits are: +<pre> + PCRE_ANCHORED Force pattern anchoring + PCRE_AUTO_CALLOUT Compile automatic callouts + PCRE_CASELESS Do caseless matching + PCRE_DOLLAR_ENDONLY $ not to match newline at end + PCRE_DOTALL . matches anything including NL + PCRE_EXTENDED Ignore whitespace and # comments + PCRE_EXTRA PCRE extra features + (not much use currently) + PCRE_FIRSTLINE Force matching to be before newline + PCRE_MULTILINE ^ and $ match newlines within data + PCRE_NO_AUTO_CAPTURE Disable numbered capturing paren- + theses (named ones available) + PCRE_UNGREEDY Invert greediness of quantifiers + PCRE_UTF8 Run in UTF-8 mode + PCRE_NO_UTF8_CHECK Do not check the pattern for UTF-8 + validity (only relevant if + PCRE_UTF8 is set) +</pre> +PCRE must be built with UTF-8 support in order to use PCRE_UTF8 and +PCRE_NO_UTF8_CHECK. +</P> +<P> +The yield of the function is a pointer to a private data structure that +contains the compiled pattern, or NULL if an error was detected. +</P> +<P> +There is a complete description of the PCRE native API in the +<a href="pcreapi.html"><b>pcreapi</b></a> +page and a description of the POSIX API in the +<a href="pcreposix.html"><b>pcreposix</b></a> +page. +<p> +Return to the <a href="index.html">PCRE index page</a>. +</p> diff --git a/doc/html/pcre_dfa_exec.html b/doc/html/pcre_dfa_exec.html new file mode 100644 index 0000000..ceadb8b --- /dev/null +++ b/doc/html/pcre_dfa_exec.html @@ -0,0 +1,88 @@ +<html> +<head> +<title>pcre_dfa_exec specification</title> +</head> +<body bgcolor="#FFFFFF" text="#00005A" link="#0066FF" alink="#3399FF" vlink="#2222BB"> +<h1>pcre_dfa_exec man page</h1> +<p> +Return to the <a href="index.html">PCRE index page</a>. +</p> +<p> +This page is part of the PCRE HTML documentation. It was generated automatically +from the original man page. If there is any nonsense in it, please consult the +man page, in case the conversion went wrong. +<br> +<br><b> +SYNOPSIS +</b><br> +<P> +<b>#include <pcre.h></b> +</P> +<P> +<b>int pcre_dfa_exec(const pcre *<i>code</i>, const pcre_extra *<i>extra</i>,</b> +<b>const char *<i>subject</i>, int <i>length</i>, int <i>startoffset</i>,</b> +<b>int <i>options</i>, int *<i>ovector</i>, int <i>ovecsize</i>,</b> +<b>int *<i>workspace</i>, int <i>wscount</i>);</b> +</P> +<br><b> +DESCRIPTION +</b><br> +<P> +This function matches a compiled regular expression against a given subject +string, using a DFA matching algorithm (<i>not</i> Perl-compatible). Note that +the main, Perl-compatible, matching function is <b>pcre_exec()</b>. The +arguments for this function are: +<pre> + <i>code</i> Points to the compiled pattern + <i>extra</i> Points to an associated <b>pcre_extra</b> structure, + or is NULL + <i>subject</i> Points to the subject string + <i>length</i> Length of the subject string, in bytes + <i>startoffset</i> Offset in bytes in the subject at which to + start matching + <i>options</i> Option bits + <i>ovector</i> Points to a vector of ints for result offsets + <i>ovecsize</i> Number of elements in the vector + <i>workspace</i> Points to a vector of ints used as working space + <i>wscount</i> Number of elements in the vector +</pre> +The options are: +<pre> + PCRE_ANCHORED Match only at the first position + PCRE_NOTBOL Subject is not the beginning of a line + PCRE_NOTEOL Subject is not the end of a line + PCRE_NOTEMPTY An empty string is not a valid match + PCRE_NO_UTF8_CHECK Do not check the subject for UTF-8 + validity (only relevant if PCRE_UTF8 + was set at compile time) + PCRE_PARTIAL Return PCRE_ERROR_PARTIAL for a partial match + PCRE_DFA_SHORTEST Return only the shortest match + PCRE_DFA_RESTART This is a restart after a partial match +</pre> +There are restrictions on what may appear in a pattern when matching using the +DFA algorithm is requested. Details are given in the +<a href="pcrematching.html"><b>pcrematching</b></a> +documentation. +</P> +<P> +A <b>pcre_extra</b> structure contains the following fields: +<pre> + <i>flags</i> Bits indicating which fields are set + <i>study_data</i> Opaque data from <b>pcre_study()</b> + <i>match_limit</i> Limit on internal recursion + <i>callout_data</i> Opaque data passed back to callouts + <i>tables</i> Points to character tables or is NULL +</pre> +The flag bits are PCRE_EXTRA_STUDY_DATA, PCRE_EXTRA_MATCH_LIMIT, +PCRE_EXTRA_CALLOUT_DATA, and PCRE_EXTRA_TABLES. For DFA matching, the +<i>match_limit</i> field is not used, and must not be set. +</P> +<P> +There is a complete description of the PCRE native API in the +<a href="pcreapi.html"><b>pcreapi</b></a> +page and a description of the POSIX API in the +<a href="pcreposix.html"><b>pcreposix</b></a> +page. +<p> +Return to the <a href="index.html">PCRE index page</a>. +</p> diff --git a/doc/html/pcre_exec.html b/doc/html/pcre_exec.html index fc3d322..5fae92f 100644 --- a/doc/html/pcre_exec.html +++ b/doc/html/pcre_exec.html @@ -28,7 +28,8 @@ DESCRIPTION </b><br> <P> This function matches a compiled regular expression against a given subject -string, and returns offsets to capturing subexpressions. Its arguments are: +string, using a matching algorithm that is similar to Perl's. It returns +offsets to captured substrings. Its arguments are: <pre> <i>code</i> Points to the compiled pattern <i>extra</i> Points to an associated <b>pcre_extra</b> structure, diff --git a/doc/html/pcre_refcount.html b/doc/html/pcre_refcount.html new file mode 100644 index 0000000..ba53112 --- /dev/null +++ b/doc/html/pcre_refcount.html @@ -0,0 +1,45 @@ +<html> +<head> +<title>pcre_refcount specification</title> +</head> +<body bgcolor="#FFFFFF" text="#00005A" link="#0066FF" alink="#3399FF" vlink="#2222BB"> +<h1>pcre_refcount man page</h1> +<p> +Return to the <a href="index.html">PCRE index page</a>. +</p> +<p> +This page is part of the PCRE HTML documentation. It was generated automatically +from the original man page. If there is any nonsense in it, please consult the +man page, in case the conversion went wrong. +<br> +<br><b> +SYNOPSIS +</b><br> +<P> +<b>#include <pcre.h></b> +</P> +<P> +<b>int pcre_info(pcre *<i>code</i>, int <i>adjust</i>);</b> +</P> +<br><b> +DESCRIPTION +</b><br> +<P> +This function is used to maintain a reference count inside a data block that +contains a compiled pattern. Its arguments are: +<pre> + <i>code</i> Compiled regular expression + <i>adjust</i> Adjustment to reference value +</pre> +The yield of the function is the adjusted reference value, which is constrained +to lie between 0 and 65535. +</P> +<P> +There is a complete description of the PCRE native API in the +<a href="pcreapi.html"><b>pcreapi</b></a> +page and a description of the POSIX API in the +<a href="pcreposix.html"><b>pcreposix</b></a> +page. +<p> +Return to the <a href="index.html">PCRE index page</a>. +</p> diff --git a/doc/html/pcreapi.html b/doc/html/pcreapi.html index 72639f4..4d5f866 100644 --- a/doc/html/pcreapi.html +++ b/doc/html/pcreapi.html @@ -19,13 +19,17 @@ man page, in case the conversion went wrong. <li><a name="TOC4" href="#SEC4">SAVING PRECOMPILED PATTERNS FOR LATER USE</a> <li><a name="TOC5" href="#SEC5">CHECKING BUILD-TIME OPTIONS</a> <li><a name="TOC6" href="#SEC6">COMPILING A PATTERN</a> -<li><a name="TOC7" href="#SEC7">STUDYING A PATTERN</a> -<li><a name="TOC8" href="#SEC8">LOCALE SUPPORT</a> -<li><a name="TOC9" href="#SEC9">INFORMATION ABOUT A PATTERN</a> -<li><a name="TOC10" href="#SEC10">OBSOLETE INFO FUNCTION</a> -<li><a name="TOC11" href="#SEC11">MATCHING A PATTERN</a> -<li><a name="TOC12" href="#SEC12">EXTRACTING CAPTURED SUBSTRINGS BY NUMBER</a> -<li><a name="TOC13" href="#SEC13">EXTRACTING CAPTURED SUBSTRINGS BY NAME</a> +<li><a name="TOC7" href="#SEC7">COMPILATION ERROR CODES</a> +<li><a name="TOC8" href="#SEC8">STUDYING A PATTERN</a> +<li><a name="TOC9" href="#SEC9">LOCALE SUPPORT</a> +<li><a name="TOC10" href="#SEC10">INFORMATION ABOUT A PATTERN</a> +<li><a name="TOC11" href="#SEC11">OBSOLETE INFO FUNCTION</a> +<li><a name="TOC12" href="#SEC12">REFERENCE COUNTS</a> +<li><a name="TOC13" href="#SEC13">MATCHING A PATTERN: THE TRADITIONAL FUNCTION</a> +<li><a name="TOC14" href="#SEC14">EXTRACTING CAPTURED SUBSTRINGS BY NUMBER</a> +<li><a name="TOC15" href="#SEC15">EXTRACTING CAPTURED SUBSTRINGS BY NAME</a> +<li><a name="TOC16" href="#SEC16">FINDING ALL POSSIBLE MATCHES</a> +<li><a name="TOC17" href="#SEC17">MATCHING A PATTERN: THE ALTERNATIVE FUNCTION</a> </ul> <br><a name="SEC1" href="#TOC1">PCRE NATIVE API</a><br> <P> @@ -37,6 +41,12 @@ man page, in case the conversion went wrong. <b>const unsigned char *<i>tableptr</i>);</b> </P> <P> +<b>pcre *pcre_compile2(const char *<i>pattern</i>, int <i>options</i>,</b> +<b>int *<i>errorcodeptr</i>,</b> +<b>const char **<i>errptr</i>, int *<i>erroffset</i>,</b> +<b>const unsigned char *<i>tableptr</i>);</b> +</P> +<P> <b>pcre_extra *pcre_study(const pcre *<i>code</i>, int <i>options</i>,</b> <b>const char **<i>errptr</i>);</b> </P> @@ -46,6 +56,12 @@ man page, in case the conversion went wrong. <b>int <i>options</i>, int *<i>ovector</i>, int <i>ovecsize</i>);</b> </P> <P> +<b>int pcre_dfa_exec(const pcre *<i>code</i>, const pcre_extra *<i>extra</i>,</b> +<b>const char *<i>subject</i>, int <i>length</i>, int <i>startoffset</i>,</b> +<b>int <i>options</i>, int *<i>ovector</i>, int <i>ovecsize</i>,</b> +<b>int *<i>workspace</i>, int <i>wscount</i>);</b> +</P> +<P> <b>int pcre_copy_named_substring(const pcre *<i>code</i>,</b> <b>const char *<i>subject</i>, int *<i>ovector</i>,</b> <b>int <i>stringcount</i>, const char *<i>stringname</i>,</b> @@ -93,6 +109,9 @@ man page, in case the conversion went wrong. <b>*<i>firstcharptr</i>);</b> </P> <P> +<b>int pcre_refcount(pcre *<i>code</i>, int <i>adjust</i>);</b> +</P> +<P> <b>int pcre_config(int <i>what</i>, void *<i>where</i>);</b> </P> <P> @@ -115,32 +134,46 @@ man page, in case the conversion went wrong. </P> <br><a name="SEC2" href="#TOC1">PCRE API OVERVIEW</a><br> <P> -PCRE has its own native API, which is described in this document. There is also -a set of wrapper functions that correspond to the POSIX regular expression API. -These are described in the +PCRE has its own native API, which is described in this document. There is +also a set of wrapper functions that correspond to the POSIX regular expression +API. These are described in the <a href="pcreposix.html"><b>pcreposix</b></a> -documentation. +documentation. Both of these APIs define a set of C function calls. A C++ +wrapper is distributed with PCRE. It is documented in the +<a href="pcrecpp.html"><b>pcrecpp</b></a> +page. </P> <P> -The native API function prototypes are defined in the header file <b>pcre.h</b>, -and on Unix systems the library itself is called <b>libpcre</b>. It can -normally be accessed by adding <b>-lpcre</b> to the command for linking an -application that uses PCRE. The header file defines the macros PCRE_MAJOR and -PCRE_MINOR to contain the major and minor release numbers for the library. +The native API C function prototypes are defined in the header file +<b>pcre.h</b>, and on Unix systems the library itself is called <b>libpcre</b>. +It can normally be accessed by adding <b>-lpcre</b> to the command for linking +an application that uses PCRE. The header file defines the macros PCRE_MAJOR +and PCRE_MINOR to contain the major and minor release numbers for the library. Applications can use these to include support for different releases of PCRE. </P> <P> -The functions <b>pcre_compile()</b>, <b>pcre_study()</b>, and <b>pcre_exec()</b> -are used for compiling and matching regular expressions. A sample program that -demonstrates the simplest way of using them is provided in the file called -<i>pcredemo.c</i> in the source distribution. The +The functions <b>pcre_compile()</b>, <b>pcre_compile2()</b>, <b>pcre_study()</b>, +and <b>pcre_exec()</b> are used for compiling and matching regular expressions +in a Perl-compatible manner. A sample program that demonstrates the simplest +way of using them is provided in the file called <i>pcredemo.c</i> in the source +distribution. The <a href="pcresample.html"><b>pcresample</b></a> documentation describes how to run it. </P> <P> +A second matching function, <b>pcre_dfa_exec()</b>, which is not +Perl-compatible, is also provided. This uses a different algorithm for the +matching. This allows it to find all possible matches (at a given point in the +subject), not just one. However, this algorithm does not return captured +substrings. A description of the two matching algorithms and their advantages +and disadvantages is given in the +<a href="pcrematching.html"><b>pcrematching</b></a> +documentation. +</P> +<P> In addition to the main compiling and matching functions, there are convenience -functions for extracting captured substrings from a matched subject string. -They are: +functions for extracting captured substrings from a subject string that is +matched by <b>pcre_exec()</b>. They are: <pre> <b>pcre_copy_substring()</b> <b>pcre_copy_named_substring()</b> @@ -154,10 +187,10 @@ provided, to free the memory used for extracted strings. </P> <P> The function <b>pcre_maketables()</b> is used to build a set of character tables -in the current locale for passing to <b>pcre_compile()</b> or <b>pcre_exec()</b>. -This is an optional facility that is provided for specialist use. Most -commonly, no special tables are passed, in which case internal tables that are -generated when PCRE is built are used. +in the current locale for passing to <b>pcre_compile()</b>, <b>pcre_exec()</b>, +or <b>pcre_dfa_exec()</b>. This is an optional facility that is provided for +specialist use. Most commonly, no special tables are passed, in which case +internal tables that are generated when PCRE is built are used. </P> <P> The function <b>pcre_fullinfo()</b> is used to find out information about a @@ -167,6 +200,11 @@ The function <b>pcre_version()</b> returns a pointer to a string containing the version of PCRE and its date of release. </P> <P> +The function <b>pcre_refcount()</b> maintains a reference count in a data block +containing a compiled pattern. This is provided for the benefit of +object-oriented applications. +</P> +<P> The global variables <b>pcre_malloc</b> and <b>pcre_free</b> initially contain the entry points of the standard <b>malloc()</b> and <b>free()</b> functions, respectively. PCRE calls the memory management functions via these variables, @@ -177,12 +215,13 @@ should be done before calling any PCRE functions. The global variables <b>pcre_stack_malloc</b> and <b>pcre_stack_free</b> are also indirections to memory management functions. These special functions are used only when PCRE is compiled to use the heap for remembering data, instead of -recursive function calls. This is a non-standard way of building PCRE, for use -in environments that have limited stacks. Because of the greater use of memory -management, it runs more slowly. Separate functions are provided so that -special-purpose external code can be used for this case. When used, these -functions are always called in a stack-like manner (last obtained, first -freed), and always for memory blocks of the same size. +recursive function calls, when running the <b>pcre_exec()</b> function. This is +a non-standard way of building PCRE, for use in environments that have limited +stacks. Because of the greater use of memory management, it runs more slowly. +Separate functions are provided so that special-purpose external code can be +used for this case. When used, these functions are always called in a +stack-like manner (last obtained, first freed), and always for memory blocks of +the same size. </P> <P> The global variable <b>pcre_callout</b> initially contains NULL. It can be set @@ -265,27 +304,37 @@ details are given with <b>pcre_exec()</b> below. <pre> PCRE_CONFIG_STACKRECURSE </pre> -The output is an integer that is set to one if internal recursion is -implemented by recursive function calls that use the stack to remember their -state. This is the usual way that PCRE is compiled. The output is zero if PCRE -was compiled to use blocks of data on the heap instead of recursive function -calls. In this case, <b>pcre_stack_malloc</b> and <b>pcre_stack_free</b> are -called to manage memory blocks on the heap, thus avoiding the use of the stack. +The output is an integer that is set to one if internal recursion when running +<b>pcre_exec()</b> is implemented by recursive function calls that use the stack +to remember their state. This is the usual way that PCRE is compiled. The +output is zero if PCRE was compiled to use blocks of data on the heap instead +of recursive function calls. In this case, <b>pcre_stack_malloc</b> and +<b>pcre_stack_free</b> are called to manage memory blocks on the heap, thus +avoiding the use of the stack. </P> <br><a name="SEC6" href="#TOC1">COMPILING A PATTERN</a><br> <P> <b>pcre *pcre_compile(const char *<i>pattern</i>, int <i>options</i>,</b> <b>const char **<i>errptr</i>, int *<i>erroffset</i>,</b> <b>const unsigned char *<i>tableptr</i>);</b> +<b>pcre *pcre_compile2(const char *<i>pattern</i>, int <i>options</i>,</b> +<b>int *<i>errorcodeptr</i>,</b> +<b>const char **<i>errptr</i>, int *<i>erroffset</i>,</b> +<b>const unsigned char *<i>tableptr</i>);</b> +</P> +<P> +Either of the functions <b>pcre_compile()</b> or <b>pcre_compile2()</b> can be +called to compile a pattern into an internal form. The only difference between +the two interfaces is that <b>pcre_compile2()</b> has an additional argument, +<i>errorcodeptr</i>, via which a numerical error code can be returned. </P> <P> -The function <b>pcre_compile()</b> is called to compile a pattern into an -internal form. The pattern is a C string terminated by a binary zero, and -is passed in the <i>pattern</i> argument. A pointer to a single block of memory -that is obtained via <b>pcre_malloc</b> is returned. This contains the compiled -code and related data. The <b>pcre</b> type is defined for the returned block; -this is a typedef for a structure whose contents are not externally defined. It -is up to the caller to free the memory when it is no longer required. +The pattern is a C string terminated by a binary zero, and is passed in the +<i>pattern</i> argument. A pointer to a single block of memory that is obtained +via <b>pcre_malloc</b> is returned. This contains the compiled code and related +data. The <b>pcre</b> type is defined for the returned block; this is a typedef +for a structure whose contents are not externally defined. It is up to the +caller to free the memory when it is no longer required. </P> <P> Although the compiled code of a PCRE regex is relocatable, that is, it does not @@ -314,6 +363,12 @@ the error was discovered is placed in the variable pointed to by <i>erroffset</i>, which must not be NULL. If it is, an immediate error is given. </P> <P> +If <b>pcre_compile2()</b> is used instead of <b>pcre_compile()</b>, and the +<i>errorcodeptr</i> argument is not NULL, a non-zero error code number is +returned via this argument in the event of an error. This is in addition to the +textual error message. Error codes and messages are listed below. +</P> +<P> If the final argument, <i>tableptr</i>, is NULL, PCRE uses a default set of character tables that are built when PCRE is compiled, using the default C locale. Otherwise, <i>tableptr</i> must be an address that is the result of a @@ -357,9 +412,13 @@ documentation. </pre> If this bit is set, letters in the pattern match both upper and lower case letters. It is equivalent to Perl's /i option, and it can be changed within a -pattern by a (?i) option setting. When running in UTF-8 mode, case support for -high-valued characters is available only when PCRE is built with Unicode -character property support. +pattern by a (?i) option setting. In UTF-8 mode, PCRE always understands the +concept of case for characters whose values are less than 128, so caseless +matching is always possible. For characters with higher values, the concept of +case is supported if PCRE is compiled with Unicode property support, but not +otherwise. If you want to use caseless matching for characters 128 and above, +you must ensure that PCRE is compiled with Unicode property support as well as +with UTF-8 support. <pre> PCRE_DOLLAR_ENDONLY </pre> @@ -404,6 +463,12 @@ special meaning is treated as a literal. There are at present no other features controlled by this option. It can also be set by a (?X) option setting within a pattern. <pre> + PCRE_FIRSTLINE +</pre> +If this option is set, an unanchored pattern is required to match before or at +the first newline character in the subject string, though the matched text may +continue over the newline. +<pre> PCRE_MULTILINE </pre> By default, PCRE treats the subject string as consisting of a single line of @@ -455,12 +520,69 @@ automatically checked. If an invalid UTF-8 sequence of bytes is found, valid, and you want to skip this check for performance reasons, you can set the PCRE_NO_UTF8_CHECK option. When it is set, the effect of passing an invalid UTF-8 string as a pattern is undefined. It may cause your program to crash. -Note that this option can also be passed to <b>pcre_exec()</b>, to suppress the -UTF-8 validity checking of subject strings. -</P> -<br><a name="SEC7" href="#TOC1">STUDYING A PATTERN</a><br> -<P> -<b>pcre_extra *pcre_study(const pcre *<i>code</i>, int <i>options</i>,</b> +Note that this option can also be passed to <b>pcre_exec()</b> and +<b>pcre_dfa_exec()</b>, to suppress the UTF-8 validity checking of subject +strings. +</P> +<br><a name="SEC7" href="#TOC1">COMPILATION ERROR CODES</a><br> +<P> +The following table lists the error codes than may be returned by +<b>pcre_compile2()</b>, along with the error messages that may be returned by +both compiling functions. +<pre> + 0 no error + 1 \ at end of pattern + 2 \c at end of pattern + 3 unrecognized character follows \ + 4 numbers out of order in {} quantifier + 5 number too big in {} quantifier + 6 missing terminating ] for character class + 7 invalid escape sequence in character class + 8 range out of order in character class + 9 nothing to repeat + 10 operand of unlimited repeat could match the empty string + 11 internal error: unexpected repeat + 12 unrecognized character after (? + 13 POSIX named classes are supported only within a class + 14 missing ) + 15 reference to non-existent subpattern + 16 erroffset passed as NULL + 17 unknown option bit(s) set + 18 missing ) after comment + 19 parentheses nested too deeply + 20 regular expression too large + 21 failed to get memory + 22 unmatched parentheses + 23 internal error: code overflow + 24 unrecognized character after (?< + 25 lookbehind assertion is not fixed length + 26 malformed number after (?( + 27 conditional group contains more than two branches + 28 assertion expected after (?( + 29 (?R or (?digits must be followed by ) + 30 unknown POSIX class name + 31 POSIX collating elements are not supported + 32 this version of PCRE is not compiled with PCRE_UTF8 support + 33 spare error + 34 character value in \x{...} sequence is too large + 35 invalid condition (?(0) + 36 \C not allowed in lookbehind assertion + 37 PCRE does not support \L, \l, \N, \U, or \u + 38 number after (?C is > 255 + 39 closing ) for (?C expected + 40 recursive call could loop indefinitely + 41 unrecognized character after (?P + 42 syntax error after (?P + 43 two named groups have the same name + 44 invalid UTF-8 string + 45 support for \P, \p, and \X has not been compiled + 46 malformed \P or \p sequence + 47 unknown property name after \P or \p +</PRE> +</P> +<br><a name="SEC8" href="#TOC1">STUDYING A PATTERN</a><br> +<P> +<b>pcre_extra *pcre_study(const pcre *<i>code</i>, int <i>options</i></b> <b>const char **<i>errptr</i>);</b> </P> <P> @@ -481,7 +603,7 @@ described in the section on matching a pattern. </P> <P> -If studying the pattern does not produce any additional information, +If studying the pattern does not produce any additional information <b>pcre_study()</b> returns NULL. In that circumstance, if the calling program wants to pass any of the other fields to <b>pcre_exec()</b>, it must set up its own <b>pcre_extra</b> block. @@ -510,14 +632,14 @@ At present, studying a pattern is useful only for non-anchored patterns that do not have a single fixed starting character. A bitmap of possible starting bytes is created. <a name="localesupport"></a></P> -<br><a name="SEC8" href="#TOC1">LOCALE SUPPORT</a><br> +<br><a name="SEC9" href="#TOC1">LOCALE SUPPORT</a><br> <P> -PCRE handles caseless matching, and determines whether characters are letters, +PCRE handles caseless matching, and determines whether characters are letters digits, or whatever, by reference to a set of tables, indexed by character -value. (When running in UTF-8 mode, this applies only to characters with codes +value. When running in UTF-8 mode, this applies only to characters with codes less than 128. Higher-valued codes never match escapes such as \w or \d, but can be tested with \p if PCRE is built with Unicode character property -support.) +support. </P> <P> An internal set of tables is created in the default C locale when PCRE is @@ -558,7 +680,7 @@ this facility could be used to match a pattern in a different locale from the one in which it was compiled. Passing table pointers at run time is discussed below in the section on matching a pattern. </P> -<br><a name="SEC9" href="#TOC1">INFORMATION ABOUT A PATTERN</a><br> +<br><a name="SEC10" href="#TOC1">INFORMATION ABOUT A PATTERN</a><br> <P> <b>int pcre_fullinfo(const pcre *<i>code</i>, const pcre_extra *<i>extra</i>,</b> <b>int <i>what</i>, void *<i>where</i>);</b> @@ -607,7 +729,7 @@ no back references. Return the number of capturing subpatterns in the pattern. The fourth argument should point to an <b>int</b> variable. <pre> - PCRE_INFO_DEFAULTTABLES + PCRE_INFO_DEFAULT_TABLES </pre> Return a pointer to the internal default character tables within PCRE. The fourth argument should point to an <b>unsigned char *</b> variable. This @@ -729,7 +851,7 @@ a <b>pcre_extra</b> block. That is, it is the value that was passed to created by <b>pcre_study()</b>. The fourth argument should point to a <b>size_t</b> variable. </P> -<br><a name="SEC10" href="#TOC1">OBSOLETE INFO FUNCTION</a><br> +<br><a name="SEC11" href="#TOC1">OBSOLETE INFO FUNCTION</a><br> <P> <b>int pcre_info(const pcre *<i>code</i>, int *<i>optptr</i>, int</b> <b>*<i>firstcharptr</i>);</b> @@ -753,7 +875,31 @@ If the pattern is not anchored and the <i>firstcharptr</i> argument is not NULL, it is used to pass back information about the first character of any matched string (see PCRE_INFO_FIRSTBYTE above). </P> -<br><a name="SEC11" href="#TOC1">MATCHING A PATTERN</a><br> +<br><a name="SEC12" href="#TOC1">REFERENCE COUNTS</a><br> +<P> +<b>int pcre_refcount(pcre *<i>code</i>, int <i>adjust</i>);</b> +</P> +<P> +The <b>pcre_refcount()</b> function is used to maintain a reference count in the +data block that contains a compiled pattern. It is provided for the benefit of +applications that operate in an object-oriented manner, where different parts +of the application may be using the same compiled pattern, but you want to free +the block when they are all done. +</P> +<P> +When a pattern is compiled, the reference count field is initialized to zero. +It is changed only by calling this function, whose action is to add the +<i>adjust</i> value (which may be positive or negative) to it. The yield of the +function is the new value. However, the value of the count is constrained to +lie between 0 and 65535, inclusive. If the new value is outside these limits, +it is forced to the appropriate limit value. +</P> +<P> +Except when it is zero, the reference count is not correctly preserved if a +pattern is compiled on one host and then transferred to a host whose byte-order +is different. (This seems a highly unlikely scenario.) +</P> +<br><a name="SEC13" href="#TOC1">MATCHING A PATTERN: THE TRADITIONAL FUNCTION</a><br> <P> <b>int pcre_exec(const pcre *<i>code</i>, const pcre_extra *<i>extra</i>,</b> <b>const char *<i>subject</i>, int <i>length</i>, int <i>startoffset</i>,</b> @@ -763,7 +909,11 @@ string (see PCRE_INFO_FIRSTBYTE above). The function <b>pcre_exec()</b> is called to match a subject string against a compiled pattern, which is passed in the <i>code</i> argument. If the pattern has been studied, the result of the study should be passed in the -<i>extra</i> argument. +<i>extra</i> argument. This function is the main matching facility of the +library, and it operates in a Perl-like manner. For specialist use there is +also an alternative matching function, which is described +<a href="#dfamatch">below</a> +in the section about the <b>pcre_dfa_exec()</b> function. </P> <P> In most applications, the pattern will have been compiled (and optionally @@ -787,7 +937,7 @@ Here is an example of a simple call to <b>pcre_exec()</b>: 0, /* start at offset 0 in the subject */ 0, /* default options */ ovector, /* vector of integers for substring information */ - 30); /* number of elements in the vector (NOT size in bytes) */ + 30); /* number of elements (NOT size in bytes) */ <a name="extradata"></a></PRE> </P> <br><b> @@ -1046,7 +1196,7 @@ Note that <b>pcre_info()</b> can be used to find out how many capturing subpatterns there are in a compiled pattern. The smallest size for <i>ovector</i> that will allow for <i>n</i> captured substrings, in addition to the offsets of the substring matched by the whole pattern, is (<i>n</i>+1)*3. -</P> +<a name="errorlist"></a></P> <br><b> Return values from <b>pcre_exec()</b> </b><br> @@ -1117,29 +1267,29 @@ A string that contains an invalid UTF-8 byte sequence was passed as a subject. The UTF-8 byte sequence that was passed as a subject was valid, but the value of <i>startoffset</i> did not point to the beginning of a UTF-8 character. <pre> - PCRE_ERROR_PARTIAL (-12) + PCRE_ERROR_PARTIAL (-12) </pre> The subject string did not match, but it did match partially. See the <a href="pcrepartial.html"><b>pcrepartial</b></a> documentation for details of partial matching. <pre> - PCRE_ERROR_BAD_PARTIAL (-13) + PCRE_ERROR_BADPARTIAL (-13) </pre> The PCRE_PARTIAL option was used with a compiled pattern containing items that are not supported for partial matching. See the <a href="pcrepartial.html"><b>pcrepartial</b></a> documentation for details of partial matching. <pre> - PCRE_ERROR_INTERNAL (-14) + PCRE_ERROR_INTERNAL (-14) </pre> An unexpected internal error has occurred. This error could be caused by a bug in PCRE or by overwriting of the compiled pattern. <pre> - PCRE_ERROR_BADCOUNT (-15) + PCRE_ERROR_BADCOUNT (-15) </pre> This error is given if the value of the <i>ovecsize</i> argument is negative. </P> -<br><a name="SEC12" href="#TOC1">EXTRACTING CAPTURED SUBSTRINGS BY NUMBER</a><br> +<br><a name="SEC14" href="#TOC1">EXTRACTING CAPTURED SUBSTRINGS BY NUMBER</a><br> <P> <b>int pcre_copy_substring(const char *<i>subject</i>, int *<i>ovector</i>,</b> <b>int <i>stringcount</i>, int <i>stringnumber</i>, char *<i>buffer</i>,</b> @@ -1227,7 +1377,7 @@ linked via a special interface to another programming language which cannot use <b>pcre_free</b> directly; it is for these cases that the functions are provided. </P> -<br><a name="SEC13" href="#TOC1">EXTRACTING CAPTURED SUBSTRINGS BY NAME</a><br> +<br><a name="SEC15" href="#TOC1">EXTRACTING CAPTURED SUBSTRINGS BY NAME</a><br> <P> <b>int pcre_get_stringnumber(const pcre *<i>code</i>,</b> <b>const char *<i>name</i>);</b> @@ -1278,10 +1428,191 @@ These functions call <b>pcre_get_stringnumber()</b>, and if it succeeds, they then call <i>pcre_copy_substring()</i> or <i>pcre_get_substring()</i>, as appropriate. </P> +<br><a name="SEC16" href="#TOC1">FINDING ALL POSSIBLE MATCHES</a><br> +<P> +The traditional matching function uses a similar algorithm to Perl, which stops +when it finds the first match, starting at a given point in the subject. If you +want to find all possible matches, or the longest possible match, consider +using the alternative matching function (see below) instead. If you cannot use +the alternative function, but still need to find all possible matches, you +can kludge it up by making use of the callout facility, which is described in +the +<a href="pcrecallout.html"><b>pcrecallout</b></a> +documentation. +</P> +<P> +What you have to do is to insert a callout right at the end of the pattern. +When your callout function is called, extract and save the current matched +substring. Then return 1, which forces <b>pcre_exec()</b> to backtrack and try +other alternatives. Ultimately, when it runs out of matches, <b>pcre_exec()</b> +will yield PCRE_ERROR_NOMATCH. +<a name="dfamatch"></a></P> +<br><a name="SEC17" href="#TOC1">MATCHING A PATTERN: THE ALTERNATIVE FUNCTION</a><br> +<P> +<b>int pcre_dfa_exec(const pcre *<i>code</i>, const pcre_extra *<i>extra</i>,</b> +<b>const char *<i>subject</i>, int <i>length</i>, int <i>startoffset</i>,</b> +<b>int <i>options</i>, int *<i>ovector</i>, int <i>ovecsize</i>,</b> +<b>int *<i>workspace</i>, int <i>wscount</i>);</b> +</P> +<P> +The function <b>pcre_dfa_exec()</b> is called to match a subject string against +a compiled pattern, using a "DFA" matching algorithm. This has different +characteristics to the normal algorithm, and is not compatible with Perl. Some +of the features of PCRE patterns are not supported. Nevertheless, there are +times when this kind of matching can be useful. For a discussion of the two +matching algorithms, see the +<a href="pcrematching.html"><b>pcrematching</b></a> +documentation. +</P> +<P> +The arguments for the <b>pcre_dfa_exec()</b> function are the same as for +<b>pcre_exec()</b>, plus two extras. The <i>ovector</i> argument is used in a +different way, and this is described below. The other common arguments are used +in the same way as for <b>pcre_exec()</b>, so their description is not repeated +here. +</P> +<P> +The two additional arguments provide workspace for the function. The workspace +vector should contain at least 20 elements. It is used for keeping track of +multiple paths through the pattern tree. More workspace will be needed for +patterns and subjects where there are a lot of possible matches. +</P> +<P> +Here is an example of a simple call to <b>pcre_exec()</b>: +<pre> + int rc; + int ovector[10]; + int wspace[20]; + rc = pcre_exec( + re, /* result of pcre_compile() */ + NULL, /* we didn't study the pattern */ + "some string", /* the subject string */ + 11, /* the length of the subject string */ + 0, /* start at offset 0 in the subject */ + 0, /* default options */ + ovector, /* vector of integers for substring information */ + 10, /* number of elements (NOT size in bytes) */ + wspace, /* working space vector */ + 20); /* number of elements (NOT size in bytes) */ +</PRE> +</P> +<br><b> +Option bits for <b>pcre_dfa_exec()</b> +</b><br> +<P> +The unused bits of the <i>options</i> argument for <b>pcre_dfa_exec()</b> must be +zero. The only bits that may be set are PCRE_ANCHORED, PCRE_NOTBOL, +PCRE_NOTEOL, PCRE_NOTEMPTY, PCRE_NO_UTF8_CHECK, PCRE_PARTIAL, +PCRE_DFA_SHORTEST, and PCRE_DFA_RESTART. All but the last three of these are +the same as for <b>pcre_exec()</b>, so their description is not repeated here. +<pre> + PCRE_PARTIAL +</pre> +This has the same general effect as it does for <b>pcre_exec()</b>, but the +details are slightly different. When PCRE_PARTIAL is set for +<b>pcre_dfa_exec()</b>, the return code PCRE_ERROR_NOMATCH is converted into +PCRE_ERROR_PARTIAL if the end of the subject is reached, there have been no +complete matches, but there is still at least one matching possibility. The +portion of the string that provided the partial match is set as the first +matching string. +<pre> + PCRE_DFA_SHORTEST +</pre> +Setting the PCRE_DFA_SHORTEST option causes the matching algorithm to stop as +soon as it has found one match. Because of the way the DFA algorithm works, +this is necessarily the shortest possible match at the first possible matching +point in the subject string. +<pre> + PCRE_DFA_RESTART +</pre> +When <b>pcre_dfa_exec()</b> is called with the PCRE_PARTIAL option, and returns +a partial match, it is possible to call it again, with additional subject +characters, and have it continue with the same match. The PCRE_DFA_RESTART +option requests this action; when it is set, the <i>workspace</i> and +<i>wscount</i> options must reference the same vector as before because data +about the match so far is left in them after a partial match. There is more +discussion of this facility in the +<a href="pcrepartial.html"><b>pcrepartial</b></a> +documentation. +</P> +<br><b> +Successful returns from <b>pcre_dfa_exec()</b> +</b><br> +<P> +When <b>pcre_dfa_exec()</b> succeeds, it may have matched more than one +substring in the subject. Note, however, that all the matches from one run of +the function start at the same point in the subject. The shorter matches are +all initial substrings of the longer matches. For example, if the pattern +<pre> + <.*> +</pre> +is matched against the string +<pre> + This is <something> <something else> <something further> no more +</pre> +the three matched strings are +<pre> + <something> + <something> <something else> + <something> <something else> <something further> +</pre> +On success, the yield of the function is a number greater than zero, which is +the number of matched substrings. The substrings themselves are returned in +<i>ovector</i>. Each string uses two elements; the first is the offset to the +start, and the second is the offset to the end. All the strings have the same +start offset. (Space could have been saved by giving this only once, but it was +decided to retain some compatibility with the way <b>pcre_exec()</b> returns +data, even though the meaning of the strings is different.) +</P> +<P> +The strings are returned in reverse order of length; that is, the longest +matching string is given first. If there were too many matches to fit into +<i>ovector</i>, the yield of the function is zero, and the vector is filled with +the longest matches. +</P> +<br><b> +Error returns from <b>pcre_dfa_exec()</b> +</b><br> +<P> +The <b>pcre_dfa_exec()</b> function returns a negative number when it fails. +Many of the errors are the same as for <b>pcre_exec()</b>, and these are +described +<a href="#errorlist">above.</a> +There are in addition the following errors that are specific to +<b>pcre_dfa_exec()</b>: +<pre> + PCRE_ERROR_DFA_UITEM (-16) +</pre> +This return is given if <b>pcre_dfa_exec()</b> encounters an item in the pattern +that it does not support, for instance, the use of \C or a back reference. +<pre> + PCRE_ERROR_DFA_UCOND (-17) +</pre> +This return is given if <b>pcre_dfa_exec()</b> encounters a condition item in a +pattern that uses a back reference for the condition. This is not supported. +<pre> + PCRE_ERROR_DFA_UMLIMIT (-18) +</pre> +This return is given if <b>pcre_dfa_exec()</b> is called with an <i>extra</i> +block that contains a setting of the <i>match_limit</i> field. This is not +supported (it is meaningless). +<pre> + PCRE_ERROR_DFA_WSSIZE (-19) +</pre> +This return is given if <b>pcre_dfa_exec()</b> runs out of space in the +<i>workspace</i> vector. +<pre> + PCRE_ERROR_DFA_RECURSE (-20) +</pre> +When a recursive subpattern is processed, the matching function calls itself +recursively, using private vectors for <i>ovector</i> and <i>workspace</i>. This +error is given if the output vector is not large enough. This should be +extremely rare, as a vector of size 1000 is used. +</P> <P> -Last updated: 09 September 2004 +Last updated: 16 May 2005 <br> -Copyright © 1997-2004 University of Cambridge. +Copyright © 1997-2005 University of Cambridge. <p> Return to the <a href="index.html">PCRE index page</a>. </p> diff --git a/doc/html/pcrebuild.html b/doc/html/pcrebuild.html index 98c7d27..4f9ff8b 100644 --- a/doc/html/pcrebuild.html +++ b/doc/html/pcrebuild.html @@ -113,17 +113,19 @@ to the <b>configure</b> command. <br><a name="SEC7" href="#TOC1">LIMITING PCRE RESOURCE USAGE</a><br> <P> Internally, PCRE has a function called <b>match()</b>, which it calls repeatedly -(possibly recursively) when matching a pattern. By controlling the maximum -number of times this function may be called during a single matching operation, -a limit can be placed on the resources used by a single call to -<b>pcre_exec()</b>. The limit can be changed at run time, as described in the +(possibly recursively) when matching a pattern with the <b>pcre_exec()</b> +function. By controlling the maximum number of times this function may be +called during a single matching operation, a limit can be placed on the +resources used by a single call to <b>pcre_exec()</b>. The limit can be changed +at run time, as described in the <a href="pcreapi.html"><b>pcreapi</b></a> documentation. The default is 10 million, but this can be changed by adding a setting such as <pre> --with-match-limit=500000 </pre> -to the <b>configure</b> command. +to the <b>configure</b> command. This setting has no effect on the +<b>pcre_dfa_exec()</b> matching function. </P> <br><a name="SEC8" href="#TOC1">HANDLING VERY LARGE PATTERNS</a><br> <P> @@ -148,13 +150,14 @@ of the compiled pattern, and this changes with the link size. </P> <br><a name="SEC9" href="#TOC1">AVOIDING EXCESSIVE STACK USAGE</a><br> <P> -PCRE implements backtracking while matching by making recursive calls to an -internal function called <b>match()</b>. In environments where the size of the -stack is limited, this can severely limit PCRE's operation. (The Unix -environment does not usually suffer from this problem.) An alternative approach -that uses memory from the heap to remember data, instead of using recursive -function calls, has been implemented to work round this problem. If you want to -build a version of PCRE that works this way, add +When matching with the <b>pcre_exec()</b> function, PCRE implements backtracking +by making recursive calls to an internal function called <b>match()</b>. In +environments where the size of the stack is limited, this can severely limit +PCRE's operation. (The Unix environment does not usually suffer from this +problem.) An alternative approach that uses memory from the heap to remember +data, instead of using recursive function calls, has been implemented to work +round this problem. If you want to build a version of PCRE that works this way, +add <pre> --disable-stack-for-recursion </pre> @@ -165,7 +168,8 @@ predictable: the block sizes requested are always the same, and the blocks are always freed in reverse order. A calling program might be able to implement optimized functions that perform better than the standard <b>malloc()</b> and <b>free()</b> functions. PCRE runs noticeably more slowly when built in this -way. +way. This option affects only the <b>pcre_exec()</b> function; it is not +relevant for the the <b>pcre_dfa_exec()</b> function. </P> <br><a name="SEC10" href="#TOC1">USING EBCDIC CODE</a><br> <P> @@ -178,9 +182,9 @@ compiled to run in an EBCDIC environment by adding to the <b>configure</b> command. </P> <P> -Last updated: 09 September 2004 +Last updated: 28 February 2005 <br> -Copyright © 1997-2004 University of Cambridge. +Copyright © 1997-2005 University of Cambridge. <p> Return to the <a href="index.html">PCRE index page</a>. </p> diff --git a/doc/html/pcrecallout.html b/doc/html/pcrecallout.html index dc2ef51..5ccfd5c 100644 --- a/doc/html/pcrecallout.html +++ b/doc/html/pcrecallout.html @@ -72,9 +72,10 @@ no match, the callout is obeyed. <br><a name="SEC3" href="#TOC1">THE CALLOUT INTERFACE</a><br> <P> During matching, when PCRE reaches a callout point, the external function -defined by <i>pcre_callout</i> is called (if it is set). The only argument is a -pointer to a <b>pcre_callout</b> block. This structure contains the following -fields: +defined by <i>pcre_callout</i> is called (if it is set). This applies to both +the <b>pcre_exec()</b> and the <b>pcre_dfa_exec()</b> matching functions. The +only argument to the callout function is a pointer to a <b>pcre_callout</b> +block. This structure contains the following fields: <pre> int <i>version</i>; int <i>callout_number</i>; @@ -101,9 +102,11 @@ automatically generated callouts). </P> <P> The <i>offset_vector</i> field is a pointer to the vector of offsets that was -passed by the caller to <b>pcre_exec()</b>. The contents can be inspected in -order to extract substrings that have been matched so far, in the same way as -for extracting substrings after a match has completed. +passed by the caller to <b>pcre_exec()</b> or <b>pcre_dfa_exec()</b>. When +<b>pcre_exec()</b> is used, the contents can be inspected in order to extract +substrings that have been matched so far, in the same way as for extracting +substrings after a match has completed. For <b>pcre_dfa_exec()</b> this field is +not useful. </P> <P> The <i>subject</i> and <i>subject_length</i> fields contain copies of the values @@ -120,21 +123,24 @@ The <i>current_position</i> field contains the offset within the subject of the current match pointer. </P> <P> -The <i>capture_top</i> field contains one more than the number of the highest -numbered captured substring so far. If no substrings have been captured, -the value of <i>capture_top</i> is one. +When the <b>pcre_exec()</b> function is used, the <i>capture_top</i> field +contains one more than the number of the highest numbered captured substring so +far. If no substrings have been captured, the value of <i>capture_top</i> is +one. This is always the case when <b>pcre_dfa_exec()</b> is used, because it +does not support captured substrings. </P> <P> The <i>capture_last</i> field contains the number of the most recently captured -substring. If no substrings have been captured, its value is -1. +substring. If no substrings have been captured, its value is -1. This is always +the case when <b>pcre_dfa_exec()</b> is used. </P> <P> The <i>callout_data</i> field contains a value that is passed to -<b>pcre_exec()</b> by the caller specifically so that it can be passed back in -callouts. It is passed in the <i>pcre_callout</i> field of the <b>pcre_extra</b> -data structure. If no such data was passed, the value of <i>callout_data</i> in -a <b>pcre_callout</b> block is NULL. There is a description of the -<b>pcre_extra</b> structure in the +<b>pcre_exec()</b> or <b>pcre_dfa_exec()</b> specifically so that it can be +passed back in callouts. It is passed in the <i>pcre_callout</i> field of the +<b>pcre_extra</b> data structure. If no such data was passed, the value of +<i>callout_data</i> in a <b>pcre_callout</b> block is NULL. There is a +description of the <b>pcre_extra</b> structure in the <a href="pcreapi.html"><b>pcreapi</b></a> documentation. </P> @@ -160,10 +166,10 @@ same callout number. However, they are set for all callouts. <P> The external callout function returns an integer to PCRE. If the value is zero, matching proceeds as normal. If the value is greater than zero, matching fails -at the current point, but backtracking to test other matching possibilities -goes ahead, just as if a lookahead assertion had failed. If the value is less -than zero, the match is abandoned, and <b>pcre_exec()</b> returns the negative -value. +at the current point, but the testing of other matching possibilities goes +ahead, just as if a lookahead assertion had failed. If the value is less than +zero, the match is abandoned, and <b>pcre_exec()</b> (or <b>pcre_dfa_exec()</b>) +returns the negative value. </P> <P> Negative values should normally be chosen from the set of PCRE_ERROR_xxx @@ -172,9 +178,9 @@ The error number PCRE_ERROR_CALLOUT is reserved for use by callout functions; it will never be used by PCRE itself. </P> <P> -Last updated: 09 September 2004 +Last updated: 28 February 2005 <br> -Copyright © 1997-2004 University of Cambridge. +Copyright © 1997-2005 University of Cambridge. <p> Return to the <a href="index.html">PCRE index page</a>. </p> diff --git a/doc/html/pcrecompat.html b/doc/html/pcrecompat.html index 6529c09..d15b3b0 100644 --- a/doc/html/pcrecompat.html +++ b/doc/html/pcrecompat.html @@ -140,11 +140,15 @@ package. <br> (m) Patterns compiled by PCRE can be saved and re-used at a later time, even on different hosts that have the other endianness. +<br> +<br> +(n) The alternative matching function (<b>pcre_dfa_exec()</b>) matches in a +different way and is not Perl-compatible. </P> <P> -Last updated: 09 September 2004 +Last updated: 28 February 2005 <br> -Copyright © 1997-2004 University of Cambridge. +Copyright © 1997-2005 University of Cambridge. <p> Return to the <a href="index.html">PCRE index page</a>. </p> diff --git a/doc/html/pcrecpp.html b/doc/html/pcrecpp.html new file mode 100644 index 0000000..3f597b1 --- /dev/null +++ b/doc/html/pcrecpp.html @@ -0,0 +1,241 @@ +<html> +<head> +<title>pcrecpp specification</title> +</head> +<body bgcolor="#FFFFFF" text="#00005A" link="#0066FF" alink="#3399FF" vlink="#2222BB"> +<h1>pcrecpp man page</h1> +<p> +Return to the <a href="index.html">PCRE index page</a>. +</p> +<p> +This page is part of the PCRE HTML documentation. It was generated automatically +from the original man page. If there is any nonsense in it, please consult the +man page, in case the conversion went wrong. +<br> +<ul> +<li><a name="TOC1" href="#SEC1">SYNOPSIS OF C++ WRAPPER</a> +<li><a name="TOC2" href="#SEC2">DESCRIPTION</a> +<li><a name="TOC3" href="#SEC3">MATCHING INTERFACE</a> +<li><a name="TOC4" href="#SEC4">PARTIAL MATCHES</a> +<li><a name="TOC5" href="#SEC5">UTF-8 AND THE MATCHING INTERFACE</a> +<li><a name="TOC6" href="#SEC6">SCANNING TEXT INCREMENTALLY</a> +<li><a name="TOC7" href="#SEC7">PARSING HEX/OCTAL/C-RADIX NUMBERS</a> +<li><a name="TOC8" href="#SEC8">REPLACING PARTS OF STRINGS</a> +<li><a name="TOC9" href="#SEC9">AUTHOR</a> +</ul> +<br><a name="SEC1" href="#TOC1">SYNOPSIS OF C++ WRAPPER</a><br> +<P> +<b>#include <pcrecpp.h></b> +</P> +<P> +</P> +<br><a name="SEC2" href="#TOC1">DESCRIPTION</a><br> +<P> +The C++ wrapper for PCRE was provided by Google Inc. This brief man page was +constructed from the notes in the <i>pcrecpp.h</i> file, which should be +consulted for further details. +</P> +<br><a name="SEC3" href="#TOC1">MATCHING INTERFACE</a><br> +<P> +The "FullMatch" operation checks that supplied text matches a supplied pattern +exactly. If pointer arguments are supplied, it copies matched sub-strings that +match sub-patterns into them. +<pre> + Example: successful match + pcrecpp::RE re("h.*o"); + re.FullMatch("hello"); + + Example: unsuccessful match (requires full match): + pcrecpp::RE re("e"); + !re.FullMatch("hello"); + + Example: creating a temporary RE object: + pcrecpp::RE("h.*o").FullMatch("hello"); +</pre> +You can pass in a "const char*" or a "string" for "text". The examples below +tend to use a const char*. You can, as in the different examples above, store +the RE object explicitly in a variable or use a temporary RE object. The +examples below use one mode or the other arbitrarily. Either could correctly be +used for any of these examples. +</P> +<P> +You must supply extra pointer arguments to extract matched subpieces. +<pre> + Example: extracts "ruby" into "s" and 1234 into "i" + int i; + string s; + pcrecpp::RE re("(\\w+):(\\d+)"); + re.FullMatch("ruby:1234", &s, &i); + + Example: does not try to extract any extra sub-patterns + re.FullMatch("ruby:1234", &s); + + Example: does not try to extract into NULL + re.FullMatch("ruby:1234", NULL, &i); + + Example: integer overflow causes failure + !re.FullMatch("ruby:1234567891234", NULL, &i); + + Example: fails because there aren't enough sub-patterns: + !pcrecpp::RE("\\w+:\\d+").FullMatch("ruby:1234", &s); + + Example: fails because string cannot be stored in integer + !pcrecpp::RE("(.*)").FullMatch("ruby", &i); +</pre> +The provided pointer arguments can be pointers to any scalar numeric +type, or one of: +<pre> + string (matched piece is copied to string) + StringPiece (StringPiece is mutated to point to matched piece) + T (where "bool T::ParseFrom(const char*, int)" exists) + NULL (the corresponding matched sub-pattern is not copied) +</pre> +The function returns true iff all of the following conditions are satisfied: +<pre> + a. "text" matches "pattern" exactly; + + b. The number of matched sub-patterns is >= number of supplied + pointers; + + c. The "i"th argument has a suitable type for holding the + string captured as the "i"th sub-pattern. If you pass in + NULL for the "i"th argument, or pass fewer arguments than + number of sub-patterns, "i"th captured sub-pattern is + ignored. +</pre> +The matching interface supports at most 16 arguments per call. +If you need more, consider using the more general interface +<b>pcrecpp::RE::DoMatch</b>. See <b>pcrecpp.h</b> for the signature for +<b>DoMatch</b>. +</P> +<br><a name="SEC4" href="#TOC1">PARTIAL MATCHES</a><br> +<P> +You can use the "PartialMatch" operation when you want the pattern +to match any substring of the text. +<pre> + Example: simple search for a string: + pcrecpp::RE("ell").PartialMatch("hello"); + + Example: find first number in a string: + int number; + pcrecpp::RE re("(\\d+)"); + re.PartialMatch("x*100 + 20", &number); + assert(number == 100); +</PRE> +</P> +<br><a name="SEC5" href="#TOC1">UTF-8 AND THE MATCHING INTERFACE</a><br> +<P> +By default, pattern and text are plain text, one byte per character. The UTF8 +flag, passed to the constructor, causes both pattern and string to be treated +as UTF-8 text, still a byte stream but potentially multiple bytes per +character. In practice, the text is likelier to be UTF-8 than the pattern, but +the match returned may depend on the UTF8 flag, so always use it when matching +UTF8 text. For example, "." will match one byte normally but with UTF8 set may +match up to three bytes of a multi-byte character. +<pre> + Example: + pcrecpp::RE_Options options; + options.set_utf8(); + pcrecpp::RE re(utf8_pattern, options); + re.FullMatch(utf8_string); + + Example: using the convenience function UTF8(): + pcrecpp::RE re(utf8_pattern, pcrecpp::UTF8()); + re.FullMatch(utf8_string); +</pre> +NOTE: The UTF8 flag is ignored if pcre was not configured with the +<pre> + --enable-utf8 flag. +</PRE> +</P> +<br><a name="SEC6" href="#TOC1">SCANNING TEXT INCREMENTALLY</a><br> +<P> +The "Consume" operation may be useful if you want to repeatedly +match regular expressions at the front of a string and skip over +them as they match. This requires use of the "StringPiece" type, +which represents a sub-range of a real string. Like RE, StringPiece +is defined in the pcrecpp namespace. +<pre> + Example: read lines of the form "var = value" from a string. + string contents = ...; // Fill string somehow + pcrecpp::StringPiece input(contents); // Wrap in a StringPiece +</PRE> +</P> +<P> +<pre> + string var; + int value; + pcrecpp::RE re("(\\w+) = (\\d+)\n"); + while (re.Consume(&input, &var, &value)) { + ...; + } +</pre> +Each successful call to "Consume" will set "var/value", and also +advance "input" so it points past the matched text. +</P> +<P> +The "FindAndConsume" operation is similar to "Consume" but does not +anchor your match at the beginning of the string. For example, you +could extract all words from a string by repeatedly calling +<pre> + pcrecpp::RE("(\\w+)").FindAndConsume(&input, &word) +</PRE> +</P> +<br><a name="SEC7" href="#TOC1">PARSING HEX/OCTAL/C-RADIX NUMBERS</a><br> +<P> +By default, if you pass a pointer to a numeric value, the +corresponding text is interpreted as a base-10 number. You can +instead wrap the pointer with a call to one of the operators Hex(), +Octal(), or CRadix() to interpret the text in another base. The +CRadix operator interprets C-style "0" (base-8) and "0x" (base-16) +prefixes, but defaults to base-10. +<pre> + Example: + int a, b, c, d; + pcrecpp::RE re("(.*) (.*) (.*) (.*)"); + re.FullMatch("100 40 0100 0x40", + pcrecpp::Octal(&a), pcrecpp::Hex(&b), + pcrecpp::CRadix(&c), pcrecpp::CRadix(&d)); +</pre> +will leave 64 in a, b, c, and d. +</P> +<br><a name="SEC8" href="#TOC1">REPLACING PARTS OF STRINGS</a><br> +<P> +You can replace the first match of "pattern" in "str" with "rewrite". +Within "rewrite", backslash-escaped digits (\1 to \9) can be +used to insert text matching corresponding parenthesized group +from the pattern. \0 in "rewrite" refers to the entire matching +text. For example: +<pre> + string s = "yabba dabba doo"; + pcrecpp::RE("b+").Replace("d", &s); +</pre> +will leave "s" containing "yada dabba doo". The result is true if the pattern +matches and a replacement occurs, false otherwise. +</P> +<P> +<b>GlobalReplace</b> is like <b>Replace</b> except that it replaces all +occurrences of the pattern in the string with the rewrite. Replacements are +not subject to re-matching. For example: +<pre> + string s = "yabba dabba doo"; + pcrecpp::RE("b+").GlobalReplace("d", &s); +</pre> +will leave "s" containing "yada dada doo". It returns the number of +replacements made. +</P> +<P> +<b>Extract</b> is like <b>Replace</b>, except that if the pattern matches, +"rewrite" is copied into "out" (an additional argument) with substitutions. +The non-matching portions of "text" are ignored. Returns true iff a match +occurred and the extraction happened successfully; if no match occurs, the +string is left unaffected. +</P> +<br><a name="SEC9" href="#TOC1">AUTHOR</a><br> +<P> +The C++ wrapper was contributed by Google Inc. +<br> +Copyright © 2005 Google Inc. +<p> +Return to the <a href="index.html">PCRE index page</a>. +</p> diff --git a/doc/html/pcregrep.html b/doc/html/pcregrep.html index 922487d..614034d 100644 --- a/doc/html/pcregrep.html +++ b/doc/html/pcregrep.html @@ -17,12 +17,13 @@ man page, in case the conversion went wrong. <li><a name="TOC2" href="#SEC2">DESCRIPTION</a> <li><a name="TOC3" href="#SEC3">OPTIONS</a> <li><a name="TOC4" href="#SEC4">LONG OPTIONS</a> -<li><a name="TOC5" href="#SEC5">DIAGNOSTICS</a> -<li><a name="TOC6" href="#SEC6">AUTHOR</a> +<li><a name="TOC5" href="#SEC5">OPTIONS WITH DATA</a> +<li><a name="TOC6" href="#SEC6">DIAGNOSTICS</a> +<li><a name="TOC7" href="#SEC7">AUTHOR</a> </ul> <br><a name="SEC1" href="#TOC1">SYNOPSIS</a><br> <P> -<b>pcregrep [-Vcfhilnrsuvx] [long options] [pattern] [file1 file2 ...]</b> +<b>pcregrep [options] [long options] [pattern] [file1 file2 ...]</b> </P> <br><a name="SEC2" href="#TOC1">DESCRIPTION</a><br> <P> @@ -38,21 +39,50 @@ A pattern must be specified on the command line unless the <b>-f</b> option is used (see below). </P> <P> -If no files are specified, <b>pcregrep</b> reads the standard input. By default, -each line that matches the pattern is copied to the standard output, and if -there is more than one file, the file name is printed before each line of -output. However, there are options that can change how <b>pcregrep</b> behaves. +If no files are specified, <b>pcregrep</b> reads the standard input. The +standard input can also be referenced by a name consisting of a single hyphen. +For example: +<pre> + pcregrep some-pattern /file1 - /file3 +</pre> +By default, each line that matches the pattern is copied to the standard +output, and if there is more than one file, the file name is printed before +each line of output. However, there are options that can change how +<b>pcregrep</b> behaves. In particular, the <b>-M</b> option makes it possible to +search for patterns that span line boundaries. </P> <P> -Lines are limited to BUFSIZ characters. BUFSIZ is defined in <b><stdio.h></b>. -The newline character is removed from the end of each line before it is matched -against the pattern. +Patterns are limited to 8K or BUFSIZ characters, whichever is the greater. +BUFSIZ is defined in <b><stdio.h></b>. </P> <br><a name="SEC3" href="#TOC1">OPTIONS</a><br> <P> -<b>-V</b> -Write the version number of the PCRE library being used to the standard error -stream. +<b>--</b> +This terminate the list of options. It is useful if the next item on the +command line starts with a hyphen, but is not an option. +</P> +<P> +<b>-A</b> <i>number</i> +Print <i>number</i> lines of context after each matching line. If file names +and/or line numbers are being printed, a hyphen separator is used instead of a +colon for the context lines. A line containing "--" is printed between each +group of lines, unless they are in fact contiguous in the input file. The value +of <i>number</i> is expected to be relatively small. However, <b>pcregrep</b> +guarantees to have up to 8K of following text available for context printing. +</P> +<P> +<b>-B</b> <i>number</i> +Print <i>number</i> lines of context before each matching line. If file names +and/or line numbers are being printed, a hyphen separator is used instead of a +colon for the context lines. A line containing "--" is printed between each +group of lines, unless they are in fact contiguous in the input file. The value +of <i>number</i> is expected to be relatively small. However, <b>pcregrep</b> +guarantees to have up to 8K of preceding text available for context printing. +</P> +<P> +<b>-C</b> <i>number</i> +Print <i>number</i> lines of context both before and after each matching line. +This is equivalent to setting both <b>-A</b> and <b>-B</b> to the same value. </P> <P> <b>-c</b> @@ -61,6 +91,14 @@ lines that would otherwise have been printed. If several files are given, a count is printed for each of them. </P> <P> +<b>--exclude</b>=<i>pattern</i> +When <b>pcregrep</b> is searching the files in a directory as a consequence of +the <b>-r</b> (recursive search) option, any files whose names match the pattern +are excluded. The pattern is a PCRE regular expression. If a file name matches +both <b>--include</b> and <b>--exclude</b>, it is excluded. There is no short +form for this option. +</P> +<P> <b>-f</b><i>filename</i> Read a number of patterns from the file, one per line, and match all of them against each line of input. A line is output if any of the patterns match it. @@ -78,24 +116,64 @@ Suppress printing of filenames when searching multiple files. Ignore upper/lower case distinctions during comparisons. </P> <P> +<b>--include</b>=<i>pattern</i> +When <b>pcregrep</b> is searching the files in a directory as a consequence of +the <b>-r</b> (recursive search) option, only files whose names match the +pattern are included. The pattern is a PCRE regular expression. If a file name +matches both <b>--include</b> and <b>--exclude</b>, it is excluded. There is no +short form for this option. +</P> +<P> +<b>-L</b> +Instead of printing lines from the files, just print the names of the files +that do not contain any lines that would have been printed. Each file name is +printed once, on a separate line. +</P> +<P> <b>-l</b> Instead of printing lines from the files, just print the names of the files containing lines that would have been printed. Each file name is printed once, on a separate line. </P> <P> +<b>--label</b>=<i>name</i> +This option supplies a name to be used for the standard input when file names +are being printed. If not supplied, "(standard input)" is used. There is no +short form for this option. +</P> +<P> +<b>-M</b> +Allow patterns to match more than one line. When this option is given, patterns +may usefully contain literal newline characters and internal occurrences of ^ +and $ characters. The output for any one match may consist of more than one +line. When this option is set, the PCRE library is called in "multiline" mode. +There is a limit to the number of lines that can be matched, imposed by the way +that <b>pcregrep</b> buffers the input file as it scans it. However, +<b>pcregrep</b> ensures that at least 8K characters or the rest of the document +(whichever is the shorter) are available for forward matching, and similarly +the previous 8K characters (or all the previous characters, if fewer than 8K) +are guaranteed to be available for lookbehind assertions. +</P> +<P> <b>-n</b> Precede each line by its line number in the file. </P> <P> +<b>-q</b> +Work quietly, that is, display nothing except error messages. +The exit status indicates whether or not any matches were found. +</P> +<P> <b>-r</b> -If any file is a directory, recursively scan the files it contains. Without +If any given path is a directory, recursively scan the files it contains, +taking note of any <b>--include</b> and <b>--exclude</b> settings. Without <b>-r</b> a directory is scanned as a normal file. </P> <P> <b>-s</b> -Work silently, that is, display nothing except error messages. -The exit status indicates whether any matches were found. +Suppress error messages about non-existent or unreadable files. Such files are +quietly skipped. However, the return code is still 2, even if matches were +found in other files. </P> <P> <b>-u</b> @@ -104,9 +182,19 @@ with UTF-8 support. Both the pattern and each subject line must be valid strings of UTF-8 characters. </P> <P> +<b>-V</b> +Write the version numbers of <b>pcregrep</b> and the PCRE library that is being +used to the standard error stream. +</P> +<P> <b>-v</b> Invert the sense of the match, so that lines which do <i>not</i> match the -pattern are now the ones that are found. +pattern are the ones that are found. +</P> +<P> +<b>-w</b> +Force the pattern to match only whole words. This is equivalent to having \b +at the start and end of the pattern. </P> <P> <b>-x</b> @@ -120,39 +208,67 @@ alternative branch in the regular expression. Long forms of all the options are available, as in GNU grep. They are shown in the following table: <pre> + -A --after-context + -B --before-context + -C --context -c --count + --exclude (no short form) + -f --file -h --no-filename + --help (no short form) -i --ignore-case + --include (no short form) + -L --files-without-match -l --files-with-matches + --label (no short form) -n --line-number -r --recursive + -q --quiet -s --no-messages -u --utf-8 -V --version -v --invert-match -x --line-regex -x --line-regexp +</PRE> +</P> +<br><a name="SEC5" href="#TOC1">OPTIONS WITH DATA</a><br> +<P> +There are four different ways in which an option with data can be specified. +If a short form option is used, the data may follow immediately, or in the next +command line item. For example: +<pre> + -f/some/file + -f /some/file </pre> -In addition, --file=<i>filename</i> is equivalent to -f<i>filename</i>, and ---help shows the list of options and then exits. +If a long form option is used, the data may appear in the same command line +item, separated by an = character, or it may appear in the next command line +item. For example: +<pre> + --file=/some/file + --file /some/file + +</PRE> </P> -<br><a name="SEC5" href="#TOC1">DIAGNOSTICS</a><br> +<br><a name="SEC6" href="#TOC1">DIAGNOSTICS</a><br> <P> Exit status is 0 if any matches were found, 1 if no matches were found, and 2 -for syntax errors or inacessible files (even if matches were found). +for syntax errors and non-existent or inacessible files (even if matches were +found in other files). Using the <b>-s</b> option to suppress error messages +about inaccessble files does not affect the return code. </P> -<br><a name="SEC6" href="#TOC1">AUTHOR</a><br> +<br><a name="SEC7" href="#TOC1">AUTHOR</a><br> <P> -Philip Hazel <ph10@cam.ac.uk> +Philip Hazel <br> University Computing Service <br> Cambridge CB2 3QG, England. </P> <P> -Last updated: 09 September 2004 +Last updated: 16 May 2005 <br> -Copyright © 1997-2004 University of Cambridge. +Copyright © 1997-2005 University of Cambridge. <p> Return to the <a href="index.html">PCRE index page</a>. </p> diff --git a/doc/html/pcrematching.html b/doc/html/pcrematching.html new file mode 100644 index 0000000..eb381b7 --- /dev/null +++ b/doc/html/pcrematching.html @@ -0,0 +1,192 @@ +<html> +<head> +<title>pcrematching specification</title> +</head> +<body bgcolor="#FFFFFF" text="#00005A" link="#0066FF" alink="#3399FF" vlink="#2222BB"> +<h1>pcrematching man page</h1> +<p> +Return to the <a href="index.html">PCRE index page</a>. +</p> +<p> +This page is part of the PCRE HTML documentation. It was generated automatically +from the original man page. If there is any nonsense in it, please consult the +man page, in case the conversion went wrong. +<br> +<ul> +<li><a name="TOC1" href="#SEC1">PCRE MATCHING ALGORITHMS</a> +<li><a name="TOC2" href="#SEC2">REGULAR EXPRESSIONS AS TREES</a> +<li><a name="TOC3" href="#SEC3">THE STANDARD MATCHING ALGORITHM</a> +<li><a name="TOC4" href="#SEC4">THE DFA MATCHING ALGORITHM</a> +<li><a name="TOC5" href="#SEC5">ADVANTAGES OF THE DFA ALGORITHM</a> +<li><a name="TOC6" href="#SEC6">DISADVANTAGES OF THE DFA ALGORITHM</a> +</ul> +<br><a name="SEC1" href="#TOC1">PCRE MATCHING ALGORITHMS</a><br> +<P> +This document describes the two different algorithms that are available in PCRE +for matching a compiled regular expression against a given subject string. The +"standard" algorithm is the one provided by the <b>pcre_exec()</b> function. +This works in the same was as Perl's matching function, and provides a +Perl-compatible matching operation. +</P> +<P> +An alternative algorithm is provided by the <b>pcre_dfa_exec()</b> function; +this operates in a different way, and is not Perl-compatible. It has advantages +and disadvantages compared with the standard algorithm, and these are described +below. +</P> +<P> +When there is only one possible way in which a given subject string can match a +pattern, the two algorithms give the same answer. A difference arises, however, +when there are multiple possibilities. For example, if the pattern +<pre> + ^<.*> +</pre> +is matched against the string +<pre> + <something> <something else> <something further> +</pre> +there are three possible answers. The standard algorithm finds only one of +them, whereas the DFA algorithm finds all three. +</P> +<br><a name="SEC2" href="#TOC1">REGULAR EXPRESSIONS AS TREES</a><br> +<P> +The set of strings that are matched by a regular expression can be represented +as a tree structure. An unlimited repetition in the pattern makes the tree of +infinite size, but it is still a tree. Matching the pattern to a given subject +string (from a given starting point) can be thought of as a search of the tree. +There are two standard ways to search a tree: depth-first and breadth-first, +and these correspond to the two matching algorithms provided by PCRE. +</P> +<br><a name="SEC3" href="#TOC1">THE STANDARD MATCHING ALGORITHM</a><br> +<P> +In the terminology of Jeffrey Friedl's book \fIMastering Regular +Expressions\fP, the standard algorithm is an "NFA algorithm". It conducts a +depth-first search of the pattern tree. That is, it proceeds along a single +path through the tree, checking that the subject matches what is required. When +there is a mismatch, the algorithm tries any alternatives at the current point, +and if they all fail, it backs up to the previous branch point in the tree, and +tries the next alternative branch at that level. This often involves backing up +(moving to the left) in the subject string as well. The order in which +repetition branches are tried is controlled by the greedy or ungreedy nature of +the quantifier. +</P> +<P> +If a leaf node is reached, a matching string has been found, and at that point +the algorithm stops. Thus, if there is more than one possible match, this +algorithm returns the first one that it finds. Whether this is the shortest, +the longest, or some intermediate length depends on the way the greedy and +ungreedy repetition quantifiers are specified in the pattern. +</P> +<P> +Because it ends up with a single path through the tree, it is relatively +straightforward for this algorithm to keep track of the substrings that are +matched by portions of the pattern in parentheses. This provides support for +capturing parentheses and back references. +</P> +<br><a name="SEC4" href="#TOC1">THE DFA MATCHING ALGORITHM</a><br> +<P> +DFA stands for "deterministic finite automaton", but you do not need to +understand the origins of that name. This algorithm conducts a breadth-first +search of the tree. Starting from the first matching point in the subject, it +scans the subject string from left to right, once, character by character, and +as it does this, it remembers all the paths through the tree that represent +valid matches. +</P> +<P> +The scan continues until either the end of the subject is reached, or there are +no more unterminated paths. At this point, terminated paths represent the +different matching possibilities (if there are none, the match has failed). +Thus, if there is more than one possible match, this algorithm finds all of +them, and in particular, it finds the longest. In PCRE, there is an option to +stop the algorithm after the first match (which is necessarily the shortest) +has been found. +</P> +<P> +Note that all the matches that are found start at the same point in the +subject. If the pattern +<pre> + cat(er(pillar)?) +</pre> +is matched against the string "the caterpillar catchment", the result will be +the three strings "cat", "cater", and "caterpillar" that start at the fourth +character of the subject. The algorithm does not automatically move on to find +matches that start at later positions. +</P> +<P> +There are a number of features of PCRE regular expressions that are not +supported by the DFA matching algorithm. They are as follows: +</P> +<P> +1. Because the algorithm finds all possible matches, the greedy or ungreedy +nature of repetition quantifiers is not relevant. Greedy and ungreedy +quantifiers are treated in exactly the same way. +</P> +<P> +2. When dealing with multiple paths through the tree simultaneously, it is not +straightforward to keep track of captured substrings for the different matching +possibilities, and PCRE's implementation of this algorithm does not attempt to +do this. This means that no captured substrings are available. +</P> +<P> +3. Because no substrings are captured, back references within the pattern are +not supported, and cause errors if encountered. +</P> +<P> +4. For the same reason, conditional expressions that use a backreference as the +condition are not supported. +</P> +<P> +5. Callouts are supported, but the value of the <i>capture_top</i> field is +always 1, and the value of the <i>capture_last</i> field is always -1. +</P> +<P> +6. +The \C escape sequence, which (in the standard algorithm) matches a single +byte, even in UTF-8 mode, is not supported because the DFA algorithm moves +through the subject string one character at a time, for all active paths +through the tree. +</P> +<br><a name="SEC5" href="#TOC1">ADVANTAGES OF THE DFA ALGORITHM</a><br> +<P> +Using the DFA matching algorithm provides the following advantages: +</P> +<P> +1. All possible matches (at a single point in the subject) are automatically +found, and in particular, the longest match is found. To find more than one +match using the standard algorithm, you have to do kludgy things with +callouts. +</P> +<P> +2. There is much better support for partial matching. The restrictions on the +content of the pattern that apply when using the standard algorithm for partial +matching do not apply to the DFA algorithm. For non-anchored patterns, the +starting position of a partial match is available. +</P> +<P> +3. Because the DFA algorithm scans the subject string just once, and never +needs to backtrack, it is possible to pass very long subject strings to the +matching function in several pieces, checking for partial matching each time. +</P> +<br><a name="SEC6" href="#TOC1">DISADVANTAGES OF THE DFA ALGORITHM</a><br> +<P> +The DFA algorithm suffers from a number of disadvantages: +</P> +<P> +1. It is substantially slower than the standard algorithm. This is partly +because it has to search for all possible matches, but is also because it is +less susceptible to optimization. +</P> +<P> +2. Capturing parentheses and back references are not supported. +</P> +<P> +3. The "atomic group" feature of PCRE regular expressions is supported, but +does not provide the advantage that it does for the standard algorithm. +</P> +<P> +Last updated: 28 February 2005 +<br> +Copyright © 1997-2005 University of Cambridge. +<p> +Return to the <a href="index.html">PCRE index page</a>. +</p> diff --git a/doc/html/pcrepartial.html b/doc/html/pcrepartial.html index c4dd886..0dbfc02 100644 --- a/doc/html/pcrepartial.html +++ b/doc/html/pcrepartial.html @@ -16,14 +16,15 @@ man page, in case the conversion went wrong. <li><a name="TOC1" href="#SEC1">PARTIAL MATCHING IN PCRE</a> <li><a name="TOC2" href="#SEC2">RESTRICTED PATTERNS FOR PCRE_PARTIAL</a> <li><a name="TOC3" href="#SEC3">EXAMPLE OF PARTIAL MATCHING USING PCRETEST</a> +<li><a name="TOC4" href="#SEC4">MULTI-SEGMENT MATCHING WITH pcre_dfa_exec()</a> </ul> <br><a name="SEC1" href="#TOC1">PARTIAL MATCHING IN PCRE</a><br> <P> In normal use of PCRE, if the subject string that is passed to -<b>pcre_exec()</b> matches as far as it goes, but is too short to match the -entire pattern, PCRE_ERROR_NOMATCH is returned. There are circumstances where -it might be helpful to distinguish this case from other cases in which there is -no match. +<b>pcre_exec()</b> or <b>pcre_dfa_exec()</b> matches as far as it goes, but is +too short to match the entire pattern, PCRE_ERROR_NOMATCH is returned. There +are circumstances where it might be helpful to distinguish this case from other +cases in which there is no match. </P> <P> Consider, for example, an application where a human is required to type in data @@ -41,10 +42,20 @@ entered. </P> <P> PCRE supports the concept of partial matching by means of the PCRE_PARTIAL -option, which can be set when calling <b>pcre_exec()</b>. When this is done, the -return code PCRE_ERROR_NOMATCH is converted into PCRE_ERROR_PARTIAL if at any -time during the matching process the entire subject string matched part of the -pattern. No captured data is set when this occurs. +option, which can be set when calling <b>pcre_exec()</b> or +<b>pcre_dfa_exec()</b>. When this flag is set for <b>pcre_exec()</b>, the return +code PCRE_ERROR_NOMATCH is converted into PCRE_ERROR_PARTIAL if at any time +during the matching process the last part of the subject string matched part of +the pattern. Unfortunately, for non-anchored matching, it is not possible to +obtain the position of the start of the partial match. No captured data is set +when PCRE_ERROR_PARTIAL is returned. +</P> +<P> +When PCRE_PARTIAL is set for <b>pcre_dfa_exec()</b>, the return code +PCRE_ERROR_NOMATCH is converted into PCRE_ERROR_PARTIAL if the end of the +subject is reached, there have been no complete matches, but there is still at +least one matching possibility. The portion of the string that provided the +partial match is set as the first matching string. </P> <P> Using PCRE_PARTIAL disables one of PCRE's optimizations. PCRE remembers the @@ -54,9 +65,10 @@ for a subject string that might match only partially. </P> <br><a name="SEC2" href="#TOC1">RESTRICTED PATTERNS FOR PCRE_PARTIAL</a><br> <P> -Because of the way certain internal optimizations are implemented in PCRE, the -PCRE_PARTIAL option cannot be used with all patterns. Repeated single -characters such as +Because of the way certain internal optimizations are implemented in the +<b>pcre_exec()</b> function, the PCRE_PARTIAL option cannot be used with all +patterns. These restrictions do not apply when <b>pcre_dfa_exec()</b> is used. +For <b>pcre_exec()</b>, repeated single characters such as <pre> a{2,4} </pre> @@ -100,12 +112,94 @@ uses the date example quoted above: </pre> The first data string is matched completely, so <b>pcretest</b> shows the matched substrings. The remaining four strings do not match the complete -pattern, but the first two are partial matches. +pattern, but the first two are partial matches. The same test, using DFA +matching (by means of the \D escape sequence), produces the following output: +<pre> + re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/ + data> 25jun04\P\D + 0: 25jun04 + data> 23dec3\P\D + Partial match: 23dec3 + data> 3ju\P\D + Partial match: 3ju + data> 3juj\P\D + No match + data> j\P\D + No match +</pre> +Notice that in this case the portion of the string that was matched is made +available. +</P> +<br><a name="SEC4" href="#TOC1">MULTI-SEGMENT MATCHING WITH pcre_dfa_exec()</a><br> +<P> +When a partial match has been found using <b>pcre_dfa_exec()</b>, it is possible +to continue the match by providing additional subject data and calling +<b>pcre_dfa_exec()</b> again with the PCRE_DFA_RESTART option and the same +working space (where details of the previous partial match are stored). Here is +an example using <b>pcretest</b>, where the \R escape sequence sets the +PCRE_DFA_RESTART option and the \D escape sequence requests the use of +<b>pcre_dfa_exec()</b>: +<pre> + re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/ + data> 23ja\P\D + Partial match: 23ja + data> n05\R\D + 0: n05 +</pre> +The first call has "23ja" as the subject, and requests partial matching; the +second call has "n05" as the subject for the continued (restarted) match. +Notice that when the match is complete, only the last part is shown; PCRE does +not retain the previously partially-matched string. It is up to the calling +program to do that if it needs to. +</P> +<P> +This facility can be used to pass very long subject strings to +<b>pcre_dfa_exec()</b>. However, some care is needed for certain types of +pattern. +</P> +<P> +1. If the pattern contains tests for the beginning or end of a line, you need +to pass the PCRE_NOTBOL or PCRE_NOTEOL options, as appropriate, when the +subject string for any call does not contain the beginning or end of a line. +</P> +<P> +2. If the pattern contains backward assertions (including \b or \B), you need +to arrange for some overlap in the subject strings to allow for this. For +example, you could pass the subject in chunks that were 500 bytes long, but in +a buffer of 700 bytes, with the starting offset set to 200 and the previous 200 +bytes at the start of the buffer. +</P> +<P> +3. Matching a subject string that is split into multiple segments does not +always produce exactly the same result as matching over one single long string. +The difference arises when there are multiple matching possibilities, because a +partial match result is given only when there are no completed matches in a +call to fBpcre_dfa_exec()\fP. This means that as soon as the shortest match has +been found, continuation to a new subject segment is no longer possible. +Consider this <b>pcretest</b> example: +<pre> + re> /dog(sbody)?/ + data> do\P\D + Partial match: do + data> gsb\R\P\D + 0: g + data> dogsbody\D + 0: dogsbody + 1: dog +</pre> +The pattern matches the words "dog" or "dogsbody". When the subject is +presented in several parts ("do" and "gsb" being the first two) the match stops +when "dog" has been found, and it is not possible to continue. On the other +hand, if "dogsbody" is presented as a single string, both matches are found. +</P> +<P> +Because of this phenomenon, it does not usually make sense to end a pattern +that is going to be matched in this way with a variable repeat. </P> <P> -Last updated: 08 September 2004 +Last updated: 28 February 2005 <br> -Copyright © 1997-2004 University of Cambridge. +Copyright © 1997-2005 University of Cambridge. <p> Return to the <a href="index.html">PCRE index page</a>. </p> diff --git a/doc/html/pcrepattern.html b/doc/html/pcrepattern.html index 1220eb7..0f77b32 100644 --- a/doc/html/pcrepattern.html +++ b/doc/html/pcrepattern.html @@ -55,15 +55,35 @@ in the main page. </P> <P> +The remainder of this document discusses the patterns that are supported by +PCRE when its main matching function, <b>pcre_exec()</b>, is used. +From release 6.0, PCRE offers a second matching function, +<b>pcre_dfa_exec()</b>, which matches using a different algorithm that is not +Perl-compatible. The advantages and disadvantages of the alternative function, +and how it differs from the normal function, are discussed in the +<a href="pcrematching.html"><b>pcrematching</b></a> +page. +</P> +<P> A regular expression is a pattern that is matched against a subject string from left to right. Most characters stand for themselves in a pattern, and match the corresponding characters in the subject. As a trivial example, the pattern <pre> The quick brown fox </pre> -matches a portion of a subject string that is identical to itself. The power of -regular expressions comes from the ability to include alternatives and -repetitions in the pattern. These are encoded in the pattern by the use of +matches a portion of a subject string that is identical to itself. When +caseless matching is specified (the PCRE_CASELESS option), letters are matched +independently of case. In UTF-8 mode, PCRE always understands the concept of +case for characters whose values are less than 128, so caseless matching is +always possible. For characters with higher values, the concept of case is +supported if PCRE is compiled with Unicode property support, but not otherwise. +If you want to use caseless matching for characters 128 and above, you must +ensure that PCRE is compiled with Unicode property support as well as with +UTF-8 support. +</P> +<P> +The power of regular expressions comes from the ability to include alternatives +and repetitions in the pattern. These are encoded in the pattern by the use of <i>metacharacters</i>, which do not stand for themselves but instead are interpreted in some special way. </P> @@ -536,9 +556,13 @@ class as a literal string of bytes, or by using the \x{ escaping mechanism. When caseless matching is set, any letters in a class represent both their upper case and lower case versions, so for example, a caseless [aeiou] matches "A" as well as "a", and a caseless [^aeiou] does not match "A", whereas a -caseful version would. When running in UTF-8 mode, PCRE supports the concept of -case for characters with values greater than 128 only when it is compiled with -Unicode property support. +caseful version would. In UTF-8 mode, PCRE always understands the concept of +case for characters whose values are less than 128, so caseless matching is +always possible. For characters with higher values, the concept of case is +supported if PCRE is compiled with Unicode property support, but not otherwise. +If you want to use caseless matching for characters 128 and above, you must +ensure that PCRE is compiled with Unicode property support as well as with +UTF-8 support. </P> <P> The newline character is never treated in any special way in character classes, @@ -1462,9 +1486,9 @@ description of the interface to the callout function is given in the documentation. </P> <P> -Last updated: 09 September 2004 +Last updated: 28 February 2005 <br> -Copyright © 1997-2004 University of Cambridge. +Copyright © 1997-2005 University of Cambridge. <p> Return to the <a href="index.html">PCRE index page</a>. </p> diff --git a/doc/html/pcreperform.html b/doc/html/pcreperform.html index f0ffa68..e97e748 100644 --- a/doc/html/pcreperform.html +++ b/doc/html/pcreperform.html @@ -50,8 +50,8 @@ this, PCRE has to retry the match starting after every newline in the subject. <P> If you are using such a pattern with subject strings that do not contain newlines, the best performance is obtained by setting PCRE_DOTALL, or starting -the pattern with ^.* to indicate explicit anchoring. That saves PCRE from -having to scan along the subject looking for a newline to restart at. +the pattern with ^.* or ^.*? to indicate explicit anchoring. That saves PCRE +from having to scan along the subject looking for a newline to restart at. </P> <P> Beware of patterns that contain nested indefinite repeats. These can take a @@ -89,9 +89,9 @@ In many cases, the solution to this kind of performance issue is to use an atomic group or a possessive quantifier. </P> <P> -Last updated: 09 September 2004 +Last updated: 28 February 2005 <br> -Copyright © 1997-2004 University of Cambridge. +Copyright © 1997-2005 University of Cambridge. <p> Return to the <a href="index.html">PCRE index page</a>. </p> diff --git a/doc/html/pcreposix.html b/doc/html/pcreposix.html index 2f7aaa9..53ea2aa 100644 --- a/doc/html/pcreposix.html +++ b/doc/html/pcreposix.html @@ -46,8 +46,8 @@ man page, in case the conversion went wrong. This set of functions provides a POSIX-style API to the PCRE regular expression package. See the <a href="pcreapi.html"><b>pcreapi</b></a> -documentation for a description of PCRE's native API, which contains additional -functionality. +documentation for a description of PCRE's native API, which contains much +additional functionality. </P> <P> The functions described here are just wrapper functions that ultimately call @@ -95,6 +95,11 @@ about the compiled expression. The argument <i>cflags</i> is either zero, or contains one or more of the bits defined by the following macros: <pre> + REG_DOTALL +</pre> +The PCRE_DOTALL option is set when the expression is passed for compilation to +the native function. Note that REG_DOTALL is not part of the POSIX standard. +<pre> REG_ICASE </pre> The PCRE_CASELESS option is set when the expression is passed for compilation @@ -203,16 +208,16 @@ memory, after which <i>preg</i> may no longer be used as a compiled expression. </P> <br><a name="SEC8" href="#TOC1">AUTHOR</a><br> <P> -Philip Hazel <ph10@cam.ac.uk> +Philip Hazel <br> University Computing Service, <br> Cambridge CB2 3QG, England. </P> <P> -Last updated: 07 September 2004 +Last updated: 28 February 2005 <br> -Copyright © 1997-2004 University of Cambridge. +Copyright © 1997-2005 University of Cambridge. <p> Return to the <a href="index.html">PCRE index page</a>. </p> diff --git a/doc/html/pcreprecompile.html b/doc/html/pcreprecompile.html index f1c109e..10be496 100644 --- a/doc/html/pcreprecompile.html +++ b/doc/html/pcreprecompile.html @@ -88,16 +88,17 @@ return a non-NULL value before trying to save the study data. <br><a name="SEC3" href="#TOC1">RE-USING A PRECOMPILED PATTERN</a><br> <P> Re-using a precompiled pattern is straightforward. Having reloaded it into main -memory, you pass its pointer to <b>pcre_exec()</b> in the usual way. This should -work even on another host, and even if that host has the opposite endianness to -the one where the pattern was compiled. +memory, you pass its pointer to <b>pcre_exec()</b> or <b>pcre_dfa_exec()</b> in +the usual way. This should work even on another host, and even if that host has +the opposite endianness to the one where the pattern was compiled. </P> <P> However, if you passed a pointer to custom character tables when the pattern was compiled (the <i>tableptr</i> argument of <b>pcre_compile()</b>), you must -now pass a similar pointer to <b>pcre_exec()</b>, because the value saved with -the compiled pattern will obviously be nonsense. A field in a -<b>pcre_extra()</b> block is used to pass this data, as described in the +now pass a similar pointer to <b>pcre_exec()</b> or <b>pcre_dfa_exec()</b>, +because the value saved with the compiled pattern will obviously be nonsense. A +field in a <b>pcre_extra()</b> block is used to pass this data, as described in +the <a href="pcreapi.html#extradata">section on matching a pattern</a> in the <a href="pcreapi.html"><b>pcreapi</b></a> @@ -114,7 +115,8 @@ If you saved study data with the compiled pattern, you need to create your own <b>pcre_extra</b> data block and set the <i>study_data</i> field to point to the reloaded study data. You must also set the PCRE_EXTRA_STUDY_DATA bit in the <i>flags</i> field to indicate that study data is present. Then pass the -<b>pcre_extra</b> block to <b>pcre_exec()</b> in the usual way. +<b>pcre_extra</b> block to <b>pcre_exec()</b> or <b>pcre_dfa_exec()</b> in the +usual way. </P> <br><a name="SEC4" href="#TOC1">COMPATIBILITY WITH DIFFERENT PCRE RELEASES</a><br> <P> @@ -125,9 +127,9 @@ advertised), you will have to recompile them for release 5.0. However, from now on, it should be possible to make changes in a compabible manner. </P> <P> -Last updated: 10 September 2004 +Last updated: 28 February 2005 <br> -Copyright © 1997-2004 University of Cambridge. +Copyright © 1997-2005 University of Cambridge. <p> Return to the <a href="index.html">PCRE index page</a>. </p> diff --git a/doc/html/pcretest.html b/doc/html/pcretest.html index d82dfcc..c43c8cb 100644 --- a/doc/html/pcretest.html +++ b/doc/html/pcretest.html @@ -18,14 +18,17 @@ man page, in case the conversion went wrong. <li><a name="TOC3" href="#SEC3">DESCRIPTION</a> <li><a name="TOC4" href="#SEC4">PATTERN MODIFIERS</a> <li><a name="TOC5" href="#SEC5">DATA LINES</a> -<li><a name="TOC6" href="#SEC6">OUTPUT FROM PCRETEST</a> -<li><a name="TOC7" href="#SEC7">CALLOUTS</a> -<li><a name="TOC8" href="#SEC8">SAVING AND RELOADING COMPILED PATTERNS</a> -<li><a name="TOC9" href="#SEC9">AUTHOR</a> +<li><a name="TOC6" href="#SEC6">THE ALTERNATIVE MATCHING FUNCTION</a> +<li><a name="TOC7" href="#SEC7">DEFAULT OUTPUT FROM PCRETEST</a> +<li><a name="TOC8" href="#SEC8">OUTPUT FROM THE ALTERNATIVE MATCHING FUNCTION</a> +<li><a name="TOC9" href="#SEC9">RESTARTING AFTER A PARTIAL MATCH</a> +<li><a name="TOC10" href="#SEC10">CALLOUTS</a> +<li><a name="TOC11" href="#SEC11">SAVING AND RELOADING COMPILED PATTERNS</a> +<li><a name="TOC12" href="#SEC12">AUTHOR</a> </ul> <br><a name="SEC1" href="#TOC1">SYNOPSIS</a><br> <P> -<b>pcretest [-C] [-d] [-i] [-m] [-o osize] [-p] [-t] [source]</b> +<b>pcretest [-C] [-d] [-dfa] [-i] [-m] [-o osize] [-p] [-t] [source]</b> <b>[destination]</b> </P> <P> @@ -47,12 +50,18 @@ about the optional features that are included, and then exit. </P> <P> <b>-d</b> -Behave as if each regex had the <b>/D</b> (debug) modifier; the internal +Behave as if each regex has the <b>/D</b> (debug) modifier; the internal form is output after compilation. </P> <P> +<b>-dfa</b> +Behave as if each data line contains the \D escape sequence; this causes the +alternative matching function, <b>pcre_dfa_exec()</b>, to be used instead of the +standard <b>pcre_exec()</b> function (more detail is given below). +</P> +<P> <b>-i</b> -Behave as if each regex had the <b>/I</b> modifier; information about the +Behave as if each regex has the <b>/I</b> modifier; information about the compiled pattern is given after compilation. </P> <P> @@ -70,8 +79,9 @@ matching calls by including \O in the data line (see below). </P> <P> <b>-p</b> -Behave as if each regex has <b>/P</b> modifier; the POSIX wrapper API is used -to call PCRE. None of the other options has any effect when <b>-p</b> is set. +Behave as if each regex has the <b>/P</b> modifier; the POSIX wrapper API is +used to call PCRE. None of the other options has any effect when <b>-p</b> is +set. </P> <P> <b>-t</b> @@ -152,6 +162,7 @@ not correspond to anything in Perl: <b>/A</b> PCRE_ANCHORED <b>/C</b> PCRE_AUTO_CALLOUT <b>/E</b> PCRE_DOLLAR_ENDONLY + <b>/f</b> PCRE_FIRSTLINE <b>/N</b> PCRE_NO_AUTO_CAPTURE <b>/U</b> PCRE_UNGREEDY <b>/X</b> PCRE_EXTRA @@ -274,6 +285,8 @@ recognized: \C!n return 1 instead of 0 when callout number n is reached \C!n!m return 1 instead of 0 when callout number n is reached for the nth time \C*n pass the number n (may be negative) as callout data; this is used as the callout return value + \D use the <b>pcre_dfa_exec()</b> match function + \F only shortest match for <b>pcre_dfa_exec()</b> \Gdd call pcre_get_substring() for substring dd after a successful match (number less than 32) \Gname call pcre_get_named_substring() for substring "name" after a successful match (name termin- ated by next non-alphanumeric character) @@ -281,7 +294,8 @@ recognized: \M discover the minimum MATCH_LIMIT setting \N pass the PCRE_NOTEMPTY option to <b>pcre_exec()</b> \Odd set the size of the output vector passed to <b>pcre_exec()</b> to dd (any number of digits) - \P pass the PCRE_PARTIAL option to <b>pcre_exec()</b> + \P pass the PCRE_PARTIAL option to <b>pcre_exec()</b> or <b>pcre_dfa_exec()</b> + \R pass the PCRE_DFA_RESTART option to <b>pcre_dfa_exec()</b> \S output details of memory get/free calls during matching \Z pass the PCRE_NOTEOL option to <b>pcre_exec()</b> \? pass the PCRE_NO_UTF8_CHECK option to <b>pcre_exec()</b> @@ -318,14 +332,35 @@ of the <b>/8</b> modifier on the pattern. It is recognized always. There may be any number of hexadecimal digits inside the braces. The result is from one to six bytes, encoded according to the UTF-8 rules. </P> -<br><a name="SEC6" href="#TOC1">OUTPUT FROM PCRETEST</a><br> +<br><a name="SEC6" href="#TOC1">THE ALTERNATIVE MATCHING FUNCTION</a><br> +<P> +By default, <b>pcretest</b> uses the standard PCRE matching function, +<b>pcre_exec()</b> to match each data line. From release 6.0, PCRE supports an +alternative matching function, <b>pcre_dfa_test()</b>, which operates in a +different way, and has some restrictions. The differences between the two +functions are described in the +<a href="pcrematching.html"><b>pcrematching</b></a> +documentation. +</P> +<P> +If a data line contains the \D escape sequence, or if the command line +contains the <b>-dfa</b> option, the alternative matching function is called. +This function finds all possible matches at a given point. If, however, the \F +escape sequence is present in the data line, it stops after the first match is +found. This is always the shortest possible match. +</P> +<br><a name="SEC7" href="#TOC1">DEFAULT OUTPUT FROM PCRETEST</a><br> +<P> +This section describes the output when the normal matching function, +<b>pcre_exec()</b>, is being used. +</P> <P> When a match succeeds, pcretest outputs the list of captured substrings that <b>pcre_exec()</b> returns, starting with number 0 for the string that matched the whole pattern. Otherwise, it outputs "No match" or "Partial match" when <b>pcre_exec()</b> returns PCRE_ERROR_NOMATCH or PCRE_ERROR_PARTIAL, respectively, and otherwise the PCRE negative error number. Here is an example -of an interactive pcretest run. +of an interactive <b>pcretest</b> run. <pre> $ pcretest PCRE version 5.00 07-Sep-2004 @@ -375,12 +410,62 @@ Note that while patterns can be continued over several lines (a plain ">" prompt is used for continuations), data lines may not. However newlines can be included in data by means of the \n escape. </P> -<br><a name="SEC7" href="#TOC1">CALLOUTS</a><br> +<br><a name="SEC8" href="#TOC1">OUTPUT FROM THE ALTERNATIVE MATCHING FUNCTION</a><br> +<P> +When the alternative matching function, <b>pcre_dfa_exec()</b>, is used (by +means of the \D escape sequence or the <b>-dfa</b> command line option), the +output consists of a list of all the matches that start at the first point in +the subject where there is at least one match. For example: +<pre> + re> /(tang|tangerine|tan)/ + data> yellow tangerine\D + 0: tangerine + 1: tang + 2: tan +</pre> +(Using the normal matching function on this data finds only "tang".) The +longest matching string is always given first (and numbered zero). +</P> +<P> +If \fB/g\P is present on the pattern, the search for further matches resumes +at the end of the longest match. For example: +<pre> + re> /(tang|tangerine|tan)/g + data> yellow tangerine and tangy sultana\D + 0: tangerine + 1: tang + 2: tan + 0: tang + 1: tan + 0: tan +</pre> +Since the matching function does not support substring capture, the escape +sequences that are concerned with captured substrings are not relevant. +</P> +<br><a name="SEC9" href="#TOC1">RESTARTING AFTER A PARTIAL MATCH</a><br> +<P> +When the alternative matching function has given the PCRE_ERROR_PARTIAL return, +indicating that the subject partially matched the pattern, you can restart the +match with additional subject data by means of the \R escape sequence. For +example: +<pre> + re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/ + data> 23ja\P\D + Partial match: 23ja + data> n05\R\D + 0: n05 +</pre> +For further information about partial matching, see the +<a href="pcrepartial.html"><b>pcrepartial</b></a> +documentation. +</P> +<br><a name="SEC10" href="#TOC1">CALLOUTS</a><br> <P> If the pattern contains any callout requests, <b>pcretest</b>'s callout function -is called during matching. By default, it displays the callout number, the -start and current positions in the text at the callout time, and the next -pattern item to be tested. For example, the output +is called during matching. This works with both matching functions. By default, +the called function displays the callout number, the start and current +positions in the text at the callout time, and the next pattern item to be +tested. For example, the output <pre> --->pqrabcdef 0 ^ ^ \d @@ -406,7 +491,7 @@ example: 0: E* </pre> The callout function in <b>pcretest</b> returns zero (carry on matching) by -default, but you can use an \C item in a data line (as described above) to +default, but you can use a \C item in a data line (as described above) to change this. </P> <P> @@ -416,7 +501,7 @@ the <a href="pcrecallout.html"><b>pcrecallout</b></a> documentation. </P> -<br><a name="SEC8" href="#TOC1">SAVING AND RELOADING COMPILED PATTERNS</a><br> +<br><a name="SEC11" href="#TOC1">SAVING AND RELOADING COMPILED PATTERNS</a><br> <P> The facilities described in this section are not available when the POSIX inteface to PCRE is being used, that is, when the <b>/P</b> pattern modifier is @@ -478,18 +563,18 @@ string using a reloaded pattern is likely to cause <b>pcretest</b> to crash. Finally, if you attempt to load a file that is not in the correct format, the result is undefined. </P> -<br><a name="SEC9" href="#TOC1">AUTHOR</a><br> +<br><a name="SEC12" href="#TOC1">AUTHOR</a><br> <P> -Philip Hazel <ph10@cam.ac.uk> +Philip Hazel <br> University Computing Service, <br> Cambridge CB2 3QG, England. </P> <P> -Last updated: 10 September 2004 +Last updated: 28 February 2005 <br> -Copyright © 1997-2004 University of Cambridge. +Copyright © 1997-2005 University of Cambridge. <p> Return to the <a href="index.html">PCRE index page</a>. </p> @@ -6,15 +6,29 @@ PCRE - Perl-compatible regular expressions .sp The PCRE library is a set of functions that implement regular expression pattern matching using the same syntax and semantics as Perl, with just a few -differences. The current implementation of PCRE (release 5.x) corresponds +differences. The current implementation of PCRE (release 6.x) corresponds approximately with Perl 5.8, including support for UTF-8 encoded strings and Unicode general category properties. However, this support has to be explicitly enabled; it is not the default. .P +In addition to the Perl-compatible matching function, PCRE also contains an +alternative matching function that matches the same compiled patterns in a +different way. In certain circumstances, the alternative function has some +advantages. For a discussion of the two matching algorithms, see the +.\" HREF +\fBpcrematching\fP +.\" +page. +.P PCRE is written in C and released as a C library. A number of people have -written wrappers and interfaces of various kinds. A C++ class is included in -these contributions, which can be found in the \fIContrib\fR directory at the -primary FTP site, which is: +written wrappers and interfaces of various kinds. In particular, Google Inc. +have provided a comprehensive C++ wrapper. This is now included as part of the +PCRE distribution. The +.\" HREF +\fBpcrecpp\fP +.\" +page has details of this interface. Other people's contributions can be found +in the \fIContrib\fR directory at the primary FTP site, which is: .sp .\" HTML <a href="ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre"> .\" </a> @@ -43,6 +57,11 @@ available. The features themselves are described in the .\" page. Documentation about building PCRE for various operating systems can be found in the \fBREADME\fP file in the source distribution. +.P +The library contains a number of undocumented internal functions and data +tables that are used by more than one of the exported external functions, but +which are not intended for use by external callers. Their names all begin with +"_pcre_", which hopefully will not provoke any name clashes. . . .SH "USER DOCUMENTATION" @@ -55,23 +74,25 @@ all the sections are concatenated, for ease of searching. The sections are as follows: .sp pcre this document - pcreapi details of PCRE's native API + pcreapi details of PCRE's native C API pcrebuild options for building PCRE pcrecallout details of the callout feature pcrecompat discussion of Perl compatibility + pcrecpp details of the C++ wrapper pcregrep description of the \fBpcregrep\fP command + pcrematching discussion of the two matching algorithms pcrepartial details of the partial matching facility .\" JOIN pcrepattern syntax and semantics of supported regular expressions pcreperform discussion of performance issues - pcreposix the POSIX-compatible API + pcreposix the POSIX-compatible C API pcreprecompile details of saving and re-using precompiled patterns pcresample discussion of the sample program pcretest description of the \fBpcretest\fP testing command .sp In addition, in the "man" and HTML formats, there is a short page for each -library function, listing its arguments and results. +C library function, listing its arguments and results. . . .SH LIMITATIONS @@ -99,9 +120,10 @@ depth of nesting of all kinds of parenthesized subpattern, including capturing subpatterns, assertions, and other types of subpattern, is 200. .P The maximum length of a subject string is the largest positive number that an -integer variable can hold. However, PCRE uses recursion to handle subpatterns -and indefinite repetition. This means that the available stack space may limit -the size of a subject string that can be processed by certain patterns. +integer variable can hold. However, when using the traditional matching +function, PCRE uses recursion to handle subpatterns and indefinite repetition. +This means that the available stack space may limit the size of a subject +string that can be processed by certain patterns. .sp .\" HTML <a name="utf8support"></a> . @@ -167,7 +189,8 @@ bytes, for example: \ex{100}{3}. 5. The dot metacharacter matches one UTF-8 character instead of a single byte. .P 6. The escape sequence \eC can be used to match a single byte in UTF-8 mode, -but its use can lead to some strange effects. +but its use can lead to some strange effects. This facility is not available in +the alternative matching function, \fBpcre_dfa_exec()\fP. .P 7. The character escapes \eb, \eB, \ed, \eD, \es, \eS, \ew, and \eW correctly test characters of any code value, but the characters that PCRE recognizes as @@ -190,15 +213,17 @@ values. .SH AUTHOR .rs .sp -Philip Hazel <ph10@cam.ac.uk> +Philip Hazel .br University Computing Service, .br Cambridge CB2 3QG, England. -.br -Phone: +44 1223 334714 +.P +Putting an actual email address here seems to have been a spam magnet, so I've +taken it away. If you want to email me, use my initial and surname, separated +by a dot, at the domain ucs.cam.ac.uk. .sp .in 0 -Last updated: 09 September 2004 +Last updated: 07 March 2005 .br -Copyright (c) 1997-2004 University of Cambridge. +Copyright (c) 1997-2005 University of Cambridge. diff --git a/doc/pcre.txt b/doc/pcre.txt index fdf0d6f..9de0d62 100644 --- a/doc/pcre.txt +++ b/doc/pcre.txt @@ -6,26 +6,33 @@ synopses of each function in the library have not been included. There are separate text files for the pcregrep and pcretest commands. ----------------------------------------------------------------------------- -PCRE(3) PCRE(3) - NAME PCRE - Perl-compatible regular expressions + INTRODUCTION The PCRE library is a set of functions that implement regular expres- sion pattern matching using the same syntax and semantics as Perl, with just a few differences. The current implementation of PCRE (release - 5.x) corresponds approximately with Perl 5.8, including support for + 6.x) corresponds approximately with Perl 5.8, including support for UTF-8 encoded strings and Unicode general category properties. However, this support has to be explicitly enabled; it is not the default. + In addition to the Perl-compatible matching function, PCRE also con- + tains an alternative matching function that matches the same compiled + patterns in a different way. In certain circumstances, the alternative + function has some advantages. For a discussion of the two matching + algorithms, see the pcrematching page. + PCRE is written in C and released as a C library. A number of people - have written wrappers and interfaces of various kinds. A C++ class is - included in these contributions, which can be found in the Contrib - directory at the primary FTP site, which is: + have written wrappers and interfaces of various kinds. In particular, + Google Inc. have provided a comprehensive C++ wrapper. This is now + included as part of the PCRE distribution. The pcrecpp page has details + of this interface. Other people's contributions can be found in the + Contrib directory at the primary FTP site, which is: ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre @@ -40,6 +47,12 @@ INTRODUCTION ing PCRE for various operating systems can be found in the README file in the source distribution. + The library contains a number of undocumented internal functions and + data tables that are used by more than one of the exported external + functions, but which are not intended for use by external callers. + Their names all begin with "_pcre_", which hopefully will not provoke + any name clashes. + USER DOCUMENTATION @@ -50,22 +63,24 @@ USER DOCUMENTATION of searching. The sections are as follows: pcre this document - pcreapi details of PCRE's native API + pcreapi details of PCRE's native C API pcrebuild options for building PCRE pcrecallout details of the callout feature pcrecompat discussion of Perl compatibility + pcrecpp details of the C++ wrapper pcregrep description of the pcregrep command + pcrematching discussion of the two matching algorithms pcrepartial details of the partial matching facility pcrepattern syntax and semantics of supported regular expressions pcreperform discussion of performance issues - pcreposix the POSIX-compatible API + pcreposix the POSIX-compatible C API pcreprecompile details of saving and re-using precompiled patterns pcresample discussion of the sample program pcretest description of the pcretest testing command In addition, in the "man" and HTML formats, there is a short page for - each library function, listing its arguments and results. + each C library function, listing its arguments and results. LIMITATIONS @@ -90,70 +105,71 @@ LIMITATIONS tern, is 200. The maximum length of a subject string is the largest positive number - that an integer variable can hold. However, PCRE uses recursion to han- - dle subpatterns and indefinite repetition. This means that the avail- - able stack space may limit the size of a subject string that can be - processed by certain patterns. + that an integer variable can hold. However, when using the traditional + matching function, PCRE uses recursion to handle subpatterns and indef- + inite repetition. This means that the available stack space may limit + the size of a subject string that can be processed by certain patterns. UTF-8 AND UNICODE PROPERTY SUPPORT - From release 3.3, PCRE has had some support for character strings - encoded in the UTF-8 format. For release 4.0 this was greatly extended - to cover most common requirements, and in release 5.0 additional sup- + From release 3.3, PCRE has had some support for character strings + encoded in the UTF-8 format. For release 4.0 this was greatly extended + to cover most common requirements, and in release 5.0 additional sup- port for Unicode general category properties was added. - In order process UTF-8 strings, you must build PCRE to include UTF-8 - support in the code, and, in addition, you must call pcre_compile() - with the PCRE_UTF8 option flag. When you do this, both the pattern and - any subject strings that are matched against it are treated as UTF-8 + In order process UTF-8 strings, you must build PCRE to include UTF-8 + support in the code, and, in addition, you must call pcre_compile() + with the PCRE_UTF8 option flag. When you do this, both the pattern and + any subject strings that are matched against it are treated as UTF-8 strings instead of just strings of bytes. - If you compile PCRE with UTF-8 support, but do not use it at run time, - the library will be a bit bigger, but the additional run time overhead - is limited to testing the PCRE_UTF8 flag in several places, so should + If you compile PCRE with UTF-8 support, but do not use it at run time, + the library will be a bit bigger, but the additional run time overhead + is limited to testing the PCRE_UTF8 flag in several places, so should not be very large. If PCRE is built with Unicode character property support (which implies - UTF-8 support), the escape sequences \p{..}, \P{..}, and \X are sup- + UTF-8 support), the escape sequences \p{..}, \P{..}, and \X are sup- ported. The available properties that can be tested are limited to the - general category properties such as Lu for an upper case letter or Nd - for a decimal number. A full list is given in the pcrepattern documen- + general category properties such as Lu for an upper case letter or Nd + for a decimal number. A full list is given in the pcrepattern documen- tation. The PCRE library is increased in size by about 90K when Unicode property support is included. The following comments apply when PCRE is running in UTF-8 mode: - 1. When you set the PCRE_UTF8 flag, the strings passed as patterns and - subjects are checked for validity on entry to the relevant functions. + 1. When you set the PCRE_UTF8 flag, the strings passed as patterns and + subjects are checked for validity on entry to the relevant functions. If an invalid UTF-8 string is passed, an error return is given. In some - situations, you may already know that your strings are valid, and + situations, you may already know that your strings are valid, and therefore want to skip these checks in order to improve performance. If - you set the PCRE_NO_UTF8_CHECK flag at compile time or at run time, - PCRE assumes that the pattern or subject it is given (respectively) - contains only valid UTF-8 codes. In this case, it does not diagnose an - invalid UTF-8 string. If you pass an invalid UTF-8 string to PCRE when - PCRE_NO_UTF8_CHECK is set, the results are undefined. Your program may + you set the PCRE_NO_UTF8_CHECK flag at compile time or at run time, + PCRE assumes that the pattern or subject it is given (respectively) + contains only valid UTF-8 codes. In this case, it does not diagnose an + invalid UTF-8 string. If you pass an invalid UTF-8 string to PCRE when + PCRE_NO_UTF8_CHECK is set, the results are undefined. Your program may crash. 2. In a pattern, the escape sequence \x{...}, where the contents of the - braces is a string of hexadecimal digits, is interpreted as a UTF-8 - character whose code number is the given hexadecimal number, for exam- - ple: \x{1234}. If a non-hexadecimal digit appears between the braces, + braces is a string of hexadecimal digits, is interpreted as a UTF-8 + character whose code number is the given hexadecimal number, for exam- + ple: \x{1234}. If a non-hexadecimal digit appears between the braces, the item is not recognized. This escape sequence can be used either as a literal, or within a character class. - 3. The original hexadecimal escape sequence, \xhh, matches a two-byte + 3. The original hexadecimal escape sequence, \xhh, matches a two-byte UTF-8 character if the value is greater than 127. - 4. Repeat quantifiers apply to complete UTF-8 characters, not to indi- + 4. Repeat quantifiers apply to complete UTF-8 characters, not to indi- vidual bytes, for example: \x{100}{3}. - 5. The dot metacharacter matches one UTF-8 character instead of a sin- + 5. The dot metacharacter matches one UTF-8 character instead of a sin- gle byte. - 6. The escape sequence \C can be used to match a single byte in UTF-8 - mode, but its use can lead to some strange effects. + 6. The escape sequence \C can be used to match a single byte in UTF-8 + mode, but its use can lead to some strange effects. This facility is + not available in the alternative matching function, pcre_dfa_exec(). 7. The character escapes \b, \B, \d, \D, \s, \S, \w, and \W correctly test characters of any code value, but the characters that PCRE recog- @@ -177,22 +193,24 @@ UTF-8 AND UNICODE PROPERTY SUPPORT AUTHOR - Philip Hazel <ph10@cam.ac.uk> + Philip Hazel University Computing Service, Cambridge CB2 3QG, England. - Phone: +44 1223 334714 -Last updated: 09 September 2004 -Copyright (c) 1997-2004 University of Cambridge. ------------------------------------------------------------------------------ + Putting an actual email address here seems to have been a spam magnet, + so I've taken it away. If you want to email me, use my initial and sur- + name, separated by a dot, at the domain ucs.cam.ac.uk. -PCRE(3) PCRE(3) +Last updated: 07 March 2005 +Copyright (c) 1997-2005 University of Cambridge. +----------------------------------------------------------------------------- NAME PCRE - Perl-compatible regular expressions + PCRE BUILD-TIME OPTIONS This document describes the optional features of PCRE that can be @@ -287,16 +305,18 @@ POSIX MALLOC USAGE LIMITING PCRE RESOURCE USAGE Internally, PCRE has a function called match(), which it calls repeat- - edly (possibly recursively) when matching a pattern. By controlling the - maximum number of times this function may be called during a single - matching operation, a limit can be placed on the resources used by a - single call to pcre_exec(). The limit can be changed at run time, as - described in the pcreapi documentation. The default is 10 million, but - this can be changed by adding a setting such as + edly (possibly recursively) when matching a pattern with the + pcre_exec() function. By controlling the maximum number of times this + function may be called during a single matching operation, a limit can + be placed on the resources used by a single call to pcre_exec(). The + limit can be changed at run time, as described in the pcreapi documen- + tation. The default is 10 million, but this can be changed by adding a + setting such as --with-match-limit=500000 - to the configure command. + to the configure command. This setting has no effect on the + pcre_dfa_exec() matching function. HANDLING VERY LARGE PATTERNS @@ -324,14 +344,14 @@ HANDLING VERY LARGE PATTERNS AVOIDING EXCESSIVE STACK USAGE - PCRE implements backtracking while matching by making recursive calls - to an internal function called match(). In environments where the size - of the stack is limited, this can severely limit PCRE's operation. (The - Unix environment does not usually suffer from this problem.) An alter- - native approach that uses memory from the heap to remember data, - instead of using recursive function calls, has been implemented to work - round this problem. If you want to build a version of PCRE that works - this way, add + When matching with the pcre_exec() function, PCRE implements backtrack- + ing by making recursive calls to an internal function called match(). + In environments where the size of the stack is limited, this can se- + verely limit PCRE's operation. (The Unix environment does not usually + suffer from this problem.) An alternative approach that uses memory + from the heap to remember data, instead of using recursive function + calls, has been implemented to work round this problem. If you want to + build a version of PCRE that works this way, add --disable-stack-for-recursion @@ -342,31 +362,197 @@ AVOIDING EXCESSIVE STACK USAGE the blocks are always freed in reverse order. A calling program might be able to implement optimized functions that perform better than the standard malloc() and free() functions. PCRE runs noticeably more - slowly when built in this way. + slowly when built in this way. This option affects only the pcre_exec() + function; it is not relevant for the the pcre_dfa_exec() function. USING EBCDIC CODE - PCRE assumes by default that it will run in an environment where the - character code is ASCII (or Unicode, which is a superset of ASCII). - PCRE can, however, be compiled to run in an EBCDIC environment by + PCRE assumes by default that it will run in an environment where the + character code is ASCII (or Unicode, which is a superset of ASCII). + PCRE can, however, be compiled to run in an EBCDIC environment by adding --enable-ebcdic to the configure command. -Last updated: 09 September 2004 -Copyright (c) 1997-2004 University of Cambridge. +Last updated: 28 February 2005 +Copyright (c) 1997-2005 University of Cambridge. ----------------------------------------------------------------------------- -PCRE(3) PCRE(3) + + +NAME + PCRE - Perl-compatible regular expressions + + +PCRE MATCHING ALGORITHMS + + This document describes the two different algorithms that are available + in PCRE for matching a compiled regular expression against a given sub- + ject string. The "standard" algorithm is the one provided by the + pcre_exec() function. This works in the same was as Perl's matching + function, and provides a Perl-compatible matching operation. + + An alternative algorithm is provided by the pcre_dfa_exec() function; + this operates in a different way, and is not Perl-compatible. It has + advantages and disadvantages compared with the standard algorithm, and + these are described below. + + When there is only one possible way in which a given subject string can + match a pattern, the two algorithms give the same answer. A difference + arises, however, when there are multiple possibilities. For example, if + the pattern + + ^<.*> + + is matched against the string + + <something> <something else> <something further> + + there are three possible answers. The standard algorithm finds only one + of them, whereas the DFA algorithm finds all three. + + +REGULAR EXPRESSIONS AS TREES + + The set of strings that are matched by a regular expression can be rep- + resented as a tree structure. An unlimited repetition in the pattern + makes the tree of infinite size, but it is still a tree. Matching the + pattern to a given subject string (from a given starting point) can be + thought of as a search of the tree. There are two standard ways to + search a tree: depth-first and breadth-first, and these correspond to + the two matching algorithms provided by PCRE. + + +THE STANDARD MATCHING ALGORITHM + + In the terminology of Jeffrey Friedl's book Mastering Regular Expres- + sions, the standard algorithm is an "NFA algorithm". It conducts a + depth-first search of the pattern tree. That is, it proceeds along a + single path through the tree, checking that the subject matches what is + required. When there is a mismatch, the algorithm tries any alterna- + tives at the current point, and if they all fail, it backs up to the + previous branch point in the tree, and tries the next alternative + branch at that level. This often involves backing up (moving to the + left) in the subject string as well. The order in which repetition + branches are tried is controlled by the greedy or ungreedy nature of + the quantifier. + + If a leaf node is reached, a matching string has been found, and at + that point the algorithm stops. Thus, if there is more than one possi- + ble match, this algorithm returns the first one that it finds. Whether + this is the shortest, the longest, or some intermediate length depends + on the way the greedy and ungreedy repetition quantifiers are specified + in the pattern. + + Because it ends up with a single path through the tree, it is rela- + tively straightforward for this algorithm to keep track of the sub- + strings that are matched by portions of the pattern in parentheses. + This provides support for capturing parentheses and back references. + + +THE DFA MATCHING ALGORITHM + + DFA stands for "deterministic finite automaton", but you do not need to + understand the origins of that name. This algorithm conducts a breadth- + first search of the tree. Starting from the first matching point in the + subject, it scans the subject string from left to right, once, charac- + ter by character, and as it does this, it remembers all the paths + through the tree that represent valid matches. + + The scan continues until either the end of the subject is reached, or + there are no more unterminated paths. At this point, terminated paths + represent the different matching possibilities (if there are none, the + match has failed). Thus, if there is more than one possible match, + this algorithm finds all of them, and in particular, it finds the long- + est. In PCRE, there is an option to stop the algorithm after the first + match (which is necessarily the shortest) has been found. + + Note that all the matches that are found start at the same point in the + subject. If the pattern + + cat(er(pillar)?) + + is matched against the string "the caterpillar catchment", the result + will be the three strings "cat", "cater", and "caterpillar" that start + at the fourth character of the subject. The algorithm does not automat- + ically move on to find matches that start at later positions. + + There are a number of features of PCRE regular expressions that are not + supported by the DFA matching algorithm. They are as follows: + + 1. Because the algorithm finds all possible matches, the greedy or + ungreedy nature of repetition quantifiers is not relevant. Greedy and + ungreedy quantifiers are treated in exactly the same way. + + 2. When dealing with multiple paths through the tree simultaneously, it + is not straightforward to keep track of captured substrings for the + different matching possibilities, and PCRE's implementation of this + algorithm does not attempt to do this. This means that no captured sub- + strings are available. + + 3. Because no substrings are captured, back references within the pat- + tern are not supported, and cause errors if encountered. + + 4. For the same reason, conditional expressions that use a backrefer- + ence as the condition are not supported. + + 5. Callouts are supported, but the value of the capture_top field is + always 1, and the value of the capture_last field is always -1. + + 6. The \C escape sequence, which (in the standard algorithm) matches a + single byte, even in UTF-8 mode, is not supported because the DFA algo- + rithm moves through the subject string one character at a time, for all + active paths through the tree. + + +ADVANTAGES OF THE DFA ALGORITHM + + Using the DFA matching algorithm provides the following advantages: + + 1. All possible matches (at a single point in the subject) are automat- + ically found, and in particular, the longest match is found. To find + more than one match using the standard algorithm, you have to do kludgy + things with callouts. + + 2. There is much better support for partial matching. The restrictions + on the content of the pattern that apply when using the standard algo- + rithm for partial matching do not apply to the DFA algorithm. For non- + anchored patterns, the starting position of a partial match is avail- + able. + + 3. Because the DFA algorithm scans the subject string just once, and + never needs to backtrack, it is possible to pass very long subject + strings to the matching function in several pieces, checking for par- + tial matching each time. + + +DISADVANTAGES OF THE DFA ALGORITHM + + The DFA algorithm suffers from a number of disadvantages: + + 1. It is substantially slower than the standard algorithm. This is + partly because it has to search for all possible matches, but is also + because it is less susceptible to optimization. + + 2. Capturing parentheses and back references are not supported. + + 3. The "atomic group" feature of PCRE regular expressions is supported, + but does not provide the advantage that it does for the standard algo- + rithm. + +Last updated: 28 February 2005 +Copyright (c) 1997-2005 University of Cambridge. +----------------------------------------------------------------------------- NAME PCRE - Perl-compatible regular expressions + PCRE NATIVE API #include <pcre.h> @@ -375,6 +561,11 @@ PCRE NATIVE API const char **errptr, int *erroffset, const unsigned char *tableptr); + pcre *pcre_compile2(const char *pattern, int options, + int *errorcodeptr, + const char **errptr, int *erroffset, + const unsigned char *tableptr); + pcre_extra *pcre_study(const pcre *code, int options, const char **errptr); @@ -382,6 +573,11 @@ PCRE NATIVE API const char *subject, int length, int startoffset, int options, int *ovector, int ovecsize); + int pcre_dfa_exec(const pcre *code, const pcre_extra *extra, + const char *subject, int length, int startoffset, + int options, int *ovector, int ovecsize, + int *workspace, int wscount); + int pcre_copy_named_substring(const pcre *code, const char *subject, int *ovector, int stringcount, const char *stringname, @@ -417,6 +613,8 @@ PCRE NATIVE API int pcre_info(const pcre *code, int *optptr, int *firstcharptr); + int pcre_refcount(pcre *code, int adjust); + int pcre_config(int what, void *where); char *pcre_version(void); @@ -436,25 +634,36 @@ PCRE API OVERVIEW PCRE has its own native API, which is described in this document. There is also a set of wrapper functions that correspond to the POSIX regular - expression API. These are described in the pcreposix documentation. + expression API. These are described in the pcreposix documentation. + Both of these APIs define a set of C function calls. A C++ wrapper is + distributed with PCRE. It is documented in the pcrecpp page. - The native API function prototypes are defined in the header file - pcre.h, and on Unix systems the library itself is called libpcre. It + The native API C function prototypes are defined in the header file + pcre.h, and on Unix systems the library itself is called libpcre. It can normally be accessed by adding -lpcre to the command for linking an application that uses PCRE. The header file defines the macros PCRE_MAJOR and PCRE_MINOR to contain the major and minor release num- bers for the library. Applications can use these to include support for different releases of PCRE. - The functions pcre_compile(), pcre_study(), and pcre_exec() are used - for compiling and matching regular expressions. A sample program that - demonstrates the simplest way of using them is provided in the file - called pcredemo.c in the source distribution. The pcresample documenta- - tion describes how to run it. - - In addition to the main compiling and matching functions, there are - convenience functions for extracting captured substrings from a matched - subject string. They are: + The functions pcre_compile(), pcre_compile2(), pcre_study(), and + pcre_exec() are used for compiling and matching regular expressions in + a Perl-compatible manner. A sample program that demonstrates the sim- + plest way of using them is provided in the file called pcredemo.c in + the source distribution. The pcresample documentation describes how to + run it. + + A second matching function, pcre_dfa_exec(), which is not Perl-compati- + ble, is also provided. This uses a different algorithm for the match- + ing. This allows it to find all possible matches (at a given point in + the subject), not just one. However, this algorithm does not return + captured substrings. A description of the two matching algorithms and + their advantages and disadvantages is given in the pcrematching docu- + mentation. + + In addition to the main compiling and matching functions, there are + convenience functions for extracting captured substrings from a subject + string that is matched by pcre_exec(). They are: pcre_copy_substring() pcre_copy_named_substring() @@ -466,11 +675,12 @@ PCRE API OVERVIEW pcre_free_substring() and pcre_free_substring_list() are also provided, to free the memory used for extracted strings. - The function pcre_maketables() is used to build a set of character - tables in the current locale for passing to pcre_compile() or - pcre_exec(). This is an optional facility that is provided for spe- - cialist use. Most commonly, no special tables are passed, in which case - internal tables that are generated when PCRE is built are used. + The function pcre_maketables() is used to build a set of character + tables in the current locale for passing to pcre_compile(), + pcre_exec(), or pcre_dfa_exec(). This is an optional facility that is + provided for specialist use. Most commonly, no special tables are + passed, in which case internal tables that are generated when PCRE is + built are used. The function pcre_fullinfo() is used to find out information about a compiled pattern; pcre_info() is an obsolete version that returns only @@ -478,6 +688,10 @@ PCRE API OVERVIEW patibility. The function pcre_version() returns a pointer to a string containing the version of PCRE and its date of release. + The function pcre_refcount() maintains a reference count in a data + block containing a compiled pattern. This is provided for the benefit + of object-oriented applications. + The global variables pcre_malloc and pcre_free initially contain the entry points of the standard malloc() and free() functions, respec- tively. PCRE calls the memory management functions via these variables, @@ -487,28 +701,28 @@ PCRE API OVERVIEW The global variables pcre_stack_malloc and pcre_stack_free are also indirections to memory management functions. These special functions are used only when PCRE is compiled to use the heap for remembering - data, instead of recursive function calls. This is a non-standard way - of building PCRE, for use in environments that have limited stacks. - Because of the greater use of memory management, it runs more slowly. - Separate functions are provided so that special-purpose external code - can be used for this case. When used, these functions are always called - in a stack-like manner (last obtained, first freed), and always for - memory blocks of the same size. + data, instead of recursive function calls, when running the pcre_exec() + function. This is a non-standard way of building PCRE, for use in envi- + ronments that have limited stacks. Because of the greater use of memory + management, it runs more slowly. Separate functions are provided so + that special-purpose external code can be used for this case. When + used, these functions are always called in a stack-like manner (last + obtained, first freed), and always for memory blocks of the same size. The global variable pcre_callout initially contains NULL. It can be set - by the caller to a "callout" function, which PCRE will then call at - specified points during a matching operation. Details are given in the + by the caller to a "callout" function, which PCRE will then call at + specified points during a matching operation. Details are given in the pcrecallout documentation. MULTITHREADING - The PCRE functions can be used in multi-threading applications, with + The PCRE functions can be used in multi-threading applications, with the proviso that the memory management functions pointed to by pcre_malloc, pcre_free, pcre_stack_malloc, and pcre_stack_free, and the callout function pointed to by pcre_callout, are shared by all threads. - The compiled form of a regular expression is not altered during match- + The compiled form of a regular expression is not altered during match- ing, so the same compiled pattern can safely be used by several threads at once. @@ -516,8 +730,8 @@ MULTITHREADING SAVING PRECOMPILED PATTERNS FOR LATER USE The compiled form of a regular expression can be saved and re-used at a - later time, possibly by a different program, and even on a host other - than the one on which it was compiled. Details are given in the + later time, possibly by a different program, and even on a host other + than the one on which it was compiled. Details are given in the pcreprecompile documentation. @@ -525,63 +739,63 @@ CHECKING BUILD-TIME OPTIONS int pcre_config(int what, void *where); - The function pcre_config() makes it possible for a PCRE client to dis- + The function pcre_config() makes it possible for a PCRE client to dis- cover which optional features have been compiled into the PCRE library. - The pcrebuild documentation has more details about these optional fea- + The pcrebuild documentation has more details about these optional fea- tures. - The first argument for pcre_config() is an integer, specifying which + The first argument for pcre_config() is an integer, specifying which information is required; the second argument is a pointer to a variable - into which the information is placed. The following information is + into which the information is placed. The following information is available: PCRE_CONFIG_UTF8 - The output is an integer that is set to one if UTF-8 support is avail- + The output is an integer that is set to one if UTF-8 support is avail- able; otherwise it is set to zero. PCRE_CONFIG_UNICODE_PROPERTIES - The output is an integer that is set to one if support for Unicode + The output is an integer that is set to one if support for Unicode character properties is available; otherwise it is set to zero. PCRE_CONFIG_NEWLINE - The output is an integer that is set to the value of the code that is - used for the newline character. It is either linefeed (10) or carriage - return (13), and should normally be the standard character for your + The output is an integer that is set to the value of the code that is + used for the newline character. It is either linefeed (10) or carriage + return (13), and should normally be the standard character for your operating system. PCRE_CONFIG_LINK_SIZE - The output is an integer that contains the number of bytes used for + The output is an integer that contains the number of bytes used for internal linkage in compiled regular expressions. The value is 2, 3, or - 4. Larger values allow larger regular expressions to be compiled, at - the expense of slower matching. The default value of 2 is sufficient - for all but the most massive patterns, since it allows the compiled + 4. Larger values allow larger regular expressions to be compiled, at + the expense of slower matching. The default value of 2 is sufficient + for all but the most massive patterns, since it allows the compiled pattern to be up to 64K in size. PCRE_CONFIG_POSIX_MALLOC_THRESHOLD - The output is an integer that contains the threshold above which the - POSIX interface uses malloc() for output vectors. Further details are + The output is an integer that contains the threshold above which the + POSIX interface uses malloc() for output vectors. Further details are given in the pcreposix documentation. PCRE_CONFIG_MATCH_LIMIT The output is an integer that gives the default limit for the number of - internal matching function calls in a pcre_exec() execution. Further + internal matching function calls in a pcre_exec() execution. Further details are given with pcre_exec() below. PCRE_CONFIG_STACKRECURSE - The output is an integer that is set to one if internal recursion is - implemented by recursive function calls that use the stack to remember - their state. This is the usual way that PCRE is compiled. The output is - zero if PCRE was compiled to use blocks of data on the heap instead of - recursive function calls. In this case, pcre_stack_malloc and - pcre_stack_free are called to manage memory blocks on the heap, thus - avoiding the use of the stack. + The output is an integer that is set to one if internal recursion when + running pcre_exec() is implemented by recursive function calls that use + the stack to remember their state. This is the usual way that PCRE is + compiled. The output is zero if PCRE was compiled to use blocks of data + on the heap instead of recursive function calls. In this case, + pcre_stack_malloc and pcre_stack_free are called to manage memory + blocks on the heap, thus avoiding the use of the stack. COMPILING A PATTERN @@ -590,38 +804,52 @@ COMPILING A PATTERN const char **errptr, int *erroffset, const unsigned char *tableptr); - The function pcre_compile() is called to compile a pattern into an - internal form. The pattern is a C string terminated by a binary zero, - and is passed in the pattern argument. A pointer to a single block of - memory that is obtained via pcre_malloc is returned. This contains the - compiled code and related data. The pcre type is defined for the - returned block; this is a typedef for a structure whose contents are - not externally defined. It is up to the caller to free the memory when - it is no longer required. + pcre *pcre_compile2(const char *pattern, int options, + int *errorcodeptr, + const char **errptr, int *erroffset, + const unsigned char *tableptr); + + Either of the functions pcre_compile() or pcre_compile2() can be called + to compile a pattern into an internal form. The only difference between + the two interfaces is that pcre_compile2() has an additional argument, + errorcodeptr, via which a numerical error code can be returned. + + The pattern is a C string terminated by a binary zero, and is passed in + the pattern argument. A pointer to a single block of memory that is + obtained via pcre_malloc is returned. This contains the compiled code + and related data. The pcre type is defined for the returned block; this + is a typedef for a structure whose contents are not externally defined. + It is up to the caller to free the memory when it is no longer + required. - Although the compiled code of a PCRE regex is relocatable, that is, it + Although the compiled code of a PCRE regex is relocatable, that is, it does not depend on memory location, the complete pcre data block is not - fully relocatable, because it may contain a copy of the tableptr argu- + fully relocatable, because it may contain a copy of the tableptr argu- ment, which is an address (see below). The options argument contains independent bits that affect the compila- - tion. It should be zero if no options are required. The available - options are described below. Some of them, in particular, those that - are compatible with Perl, can also be set and unset from within the - pattern (see the detailed description in the pcrepattern documenta- - tion). For these options, the contents of the options argument speci- - fies their initial settings at the start of compilation and execution. - The PCRE_ANCHORED option can be set at the time of matching as well as + tion. It should be zero if no options are required. The available + options are described below. Some of them, in particular, those that + are compatible with Perl, can also be set and unset from within the + pattern (see the detailed description in the pcrepattern documenta- + tion). For these options, the contents of the options argument speci- + fies their initial settings at the start of compilation and execution. + The PCRE_ANCHORED option can be set at the time of matching as well as at compile time. If errptr is NULL, pcre_compile() returns NULL immediately. Otherwise, - if compilation of a pattern fails, pcre_compile() returns NULL, and + if compilation of a pattern fails, pcre_compile() returns NULL, and sets the variable pointed to by errptr to point to a textual error mes- - sage. The offset from the start of the pattern to the character where - the error was discovered is placed in the variable pointed to by - erroffset, which must not be NULL. If it is, an immediate error is + sage. The offset from the start of the pattern to the character where + the error was discovered is placed in the variable pointed to by + erroffset, which must not be NULL. If it is, an immediate error is given. + If pcre_compile2() is used instead of pcre_compile(), and the error- + codeptr argument is not NULL, a non-zero error code number is returned + via this argument in the event of an error. This is in addition to the + textual error message. Error codes and messages are listed below. + If the final argument, tableptr, is NULL, PCRE uses a default set of character tables that are built when PCRE is compiled, using the default C locale. Otherwise, tableptr must be an address that is the @@ -664,139 +892,206 @@ COMPILING A PATTERN If this bit is set, letters in the pattern match both upper and lower case letters. It is equivalent to Perl's /i option, and it can be - changed within a pattern by a (?i) option setting. When running in - UTF-8 mode, case support for high-valued characters is available only - when PCRE is built with Unicode character property support. + changed within a pattern by a (?i) option setting. In UTF-8 mode, PCRE + always understands the concept of case for characters whose values are + less than 128, so caseless matching is always possible. For characters + with higher values, the concept of case is supported if PCRE is com- + piled with Unicode property support, but not otherwise. If you want to + use caseless matching for characters 128 and above, you must ensure + that PCRE is compiled with Unicode property support as well as with + UTF-8 support. PCRE_DOLLAR_ENDONLY - If this bit is set, a dollar metacharacter in the pattern matches only - at the end of the subject string. Without this option, a dollar also - matches immediately before the final character if it is a newline (but - not before any other newlines). The PCRE_DOLLAR_ENDONLY option is + If this bit is set, a dollar metacharacter in the pattern matches only + at the end of the subject string. Without this option, a dollar also + matches immediately before the final character if it is a newline (but + not before any other newlines). The PCRE_DOLLAR_ENDONLY option is ignored if PCRE_MULTILINE is set. There is no equivalent to this option in Perl, and no way to set it within a pattern. PCRE_DOTALL If this bit is set, a dot metacharater in the pattern matches all char- - acters, including newlines. Without it, newlines are excluded. This - option is equivalent to Perl's /s option, and it can be changed within - a pattern by a (?s) option setting. A negative class such as [^a] - always matches a newline character, independent of the setting of this + acters, including newlines. Without it, newlines are excluded. This + option is equivalent to Perl's /s option, and it can be changed within + a pattern by a (?s) option setting. A negative class such as [^a] + always matches a newline character, independent of the setting of this option. PCRE_EXTENDED - If this bit is set, whitespace data characters in the pattern are - totally ignored except when escaped or inside a character class. - Whitespace does not include the VT character (code 11). In addition, - characters between an unescaped # outside a character class and the - next newline character, inclusive, are also ignored. This is equivalent - to Perl's /x option, and it can be changed within a pattern by a (?x) + If this bit is set, whitespace data characters in the pattern are + totally ignored except when escaped or inside a character class. White- + space does not include the VT character (code 11). In addition, charac- + ters between an unescaped # outside a character class and the next new- + line character, inclusive, are also ignored. This is equivalent to + Perl's /x option, and it can be changed within a pattern by a (?x) option setting. - This option makes it possible to include comments inside complicated - patterns. Note, however, that this applies only to data characters. - Whitespace characters may never appear within special character - sequences in a pattern, for example within the sequence (?( which + This option makes it possible to include comments inside complicated + patterns. Note, however, that this applies only to data characters. + Whitespace characters may never appear within special character + sequences in a pattern, for example within the sequence (?( which introduces a conditional subpattern. PCRE_EXTRA - This option was invented in order to turn on additional functionality - of PCRE that is incompatible with Perl, but it is currently of very - little use. When set, any backslash in a pattern that is followed by a - letter that has no special meaning causes an error, thus reserving - these combinations for future expansion. By default, as in Perl, a - backslash followed by a letter with no special meaning is treated as a - literal. There are at present no other features controlled by this + This option was invented in order to turn on additional functionality + of PCRE that is incompatible with Perl, but it is currently of very + little use. When set, any backslash in a pattern that is followed by a + letter that has no special meaning causes an error, thus reserving + these combinations for future expansion. By default, as in Perl, a + backslash followed by a letter with no special meaning is treated as a + literal. There are at present no other features controlled by this option. It can also be set by a (?X) option setting within a pattern. + PCRE_FIRSTLINE + + If this option is set, an unanchored pattern is required to match + before or at the first newline character in the subject string, though + the matched text may continue over the newline. + PCRE_MULTILINE - By default, PCRE treats the subject string as consisting of a single - line of characters (even if it actually contains newlines). The "start - of line" metacharacter (^) matches only at the start of the string, - while the "end of line" metacharacter ($) matches only at the end of + By default, PCRE treats the subject string as consisting of a single + line of characters (even if it actually contains newlines). The "start + of line" metacharacter (^) matches only at the start of the string, + while the "end of line" metacharacter ($) matches only at the end of the string, or before a terminating newline (unless PCRE_DOLLAR_ENDONLY is set). This is the same as Perl. - When PCRE_MULTILINE it is set, the "start of line" and "end of line" - constructs match immediately following or immediately before any new- - line in the subject string, respectively, as well as at the very start - and end. This is equivalent to Perl's /m option, and it can be changed + When PCRE_MULTILINE it is set, the "start of line" and "end of line" + constructs match immediately following or immediately before any new- + line in the subject string, respectively, as well as at the very start + and end. This is equivalent to Perl's /m option, and it can be changed within a pattern by a (?m) option setting. If there are no "\n" charac- - ters in a subject string, or no occurrences of ^ or $ in a pattern, + ters in a subject string, or no occurrences of ^ or $ in a pattern, setting PCRE_MULTILINE has no effect. PCRE_NO_AUTO_CAPTURE If this option is set, it disables the use of numbered capturing paren- - theses in the pattern. Any opening parenthesis that is not followed by - ? behaves as if it were followed by ?: but named parentheses can still - be used for capturing (and they acquire numbers in the usual way). + theses in the pattern. Any opening parenthesis that is not followed by + ? behaves as if it were followed by ?: but named parentheses can still + be used for capturing (and they acquire numbers in the usual way). There is no equivalent of this option in Perl. PCRE_UNGREEDY - This option inverts the "greediness" of the quantifiers so that they - are not greedy by default, but become greedy if followed by "?". It is - not compatible with Perl. It can also be set by a (?U) option setting + This option inverts the "greediness" of the quantifiers so that they + are not greedy by default, but become greedy if followed by "?". It is + not compatible with Perl. It can also be set by a (?U) option setting within the pattern. PCRE_UTF8 - This option causes PCRE to regard both the pattern and the subject as - strings of UTF-8 characters instead of single-byte character strings. - However, it is available only when PCRE is built to include UTF-8 sup- - port. If not, the use of this option provokes an error. Details of how - this option changes the behaviour of PCRE are given in the section on + This option causes PCRE to regard both the pattern and the subject as + strings of UTF-8 characters instead of single-byte character strings. + However, it is available only when PCRE is built to include UTF-8 sup- + port. If not, the use of this option provokes an error. Details of how + this option changes the behaviour of PCRE are given in the section on UTF-8 support in the main pcre page. PCRE_NO_UTF8_CHECK When PCRE_UTF8 is set, the validity of the pattern as a UTF-8 string is - automatically checked. If an invalid UTF-8 sequence of bytes is found, - pcre_compile() returns an error. If you already know that your pattern - is valid, and you want to skip this check for performance reasons, you - can set the PCRE_NO_UTF8_CHECK option. When it is set, the effect of + automatically checked. If an invalid UTF-8 sequence of bytes is found, + pcre_compile() returns an error. If you already know that your pattern + is valid, and you want to skip this check for performance reasons, you + can set the PCRE_NO_UTF8_CHECK option. When it is set, the effect of passing an invalid UTF-8 string as a pattern is undefined. It may cause - your program to crash. Note that this option can also be passed to - pcre_exec(), to suppress the UTF-8 validity checking of subject - strings. + your program to crash. Note that this option can also be passed to + pcre_exec() and pcre_dfa_exec(), to suppress the UTF-8 validity check- + ing of subject strings. + + +COMPILATION ERROR CODES + + The following table lists the error codes than may be returned by + pcre_compile2(), along with the error messages that may be returned by + both compiling functions. + + 0 no error + 1 \ at end of pattern + 2 \c at end of pattern + 3 unrecognized character follows \ + 4 numbers out of order in {} quantifier + 5 number too big in {} quantifier + 6 missing terminating ] for character class + 7 invalid escape sequence in character class + 8 range out of order in character class + 9 nothing to repeat + 10 operand of unlimited repeat could match the empty string + 11 internal error: unexpected repeat + 12 unrecognized character after (? + 13 POSIX named classes are supported only within a class + 14 missing ) + 15 reference to non-existent subpattern + 16 erroffset passed as NULL + 17 unknown option bit(s) set + 18 missing ) after comment + 19 parentheses nested too deeply + 20 regular expression too large + 21 failed to get memory + 22 unmatched parentheses + 23 internal error: code overflow + 24 unrecognized character after (?< + 25 lookbehind assertion is not fixed length + 26 malformed number after (?( + 27 conditional group contains more than two branches + 28 assertion expected after (?( + 29 (?R or (?digits must be followed by ) + 30 unknown POSIX class name + 31 POSIX collating elements are not supported + 32 this version of PCRE is not compiled with PCRE_UTF8 support + 33 spare error + 34 character value in \x{...} sequence is too large + 35 invalid condition (?(0) + 36 \C not allowed in lookbehind assertion + 37 PCRE does not support \L, \l, \N, \U, or \u + 38 number after (?C is > 255 + 39 closing ) for (?C expected + 40 recursive call could loop indefinitely + 41 unrecognized character after (?P + 42 syntax error after (?P + 43 two named groups have the same name + 44 invalid UTF-8 string + 45 support for \P, \p, and \X has not been compiled + 46 malformed \P or \p sequence + 47 unknown property name after \P or \p STUDYING A PATTERN - pcre_extra *pcre_study(const pcre *code, int options, + pcre_extra *pcre_study(const pcre *code, int options const char **errptr); - If a compiled pattern is going to be used several times, it is worth + If a compiled pattern is going to be used several times, it is worth spending more time analyzing it in order to speed up the time taken for - matching. The function pcre_study() takes a pointer to a compiled pat- + matching. The function pcre_study() takes a pointer to a compiled pat- tern as its first argument. If studying the pattern produces additional - information that will help speed up matching, pcre_study() returns a - pointer to a pcre_extra block, in which the study_data field points to + information that will help speed up matching, pcre_study() returns a + pointer to a pcre_extra block, in which the study_data field points to the results of the study. The returned value from pcre_study() can be passed directly to - pcre_exec(). However, a pcre_extra block also contains other fields - that can be set by the caller before the block is passed; these are + pcre_exec(). However, a pcre_extra block also contains other fields + that can be set by the caller before the block is passed; these are described below in the section on matching a pattern. - If studying the pattern does not produce any additional information, + If studying the pattern does not produce any additional information pcre_study() returns NULL. In that circumstance, if the calling program - wants to pass any of the other fields to pcre_exec(), it must set up + wants to pass any of the other fields to pcre_exec(), it must set up its own pcre_extra block. - The second argument of pcre_study() contains option bits. At present, + The second argument of pcre_study() contains option bits. At present, no options are defined, and this argument should always be zero. - The third argument for pcre_study() is a pointer for an error message. - If studying succeeds (even if no data is returned), the variable it - points to is set to NULL. Otherwise it points to a textual error mes- - sage. You should therefore test the error pointer for NULL after call- + The third argument for pcre_study() is a pointer for an error message. + If studying succeeds (even if no data is returned), the variable it + points to is set to NULL. Otherwise it points to a textual error mes- + sage. You should therefore test the error pointer for NULL after call- ing pcre_study(), to be sure that it has run successfully. This is a typical call to pcre_study(): @@ -808,51 +1103,51 @@ STUDYING A PATTERN &error); /* set to NULL or points to a message */ At present, studying a pattern is useful only for non-anchored patterns - that do not have a single fixed starting character. A bitmap of possi- + that do not have a single fixed starting character. A bitmap of possi- ble starting bytes is created. LOCALE SUPPORT - PCRE handles caseless matching, and determines whether characters are - letters, digits, or whatever, by reference to a set of tables, indexed - by character value. (When running in UTF-8 mode, this applies only to - characters with codes less than 128. Higher-valued codes never match - escapes such as \w or \d, but can be tested with \p if PCRE is built - with Unicode character property support.) - - An internal set of tables is created in the default C locale when PCRE - is built. This is used when the final argument of pcre_compile() is - NULL, and is sufficient for many applications. An alternative set of - tables can, however, be supplied. These may be created in a different - locale from the default. As more and more applications change to using + PCRE handles caseless matching, and determines whether characters are + letters digits, or whatever, by reference to a set of tables, indexed + by character value. When running in UTF-8 mode, this applies only to + characters with codes less than 128. Higher-valued codes never match + escapes such as \w or \d, but can be tested with \p if PCRE is built + with Unicode character property support. + + An internal set of tables is created in the default C locale when PCRE + is built. This is used when the final argument of pcre_compile() is + NULL, and is sufficient for many applications. An alternative set of + tables can, however, be supplied. These may be created in a different + locale from the default. As more and more applications change to using Unicode, the need for this locale support is expected to die away. - External tables are built by calling the pcre_maketables() function, - which has no arguments, in the relevant locale. The result can then be - passed to pcre_compile() or pcre_exec() as often as necessary. For - example, to build and use tables that are appropriate for the French - locale (where accented characters with values greater than 128 are + External tables are built by calling the pcre_maketables() function, + which has no arguments, in the relevant locale. The result can then be + passed to pcre_compile() or pcre_exec() as often as necessary. For + example, to build and use tables that are appropriate for the French + locale (where accented characters with values greater than 128 are treated as letters), the following code could be used: setlocale(LC_CTYPE, "fr_FR"); tables = pcre_maketables(); re = pcre_compile(..., tables); - When pcre_maketables() runs, the tables are built in memory that is - obtained via pcre_malloc. It is the caller's responsibility to ensure - that the memory containing the tables remains available for as long as + When pcre_maketables() runs, the tables are built in memory that is + obtained via pcre_malloc. It is the caller's responsibility to ensure + that the memory containing the tables remains available for as long as it is needed. The pointer that is passed to pcre_compile() is saved with the compiled - pattern, and the same tables are used via this pointer by pcre_study() + pattern, and the same tables are used via this pointer by pcre_study() and normally also by pcre_exec(). Thus, by default, for any single pat- tern, compilation, studying and matching all happen in the same locale, but different patterns can be compiled in different locales. - It is possible to pass a table pointer or NULL (indicating the use of - the internal tables) to pcre_exec(). Although not intended for this - purpose, this facility could be used to match a pattern in a different + It is possible to pass a table pointer or NULL (indicating the use of + the internal tables) to pcre_exec(). Although not intended for this + purpose, this facility could be used to match a pattern in a different locale from the one in which it was compiled. Passing table pointers at run time is discussed below in the section on matching a pattern. @@ -862,15 +1157,15 @@ INFORMATION ABOUT A PATTERN int pcre_fullinfo(const pcre *code, const pcre_extra *extra, int what, void *where); - The pcre_fullinfo() function returns information about a compiled pat- + The pcre_fullinfo() function returns information about a compiled pat- tern. It replaces the obsolete pcre_info() function, which is neverthe- less retained for backwards compability (and is documented below). - The first argument for pcre_fullinfo() is a pointer to the compiled - pattern. The second argument is the result of pcre_study(), or NULL if - the pattern was not studied. The third argument specifies which piece - of information is required, and the fourth argument is a pointer to a - variable to receive the data. The yield of the function is zero for + The first argument for pcre_fullinfo() is a pointer to the compiled + pattern. The second argument is the result of pcre_study(), or NULL if + the pattern was not studied. The third argument specifies which piece + of information is required, and the fourth argument is a pointer to a + variable to receive the data. The yield of the function is zero for success, or one of the following negative numbers: PCRE_ERROR_NULL the argument code was NULL @@ -878,9 +1173,9 @@ INFORMATION ABOUT A PATTERN PCRE_ERROR_BADMAGIC the "magic number" was not found PCRE_ERROR_BADOPTION the value of what was invalid - The "magic number" is placed at the start of each compiled pattern as - an simple check against passing an arbitrary memory pointer. Here is a - typical call of pcre_fullinfo(), to obtain the length of the compiled + The "magic number" is placed at the start of each compiled pattern as + an simple check against passing an arbitrary memory pointer. Here is a + typical call of pcre_fullinfo(), to obtain the length of the compiled pattern: int rc; @@ -891,64 +1186,64 @@ INFORMATION ABOUT A PATTERN PCRE_INFO_SIZE, /* what is required */ &length); /* where to put the data */ - The possible values for the third argument are defined in pcre.h, and + The possible values for the third argument are defined in pcre.h, and are as follows: PCRE_INFO_BACKREFMAX - Return the number of the highest back reference in the pattern. The - fourth argument should point to an int variable. Zero is returned if + Return the number of the highest back reference in the pattern. The + fourth argument should point to an int variable. Zero is returned if there are no back references. PCRE_INFO_CAPTURECOUNT - Return the number of capturing subpatterns in the pattern. The fourth + Return the number of capturing subpatterns in the pattern. The fourth argument should point to an int variable. - PCRE_INFO_DEFAULTTABLES + PCRE_INFO_DEFAULT_TABLES - Return a pointer to the internal default character tables within PCRE. - The fourth argument should point to an unsigned char * variable. This + Return a pointer to the internal default character tables within PCRE. + The fourth argument should point to an unsigned char * variable. This information call is provided for internal use by the pcre_study() func- - tion. External callers can cause PCRE to use its internal tables by + tion. External callers can cause PCRE to use its internal tables by passing a NULL table pointer. PCRE_INFO_FIRSTBYTE - Return information about the first byte of any matched string, for a - non-anchored pattern. (This option used to be called - PCRE_INFO_FIRSTCHAR; the old name is still recognized for backwards + Return information about the first byte of any matched string, for a + non-anchored pattern. (This option used to be called + PCRE_INFO_FIRSTCHAR; the old name is still recognized for backwards compatibility.) - If there is a fixed first byte, for example, from a pattern such as - (cat|cow|coyote), it is returned in the integer pointed to by where. + If there is a fixed first byte, for example, from a pattern such as + (cat|cow|coyote), it is returned in the integer pointed to by where. Otherwise, if either - (a) the pattern was compiled with the PCRE_MULTILINE option, and every + (a) the pattern was compiled with the PCRE_MULTILINE option, and every branch starts with "^", or (b) every branch of the pattern starts with ".*" and PCRE_DOTALL is not set (if it were set, the pattern would be anchored), - -1 is returned, indicating that the pattern matches only at the start - of a subject string or after any newline within the string. Otherwise + -1 is returned, indicating that the pattern matches only at the start + of a subject string or after any newline within the string. Otherwise -2 is returned. For anchored patterns, -2 is returned. PCRE_INFO_FIRSTTABLE - If the pattern was studied, and this resulted in the construction of a + If the pattern was studied, and this resulted in the construction of a 256-bit table indicating a fixed set of bytes for the first byte in any - matching string, a pointer to the table is returned. Otherwise NULL is - returned. The fourth argument should point to an unsigned char * vari- + matching string, a pointer to the table is returned. Otherwise NULL is + returned. The fourth argument should point to an unsigned char * vari- able. PCRE_INFO_LASTLITERAL - Return the value of the rightmost literal byte that must exist in any - matched string, other than at its start, if such a byte has been + Return the value of the rightmost literal byte that must exist in any + matched string, other than at its start, if such a byte has been recorded. The fourth argument should point to an int variable. If there - is no such byte, -1 is returned. For anchored patterns, a last literal - byte is recorded only if it follows something of variable length. For + is no such byte, -1 is returned. For anchored patterns, a last literal + byte is recorded only if it follows something of variable length. For example, for the pattern /^a\d+z\d+/ the returned value is "z", but for /^a\dz\d/ the returned value is -1. @@ -956,32 +1251,32 @@ INFORMATION ABOUT A PATTERN PCRE_INFO_NAMEENTRYSIZE PCRE_INFO_NAMETABLE - PCRE supports the use of named as well as numbered capturing parenthe- - ses. The names are just an additional way of identifying the parenthe- + PCRE supports the use of named as well as numbered capturing parenthe- + ses. The names are just an additional way of identifying the parenthe- ses, which still acquire numbers. A convenience function called - pcre_get_named_substring() is provided for extracting an individual - captured substring by name. It is also possible to extract the data - directly, by first converting the name to a number in order to access - the correct pointers in the output vector (described with pcre_exec() - below). To do the conversion, you need to use the name-to-number map, + pcre_get_named_substring() is provided for extracting an individual + captured substring by name. It is also possible to extract the data + directly, by first converting the name to a number in order to access + the correct pointers in the output vector (described with pcre_exec() + below). To do the conversion, you need to use the name-to-number map, which is described by these three values. The map consists of a number of fixed-size entries. PCRE_INFO_NAMECOUNT gives the number of entries, and PCRE_INFO_NAMEENTRYSIZE gives the size - of each entry; both of these return an int value. The entry size - depends on the length of the longest name. PCRE_INFO_NAMETABLE returns - a pointer to the first entry of the table (a pointer to char). The + of each entry; both of these return an int value. The entry size + depends on the length of the longest name. PCRE_INFO_NAMETABLE returns + a pointer to the first entry of the table (a pointer to char). The first two bytes of each entry are the number of the capturing parenthe- - sis, most significant byte first. The rest of the entry is the corre- - sponding name, zero terminated. The names are in alphabetical order. - For example, consider the following pattern (assume PCRE_EXTENDED is + sis, most significant byte first. The rest of the entry is the corre- + sponding name, zero terminated. The names are in alphabetical order. + For example, consider the following pattern (assume PCRE_EXTENDED is set, so white space - including newlines - is ignored): (?P<date> (?P<year>(\d\d)?\d\d) - (?P<month>\d\d) - (?P<day>\d\d) ) - There are four named subpatterns, so the table has four entries, and - each entry in the table is eight bytes long. The table is as follows, + There are four named subpatterns, so the table has four entries, and + each entry in the table is eight bytes long. The table is as follows, with non-printing bytes shows in hexadecimal, and undefined bytes shown as ??: @@ -990,18 +1285,18 @@ INFORMATION ABOUT A PATTERN 00 04 m o n t h 00 00 02 y e a r 00 ?? - When writing code to extract data from named subpatterns using the + When writing code to extract data from named subpatterns using the name-to-number map, remember that the length of each entry is likely to be different for each compiled pattern. PCRE_INFO_OPTIONS - Return a copy of the options with which the pattern was compiled. The - fourth argument should point to an unsigned long int variable. These + Return a copy of the options with which the pattern was compiled. The + fourth argument should point to an unsigned long int variable. These option bits are those specified in the call to pcre_compile(), modified by any top-level option settings within the pattern itself. - A pattern is automatically anchored by PCRE if all of its top-level + A pattern is automatically anchored by PCRE if all of its top-level alternatives begin with one of the following: ^ unless PCRE_MULTILINE is set @@ -1015,7 +1310,7 @@ INFORMATION ABOUT A PATTERN PCRE_INFO_SIZE - Return the size of the compiled pattern, that is, the value that was + Return the size of the compiled pattern, that is, the value that was passed as the argument to pcre_malloc() when PCRE was getting memory in which to place the compiled data. The fourth argument should point to a size_t variable. @@ -1023,9 +1318,9 @@ INFORMATION ABOUT A PATTERN PCRE_INFO_STUDYSIZE Return the size of the data block pointed to by the study_data field in - a pcre_extra block. That is, it is the value that was passed to + a pcre_extra block. That is, it is the value that was passed to pcre_malloc() when PCRE was getting memory into which to place the data - created by pcre_study(). The fourth argument should point to a size_t + created by pcre_study(). The fourth argument should point to a size_t variable. @@ -1033,34 +1328,59 @@ OBSOLETE INFO FUNCTION int pcre_info(const pcre *code, int *optptr, int *firstcharptr); - The pcre_info() function is now obsolete because its interface is too - restrictive to return all the available data about a compiled pattern. - New programs should use pcre_fullinfo() instead. The yield of - pcre_info() is the number of capturing subpatterns, or one of the fol- + The pcre_info() function is now obsolete because its interface is too + restrictive to return all the available data about a compiled pattern. + New programs should use pcre_fullinfo() instead. The yield of + pcre_info() is the number of capturing subpatterns, or one of the fol- lowing negative numbers: PCRE_ERROR_NULL the argument code was NULL PCRE_ERROR_BADMAGIC the "magic number" was not found - If the optptr argument is not NULL, a copy of the options with which - the pattern was compiled is placed in the integer it points to (see + If the optptr argument is not NULL, a copy of the options with which + the pattern was compiled is placed in the integer it points to (see PCRE_INFO_OPTIONS above). - If the pattern is not anchored and the firstcharptr argument is not - NULL, it is used to pass back information about the first character of + If the pattern is not anchored and the firstcharptr argument is not + NULL, it is used to pass back information about the first character of any matched string (see PCRE_INFO_FIRSTBYTE above). -MATCHING A PATTERN +REFERENCE COUNTS + + int pcre_refcount(pcre *code, int adjust); + + The pcre_refcount() function is used to maintain a reference count in + the data block that contains a compiled pattern. It is provided for the + benefit of applications that operate in an object-oriented manner, + where different parts of the application may be using the same compiled + pattern, but you want to free the block when they are all done. + + When a pattern is compiled, the reference count field is initialized to + zero. It is changed only by calling this function, whose action is to + add the adjust value (which may be positive or negative) to it. The + yield of the function is the new value. However, the value of the count + is constrained to lie between 0 and 65535, inclusive. If the new value + is outside these limits, it is forced to the appropriate limit value. + + Except when it is zero, the reference count is not correctly preserved + if a pattern is compiled on one host and then transferred to a host + whose byte-order is different. (This seems a highly unlikely scenario.) + + +MATCHING A PATTERN: THE TRADITIONAL FUNCTION int pcre_exec(const pcre *code, const pcre_extra *extra, const char *subject, int length, int startoffset, int options, int *ovector, int ovecsize); - The function pcre_exec() is called to match a subject string against a - compiled pattern, which is passed in the code argument. If the pattern + The function pcre_exec() is called to match a subject string against a + compiled pattern, which is passed in the code argument. If the pattern has been studied, the result of the study should be passed in the extra - argument. + argument. This function is the main matching facility of the library, + and it operates in a Perl-like manner. For specialist use there is also + an alternative matching function, which is described below in the sec- + tion about the pcre_dfa_exec() function. In most applications, the pattern will have been compiled (and option- ally studied) in the same process that calls pcre_exec(). However, it @@ -1080,15 +1400,14 @@ MATCHING A PATTERN 0, /* start at offset 0 in the subject */ 0, /* default options */ ovector, /* vector of integers for substring information */ - 30); /* number of elements in the vector (NOT size in - bytes) */ + 30); /* number of elements (NOT size in bytes) */ Extra data for pcre_exec() - If the extra argument is not NULL, it must point to a pcre_extra data - block. The pcre_study() function returns such a block (when it doesn't - return NULL), but you can also create one for yourself, and pass addi- - tional information in it. The fields in a pcre_extra block are as fol- + If the extra argument is not NULL, it must point to a pcre_extra data + block. The pcre_study() function returns such a block (when it doesn't + return NULL), but you can also create one for yourself, and pass addi- + tional information in it. The fields in a pcre_extra block are as fol- lows: unsigned long int flags; @@ -1097,7 +1416,7 @@ MATCHING A PATTERN void *callout_data; const unsigned char *tables; - The flags field is a bitmap that specifies which of the other fields + The flags field is a bitmap that specifies which of the other fields are set. The flag bits are: PCRE_EXTRA_STUDY_DATA @@ -1105,229 +1424,229 @@ MATCHING A PATTERN PCRE_EXTRA_CALLOUT_DATA PCRE_EXTRA_TABLES - Other flag bits should be set to zero. The study_data field is set in - the pcre_extra block that is returned by pcre_study(), together with + Other flag bits should be set to zero. The study_data field is set in + the pcre_extra block that is returned by pcre_study(), together with the appropriate flag bit. You should not set this yourself, but you may - add to the block by setting the other fields and their corresponding + add to the block by setting the other fields and their corresponding flag bits. The match_limit field provides a means of preventing PCRE from using up - a vast amount of resources when running patterns that are not going to - match, but which have a very large number of possibilities in their - search trees. The classic example is the use of nested unlimited + a vast amount of resources when running patterns that are not going to + match, but which have a very large number of possibilities in their + search trees. The classic example is the use of nested unlimited repeats. - Internally, PCRE uses a function called match() which it calls repeat- - edly (sometimes recursively). The limit is imposed on the number of - times this function is called during a match, which has the effect of - limiting the amount of recursion and backtracking that can take place. + Internally, PCRE uses a function called match() which it calls repeat- + edly (sometimes recursively). The limit is imposed on the number of + times this function is called during a match, which has the effect of + limiting the amount of recursion and backtracking that can take place. For patterns that are not anchored, the count starts from zero for each position in the subject string. - The default limit for the library can be set when PCRE is built; the - default default is 10 million, which handles all but the most extreme - cases. You can reduce the default by suppling pcre_exec() with a - pcre_extra block in which match_limit is set to a smaller value, and - PCRE_EXTRA_MATCH_LIMIT is set in the flags field. If the limit is + The default limit for the library can be set when PCRE is built; the + default default is 10 million, which handles all but the most extreme + cases. You can reduce the default by suppling pcre_exec() with a + pcre_extra block in which match_limit is set to a smaller value, and + PCRE_EXTRA_MATCH_LIMIT is set in the flags field. If the limit is exceeded, pcre_exec() returns PCRE_ERROR_MATCHLIMIT. - The pcre_callout field is used in conjunction with the "callout" fea- + The pcre_callout field is used in conjunction with the "callout" fea- ture, which is described in the pcrecallout documentation. - The tables field is used to pass a character tables pointer to - pcre_exec(); this overrides the value that is stored with the compiled - pattern. A non-NULL value is stored with the compiled pattern only if - custom tables were supplied to pcre_compile() via its tableptr argu- + The tables field is used to pass a character tables pointer to + pcre_exec(); this overrides the value that is stored with the compiled + pattern. A non-NULL value is stored with the compiled pattern only if + custom tables were supplied to pcre_compile() via its tableptr argu- ment. If NULL is passed to pcre_exec() using this mechanism, it forces - PCRE's internal tables to be used. This facility is helpful when re- - using patterns that have been saved after compiling with an external - set of tables, because the external tables might be at a different - address when pcre_exec() is called. See the pcreprecompile documenta- + PCRE's internal tables to be used. This facility is helpful when re- + using patterns that have been saved after compiling with an external + set of tables, because the external tables might be at a different + address when pcre_exec() is called. See the pcreprecompile documenta- tion for a discussion of saving compiled patterns for later use. Option bits for pcre_exec() - The unused bits of the options argument for pcre_exec() must be zero. - The only bits that may be set are PCRE_ANCHORED, PCRE_NOTBOL, + The unused bits of the options argument for pcre_exec() must be zero. + The only bits that may be set are PCRE_ANCHORED, PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, PCRE_NO_UTF8_CHECK and PCRE_PARTIAL. PCRE_ANCHORED - The PCRE_ANCHORED option limits pcre_exec() to matching at the first - matching position. If a pattern was compiled with PCRE_ANCHORED, or - turned out to be anchored by virtue of its contents, it cannot be made + The PCRE_ANCHORED option limits pcre_exec() to matching at the first + matching position. If a pattern was compiled with PCRE_ANCHORED, or + turned out to be anchored by virtue of its contents, it cannot be made unachored at matching time. PCRE_NOTBOL This option specifies that first character of the subject string is not - the beginning of a line, so the circumflex metacharacter should not - match before it. Setting this without PCRE_MULTILINE (at compile time) - causes circumflex never to match. This option affects only the - behaviour of the circumflex metacharacter. It does not affect \A. + the beginning of a line, so the circumflex metacharacter should not + match before it. Setting this without PCRE_MULTILINE (at compile time) + causes circumflex never to match. This option affects only the behav- + iour of the circumflex metacharacter. It does not affect \A. PCRE_NOTEOL This option specifies that the end of the subject string is not the end - of a line, so the dollar metacharacter should not match it nor (except - in multiline mode) a newline immediately before it. Setting this with- + of a line, so the dollar metacharacter should not match it nor (except + in multiline mode) a newline immediately before it. Setting this with- out PCRE_MULTILINE (at compile time) causes dollar never to match. This - option affects only the behaviour of the dollar metacharacter. It does + option affects only the behaviour of the dollar metacharacter. It does not affect \Z or \z. PCRE_NOTEMPTY An empty string is not considered to be a valid match if this option is - set. If there are alternatives in the pattern, they are tried. If all - the alternatives match the empty string, the entire match fails. For + set. If there are alternatives in the pattern, they are tried. If all + the alternatives match the empty string, the entire match fails. For example, if the pattern a?b? - is applied to a string not beginning with "a" or "b", it matches the - empty string at the start of the subject. With PCRE_NOTEMPTY set, this + is applied to a string not beginning with "a" or "b", it matches the + empty string at the start of the subject. With PCRE_NOTEMPTY set, this match is not valid, so PCRE searches further into the string for occur- rences of "a" or "b". Perl has no direct equivalent of PCRE_NOTEMPTY, but it does make a spe- - cial case of a pattern match of the empty string within its split() - function, and when using the /g modifier. It is possible to emulate + cial case of a pattern match of the empty string within its split() + function, and when using the /g modifier. It is possible to emulate Perl's behaviour after matching a null string by first trying the match again at the same offset with PCRE_NOTEMPTY and PCRE_ANCHORED, and then - if that fails by advancing the starting offset (see below) and trying + if that fails by advancing the starting offset (see below) and trying an ordinary match again. There is some code that demonstrates how to do this in the pcredemo.c sample program. PCRE_NO_UTF8_CHECK When PCRE_UTF8 is set at compile time, the validity of the subject as a - UTF-8 string is automatically checked when pcre_exec() is subsequently - called. The value of startoffset is also checked to ensure that it - points to the start of a UTF-8 character. If an invalid UTF-8 sequence + UTF-8 string is automatically checked when pcre_exec() is subsequently + called. The value of startoffset is also checked to ensure that it + points to the start of a UTF-8 character. If an invalid UTF-8 sequence of bytes is found, pcre_exec() returns the error PCRE_ERROR_BADUTF8. If - startoffset contains an invalid value, PCRE_ERROR_BADUTF8_OFFSET is + startoffset contains an invalid value, PCRE_ERROR_BADUTF8_OFFSET is returned. - If you already know that your subject is valid, and you want to skip - these checks for performance reasons, you can set the - PCRE_NO_UTF8_CHECK option when calling pcre_exec(). You might want to - do this for the second and subsequent calls to pcre_exec() if you are - making repeated calls to find all the matches in a single subject - string. However, you should be sure that the value of startoffset - points to the start of a UTF-8 character. When PCRE_NO_UTF8_CHECK is - set, the effect of passing an invalid UTF-8 string as a subject, or a - value of startoffset that does not point to the start of a UTF-8 char- + If you already know that your subject is valid, and you want to skip + these checks for performance reasons, you can set the + PCRE_NO_UTF8_CHECK option when calling pcre_exec(). You might want to + do this for the second and subsequent calls to pcre_exec() if you are + making repeated calls to find all the matches in a single subject + string. However, you should be sure that the value of startoffset + points to the start of a UTF-8 character. When PCRE_NO_UTF8_CHECK is + set, the effect of passing an invalid UTF-8 string as a subject, or a + value of startoffset that does not point to the start of a UTF-8 char- acter, is undefined. Your program may crash. PCRE_PARTIAL - This option turns on the partial matching feature. If the subject - string fails to match the pattern, but at some point during the match- - ing process the end of the subject was reached (that is, the subject - partially matches the pattern and the failure to match occurred only - because there were not enough subject characters), pcre_exec() returns - PCRE_ERROR_PARTIAL instead of PCRE_ERROR_NOMATCH. When PCRE_PARTIAL is - used, there are restrictions on what may appear in the pattern. These + This option turns on the partial matching feature. If the subject + string fails to match the pattern, but at some point during the match- + ing process the end of the subject was reached (that is, the subject + partially matches the pattern and the failure to match occurred only + because there were not enough subject characters), pcre_exec() returns + PCRE_ERROR_PARTIAL instead of PCRE_ERROR_NOMATCH. When PCRE_PARTIAL is + used, there are restrictions on what may appear in the pattern. These are discussed in the pcrepartial documentation. The string to be matched by pcre_exec() - The subject string is passed to pcre_exec() as a pointer in subject, a - length in length, and a starting byte offset in startoffset. In UTF-8 - mode, the byte offset must point to the start of a UTF-8 character. - Unlike the pattern string, the subject may contain binary zero bytes. - When the starting offset is zero, the search for a match starts at the + The subject string is passed to pcre_exec() as a pointer in subject, a + length in length, and a starting byte offset in startoffset. In UTF-8 + mode, the byte offset must point to the start of a UTF-8 character. + Unlike the pattern string, the subject may contain binary zero bytes. + When the starting offset is zero, the search for a match starts at the beginning of the subject, and this is by far the most common case. - A non-zero starting offset is useful when searching for another match - in the same subject by calling pcre_exec() again after a previous suc- - cess. Setting startoffset differs from just passing over a shortened - string and setting PCRE_NOTBOL in the case of a pattern that begins + A non-zero starting offset is useful when searching for another match + in the same subject by calling pcre_exec() again after a previous suc- + cess. Setting startoffset differs from just passing over a shortened + string and setting PCRE_NOTBOL in the case of a pattern that begins with any kind of lookbehind. For example, consider the pattern \Biss\B - which finds occurrences of "iss" in the middle of words. (\B matches - only if the current position in the subject is not a word boundary.) - When applied to the string "Mississipi" the first call to pcre_exec() - finds the first occurrence. If pcre_exec() is called again with just - the remainder of the subject, namely "issipi", it does not match, + which finds occurrences of "iss" in the middle of words. (\B matches + only if the current position in the subject is not a word boundary.) + When applied to the string "Mississipi" the first call to pcre_exec() + finds the first occurrence. If pcre_exec() is called again with just + the remainder of the subject, namely "issipi", it does not match, because \B is always false at the start of the subject, which is deemed - to be a word boundary. However, if pcre_exec() is passed the entire + to be a word boundary. However, if pcre_exec() is passed the entire string again, but with startoffset set to 4, it finds the second occur- - rence of "iss" because it is able to look behind the starting point to + rence of "iss" because it is able to look behind the starting point to discover that it is preceded by a letter. - If a non-zero starting offset is passed when the pattern is anchored, + If a non-zero starting offset is passed when the pattern is anchored, one attempt to match at the given offset is made. This can only succeed - if the pattern does not require the match to be at the start of the + if the pattern does not require the match to be at the start of the subject. How pcre_exec() returns captured substrings - In general, a pattern matches a certain portion of the subject, and in - addition, further substrings from the subject may be picked out by - parts of the pattern. Following the usage in Jeffrey Friedl's book, - this is called "capturing" in what follows, and the phrase "capturing - subpattern" is used for a fragment of a pattern that picks out a sub- - string. PCRE supports several other kinds of parenthesized subpattern + In general, a pattern matches a certain portion of the subject, and in + addition, further substrings from the subject may be picked out by + parts of the pattern. Following the usage in Jeffrey Friedl's book, + this is called "capturing" in what follows, and the phrase "capturing + subpattern" is used for a fragment of a pattern that picks out a sub- + string. PCRE supports several other kinds of parenthesized subpattern that do not cause substrings to be captured. - Captured substrings are returned to the caller via a vector of integer - offsets whose address is passed in ovector. The number of elements in - the vector is passed in ovecsize, which must be a non-negative number. + Captured substrings are returned to the caller via a vector of integer + offsets whose address is passed in ovector. The number of elements in + the vector is passed in ovecsize, which must be a non-negative number. Note: this argument is NOT the size of ovector in bytes. - The first two-thirds of the vector is used to pass back captured sub- - strings, each substring using a pair of integers. The remaining third - of the vector is used as workspace by pcre_exec() while matching cap- - turing subpatterns, and is not available for passing back information. - The length passed in ovecsize should always be a multiple of three. If + The first two-thirds of the vector is used to pass back captured sub- + strings, each substring using a pair of integers. The remaining third + of the vector is used as workspace by pcre_exec() while matching cap- + turing subpatterns, and is not available for passing back information. + The length passed in ovecsize should always be a multiple of three. If it is not, it is rounded down. - When a match is successful, information about captured substrings is - returned in pairs of integers, starting at the beginning of ovector, - and continuing up to two-thirds of its length at the most. The first + When a match is successful, information about captured substrings is + returned in pairs of integers, starting at the beginning of ovector, + and continuing up to two-thirds of its length at the most. The first element of a pair is set to the offset of the first character in a sub- - string, and the second is set to the offset of the first character - after the end of a substring. The first pair, ovector[0] and ovec- - tor[1], identify the portion of the subject string matched by the - entire pattern. The next pair is used for the first capturing subpat- - tern, and so on. The value returned by pcre_exec() is the number of - pairs that have been set. If there are no capturing subpatterns, the - return value from a successful match is 1, indicating that just the + string, and the second is set to the offset of the first character + after the end of a substring. The first pair, ovector[0] and ovec- + tor[1], identify the portion of the subject string matched by the + entire pattern. The next pair is used for the first capturing subpat- + tern, and so on. The value returned by pcre_exec() is the number of + pairs that have been set. If there are no capturing subpatterns, the + return value from a successful match is 1, indicating that just the first pair of offsets has been set. - Some convenience functions are provided for extracting the captured - substrings as separate strings. These are described in the following + Some convenience functions are provided for extracting the captured + substrings as separate strings. These are described in the following section. - It is possible for an capturing subpattern number n+1 to match some - part of the subject when subpattern n has not been used at all. For + It is possible for an capturing subpattern number n+1 to match some + part of the subject when subpattern n has not been used at all. For example, if the string "abc" is matched against the pattern (a|(z))(bc) - subpatterns 1 and 3 are matched, but 2 is not. When this happens, both + subpatterns 1 and 3 are matched, but 2 is not. When this happens, both offset values corresponding to the unused subpattern are set to -1. If a capturing subpattern is matched repeatedly, it is the last portion of the string that it matched that is returned. - If the vector is too small to hold all the captured substring offsets, + If the vector is too small to hold all the captured substring offsets, it is used as far as possible (up to two-thirds of its length), and the - function returns a value of zero. In particular, if the substring off- + function returns a value of zero. In particular, if the substring off- sets are not of interest, pcre_exec() may be called with ovector passed - as NULL and ovecsize as zero. However, if the pattern contains back - references and the ovector is not big enough to remember the related - substrings, PCRE has to get additional memory for use during matching. + as NULL and ovecsize as zero. However, if the pattern contains back + references and the ovector is not big enough to remember the related + substrings, PCRE has to get additional memory for use during matching. Thus it is usually advisable to supply an ovector. - Note that pcre_info() can be used to find out how many capturing sub- + Note that pcre_info() can be used to find out how many capturing sub- patterns there are in a compiled pattern. The smallest size for ovector - that will allow for n captured substrings, in addition to the offsets + that will allow for n captured substrings, in addition to the offsets of the substring matched by the whole pattern, is (n+1)*3. Return values from pcre_exec() - If pcre_exec() fails, it returns a negative number. The following are + If pcre_exec() fails, it returns a negative number. The following are defined in the header file: PCRE_ERROR_NOMATCH (-1) @@ -1336,7 +1655,7 @@ MATCHING A PATTERN PCRE_ERROR_NULL (-2) - Either code or subject was passed as NULL, or ovector was NULL and + Either code or subject was passed as NULL, or ovector was NULL and ovecsize was not zero. PCRE_ERROR_BADOPTION (-3) @@ -1345,74 +1664,74 @@ MATCHING A PATTERN PCRE_ERROR_BADMAGIC (-4) - PCRE stores a 4-byte "magic number" at the start of the compiled code, + PCRE stores a 4-byte "magic number" at the start of the compiled code, to catch the case when it is passed a junk pointer and to detect when a pattern that was compiled in an environment of one endianness is run in - an environment with the other endianness. This is the error that PCRE + an environment with the other endianness. This is the error that PCRE gives when the magic number is not present. PCRE_ERROR_UNKNOWN_NODE (-5) While running the pattern match, an unknown item was encountered in the - compiled pattern. This error could be caused by a bug in PCRE or by + compiled pattern. This error could be caused by a bug in PCRE or by overwriting of the compiled pattern. PCRE_ERROR_NOMEMORY (-6) - If a pattern contains back references, but the ovector that is passed + If a pattern contains back references, but the ovector that is passed to pcre_exec() is not big enough to remember the referenced substrings, - PCRE gets a block of memory at the start of matching to use for this - purpose. If the call via pcre_malloc() fails, this error is given. The + PCRE gets a block of memory at the start of matching to use for this + purpose. If the call via pcre_malloc() fails, this error is given. The memory is automatically freed at the end of matching. PCRE_ERROR_NOSUBSTRING (-7) - This error is used by the pcre_copy_substring(), pcre_get_substring(), + This error is used by the pcre_copy_substring(), pcre_get_substring(), and pcre_get_substring_list() functions (see below). It is never returned by pcre_exec(). PCRE_ERROR_MATCHLIMIT (-8) - The recursion and backtracking limit, as specified by the match_limit - field in a pcre_extra structure (or defaulted) was reached. See the + The recursion and backtracking limit, as specified by the match_limit + field in a pcre_extra structure (or defaulted) was reached. See the description above. PCRE_ERROR_CALLOUT (-9) This error is never generated by pcre_exec() itself. It is provided for - use by callout functions that want to yield a distinctive error code. + use by callout functions that want to yield a distinctive error code. See the pcrecallout documentation for details. PCRE_ERROR_BADUTF8 (-10) - A string that contains an invalid UTF-8 byte sequence was passed as a + A string that contains an invalid UTF-8 byte sequence was passed as a subject. PCRE_ERROR_BADUTF8_OFFSET (-11) The UTF-8 byte sequence that was passed as a subject was valid, but the - value of startoffset did not point to the beginning of a UTF-8 charac- + value of startoffset did not point to the beginning of a UTF-8 charac- ter. - PCRE_ERROR_PARTIAL (-12) + PCRE_ERROR_PARTIAL (-12) - The subject string did not match, but it did match partially. See the + The subject string did not match, but it did match partially. See the pcrepartial documentation for details of partial matching. - PCRE_ERROR_BAD_PARTIAL (-13) + PCRE_ERROR_BADPARTIAL (-13) - The PCRE_PARTIAL option was used with a compiled pattern containing - items that are not supported for partial matching. See the pcrepartial + The PCRE_PARTIAL option was used with a compiled pattern containing + items that are not supported for partial matching. See the pcrepartial documentation for details of partial matching. - PCRE_ERROR_INTERNAL (-14) + PCRE_ERROR_INTERNAL (-14) - An unexpected internal error has occurred. This error could be caused + An unexpected internal error has occurred. This error could be caused by a bug in PCRE or by overwriting of the compiled pattern. - PCRE_ERROR_BADCOUNT (-15) + PCRE_ERROR_BADCOUNT (-15) - This error is given if the value of the ovecsize argument is negative. + This error is given if the value of the ovecsize argument is negative. EXTRACTING CAPTURED SUBSTRINGS BY NUMBER @@ -1428,72 +1747,72 @@ EXTRACTING CAPTURED SUBSTRINGS BY NUMBER int pcre_get_substring_list(const char *subject, int *ovector, int stringcount, const char ***listptr); - Captured substrings can be accessed directly by using the offsets - returned by pcre_exec() in ovector. For convenience, the functions + Captured substrings can be accessed directly by using the offsets + returned by pcre_exec() in ovector. For convenience, the functions pcre_copy_substring(), pcre_get_substring(), and pcre_get_sub- - string_list() are provided for extracting captured substrings as new, - separate, zero-terminated strings. These functions identify substrings - by number. The next section describes functions for extracting named - substrings. A substring that contains a binary zero is correctly - extracted and has a further zero added on the end, but the result is + string_list() are provided for extracting captured substrings as new, + separate, zero-terminated strings. These functions identify substrings + by number. The next section describes functions for extracting named + substrings. A substring that contains a binary zero is correctly + extracted and has a further zero added on the end, but the result is not, of course, a C string. - The first three arguments are the same for all three of these func- - tions: subject is the subject string that has just been successfully + The first three arguments are the same for all three of these func- + tions: subject is the subject string that has just been successfully matched, ovector is a pointer to the vector of integer offsets that was passed to pcre_exec(), and stringcount is the number of substrings that - were captured by the match, including the substring that matched the + were captured by the match, including the substring that matched the entire regular expression. This is the value returned by pcre_exec() if - it is greater than zero. If pcre_exec() returned zero, indicating that - it ran out of space in ovector, the value passed as stringcount should + it is greater than zero. If pcre_exec() returned zero, indicating that + it ran out of space in ovector, the value passed as stringcount should be the number of elements in the vector divided by three. - The functions pcre_copy_substring() and pcre_get_substring() extract a - single substring, whose number is given as stringnumber. A value of - zero extracts the substring that matched the entire pattern, whereas - higher values extract the captured substrings. For pcre_copy_sub- - string(), the string is placed in buffer, whose length is given by - buffersize, while for pcre_get_substring() a new block of memory is - obtained via pcre_malloc, and its address is returned via stringptr. - The yield of the function is the length of the string, not including + The functions pcre_copy_substring() and pcre_get_substring() extract a + single substring, whose number is given as stringnumber. A value of + zero extracts the substring that matched the entire pattern, whereas + higher values extract the captured substrings. For pcre_copy_sub- + string(), the string is placed in buffer, whose length is given by + buffersize, while for pcre_get_substring() a new block of memory is + obtained via pcre_malloc, and its address is returned via stringptr. + The yield of the function is the length of the string, not including the terminating zero, or one of PCRE_ERROR_NOMEMORY (-6) - The buffer was too small for pcre_copy_substring(), or the attempt to + The buffer was too small for pcre_copy_substring(), or the attempt to get memory failed for pcre_get_substring(). PCRE_ERROR_NOSUBSTRING (-7) There is no substring whose number is stringnumber. - The pcre_get_substring_list() function extracts all available sub- - strings and builds a list of pointers to them. All this is done in a + The pcre_get_substring_list() function extracts all available sub- + strings and builds a list of pointers to them. All this is done in a single block of memory that is obtained via pcre_malloc. The address of - the memory block is returned via listptr, which is also the start of - the list of string pointers. The end of the list is marked by a NULL + the memory block is returned via listptr, which is also the start of + the list of string pointers. The end of the list is marked by a NULL pointer. The yield of the function is zero if all went well, or PCRE_ERROR_NOMEMORY (-6) if the attempt to get the memory block failed. - When any of these functions encounter a substring that is unset, which - can happen when capturing subpattern number n+1 matches some part of - the subject, but subpattern n has not been used at all, they return an + When any of these functions encounter a substring that is unset, which + can happen when capturing subpattern number n+1 matches some part of + the subject, but subpattern n has not been used at all, they return an empty string. This can be distinguished from a genuine zero-length sub- - string by inspecting the appropriate offset in ovector, which is nega- + string by inspecting the appropriate offset in ovector, which is nega- tive for unset substrings. - The two convenience functions pcre_free_substring() and pcre_free_sub- - string_list() can be used to free the memory returned by a previous + The two convenience functions pcre_free_substring() and pcre_free_sub- + string_list() can be used to free the memory returned by a previous call of pcre_get_substring() or pcre_get_substring_list(), respec- - tively. They do nothing more than call the function pointed to by - pcre_free, which of course could be called directly from a C program. - However, PCRE is used in some situations where it is linked via a spe- + tively. They do nothing more than call the function pointed to by + pcre_free, which of course could be called directly from a C program. + However, PCRE is used in some situations where it is linked via a spe- cial interface to another programming language which cannot use - pcre_free directly; it is for these cases that the functions are - provided. + pcre_free directly; it is for these cases that the functions are pro- + vided. EXTRACTING CAPTURED SUBSTRINGS BY NAME @@ -1511,47 +1830,217 @@ EXTRACTING CAPTURED SUBSTRINGS BY NAME int stringcount, const char *stringname, const char **stringptr); - To extract a substring by name, you first have to find associated num- + To extract a substring by name, you first have to find associated num- ber. For example, for this pattern (a+)b(?<xxx>\d+)... the number of the subpattern called "xxx" is 2. You can find the number from the name by calling pcre_get_stringnumber(). The first argument is - the compiled pattern, and the second is the name. The yield of the - function is the subpattern number, or PCRE_ERROR_NOSUBSTRING (-7) if + the compiled pattern, and the second is the name. The yield of the + function is the subpattern number, or PCRE_ERROR_NOSUBSTRING (-7) if there is no subpattern of that name. Given the number, you can extract the substring directly, or use one of the functions described in the previous section. For convenience, there are also two functions that do the whole job. - Most of the arguments of pcre_copy_named_substring() and - pcre_get_named_substring() are the same as those for the similarly - named functions that extract by number. As these are described in the - previous section, they are not re-described here. There are just two + Most of the arguments of pcre_copy_named_substring() and + pcre_get_named_substring() are the same as those for the similarly + named functions that extract by number. As these are described in the + previous section, they are not re-described here. There are just two differences: - First, instead of a substring number, a substring name is given. Sec- + First, instead of a substring number, a substring name is given. Sec- ond, there is an extra argument, given at the start, which is a pointer - to the compiled pattern. This is needed in order to gain access to the + to the compiled pattern. This is needed in order to gain access to the name-to-number translation table. - These functions call pcre_get_stringnumber(), and if it succeeds, they - then call pcre_copy_substring() or pcre_get_substring(), as appropri- + These functions call pcre_get_stringnumber(), and if it succeeds, they + then call pcre_copy_substring() or pcre_get_substring(), as appropri- ate. -Last updated: 09 September 2004 -Copyright (c) 1997-2004 University of Cambridge. ------------------------------------------------------------------------------ -PCRE(3) PCRE(3) +FINDING ALL POSSIBLE MATCHES + + The traditional matching function uses a similar algorithm to Perl, + which stops when it finds the first match, starting at a given point in + the subject. If you want to find all possible matches, or the longest + possible match, consider using the alternative matching function (see + below) instead. If you cannot use the alternative function, but still + need to find all possible matches, you can kludge it up by making use + of the callout facility, which is described in the pcrecallout documen- + tation. + + What you have to do is to insert a callout right at the end of the pat- + tern. When your callout function is called, extract and save the cur- + rent matched substring. Then return 1, which forces pcre_exec() to + backtrack and try other alternatives. Ultimately, when it runs out of + matches, pcre_exec() will yield PCRE_ERROR_NOMATCH. + + +MATCHING A PATTERN: THE ALTERNATIVE FUNCTION + + int pcre_dfa_exec(const pcre *code, const pcre_extra *extra, + const char *subject, int length, int startoffset, + int options, int *ovector, int ovecsize, + int *workspace, int wscount); + + The function pcre_dfa_exec() is called to match a subject string + against a compiled pattern, using a "DFA" matching algorithm. This has + different characteristics to the normal algorithm, and is not compati- + ble with Perl. Some of the features of PCRE patterns are not supported. + Nevertheless, there are times when this kind of matching can be useful. + For a discussion of the two matching algorithms, see the pcrematching + documentation. + + The arguments for the pcre_dfa_exec() function are the same as for + pcre_exec(), plus two extras. The ovector argument is used in a differ- + ent way, and this is described below. The other common arguments are + used in the same way as for pcre_exec(), so their description is not + repeated here. + + The two additional arguments provide workspace for the function. The + workspace vector should contain at least 20 elements. It is used for + keeping track of multiple paths through the pattern tree. More + workspace will be needed for patterns and subjects where there are a + lot of possible matches. + + Here is an example of a simple call to pcre_exec(): + + int rc; + int ovector[10]; + int wspace[20]; + rc = pcre_exec( + re, /* result of pcre_compile() */ + NULL, /* we didn't study the pattern */ + "some string", /* the subject string */ + 11, /* the length of the subject string */ + 0, /* start at offset 0 in the subject */ + 0, /* default options */ + ovector, /* vector of integers for substring information */ + 10, /* number of elements (NOT size in bytes) */ + wspace, /* working space vector */ + 20); /* number of elements (NOT size in bytes) */ + + Option bits for pcre_dfa_exec() + + The unused bits of the options argument for pcre_dfa_exec() must be + zero. The only bits that may be set are PCRE_ANCHORED, PCRE_NOTBOL, + PCRE_NOTEOL, PCRE_NOTEMPTY, PCRE_NO_UTF8_CHECK, PCRE_PARTIAL, + PCRE_DFA_SHORTEST, and PCRE_DFA_RESTART. All but the last three of + these are the same as for pcre_exec(), so their description is not + repeated here. + + PCRE_PARTIAL + + This has the same general effect as it does for pcre_exec(), but the + details are slightly different. When PCRE_PARTIAL is set for + pcre_dfa_exec(), the return code PCRE_ERROR_NOMATCH is converted into + PCRE_ERROR_PARTIAL if the end of the subject is reached, there have + been no complete matches, but there is still at least one matching pos- + sibility. The portion of the string that provided the partial match is + set as the first matching string. + + PCRE_DFA_SHORTEST + + Setting the PCRE_DFA_SHORTEST option causes the matching algorithm to + stop as soon as it has found one match. Because of the way the DFA + algorithm works, this is necessarily the shortest possible match at the + first possible matching point in the subject string. + + PCRE_DFA_RESTART + + When pcre_dfa_exec() is called with the PCRE_PARTIAL option, and + returns a partial match, it is possible to call it again, with addi- + tional subject characters, and have it continue with the same match. + The PCRE_DFA_RESTART option requests this action; when it is set, the + workspace and wscount options must reference the same vector as before + because data about the match so far is left in them after a partial + match. There is more discussion of this facility in the pcrepartial + documentation. + + Successful returns from pcre_dfa_exec() + + When pcre_dfa_exec() succeeds, it may have matched more than one sub- + string in the subject. Note, however, that all the matches from one run + of the function start at the same point in the subject. The shorter + matches are all initial substrings of the longer matches. For example, + if the pattern + + <.*> + + is matched against the string + + This is <something> <something else> <something further> no more + + the three matched strings are + + <something> + <something> <something else> + <something> <something else> <something further> + + On success, the yield of the function is a number greater than zero, + which is the number of matched substrings. The substrings themselves + are returned in ovector. Each string uses two elements; the first is + the offset to the start, and the second is the offset to the end. All + the strings have the same start offset. (Space could have been saved by + giving this only once, but it was decided to retain some compatibility + with the way pcre_exec() returns data, even though the meaning of the + strings is different.) + + The strings are returned in reverse order of length; that is, the long- + est matching string is given first. If there were too many matches to + fit into ovector, the yield of the function is zero, and the vector is + filled with the longest matches. + + Error returns from pcre_dfa_exec() + + The pcre_dfa_exec() function returns a negative number when it fails. + Many of the errors are the same as for pcre_exec(), and these are + described above. There are in addition the following errors that are + specific to pcre_dfa_exec(): + + PCRE_ERROR_DFA_UITEM (-16) + + This return is given if pcre_dfa_exec() encounters an item in the pat- + tern that it does not support, for instance, the use of \C or a back + reference. + + PCRE_ERROR_DFA_UCOND (-17) + + This return is given if pcre_dfa_exec() encounters a condition item in + a pattern that uses a back reference for the condition. This is not + supported. + + PCRE_ERROR_DFA_UMLIMIT (-18) + + This return is given if pcre_dfa_exec() is called with an extra block + that contains a setting of the match_limit field. This is not supported + (it is meaningless). + + PCRE_ERROR_DFA_WSSIZE (-19) + + This return is given if pcre_dfa_exec() runs out of space in the + workspace vector. + + PCRE_ERROR_DFA_RECURSE (-20) + + When a recursive subpattern is processed, the matching function calls + itself recursively, using private vectors for ovector and workspace. + This error is given if the output vector is not large enough. This + should be extremely rare, as a vector of size 1000 is used. + +Last updated: 16 May 2005 +Copyright (c) 1997-2005 University of Cambridge. +----------------------------------------------------------------------------- NAME PCRE - Perl-compatible regular expressions + PCRE CALLOUTS int (*pcre_callout)(pcre_callout_block *); @@ -1606,9 +2095,10 @@ MISSING CALLOUTS THE CALLOUT INTERFACE During matching, when PCRE reaches a callout point, the external func- - tion defined by pcre_callout is called (if it is set). The only argu- - ment is a pointer to a pcre_callout block. This structure contains the - following fields: + tion defined by pcre_callout is called (if it is set). This applies to + both the pcre_exec() and the pcre_dfa_exec() matching functions. The + only argument to the callout function is a pointer to a pcre_callout + block. This structure contains the following fields: int version; int callout_number; @@ -1623,87 +2113,91 @@ THE CALLOUT INTERFACE int pattern_position; int next_item_length; - The version field is an integer containing the version number of the - block format. The initial version was 0; the current version is 1. The - version number will change again in future if additional fields are + The version field is an integer containing the version number of the + block format. The initial version was 0; the current version is 1. The + version number will change again in future if additional fields are added, but the intention is never to remove any of the existing fields. - The callout_number field contains the number of the callout, as com- - piled into the pattern (that is, the number after ?C for manual call- + The callout_number field contains the number of the callout, as com- + piled into the pattern (that is, the number after ?C for manual call- outs, and 255 for automatically generated callouts). - The offset_vector field is a pointer to the vector of offsets that was - passed by the caller to pcre_exec(). The contents can be inspected in - order to extract substrings that have been matched so far, in the same - way as for extracting substrings after a match has completed. + The offset_vector field is a pointer to the vector of offsets that was + passed by the caller to pcre_exec() or pcre_dfa_exec(). When + pcre_exec() is used, the contents can be inspected in order to extract + substrings that have been matched so far, in the same way as for + extracting substrings after a match has completed. For pcre_dfa_exec() + this field is not useful. The subject and subject_length fields contain copies of the values that were passed to pcre_exec(). - The start_match field contains the offset within the subject at which - the current match attempt started. If the pattern is not anchored, the + The start_match field contains the offset within the subject at which + the current match attempt started. If the pattern is not anchored, the callout function may be called several times from the same point in the pattern for different starting points in the subject. - The current_position field contains the offset within the subject of + The current_position field contains the offset within the subject of the current match pointer. - The capture_top field contains one more than the number of the highest - numbered captured substring so far. If no substrings have been cap- - tured, the value of capture_top is one. - - The capture_last field contains the number of the most recently cap- - tured substring. If no substrings have been captured, its value is -1. - - The callout_data field contains a value that is passed to pcre_exec() - by the caller specifically so that it can be passed back in callouts. - It is passed in the pcre_callout field of the pcre_extra data struc- - ture. If no such data was passed, the value of callout_data in a - pcre_callout block is NULL. There is a description of the pcre_extra + When the pcre_exec() function is used, the capture_top field contains + one more than the number of the highest numbered captured substring so + far. If no substrings have been captured, the value of capture_top is + one. This is always the case when pcre_dfa_exec() is used, because it + does not support captured substrings. + + The capture_last field contains the number of the most recently cap- + tured substring. If no substrings have been captured, its value is -1. + This is always the case when pcre_dfa_exec() is used. + + The callout_data field contains a value that is passed to pcre_exec() + or pcre_dfa_exec() specifically so that it can be passed back in call- + outs. It is passed in the pcre_callout field of the pcre_extra data + structure. If no such data was passed, the value of callout_data in a + pcre_callout block is NULL. There is a description of the pcre_extra structure in the pcreapi documentation. - The pattern_position field is present from version 1 of the pcre_call- + The pattern_position field is present from version 1 of the pcre_call- out structure. It contains the offset to the next item to be matched in the pattern string. - The next_item_length field is present from version 1 of the pcre_call- + The next_item_length field is present from version 1 of the pcre_call- out structure. It contains the length of the next item to be matched in - the pattern string. When the callout immediately precedes an alterna- - tion bar, a closing parenthesis, or the end of the pattern, the length - is zero. When the callout precedes an opening parenthesis, the length + the pattern string. When the callout immediately precedes an alterna- + tion bar, a closing parenthesis, or the end of the pattern, the length + is zero. When the callout precedes an opening parenthesis, the length is that of the entire subpattern. - The pattern_position and next_item_length fields are intended to help - in distinguishing between different automatic callouts, which all have + The pattern_position and next_item_length fields are intended to help + in distinguishing between different automatic callouts, which all have the same callout number. However, they are set for all callouts. RETURN VALUES - The external callout function returns an integer to PCRE. If the value - is zero, matching proceeds as normal. If the value is greater than - zero, matching fails at the current point, but backtracking to test - other matching possibilities goes ahead, just as if a lookahead asser- - tion had failed. If the value is less than zero, the match is aban- - doned, and pcre_exec() returns the negative value. + The external callout function returns an integer to PCRE. If the value + is zero, matching proceeds as normal. If the value is greater than + zero, matching fails at the current point, but the testing of other + matching possibilities goes ahead, just as if a lookahead assertion had + failed. If the value is less than zero, the match is abandoned, and + pcre_exec() (or pcre_dfa_exec()) returns the negative value. - Negative values should normally be chosen from the set of + Negative values should normally be chosen from the set of PCRE_ERROR_xxx values. In particular, PCRE_ERROR_NOMATCH forces a stan- - dard "no match" failure. The error number PCRE_ERROR_CALLOUT is - reserved for use by callout functions; it will never be used by PCRE + dard "no match" failure. The error number PCRE_ERROR_CALLOUT is + reserved for use by callout functions; it will never be used by PCRE itself. -Last updated: 09 September 2004 -Copyright (c) 1997-2004 University of Cambridge. +Last updated: 28 February 2005 +Copyright (c) 1997-2005 University of Cambridge. ----------------------------------------------------------------------------- -PCRE(3) PCRE(3) - NAME PCRE - Perl-compatible regular expressions + DIFFERENCES BETWEEN PCRE AND PERL This document describes the differences in the ways that PCRE and Perl @@ -1808,17 +2302,19 @@ DIFFERENCES BETWEEN PCRE AND PERL (m) Patterns compiled by PCRE can be saved and re-used at a later time, even on different hosts that have the other endianness. -Last updated: 09 September 2004 -Copyright (c) 1997-2004 University of Cambridge. ------------------------------------------------------------------------------ + (n) The alternative matching function (pcre_dfa_exec()) matches in a + different way and is not Perl-compatible. -PCRE(3) PCRE(3) +Last updated: 28 February 2005 +Copyright (c) 1997-2005 University of Cambridge. +----------------------------------------------------------------------------- NAME PCRE - Perl-compatible regular expressions + PCRE REGULAR EXPRESSION DETAILS The syntax and semantics of the regular expressions supported by PCRE @@ -1836,6 +2332,14 @@ PCRE REGULAR EXPRESSION DETAILS of UTF-8 features in the section on UTF-8 support in the main pcre page. + The remainder of this document discusses the patterns that are sup- + ported by PCRE when its main matching function, pcre_exec(), is used. + From release 6.0, PCRE offers a second matching function, + pcre_dfa_exec(), which matches using a different algorithm that is not + Perl-compatible. The advantages and disadvantages of the alternative + function, and how it differs from the normal function, are discussed in + the pcrematching page. + A regular expression is a pattern that is matched against a subject string from left to right. Most characters stand for themselves in a pattern, and match the corresponding characters in the subject. As a @@ -1843,15 +2347,24 @@ PCRE REGULAR EXPRESSION DETAILS The quick brown fox - matches a portion of a subject string that is identical to itself. The - power of regular expressions comes from the ability to include alterna- - tives and repetitions in the pattern. These are encoded in the pattern - by the use of metacharacters, which do not stand for themselves but - instead are interpreted in some special way. - - There are two different sets of metacharacters: those that are recog- - nized anywhere in the pattern except within square brackets, and those - that are recognized in square brackets. Outside square brackets, the + matches a portion of a subject string that is identical to itself. When + caseless matching is specified (the PCRE_CASELESS option), letters are + matched independently of case. In UTF-8 mode, PCRE always understands + the concept of case for characters whose values are less than 128, so + caseless matching is always possible. For characters with higher val- + ues, the concept of case is supported if PCRE is compiled with Unicode + property support, but not otherwise. If you want to use caseless + matching for characters 128 and above, you must ensure that PCRE is + compiled with Unicode property support as well as with UTF-8 support. + + The power of regular expressions comes from the ability to include + alternatives and repetitions in the pattern. These are encoded in the + pattern by the use of metacharacters, which do not stand for themselves + but instead are interpreted in some special way. + + There are two different sets of metacharacters: those that are recog- + nized anywhere in the pattern except within square brackets, and those + that are recognized in square brackets. Outside square brackets, the metacharacters are as follows: \ general escape character with several uses @@ -1870,7 +2383,7 @@ PCRE REGULAR EXPRESSION DETAILS also "possessive quantifier" { start min/max quantifier - Part of a pattern that is in square brackets is called a "character + Part of a pattern that is in square brackets is called a "character class". In a character class the only metacharacters are: \ general escape character @@ -1880,33 +2393,33 @@ PCRE REGULAR EXPRESSION DETAILS syntax) ] terminates the character class - The following sections describe the use of each of the metacharacters. + The following sections describe the use of each of the metacharacters. BACKSLASH The backslash character has several uses. Firstly, if it is followed by - a non-alphanumeric character, it takes away any special meaning that - character may have. This use of backslash as an escape character + a non-alphanumeric character, it takes away any special meaning that + character may have. This use of backslash as an escape character applies both inside and outside character classes. - For example, if you want to match a * character, you write \* in the - pattern. This escaping action applies whether or not the following - character would otherwise be interpreted as a metacharacter, so it is - always safe to precede a non-alphanumeric with backslash to specify - that it stands for itself. In particular, if you want to match a back- + For example, if you want to match a * character, you write \* in the + pattern. This escaping action applies whether or not the following + character would otherwise be interpreted as a metacharacter, so it is + always safe to precede a non-alphanumeric with backslash to specify + that it stands for itself. In particular, if you want to match a back- slash, you write \\. - If a pattern is compiled with the PCRE_EXTENDED option, whitespace in - the pattern (other than in a character class) and characters between a + If a pattern is compiled with the PCRE_EXTENDED option, whitespace in + the pattern (other than in a character class) and characters between a # outside a character class and the next newline character are ignored. - An escaping backslash can be used to include a whitespace or # charac- + An escaping backslash can be used to include a whitespace or # charac- ter as part of the pattern. - If you want to remove the special meaning from a sequence of charac- - ters, you can do so by putting them between \Q and \E. This is differ- - ent from Perl in that $ and @ are handled as literals in \Q...\E - sequences in PCRE, whereas in Perl, $ and @ cause variable interpola- + If you want to remove the special meaning from a sequence of charac- + ters, you can do so by putting them between \Q and \E. This is differ- + ent from Perl in that $ and @ are handled as literals in \Q...\E + sequences in PCRE, whereas in Perl, $ and @ cause variable interpola- tion. Note the following examples: Pattern PCRE matches Perl matches @@ -1916,16 +2429,16 @@ BACKSLASH \Qabc\$xyz\E abc\$xyz abc\$xyz \Qabc\E\$\Qxyz\E abc$xyz abc$xyz - The \Q...\E sequence is recognized both inside and outside character + The \Q...\E sequence is recognized both inside and outside character classes. Non-printing characters A second use of backslash provides a way of encoding non-printing char- - acters in patterns in a visible manner. There is no restriction on the - appearance of non-printing characters, apart from the binary zero that - terminates a pattern, but when a pattern is being prepared by text - editing, it is usually easier to use one of the following escape + acters in patterns in a visible manner. There is no restriction on the + appearance of non-printing characters, apart from the binary zero that + terminates a pattern, but when a pattern is being prepared by text + editing, it is usually easier to use one of the following escape sequences than the binary character it represents: \a alarm, that is, the BEL character (hex 07) @@ -1939,44 +2452,44 @@ BACKSLASH \xhh character with hex code hh \x{hhh..} character with hex code hhh... (UTF-8 mode only) - The precise effect of \cx is as follows: if x is a lower case letter, - it is converted to upper case. Then bit 6 of the character (hex 40) is - inverted. Thus \cz becomes hex 1A, but \c{ becomes hex 3B, while \c; + The precise effect of \cx is as follows: if x is a lower case letter, + it is converted to upper case. Then bit 6 of the character (hex 40) is + inverted. Thus \cz becomes hex 1A, but \c{ becomes hex 3B, while \c; becomes hex 7B. - After \x, from zero to two hexadecimal digits are read (letters can be - in upper or lower case). In UTF-8 mode, any number of hexadecimal dig- - its may appear between \x{ and }, but the value of the character code - must be less than 2**31 (that is, the maximum hexadecimal value is - 7FFFFFFF). If characters other than hexadecimal digits appear between - \x{ and }, or if there is no terminating }, this form of escape is not - recognized. Instead, the initial \x will be interpreted as a basic hex- - adecimal escape, with no following digits, giving a character whose + After \x, from zero to two hexadecimal digits are read (letters can be + in upper or lower case). In UTF-8 mode, any number of hexadecimal dig- + its may appear between \x{ and }, but the value of the character code + must be less than 2**31 (that is, the maximum hexadecimal value is + 7FFFFFFF). If characters other than hexadecimal digits appear between + \x{ and }, or if there is no terminating }, this form of escape is not + recognized. Instead, the initial \x will be interpreted as a basic + hexadecimal escape, with no following digits, giving a character whose value is zero. Characters whose value is less than 256 can be defined by either of the - two syntaxes for \x when PCRE is in UTF-8 mode. There is no difference - in the way they are handled. For example, \xdc is exactly the same as + two syntaxes for \x when PCRE is in UTF-8 mode. There is no difference + in the way they are handled. For example, \xdc is exactly the same as \x{dc}. - After \0 up to two further octal digits are read. In both cases, if - there are fewer than two digits, just those that are present are used. - Thus the sequence \0\x\07 specifies two binary zeros followed by a BEL - character (code value 7). Make sure you supply two digits after the - initial zero if the pattern character that follows is itself an octal + After \0 up to two further octal digits are read. In both cases, if + there are fewer than two digits, just those that are present are used. + Thus the sequence \0\x\07 specifies two binary zeros followed by a BEL + character (code value 7). Make sure you supply two digits after the + initial zero if the pattern character that follows is itself an octal digit. The handling of a backslash followed by a digit other than 0 is compli- cated. Outside a character class, PCRE reads it and any following dig- - its as a decimal number. If the number is less than 10, or if there + its as a decimal number. If the number is less than 10, or if there have been at least that many previous capturing left parentheses in the - expression, the entire sequence is taken as a back reference. A - description of how this works is given later, following the discussion + expression, the entire sequence is taken as a back reference. A + description of how this works is given later, following the discussion of parenthesized subpatterns. - Inside a character class, or if the decimal number is greater than 9 - and there have not been that many capturing subpatterns, PCRE re-reads - up to three octal digits following the backslash, and generates a sin- + Inside a character class, or if the decimal number is greater than 9 + and there have not been that many capturing subpatterns, PCRE re-reads + up to three octal digits following the backslash, and generates a sin- gle byte from the least significant 8 bits of the value. Any subsequent digits stand for themselves. For example: @@ -1995,19 +2508,19 @@ BACKSLASH \81 is either a back reference, or a binary zero followed by the two characters "8" and "1" - Note that octal values of 100 or greater must not be introduced by a + Note that octal values of 100 or greater must not be introduced by a leading zero, because no more than three octal digits are ever read. - All the sequences that define a single byte value or a single UTF-8 + All the sequences that define a single byte value or a single UTF-8 character (in UTF-8 mode) can be used both inside and outside character - classes. In addition, inside a character class, the sequence \b is + classes. In addition, inside a character class, the sequence \b is interpreted as the backspace character (hex 08), and the sequence \X is - interpreted as the character "X". Outside a character class, these + interpreted as the character "X". Outside a character class, these sequences have different meanings (see below). Generic character types - The third use of backslash is for specifying generic character types. + The third use of backslash is for specifying generic character types. The following are always recognized: \d any decimal digit @@ -2018,48 +2531,48 @@ BACKSLASH \W any "non-word" character Each pair of escape sequences partitions the complete set of characters - into two disjoint sets. Any given character matches one, and only one, + into two disjoint sets. Any given character matches one, and only one, of each pair. These character type sequences can appear both inside and outside char- - acter classes. They each match one character of the appropriate type. - If the current matching point is at the end of the subject string, all + acter classes. They each match one character of the appropriate type. + If the current matching point is at the end of the subject string, all of them fail, since there is no character to match. - For compatibility with Perl, \s does not match the VT character (code - 11). This makes it different from the the POSIX "space" class. The \s + For compatibility with Perl, \s does not match the VT character (code + 11). This makes it different from the the POSIX "space" class. The \s characters are HT (9), LF (10), FF (12), CR (13), and space (32). A "word" character is an underscore or any character less than 256 that - is a letter or digit. The definition of letters and digits is con- - trolled by PCRE's low-valued character tables, and may vary if locale- - specific matching is taking place (see "Locale support" in the pcreapi - page). For example, in the "fr_FR" (French) locale, some character - codes greater than 128 are used for accented letters, and these are + is a letter or digit. The definition of letters and digits is con- + trolled by PCRE's low-valued character tables, and may vary if locale- + specific matching is taking place (see "Locale support" in the pcreapi + page). For example, in the "fr_FR" (French) locale, some character + codes greater than 128 are used for accented letters, and these are matched by \w. - In UTF-8 mode, characters with values greater than 128 never match \d, + In UTF-8 mode, characters with values greater than 128 never match \d, \s, or \w, and always match \D, \S, and \W. This is true even when Uni- code character property support is available. Unicode character properties When PCRE is built with Unicode character property support, three addi- - tional escape sequences to match generic character types are available + tional escape sequences to match generic character types are available when UTF-8 mode is selected. They are: \p{xx} a character with the xx property \P{xx} a character without the xx property \X an extended Unicode sequence - The property names represented by xx above are limited to the Unicode - general category properties. Each character has exactly one such prop- - erty, specified by a two-letter abbreviation. For compatibility with - Perl, negation can be specified by including a circumflex between the - opening brace and the property name. For example, \p{^Lu} is the same + The property names represented by xx above are limited to the Unicode + general category properties. Each character has exactly one such prop- + erty, specified by a two-letter abbreviation. For compatibility with + Perl, negation can be specified by including a circumflex between the + opening brace and the property name. For example, \p{^Lu} is the same as \P{Lu}. - If only one letter is specified with \p or \P, it includes all the + If only one letter is specified with \p or \P, it includes all the properties that start with that letter. In this case, in the absence of negation, the curly brackets in the escape sequence are optional; these two examples have the same effect: @@ -2113,33 +2626,33 @@ BACKSLASH Zp Paragraph separator Zs Space separator - Extended properties such as "Greek" or "InMusicalSymbols" are not sup- + Extended properties such as "Greek" or "InMusicalSymbols" are not sup- ported by PCRE. - Specifying caseless matching does not affect these escape sequences. + Specifying caseless matching does not affect these escape sequences. For example, \p{Lu} always matches only upper case letters. - The \X escape matches any number of Unicode characters that form an + The \X escape matches any number of Unicode characters that form an extended Unicode sequence. \X is equivalent to (?>\PM\pM*) - That is, it matches a character without the "mark" property, followed - by zero or more characters with the "mark" property, and treats the - sequence as an atomic group (see below). Characters with the "mark" + That is, it matches a character without the "mark" property, followed + by zero or more characters with the "mark" property, and treats the + sequence as an atomic group (see below). Characters with the "mark" property are typically accents that affect the preceding character. - Matching characters by Unicode property is not fast, because PCRE has - to search a structure that contains data for over fifteen thousand + Matching characters by Unicode property is not fast, because PCRE has + to search a structure that contains data for over fifteen thousand characters. That is why the traditional escape sequences such as \d and \w do not use Unicode properties in PCRE. Simple assertions The fourth use of backslash is for certain simple assertions. An asser- - tion specifies a condition that has to be met at a particular point in - a match, without consuming any characters from the subject string. The - use of subpatterns for more complicated assertions is described below. + tion specifies a condition that has to be met at a particular point in + a match, without consuming any characters from the subject string. The + use of subpatterns for more complicated assertions is described below. The backslashed assertions are: \b matches at a word boundary @@ -2149,42 +2662,42 @@ BACKSLASH \z matches at end of subject \G matches at first matching position in subject - These assertions may not appear in character classes (but note that \b + These assertions may not appear in character classes (but note that \b has a different meaning, namely the backspace character, inside a char- acter class). - A word boundary is a position in the subject string where the current - character and the previous character do not both match \w or \W (i.e. - one matches \w and the other matches \W), or the start or end of the + A word boundary is a position in the subject string where the current + character and the previous character do not both match \w or \W (i.e. + one matches \w and the other matches \W), or the start or end of the string if the first or last character matches \w, respectively. - The \A, \Z, and \z assertions differ from the traditional circumflex + The \A, \Z, and \z assertions differ from the traditional circumflex and dollar (described in the next section) in that they only ever match - at the very start and end of the subject string, whatever options are - set. Thus, they are independent of multiline mode. These three asser- + at the very start and end of the subject string, whatever options are + set. Thus, they are independent of multiline mode. These three asser- tions are not affected by the PCRE_NOTBOL or PCRE_NOTEOL options, which - affect only the behaviour of the circumflex and dollar metacharacters. - However, if the startoffset argument of pcre_exec() is non-zero, indi- + affect only the behaviour of the circumflex and dollar metacharacters. + However, if the startoffset argument of pcre_exec() is non-zero, indi- cating that matching is to start at a point other than the beginning of - the subject, \A can never match. The difference between \Z and \z is - that \Z matches before a newline that is the last character of the - string as well as at the end of the string, whereas \z matches only at + the subject, \A can never match. The difference between \Z and \z is + that \Z matches before a newline that is the last character of the + string as well as at the end of the string, whereas \z matches only at the end. - The \G assertion is true only when the current matching position is at - the start point of the match, as specified by the startoffset argument - of pcre_exec(). It differs from \A when the value of startoffset is - non-zero. By calling pcre_exec() multiple times with appropriate argu- + The \G assertion is true only when the current matching position is at + the start point of the match, as specified by the startoffset argument + of pcre_exec(). It differs from \A when the value of startoffset is + non-zero. By calling pcre_exec() multiple times with appropriate argu- ments, you can mimic Perl's /g option, and it is in this kind of imple- mentation where \G can be useful. - Note, however, that PCRE's interpretation of \G, as the start of the + Note, however, that PCRE's interpretation of \G, as the start of the current match, is subtly different from Perl's, which defines it as the - end of the previous match. In Perl, these can be different when the - previously matched string was empty. Because PCRE does just one match + end of the previous match. In Perl, these can be different when the + previously matched string was empty. Because PCRE does just one match at a time, it cannot reproduce this behaviour. - If all the alternatives of a pattern begin with \G, the expression is + If all the alternatives of a pattern begin with \G, the expression is anchored to the starting match position, and the "anchored" flag is set in the compiled regular expression. @@ -2192,73 +2705,73 @@ BACKSLASH CIRCUMFLEX AND DOLLAR Outside a character class, in the default matching mode, the circumflex - character is an assertion that is true only if the current matching - point is at the start of the subject string. If the startoffset argu- - ment of pcre_exec() is non-zero, circumflex can never match if the - PCRE_MULTILINE option is unset. Inside a character class, circumflex + character is an assertion that is true only if the current matching + point is at the start of the subject string. If the startoffset argu- + ment of pcre_exec() is non-zero, circumflex can never match if the + PCRE_MULTILINE option is unset. Inside a character class, circumflex has an entirely different meaning (see below). - Circumflex need not be the first character of the pattern if a number - of alternatives are involved, but it should be the first thing in each - alternative in which it appears if the pattern is ever to match that - branch. If all possible alternatives start with a circumflex, that is, - if the pattern is constrained to match only at the start of the sub- - ject, it is said to be an "anchored" pattern. (There are also other + Circumflex need not be the first character of the pattern if a number + of alternatives are involved, but it should be the first thing in each + alternative in which it appears if the pattern is ever to match that + branch. If all possible alternatives start with a circumflex, that is, + if the pattern is constrained to match only at the start of the sub- + ject, it is said to be an "anchored" pattern. (There are also other constructs that can cause a pattern to be anchored.) - A dollar character is an assertion that is true only if the current - matching point is at the end of the subject string, or immediately + A dollar character is an assertion that is true only if the current + matching point is at the end of the subject string, or immediately before a newline character that is the last character in the string (by - default). Dollar need not be the last character of the pattern if a - number of alternatives are involved, but it should be the last item in - any branch in which it appears. Dollar has no special meaning in a + default). Dollar need not be the last character of the pattern if a + number of alternatives are involved, but it should be the last item in + any branch in which it appears. Dollar has no special meaning in a character class. - The meaning of dollar can be changed so that it matches only at the - very end of the string, by setting the PCRE_DOLLAR_ENDONLY option at + The meaning of dollar can be changed so that it matches only at the + very end of the string, by setting the PCRE_DOLLAR_ENDONLY option at compile time. This does not affect the \Z assertion. The meanings of the circumflex and dollar characters are changed if the PCRE_MULTILINE option is set. When this is the case, they match immedi- - ately after and immediately before an internal newline character, - respectively, in addition to matching at the start and end of the sub- - ject string. For example, the pattern /^abc$/ matches the subject - string "def\nabc" (where \n represents a newline character) in multi- + ately after and immediately before an internal newline character, + respectively, in addition to matching at the start and end of the sub- + ject string. For example, the pattern /^abc$/ matches the subject + string "def\nabc" (where \n represents a newline character) in multi- line mode, but not otherwise. Consequently, patterns that are anchored - in single line mode because all branches start with ^ are not anchored - in multiline mode, and a match for circumflex is possible when the - startoffset argument of pcre_exec() is non-zero. The PCRE_DOL- + in single line mode because all branches start with ^ are not anchored + in multiline mode, and a match for circumflex is possible when the + startoffset argument of pcre_exec() is non-zero. The PCRE_DOL- LAR_ENDONLY option is ignored if PCRE_MULTILINE is set. - Note that the sequences \A, \Z, and \z can be used to match the start - and end of the subject in both modes, and if all branches of a pattern - start with \A it is always anchored, whether PCRE_MULTILINE is set or + Note that the sequences \A, \Z, and \z can be used to match the start + and end of the subject in both modes, and if all branches of a pattern + start with \A it is always anchored, whether PCRE_MULTILINE is set or not. FULL STOP (PERIOD, DOT) Outside a character class, a dot in the pattern matches any one charac- - ter in the subject, including a non-printing character, but not (by - default) newline. In UTF-8 mode, a dot matches any UTF-8 character, + ter in the subject, including a non-printing character, but not (by + default) newline. In UTF-8 mode, a dot matches any UTF-8 character, which might be more than one byte long, except (by default) newline. If - the PCRE_DOTALL option is set, dots match newlines as well. The han- - dling of dot is entirely independent of the handling of circumflex and - dollar, the only relationship being that they both involve newline + the PCRE_DOTALL option is set, dots match newlines as well. The han- + dling of dot is entirely independent of the handling of circumflex and + dollar, the only relationship being that they both involve newline characters. Dot has no special meaning in a character class. MATCHING A SINGLE BYTE Outside a character class, the escape sequence \C matches any one byte, - both in and out of UTF-8 mode. Unlike a dot, it can match a newline. - The feature is provided in Perl in order to match individual bytes in - UTF-8 mode. Because it breaks up UTF-8 characters into individual - bytes, what remains in the string may be a malformed UTF-8 string. For + both in and out of UTF-8 mode. Unlike a dot, it can match a newline. + The feature is provided in Perl in order to match individual bytes in + UTF-8 mode. Because it breaks up UTF-8 characters into individual + bytes, what remains in the string may be a malformed UTF-8 string. For this reason, the \C escape sequence is best avoided. - PCRE does not allow \C to appear in lookbehind assertions (described - below), because in UTF-8 mode this would make it impossible to calcu- + PCRE does not allow \C to appear in lookbehind assertions (described + below), because in UTF-8 mode this would make it impossible to calcu- late the length of the lookbehind. @@ -2267,35 +2780,40 @@ SQUARE BRACKETS AND CHARACTER CLASSES An opening square bracket introduces a character class, terminated by a closing square bracket. A closing square bracket on its own is not spe- cial. If a closing square bracket is required as a member of the class, - it should be the first data character in the class (after an initial + it should be the first data character in the class (after an initial circumflex, if present) or escaped with a backslash. - A character class matches a single character in the subject. In UTF-8 - mode, the character may occupy more than one byte. A matched character + A character class matches a single character in the subject. In UTF-8 + mode, the character may occupy more than one byte. A matched character must be in the set of characters defined by the class, unless the first - character in the class definition is a circumflex, in which case the - subject character must not be in the set defined by the class. If a - circumflex is actually required as a member of the class, ensure it is + character in the class definition is a circumflex, in which case the + subject character must not be in the set defined by the class. If a + circumflex is actually required as a member of the class, ensure it is not the first character, or escape it with a backslash. - For example, the character class [aeiou] matches any lower case vowel, - while [^aeiou] matches any character that is not a lower case vowel. + For example, the character class [aeiou] matches any lower case vowel, + while [^aeiou] matches any character that is not a lower case vowel. Note that a circumflex is just a convenient notation for specifying the - characters that are in the class by enumerating those that are not. A - class that starts with a circumflex is not an assertion: it still con- - sumes a character from the subject string, and therefore it fails if + characters that are in the class by enumerating those that are not. A + class that starts with a circumflex is not an assertion: it still con- + sumes a character from the subject string, and therefore it fails if the current pointer is at the end of the string. - In UTF-8 mode, characters with values greater than 255 can be included - in a class as a literal string of bytes, or by using the \x{ escaping + In UTF-8 mode, characters with values greater than 255 can be included + in a class as a literal string of bytes, or by using the \x{ escaping mechanism. - When caseless matching is set, any letters in a class represent both - their upper case and lower case versions, so for example, a caseless - [aeiou] matches "A" as well as "a", and a caseless [^aeiou] does not - match "A", whereas a caseful version would. When running in UTF-8 mode, - PCRE supports the concept of case for characters with values greater - than 128 only when it is compiled with Unicode property support. + When caseless matching is set, any letters in a class represent both + their upper case and lower case versions, so for example, a caseless + [aeiou] matches "A" as well as "a", and a caseless [^aeiou] does not + match "A", whereas a caseful version would. In UTF-8 mode, PCRE always + understands the concept of case for characters whose values are less + than 128, so caseless matching is always possible. For characters with + higher values, the concept of case is supported if PCRE is compiled + with Unicode property support, but not otherwise. If you want to use + caseless matching for characters 128 and above, you must ensure that + PCRE is compiled with Unicode property support as well as with UTF-8 + support. The newline character is never treated in any special way in character classes, whatever the setting of the PCRE_DOTALL or PCRE_MULTILINE @@ -3215,24 +3733,23 @@ CALLOUTS gether. A complete description of the interface to the callout function is given in the pcrecallout documentation. -Last updated: 09 September 2004 -Copyright (c) 1997-2004 University of Cambridge. +Last updated: 28 February 2005 +Copyright (c) 1997-2005 University of Cambridge. ----------------------------------------------------------------------------- -PCRE(3) PCRE(3) - NAME PCRE - Perl-compatible regular expressions + PARTIAL MATCHING IN PCRE In normal use of PCRE, if the subject string that is passed to - pcre_exec() matches as far as it goes, but is too short to match the - entire pattern, PCRE_ERROR_NOMATCH is returned. There are circumstances - where it might be helpful to distinguish this case from other cases in - which there is no match. + pcre_exec() or pcre_dfa_exec() matches as far as it goes, but is too + short to match the entire pattern, PCRE_ERROR_NOMATCH is returned. + There are circumstances where it might be helpful to distinguish this + case from other cases in which there is no match. Consider, for example, an application where a human is required to type in data for a field with specific formatting requirements. An example @@ -3248,11 +3765,19 @@ PARTIAL MATCHING IN PCRE until the entire string has been entered. PCRE supports the concept of partial matching by means of the PCRE_PAR- - TIAL option, which can be set when calling pcre_exec(). When this is - done, the return code PCRE_ERROR_NOMATCH is converted into - PCRE_ERROR_PARTIAL if at any time during the matching process the - entire subject string matched part of the pattern. No captured data is - set when this occurs. + TIAL option, which can be set when calling pcre_exec() or + pcre_dfa_exec(). When this flag is set for pcre_exec(), the return code + PCRE_ERROR_NOMATCH is converted into PCRE_ERROR_PARTIAL if at any time + during the matching process the last part of the subject string matched + part of the pattern. Unfortunately, for non-anchored matching, it is + not possible to obtain the position of the start of the partial match. + No captured data is set when PCRE_ERROR_PARTIAL is returned. + + When PCRE_PARTIAL is set for pcre_dfa_exec(), the return code + PCRE_ERROR_NOMATCH is converted into PCRE_ERROR_PARTIAL if the end of + the subject is reached, there have been no complete matches, but there + is still at least one matching possibility. The portion of the string + that provided the partial match is set as the first matching string. Using PCRE_PARTIAL disables one of PCRE's optimizations. PCRE remembers the last literal byte in a pattern, and abandons matching immediately @@ -3263,8 +3788,9 @@ PARTIAL MATCHING IN PCRE RESTRICTED PATTERNS FOR PCRE_PARTIAL Because of the way certain internal optimizations are implemented in - PCRE, the PCRE_PARTIAL option cannot be used with all patterns. - Repeated single characters such as + the pcre_exec() function, the PCRE_PARTIAL option cannot be used with + all patterns. These restrictions do not apply when pcre_dfa_exec() is + used. For pcre_exec(), repeated single characters such as a{2,4} @@ -3272,26 +3798,26 @@ RESTRICTED PATTERNS FOR PCRE_PARTIAL \d+ - are not permitted if the maximum number of occurrences is greater than + are not permitted if the maximum number of occurrences is greater than one. Optional items such as \d? (where the maximum is one) are permit- - ted. Quantifiers with any values are permitted after parentheses, so + ted. Quantifiers with any values are permitted after parentheses, so the invalid examples above can be coded thus: (a){2,4} (\d)+ - These constructions run more slowly, but for the kinds of application - that are envisaged for this facility, this is not felt to be a major + These constructions run more slowly, but for the kinds of application + that are envisaged for this facility, this is not felt to be a major restriction. - If PCRE_PARTIAL is set for a pattern that does not conform to the - restrictions, pcre_exec() returns the error code PCRE_ERROR_BADPARTIAL + If PCRE_PARTIAL is set for a pattern that does not conform to the + restrictions, pcre_exec() returns the error code PCRE_ERROR_BADPARTIAL (-13). EXAMPLE OF PARTIAL MATCHING USING PCRETEST - If the escape sequence \P is present in a pcretest data line, the + If the escape sequence \P is present in a pcretest data line, the PCRE_PARTIAL flag is used for the match. Here is a run of pcretest that uses the date example quoted above: @@ -3308,21 +3834,103 @@ EXAMPLE OF PARTIAL MATCHING USING PCRETEST data> jP No match - The first data string is matched completely, so pcretest shows the - matched substrings. The remaining four strings do not match the com- - plete pattern, but the first two are partial matches. + The first data string is matched completely, so pcretest shows the + matched substrings. The remaining four strings do not match the com- + plete pattern, but the first two are partial matches. The same test, + using DFA matching (by means of the \D escape sequence), produces the + following output: -Last updated: 08 September 2004 -Copyright (c) 1997-2004 University of Cambridge. ------------------------------------------------------------------------------ + re> /^?(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)$/ + data> 25jun04\P\D + 0: 25jun04 + data> 23dec3\P\D + Partial match: 23dec3 + data> 3ju\P\D + Partial match: 3ju + data> 3juj\P\D + No match + data> j\P\D + No match + + Notice that in this case the portion of the string that was matched is + made available. + + +MULTI-SEGMENT MATCHING WITH pcre_dfa_exec() + + When a partial match has been found using pcre_dfa_exec(), it is possi- + ble to continue the match by providing additional subject data and + calling pcre_dfa_exec() again with the PCRE_DFA_RESTART option and the + same working space (where details of the previous partial match are + stored). Here is an example using pcretest, where the \R escape + sequence sets the PCRE_DFA_RESTART option and the \D escape sequence + requests the use of pcre_dfa_exec(): + + re> /^?(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)$/ + data> 23ja\P\D + Partial match: 23ja + data> n05\R\D + 0: n05 + + The first call has "23ja" as the subject, and requests partial match- + ing; the second call has "n05" as the subject for the continued + (restarted) match. Notice that when the match is complete, only the + last part is shown; PCRE does not retain the previously partially- + matched string. It is up to the calling program to do that if it needs + to. + + This facility can be used to pass very long subject strings to + pcre_dfa_exec(). However, some care is needed for certain types of pat- + tern. + + 1. If the pattern contains tests for the beginning or end of a line, + you need to pass the PCRE_NOTBOL or PCRE_NOTEOL options, as appropri- + ate, when the subject string for any call does not contain the begin- + ning or end of a line. + + 2. If the pattern contains backward assertions (including \b or \B), + you need to arrange for some overlap in the subject strings to allow + for this. For example, you could pass the subject in chunks that were + 500 bytes long, but in a buffer of 700 bytes, with the starting offset + set to 200 and the previous 200 bytes at the start of the buffer. + + 3. Matching a subject string that is split into multiple segments does + not always produce exactly the same result as matching over one single + long string. The difference arises when there are multiple matching + possibilities, because a partial match result is given only when there + are no completed matches in a call to fBpcre_dfa_exec(). This means + that as soon as the shortest match has been found, continuation to a + new subject segment is no longer possible. Consider this pcretest + example: -PCRE(3) PCRE(3) + re> /dog(sbody)?/ + data> do\P\D + Partial match: do + data> gsb\R\P\D + 0: g + data> dogsbody\D + 0: dogsbody + 1: dog + + The pattern matches the words "dog" or "dogsbody". When the subject is + presented in several parts ("do" and "gsb" being the first two) the + match stops when "dog" has been found, and it is not possible to con- + tinue. On the other hand, if "dogsbody" is presented as a single + string, both matches are found. + + Because of this phenomenon, it does not usually make sense to end a + pattern that is going to be matched in this way with a variable repeat. + +Last updated: 28 February 2005 +Copyright (c) 1997-2005 University of Cambridge. +----------------------------------------------------------------------------- NAME PCRE - Perl-compatible regular expressions + SAVING AND RE-USING PRECOMPILED PCRE PATTERNS If you are running an application that uses a large number of regular @@ -3391,16 +3999,18 @@ SAVING A COMPILED PATTERN RE-USING A PRECOMPILED PATTERN Re-using a precompiled pattern is straightforward. Having reloaded it - into main memory, you pass its pointer to pcre_exec() in the usual way. - This should work even on another host, and even if that host has the - opposite endianness to the one where the pattern was compiled. - - However, if you passed a pointer to custom character tables when the - pattern was compiled (the tableptr argument of pcre_compile()), you - must now pass a similar pointer to pcre_exec(), because the value saved - with the compiled pattern will obviously be nonsense. A field in a - pcre_extra() block is used to pass this data, as described in the sec- - tion on matching a pattern in the pcreapi documentation. + into main memory, you pass its pointer to pcre_exec() or + pcre_dfa_exec() in the usual way. This should work even on another + host, and even if that host has the opposite endianness to the one + where the pattern was compiled. + + However, if you passed a pointer to custom character tables when the + pattern was compiled (the tableptr argument of pcre_compile()), you + must now pass a similar pointer to pcre_exec() or pcre_dfa_exec(), + because the value saved with the compiled pattern will obviously be + nonsense. A field in a pcre_extra() block is used to pass this data, as + described in the section on matching a pattern in the pcreapi documen- + tation. If you did not provide custom character tables when the pattern was compiled, the pointer in the compiled pattern is NULL, which causes @@ -3411,29 +4021,29 @@ RE-USING A PRECOMPILED PATTERN your own pcre_extra data block and set the study_data field to point to the reloaded study data. You must also set the PCRE_EXTRA_STUDY_DATA bit in the flags field to indicate that study data is present. Then - pass the pcre_extra block to pcre_exec() in the usual way. + pass the pcre_extra block to pcre_exec() or pcre_dfa_exec() in the + usual way. COMPATIBILITY WITH DIFFERENT PCRE RELEASES - The layout of the control block that is at the start of the data that - makes up a compiled pattern was changed for release 5.0. If you have - any saved patterns that were compiled with previous releases (not a - facility that was previously advertised), you will have to recompile - them for release 5.0. However, from now on, it should be possible to + The layout of the control block that is at the start of the data that + makes up a compiled pattern was changed for release 5.0. If you have + any saved patterns that were compiled with previous releases (not a + facility that was previously advertised), you will have to recompile + them for release 5.0. However, from now on, it should be possible to make changes in a compabible manner. -Last updated: 10 September 2004 -Copyright (c) 1997-2004 University of Cambridge. +Last updated: 28 February 2005 +Copyright (c) 1997-2005 University of Cambridge. ----------------------------------------------------------------------------- -PCRE(3) PCRE(3) - NAME PCRE - Perl-compatible regular expressions + PCRE PERFORMANCE Certain items that may appear in regular expression patterns are more @@ -3469,9 +4079,9 @@ PCRE PERFORMANCE If you are using such a pattern with subject strings that do not con- tain newlines, the best performance is obtained by setting PCRE_DOTALL, - or starting the pattern with ^.* to indicate explicit anchoring. That - saves PCRE from having to scan along the subject looking for a newline - to restart at. + or starting the pattern with ^.* or ^.*? to indicate explicit anchor- + ing. That saves PCRE from having to scan along the subject looking for + a newline to restart at. Beware of patterns that contain nested indefinite repeats. These can take a long time to run when applied to a string that does not match. @@ -3492,9 +4102,9 @@ PCRE PERFORMANCE (a+)*b where a literal character follows. Before embarking on the standard - matching procedure, PCRE checks that there is a "b" later in the - subject string, and if there is not, it fails the match immediately. - However, when there is no following literal this optimization cannot be + matching procedure, PCRE checks that there is a "b" later in the sub- + ject string, and if there is not, it fails the match immediately. How- + ever, when there is no following literal this optimization cannot be used. You can see the difference by comparing the behaviour of (a+)*\d @@ -3506,17 +4116,16 @@ PCRE PERFORMANCE In many cases, the solution to this kind of performance issue is to use an atomic group or a possessive quantifier. -Last updated: 09 September 2004 -Copyright (c) 1997-2004 University of Cambridge. +Last updated: 28 February 2005 +Copyright (c) 1997-2005 University of Cambridge. ----------------------------------------------------------------------------- -PCRE(3) PCRE(3) - NAME PCRE - Perl-compatible regular expressions. + SYNOPSIS OF POSIX API #include <pcreposix.h> @@ -3537,7 +4146,7 @@ DESCRIPTION This set of functions provides a POSIX-style API to the PCRE regular expression package. See the pcreapi documentation for a description of - PCRE's native API, which contains additional functionality. + PCRE's native API, which contains much additional functionality. The functions described here are just wrapper functions that ultimately call the PCRE native API. Their prototypes are defined in the @@ -3581,6 +4190,12 @@ COMPILING A PATTERN The argument cflags is either zero, or contains one or more of the bits defined by the following macros: + REG_DOTALL + + The PCRE_DOTALL option is set when the expression is passed for compi- + lation to the native function. Note that REG_DOTALL is not part of the + POSIX standard. + REG_ICASE The PCRE_CASELESS option is set when the expression is passed for com- @@ -3692,21 +4307,235 @@ MEMORY USAGE AUTHOR - Philip Hazel <ph10@cam.ac.uk> + Philip Hazel University Computing Service, Cambridge CB2 3QG, England. -Last updated: 07 September 2004 -Copyright (c) 1997-2004 University of Cambridge. +Last updated: 28 February 2005 +Copyright (c) 1997-2005 University of Cambridge. ----------------------------------------------------------------------------- -PCRE(3) PCRE(3) + + +NAME + PCRE - Perl-compatible regular expressions. + + +SYNOPSIS OF C++ WRAPPER + + #include <pcrecpp.h> + + +DESCRIPTION + + The C++ wrapper for PCRE was provided by Google Inc. This brief man + page was constructed from the notes in the pcrecpp.h file, which should + be consulted for further details. + + +MATCHING INTERFACE + + The "FullMatch" operation checks that supplied text matches a supplied + pattern exactly. If pointer arguments are supplied, it copies matched + sub-strings that match sub-patterns into them. + + Example: successful match + pcrecpp::RE re("h.*o"); + re.FullMatch("hello"); + + Example: unsuccessful match (requires full match): + pcrecpp::RE re("e"); + !re.FullMatch("hello"); + + Example: creating a temporary RE object: + pcrecpp::RE("h.*o").FullMatch("hello"); + + You can pass in a "const char*" or a "string" for "text". The examples + below tend to use a const char*. You can, as in the different examples + above, store the RE object explicitly in a variable or use a temporary + RE object. The examples below use one mode or the other arbitrarily. + Either could correctly be used for any of these examples. + + You must supply extra pointer arguments to extract matched subpieces. + + Example: extracts "ruby" into "s" and 1234 into "i" + int i; + string s; + pcrecpp::RE re("(\\w+):(\\d+)"); + re.FullMatch("ruby:1234", &s, &i); + + Example: does not try to extract any extra sub-patterns + re.FullMatch("ruby:1234", &s); + + Example: does not try to extract into NULL + re.FullMatch("ruby:1234", NULL, &i); + + Example: integer overflow causes failure + !re.FullMatch("ruby:1234567891234", NULL, &i); + + Example: fails because there aren't enough sub-patterns: + !pcrecpp::RE("\\w+:\\d+").FullMatch("ruby:1234", &s); + + Example: fails because string cannot be stored in integer + !pcrecpp::RE("(.*)").FullMatch("ruby", &i); + + The provided pointer arguments can be pointers to any scalar numeric + type, or one of: + + string (matched piece is copied to string) + StringPiece (StringPiece is mutated to point to matched piece) + T (where "bool T::ParseFrom(const char*, int)" exists) + NULL (the corresponding matched sub-pattern is not copied) + + The function returns true iff all of the following conditions are sat- + isfied: + + a. "text" matches "pattern" exactly; + + b. The number of matched sub-patterns is >= number of supplied + pointers; + + c. The "i"th argument has a suitable type for holding the + string captured as the "i"th sub-pattern. If you pass in + NULL for the "i"th argument, or pass fewer arguments than + number of sub-patterns, "i"th captured sub-pattern is + ignored. + + The matching interface supports at most 16 arguments per call. If you + need more, consider using the more general interface + pcrecpp::RE::DoMatch. See pcrecpp.h for the signature for DoMatch. + + +PARTIAL MATCHES + + You can use the "PartialMatch" operation when you want the pattern to + match any substring of the text. + + Example: simple search for a string: + pcrecpp::RE("ell").PartialMatch("hello"); + + Example: find first number in a string: + int number; + pcrecpp::RE re("(\\d+)"); + re.PartialMatch("x*100 + 20", &number); + assert(number == 100); + + +UTF-8 AND THE MATCHING INTERFACE + + By default, pattern and text are plain text, one byte per character. + The UTF8 flag, passed to the constructor, causes both pattern and + string to be treated as UTF-8 text, still a byte stream but potentially + multiple bytes per character. In practice, the text is likelier to be + UTF-8 than the pattern, but the match returned may depend on the UTF8 + flag, so always use it when matching UTF8 text. For example, "." will + match one byte normally but with UTF8 set may match up to three bytes + of a multi-byte character. + + Example: + pcrecpp::RE_Options options; + options.set_utf8(); + pcrecpp::RE re(utf8_pattern, options); + re.FullMatch(utf8_string); + + Example: using the convenience function UTF8(): + pcrecpp::RE re(utf8_pattern, pcrecpp::UTF8()); + re.FullMatch(utf8_string); + + NOTE: The UTF8 flag is ignored if pcre was not configured with the + --enable-utf8 flag. + + +SCANNING TEXT INCREMENTALLY + + The "Consume" operation may be useful if you want to repeatedly match + regular expressions at the front of a string and skip over them as they + match. This requires use of the "StringPiece" type, which represents a + sub-range of a real string. Like RE, StringPiece is defined in the + pcrecpp namespace. + + Example: read lines of the form "var = value" from a string. + string contents = ...; // Fill string somehow + pcrecpp::StringPiece input(contents); // Wrap in a StringPiece + + string var; + int value; + pcrecpp::RE re("(\\w+) = (\\d+)\n"); + while (re.Consume(&input, &var, &value)) { + ...; + } + + Each successful call to "Consume" will set "var/value", and also + advance "input" so it points past the matched text. + + The "FindAndConsume" operation is similar to "Consume" but does not + anchor your match at the beginning of the string. For example, you + could extract all words from a string by repeatedly calling + + pcrecpp::RE("(\\w+)").FindAndConsume(&input, &word) + + +PARSING HEX/OCTAL/C-RADIX NUMBERS + + By default, if you pass a pointer to a numeric value, the corresponding + text is interpreted as a base-10 number. You can instead wrap the + pointer with a call to one of the operators Hex(), Octal(), or CRadix() + to interpret the text in another base. The CRadix operator interprets + C-style "0" (base-8) and "0x" (base-16) prefixes, but defaults to + base-10. + + Example: + int a, b, c, d; + pcrecpp::RE re("(.*) (.*) (.*) (.*)"); + re.FullMatch("100 40 0100 0x40", + pcrecpp::Octal(&a), pcrecpp::Hex(&b), + pcrecpp::CRadix(&c), pcrecpp::CRadix(&d)); + + will leave 64 in a, b, c, and d. + + +REPLACING PARTS OF STRINGS + + You can replace the first match of "pattern" in "str" with "rewrite". + Within "rewrite", backslash-escaped digits (\1 to \9) can be used to + insert text matching corresponding parenthesized group from the pat- + tern. \0 in "rewrite" refers to the entire matching text. For example: + + string s = "yabba dabba doo"; + pcrecpp::RE("b+").Replace("d", &s); + + will leave "s" containing "yada dabba doo". The result is true if the + pattern matches and a replacement occurs, false otherwise. + + GlobalReplace is like Replace except that it replaces all occurrences + of the pattern in the string with the rewrite. Replacements are not + subject to re-matching. For example: + + string s = "yabba dabba doo"; + pcrecpp::RE("b+").GlobalReplace("d", &s); + + will leave "s" containing "yada dada doo". It returns the number of + replacements made. + + Extract is like Replace, except that if the pattern matches, "rewrite" + is copied into "out" (an additional argument) with substitutions. The + non-matching portions of "text" are ignored. Returns true iff a match + occurred and the extraction happened successfully; if no match occurs, + the string is left unaffected. + + +AUTHOR + + The C++ wrapper was contributed by Google Inc. + Copyright (c) 2005 Google Inc. +----------------------------------------------------------------------------- NAME PCRE - Perl-compatible regular expressions + PCRE SAMPLE PROGRAM A simple, complete demonstration program, to get you started with using diff --git a/doc/pcre_compile.3 b/doc/pcre_compile.3 index 37a5f2d..4d4cb5a 100644 --- a/doc/pcre_compile.3 +++ b/doc/pcre_compile.3 @@ -38,6 +38,7 @@ The option bits are: PCRE_EXTENDED Ignore whitespace and # comments PCRE_EXTRA PCRE extra features (not much use currently) + PCRE_FIRSTLINE Force matching to be before newline PCRE_MULTILINE ^ and $ match newlines within data PCRE_NO_AUTO_CAPTURE Disable numbered capturing paren- theses (named ones available) diff --git a/doc/pcre_compile2.3 b/doc/pcre_compile2.3 new file mode 100644 index 0000000..1c2d704 --- /dev/null +++ b/doc/pcre_compile2.3 @@ -0,0 +1,70 @@ +.TH PCRE 3 +.SH NAME +PCRE - Perl-compatible regular expressions +.SH SYNOPSIS +.rs +.sp +.B #include <pcre.h> +.PP +.SM +.br +.B pcre *pcre_compile2(const char *\fIpattern\fP, int \fIoptions\fP, +.ti +5n +.B int *\fIerrorcodeptr\fP, +.ti +5n +.B const char **\fIerrptr\fP, int *\fIerroffset\fP, +.ti +5n +.B const unsigned char *\fItableptr\fP); +. +.SH DESCRIPTION +.rs +.sp +This function compiles a regular expression into an internal form. It is the +same as \fBpcre_compile()\fP, except for the addition of the \fIerrorcodeptr\fP +argument. The arguments are: + +.sp + \fIpattern\fR A zero-terminated string containing the + regular expression to be compiled + \fIoptions\fR Zero or more option bits + \fIerrorcodeptr\fP Where to put an error code + \fIerrptr\fR Where to put an error message + \fIerroffset\fR Offset in pattern where error was found + \fItableptr\fR Pointer to character tables, or NULL to + use the built-in default +.sp +The option bits are: +.sp + PCRE_ANCHORED Force pattern anchoring + PCRE_AUTO_CALLOUT Compile automatic callouts + PCRE_CASELESS Do caseless matching + PCRE_DOLLAR_ENDONLY $ not to match newline at end + PCRE_DOTALL . matches anything including NL + PCRE_EXTENDED Ignore whitespace and # comments + PCRE_EXTRA PCRE extra features + (not much use currently) + PCRE_FIRSTLINE Force matching to be before newline + PCRE_MULTILINE ^ and $ match newlines within data + PCRE_NO_AUTO_CAPTURE Disable numbered capturing paren- + theses (named ones available) + PCRE_UNGREEDY Invert greediness of quantifiers + PCRE_UTF8 Run in UTF-8 mode + PCRE_NO_UTF8_CHECK Do not check the pattern for UTF-8 + validity (only relevant if + PCRE_UTF8 is set) +.sp +PCRE must be built with UTF-8 support in order to use PCRE_UTF8 and +PCRE_NO_UTF8_CHECK. +.P +The yield of the function is a pointer to a private data structure that +contains the compiled pattern, or NULL if an error was detected. +.P +There is a complete description of the PCRE native API in the +.\" HREF +\fBpcreapi\fR +.\" +page and a description of the POSIX API in the +.\" HREF +\fBpcreposix\fR +.\" +page. diff --git a/doc/pcre_dfa_exec.3 b/doc/pcre_dfa_exec.3 new file mode 100644 index 0000000..277c2e1 --- /dev/null +++ b/doc/pcre_dfa_exec.3 @@ -0,0 +1,80 @@ +.TH PCRE 3 +.SH NAME +PCRE - Perl-compatible regular expressions +.SH SYNOPSIS +.rs +.sp +.B #include <pcre.h> +.PP +.SM +.br +.B int pcre_dfa_exec(const pcre *\fIcode\fP, "const pcre_extra *\fIextra\fP," +.ti +5n +.B "const char *\fIsubject\fP," int \fIlength\fP, int \fIstartoffset\fP, +.ti +5n +.B int \fIoptions\fP, int *\fIovector\fP, int \fIovecsize\fP, +.ti +5n +.B int *\fIworkspace\fP, int \fIwscount\fP); +. +.SH DESCRIPTION +.rs +.sp +This function matches a compiled regular expression against a given subject +string, using a DFA matching algorithm (\fInot\fP Perl-compatible). Note that +the main, Perl-compatible, matching function is \fBpcre_exec()\fP. The +arguments for this function are: +.sp + \fIcode\fP Points to the compiled pattern + \fIextra\fP Points to an associated \fBpcre_extra\fP structure, + or is NULL + \fIsubject\fP Points to the subject string + \fIlength\fP Length of the subject string, in bytes + \fIstartoffset\fP Offset in bytes in the subject at which to + start matching + \fIoptions\fP Option bits + \fIovector\fP Points to a vector of ints for result offsets + \fIovecsize\fP Number of elements in the vector + \fIworkspace\fP Points to a vector of ints used as working space + \fIwscount\fP Number of elements in the vector +.sp +The options are: +.sp + PCRE_ANCHORED Match only at the first position + PCRE_NOTBOL Subject is not the beginning of a line + PCRE_NOTEOL Subject is not the end of a line + PCRE_NOTEMPTY An empty string is not a valid match + PCRE_NO_UTF8_CHECK Do not check the subject for UTF-8 + validity (only relevant if PCRE_UTF8 + was set at compile time) + PCRE_PARTIAL Return PCRE_ERROR_PARTIAL for a partial match + PCRE_DFA_SHORTEST Return only the shortest match + PCRE_DFA_RESTART This is a restart after a partial match +.sp +There are restrictions on what may appear in a pattern when matching using the +DFA algorithm is requested. Details are given in the +.\" HREF +\fBpcrematching\fP +.\" +documentation. +.P +A \fBpcre_extra\fP structure contains the following fields: +.sp + \fIflags\fP Bits indicating which fields are set + \fIstudy_data\fP Opaque data from \fBpcre_study()\fP + \fImatch_limit\fP Limit on internal recursion + \fIcallout_data\fP Opaque data passed back to callouts + \fItables\fP Points to character tables or is NULL +.sp +The flag bits are PCRE_EXTRA_STUDY_DATA, PCRE_EXTRA_MATCH_LIMIT, +PCRE_EXTRA_CALLOUT_DATA, and PCRE_EXTRA_TABLES. For DFA matching, the +\fImatch_limit\fP field is not used, and must not be set. +.P +There is a complete description of the PCRE native API in the +.\" HREF +\fBpcreapi\fP +.\" +page and a description of the POSIX API in the +.\" HREF +\fBpcreposix\fP +.\" +page. diff --git a/doc/pcre_exec.3 b/doc/pcre_exec.3 index 7e071a9..113a632 100644 --- a/doc/pcre_exec.3 +++ b/doc/pcre_exec.3 @@ -18,7 +18,8 @@ PCRE - Perl-compatible regular expressions .rs .sp This function matches a compiled regular expression against a given subject -string, and returns offsets to capturing subexpressions. Its arguments are: +string, using a matching algorithm that is similar to Perl's. It returns +offsets to captured substrings. Its arguments are: .sp \fIcode\fP Points to the compiled pattern \fIextra\fP Points to an associated \fBpcre_extra\fP structure, diff --git a/doc/pcre_refcount.3 b/doc/pcre_refcount.3 new file mode 100644 index 0000000..0b94068 --- /dev/null +++ b/doc/pcre_refcount.3 @@ -0,0 +1,33 @@ +.TH PCRE 3 +.SH NAME +PCRE - Perl-compatible regular expressions +.SH SYNOPSIS +.rs +.sp +.B #include <pcre.h> +.PP +.SM +.br +.B int pcre_info(pcre *\fIcode\fP, int \fIadjust\fP); +. +.SH DESCRIPTION +.rs +.sp +This function is used to maintain a reference count inside a data block that +contains a compiled pattern. Its arguments are: +.sp + \fIcode\fP Compiled regular expression + \fIadjust\fP Adjustment to reference value +.sp +The yield of the function is the adjusted reference value, which is constrained +to lie between 0 and 65535. +.P +There is a complete description of the PCRE native API in the +.\" HREF +\fBpcreapi\fP +.\" +page and a description of the POSIX API in the +.\" HREF +\fBpcreposix\fP +.\" +page. diff --git a/doc/pcreapi.3 b/doc/pcreapi.3 index 42a4e59..9fc8182 100644 --- a/doc/pcreapi.3 +++ b/doc/pcreapi.3 @@ -15,6 +15,15 @@ PCRE - Perl-compatible regular expressions .B const unsigned char *\fItableptr\fP); .PP .br +.B pcre *pcre_compile2(const char *\fIpattern\fP, int \fIoptions\fP, +.ti +5n +.B int *\fIerrorcodeptr\fP, +.ti +5n +.B const char **\fIerrptr\fP, int *\fIerroffset\fP, +.ti +5n +.B const unsigned char *\fItableptr\fP); +.PP +.br .B pcre_extra *pcre_study(const pcre *\fIcode\fP, int \fIoptions\fP, .ti +5n .B const char **\fIerrptr\fP); @@ -27,6 +36,15 @@ PCRE - Perl-compatible regular expressions .B int \fIoptions\fP, int *\fIovector\fP, int \fIovecsize\fP); .PP .br +.B int pcre_dfa_exec(const pcre *\fIcode\fP, "const pcre_extra *\fIextra\fP," +.ti +5n +.B "const char *\fIsubject\fP," int \fIlength\fP, int \fIstartoffset\fP, +.ti +5n +.B int \fIoptions\fP, int *\fIovector\fP, int \fIovecsize\fP, +.ti +5n +.B int *\fIworkspace\fP, int \fIwscount\fP); +.PP +.br .B int pcre_copy_named_substring(const pcre *\fIcode\fP, .ti +5n .B const char *\fIsubject\fP, int *\fIovector\fP, @@ -87,6 +105,9 @@ PCRE - Perl-compatible regular expressions .B *\fIfirstcharptr\fP); .PP .br +.B int pcre_refcount(pcre *\fIcode\fP, int \fIadjust\fP); +.PP +.br .B int pcre_config(int \fIwhat\fP, void *\fIwhere\fP); .PP .br @@ -111,33 +132,50 @@ PCRE - Perl-compatible regular expressions .SH "PCRE API OVERVIEW" .rs .sp -PCRE has its own native API, which is described in this document. There is also -a set of wrapper functions that correspond to the POSIX regular expression API. -These are described in the +PCRE has its own native API, which is described in this document. There is +also a set of wrapper functions that correspond to the POSIX regular expression +API. These are described in the .\" HREF \fBpcreposix\fP .\" -documentation. +documentation. Both of these APIs define a set of C function calls. A C++ +wrapper is distributed with PCRE. It is documented in the +.\" HREF +\fBpcrecpp\fP +.\" +page. .P -The native API function prototypes are defined in the header file \fBpcre.h\fP, -and on Unix systems the library itself is called \fBlibpcre\fP. It can -normally be accessed by adding \fB-lpcre\fP to the command for linking an -application that uses PCRE. The header file defines the macros PCRE_MAJOR and -PCRE_MINOR to contain the major and minor release numbers for the library. +The native API C function prototypes are defined in the header file +\fBpcre.h\fP, and on Unix systems the library itself is called \fBlibpcre\fP. +It can normally be accessed by adding \fB-lpcre\fP to the command for linking +an application that uses PCRE. The header file defines the macros PCRE_MAJOR +and PCRE_MINOR to contain the major and minor release numbers for the library. Applications can use these to include support for different releases of PCRE. .P -The functions \fBpcre_compile()\fP, \fBpcre_study()\fP, and \fBpcre_exec()\fP -are used for compiling and matching regular expressions. A sample program that -demonstrates the simplest way of using them is provided in the file called -\fIpcredemo.c\fP in the source distribution. The +The functions \fBpcre_compile()\fP, \fBpcre_compile2()\fP, \fBpcre_study()\fP, +and \fBpcre_exec()\fP are used for compiling and matching regular expressions +in a Perl-compatible manner. A sample program that demonstrates the simplest +way of using them is provided in the file called \fIpcredemo.c\fP in the source +distribution. The .\" HREF \fBpcresample\fP .\" documentation describes how to run it. .P +A second matching function, \fBpcre_dfa_exec()\fP, which is not +Perl-compatible, is also provided. This uses a different algorithm for the +matching. This allows it to find all possible matches (at a given point in the +subject), not just one. However, this algorithm does not return captured +substrings. A description of the two matching algorithms and their advantages +and disadvantages is given in the +.\" HREF +\fBpcrematching\fP +.\" +documentation. +.P In addition to the main compiling and matching functions, there are convenience -functions for extracting captured substrings from a matched subject string. -They are: +functions for extracting captured substrings from a subject string that is +matched by \fBpcre_exec()\fP. They are: .sp \fBpcre_copy_substring()\fP \fBpcre_copy_named_substring()\fP @@ -150,10 +188,10 @@ They are: provided, to free the memory used for extracted strings. .P The function \fBpcre_maketables()\fP is used to build a set of character tables -in the current locale for passing to \fBpcre_compile()\fP or \fBpcre_exec()\fP. -This is an optional facility that is provided for specialist use. Most -commonly, no special tables are passed, in which case internal tables that are -generated when PCRE is built are used. +in the current locale for passing to \fBpcre_compile()\fP, \fBpcre_exec()\fP, +or \fBpcre_dfa_exec()\fP. This is an optional facility that is provided for +specialist use. Most commonly, no special tables are passed, in which case +internal tables that are generated when PCRE is built are used. .P The function \fBpcre_fullinfo()\fP is used to find out information about a compiled pattern; \fBpcre_info()\fP is an obsolete version that returns only @@ -161,6 +199,10 @@ some of the available information, but is retained for backwards compatibility. The function \fBpcre_version()\fP returns a pointer to a string containing the version of PCRE and its date of release. .P +The function \fBpcre_refcount()\fP maintains a reference count in a data block +containing a compiled pattern. This is provided for the benefit of +object-oriented applications. +.P The global variables \fBpcre_malloc\fP and \fBpcre_free\fP initially contain the entry points of the standard \fBmalloc()\fP and \fBfree()\fP functions, respectively. PCRE calls the memory management functions via these variables, @@ -170,12 +212,13 @@ should be done before calling any PCRE functions. The global variables \fBpcre_stack_malloc\fP and \fBpcre_stack_free\fP are also indirections to memory management functions. These special functions are used only when PCRE is compiled to use the heap for remembering data, instead of -recursive function calls. This is a non-standard way of building PCRE, for use -in environments that have limited stacks. Because of the greater use of memory -management, it runs more slowly. Separate functions are provided so that -special-purpose external code can be used for this case. When used, these -functions are always called in a stack-like manner (last obtained, first -freed), and always for memory blocks of the same size. +recursive function calls, when running the \fBpcre_exec()\fP function. This is +a non-standard way of building PCRE, for use in environments that have limited +stacks. Because of the greater use of memory management, it runs more slowly. +Separate functions are provided so that special-purpose external code can be +used for this case. When used, these functions are always called in a +stack-like manner (last obtained, first freed), and always for memory blocks of +the same size. .P The global variable \fBpcre_callout\fP initially contains NULL. It can be set by the caller to a "callout" function, which PCRE will then call at specified @@ -268,12 +311,13 @@ details are given with \fBpcre_exec()\fP below. .sp PCRE_CONFIG_STACKRECURSE .sp -The output is an integer that is set to one if internal recursion is -implemented by recursive function calls that use the stack to remember their -state. This is the usual way that PCRE is compiled. The output is zero if PCRE -was compiled to use blocks of data on the heap instead of recursive function -calls. In this case, \fBpcre_stack_malloc\fP and \fBpcre_stack_free\fP are -called to manage memory blocks on the heap, thus avoiding the use of the stack. +The output is an integer that is set to one if internal recursion when running +\fBpcre_exec()\fP is implemented by recursive function calls that use the stack +to remember their state. This is the usual way that PCRE is compiled. The +output is zero if PCRE was compiled to use blocks of data on the heap instead +of recursive function calls. In this case, \fBpcre_stack_malloc\fP and +\fBpcre_stack_free\fP are called to manage memory blocks on the heap, thus +avoiding the use of the stack. . . .SH "COMPILING A PATTERN" @@ -284,14 +328,26 @@ called to manage memory blocks on the heap, thus avoiding the use of the stack. .B const char **\fIerrptr\fP, int *\fIerroffset\fP, .ti +5n .B const unsigned char *\fItableptr\fP); +.sp +.B pcre *pcre_compile2(const char *\fIpattern\fP, int \fIoptions\fP, +.ti +5n +.B int *\fIerrorcodeptr\fP, +.ti +5n +.B const char **\fIerrptr\fP, int *\fIerroffset\fP, +.ti +5n +.B const unsigned char *\fItableptr\fP); +.P +Either of the functions \fBpcre_compile()\fP or \fBpcre_compile2()\fP can be +called to compile a pattern into an internal form. The only difference between +the two interfaces is that \fBpcre_compile2()\fP has an additional argument, +\fIerrorcodeptr\fP, via which a numerical error code can be returned. .P -The function \fBpcre_compile()\fP is called to compile a pattern into an -internal form. The pattern is a C string terminated by a binary zero, and -is passed in the \fIpattern\fP argument. A pointer to a single block of memory -that is obtained via \fBpcre_malloc\fP is returned. This contains the compiled -code and related data. The \fBpcre\fP type is defined for the returned block; -this is a typedef for a structure whose contents are not externally defined. It -is up to the caller to free the memory when it is no longer required. +The pattern is a C string terminated by a binary zero, and is passed in the +\fIpattern\fP argument. A pointer to a single block of memory that is obtained +via \fBpcre_malloc\fP is returned. This contains the compiled code and related +data. The \fBpcre\fP type is defined for the returned block; this is a typedef +for a structure whose contents are not externally defined. It is up to the +caller to free the memory when it is no longer required. .P Although the compiled code of a PCRE regex is relocatable, that is, it does not depend on memory location, the complete \fBpcre\fP data block is not @@ -318,6 +374,11 @@ error message. The offset from the start of the pattern to the character where the error was discovered is placed in the variable pointed to by \fIerroffset\fP, which must not be NULL. If it is, an immediate error is given. .P +If \fBpcre_compile2()\fP is used instead of \fBpcre_compile()\fP, and the +\fIerrorcodeptr\fP argument is not NULL, a non-zero error code number is +returned via this argument in the event of an error. This is in addition to the +textual error message. Error codes and messages are listed below. +.P If the final argument, \fItableptr\fP, is NULL, PCRE uses a default set of character tables that are built when PCRE is compiled, using the default C locale. Otherwise, \fItableptr\fP must be an address that is the result of a @@ -362,9 +423,13 @@ documentation. .sp If this bit is set, letters in the pattern match both upper and lower case letters. It is equivalent to Perl's /i option, and it can be changed within a -pattern by a (?i) option setting. When running in UTF-8 mode, case support for -high-valued characters is available only when PCRE is built with Unicode -character property support. +pattern by a (?i) option setting. In UTF-8 mode, PCRE always understands the +concept of case for characters whose values are less than 128, so caseless +matching is always possible. For characters with higher values, the concept of +case is supported if PCRE is compiled with Unicode property support, but not +otherwise. If you want to use caseless matching for characters 128 and above, +you must ensure that PCRE is compiled with Unicode property support as well as +with UTF-8 support. .sp PCRE_DOLLAR_ENDONLY .sp @@ -408,6 +473,12 @@ special meaning is treated as a literal. There are at present no other features controlled by this option. It can also be set by a (?X) option setting within a pattern. .sp + PCRE_FIRSTLINE +.sp +If this option is set, an unanchored pattern is required to match before or at +the first newline character in the subject string, though the matched text may +continue over the newline. +.sp PCRE_MULTILINE .sp By default, PCRE treats the subject string as consisting of a single line of @@ -463,14 +534,72 @@ automatically checked. If an invalid UTF-8 sequence of bytes is found, valid, and you want to skip this check for performance reasons, you can set the PCRE_NO_UTF8_CHECK option. When it is set, the effect of passing an invalid UTF-8 string as a pattern is undefined. It may cause your program to crash. -Note that this option can also be passed to \fBpcre_exec()\fP, to suppress the -UTF-8 validity checking of subject strings. +Note that this option can also be passed to \fBpcre_exec()\fP and +\fBpcre_dfa_exec()\fP, to suppress the UTF-8 validity checking of subject +strings. +. +. +.SH "COMPILATION ERROR CODES" +.rs +.sp +The following table lists the error codes than may be returned by +\fBpcre_compile2()\fP, along with the error messages that may be returned by +both compiling functions. +.sp + 0 no error + 1 \e at end of pattern + 2 \ec at end of pattern + 3 unrecognized character follows \e + 4 numbers out of order in {} quantifier + 5 number too big in {} quantifier + 6 missing terminating ] for character class + 7 invalid escape sequence in character class + 8 range out of order in character class + 9 nothing to repeat + 10 operand of unlimited repeat could match the empty string + 11 internal error: unexpected repeat + 12 unrecognized character after (? + 13 POSIX named classes are supported only within a class + 14 missing ) + 15 reference to non-existent subpattern + 16 erroffset passed as NULL + 17 unknown option bit(s) set + 18 missing ) after comment + 19 parentheses nested too deeply + 20 regular expression too large + 21 failed to get memory + 22 unmatched parentheses + 23 internal error: code overflow + 24 unrecognized character after (?< + 25 lookbehind assertion is not fixed length + 26 malformed number after (?( + 27 conditional group contains more than two branches + 28 assertion expected after (?( + 29 (?R or (?digits must be followed by ) + 30 unknown POSIX class name + 31 POSIX collating elements are not supported + 32 this version of PCRE is not compiled with PCRE_UTF8 support + 33 spare error + 34 character value in \ex{...} sequence is too large + 35 invalid condition (?(0) + 36 \eC not allowed in lookbehind assertion + 37 PCRE does not support \eL, \el, \eN, \eU, or \eu + 38 number after (?C is > 255 + 39 closing ) for (?C expected + 40 recursive call could loop indefinitely + 41 unrecognized character after (?P + 42 syntax error after (?P + 43 two named groups have the same name + 44 invalid UTF-8 string + 45 support for \eP, \ep, and \eX has not been compiled + 46 malformed \eP or \ep sequence + 47 unknown property name after \eP or \ep . . .SH "STUDYING A PATTERN" .rs .sp -.B pcre_extra *pcre_study(const pcre *\fIcode\fP, int \fIoptions\fP, +.B pcre_extra *pcre_study(const pcre *\fIcode\fP, int \fIoptions\fP .ti +5n .B const char **\fIerrptr\fP); .PP @@ -492,7 +621,7 @@ below .\" in the section on matching a pattern. .P -If studying the pattern does not produce any additional information, +If studying the pattern does not produce any additional information \fBpcre_study()\fP returns NULL. In that circumstance, if the calling program wants to pass any of the other fields to \fBpcre_exec()\fP, it must set up its own \fBpcre_extra\fP block. @@ -523,12 +652,12 @@ bytes is created. .SH "LOCALE SUPPORT" .rs .sp -PCRE handles caseless matching, and determines whether characters are letters, +PCRE handles caseless matching, and determines whether characters are letters digits, or whatever, by reference to a set of tables, indexed by character -value. (When running in UTF-8 mode, this applies only to characters with codes +value. When running in UTF-8 mode, this applies only to characters with codes less than 128. Higher-valued codes never match escapes such as \ew or \ed, but can be tested with \ep if PCRE is built with Unicode character property -support.) +support. .P An internal set of tables is created in the default C locale when PCRE is built. This is used when the final argument of \fBpcre_compile()\fP is NULL, @@ -615,7 +744,7 @@ no back references. Return the number of capturing subpatterns in the pattern. The fourth argument should point to an \fBint\fP variable. .sp - PCRE_INFO_DEFAULTTABLES + PCRE_INFO_DEFAULT_TABLES .sp Return a pointer to the internal default character tables within PCRE. The fourth argument should point to an \fBunsigned char *\fP variable. This @@ -760,7 +889,30 @@ it is used to pass back information about the first character of any matched string (see PCRE_INFO_FIRSTBYTE above). . . -.SH "MATCHING A PATTERN" +.SH "REFERENCE COUNTS" +.rs +.sp +.B int pcre_refcount(pcre *\fIcode\fP, int \fIadjust\fP); +.PP +The \fBpcre_refcount()\fP function is used to maintain a reference count in the +data block that contains a compiled pattern. It is provided for the benefit of +applications that operate in an object-oriented manner, where different parts +of the application may be using the same compiled pattern, but you want to free +the block when they are all done. +.P +When a pattern is compiled, the reference count field is initialized to zero. +It is changed only by calling this function, whose action is to add the +\fIadjust\fP value (which may be positive or negative) to it. The yield of the +function is the new value. However, the value of the count is constrained to +lie between 0 and 65535, inclusive. If the new value is outside these limits, +it is forced to the appropriate limit value. +.P +Except when it is zero, the reference count is not correctly preserved if a +pattern is compiled on one host and then transferred to a host whose byte-order +is different. (This seems a highly unlikely scenario.) +. +. +.SH "MATCHING A PATTERN: THE TRADITIONAL FUNCTION" .rs .sp .B int pcre_exec(const pcre *\fIcode\fP, "const pcre_extra *\fIextra\fP," @@ -772,7 +924,14 @@ string (see PCRE_INFO_FIRSTBYTE above). The function \fBpcre_exec()\fP is called to match a subject string against a compiled pattern, which is passed in the \fIcode\fP argument. If the pattern has been studied, the result of the study should be passed in the -\fIextra\fP argument. +\fIextra\fP argument. This function is the main matching facility of the +library, and it operates in a Perl-like manner. For specialist use there is +also an alternative matching function, which is described +.\" HTML <a href="#dfamatch"> +.\" </a> +below +.\" +in the section about the \fBpcre_dfa_exec()\fP function. .P In most applications, the pattern will have been compiled (and optionally studied) in the same process that calls \fBpcre_exec()\fP. However, it is @@ -796,7 +955,7 @@ Here is an example of a simple call to \fBpcre_exec()\fP: 0, /* start at offset 0 in the subject */ 0, /* default options */ ovector, /* vector of integers for substring information */ - 30); /* number of elements in the vector (NOT size in bytes) */ + 30); /* number of elements (NOT size in bytes) */ . .\" HTML <a name="extradata"></a> .SS "Extra data for \fBpcre_exec()\fR" @@ -1041,6 +1200,7 @@ subpatterns there are in a compiled pattern. The smallest size for \fIovector\fP that will allow for \fIn\fP captured substrings, in addition to the offsets of the substring matched by the whole pattern, is (\fIn\fP+1)*3. . +.\" HTML <a name="errorlist"></a> .SS "Return values from \fBpcre_exec()\fP" .rs .sp @@ -1112,7 +1272,7 @@ A string that contains an invalid UTF-8 byte sequence was passed as a subject. The UTF-8 byte sequence that was passed as a subject was valid, but the value of \fIstartoffset\fP did not point to the beginning of a UTF-8 character. .sp - PCRE_ERROR_PARTIAL (-12) + PCRE_ERROR_PARTIAL (-12) .sp The subject string did not match, but it did match partially. See the .\" HREF @@ -1120,7 +1280,7 @@ The subject string did not match, but it did match partially. See the .\" documentation for details of partial matching. .sp - PCRE_ERROR_BAD_PARTIAL (-13) + PCRE_ERROR_BADPARTIAL (-13) .sp The PCRE_PARTIAL option was used with a compiled pattern containing items that are not supported for partial matching. See the @@ -1129,12 +1289,12 @@ are not supported for partial matching. See the .\" documentation for details of partial matching. .sp - PCRE_ERROR_INTERNAL (-14) + PCRE_ERROR_INTERNAL (-14) .sp An unexpected internal error has occurred. This error could be caused by a bug in PCRE or by overwriting of the compiled pattern. .sp - PCRE_ERROR_BADCOUNT (-15) + PCRE_ERROR_BADCOUNT (-15) .sp This error is given if the value of the \fIovecsize\fP argument is negative. . @@ -1281,8 +1441,196 @@ translation table. These functions call \fBpcre_get_stringnumber()\fP, and if it succeeds, they then call \fIpcre_copy_substring()\fP or \fIpcre_get_substring()\fP, as appropriate. +. +. +.SH "FINDING ALL POSSIBLE MATCHES" +.rs +.sp +The traditional matching function uses a similar algorithm to Perl, which stops +when it finds the first match, starting at a given point in the subject. If you +want to find all possible matches, or the longest possible match, consider +using the alternative matching function (see below) instead. If you cannot use +the alternative function, but still need to find all possible matches, you +can kludge it up by making use of the callout facility, which is described in +the +.\" HREF +\fBpcrecallout\fP +.\" +documentation. +.P +What you have to do is to insert a callout right at the end of the pattern. +When your callout function is called, extract and save the current matched +substring. Then return 1, which forces \fBpcre_exec()\fP to backtrack and try +other alternatives. Ultimately, when it runs out of matches, \fBpcre_exec()\fP +will yield PCRE_ERROR_NOMATCH. +. +. +.\" HTML <a name="dfamatch"></a> +.SH "MATCHING A PATTERN: THE ALTERNATIVE FUNCTION" +.rs +.sp +.B int pcre_dfa_exec(const pcre *\fIcode\fP, "const pcre_extra *\fIextra\fP," +.ti +5n +.B "const char *\fIsubject\fP," int \fIlength\fP, int \fIstartoffset\fP, +.ti +5n +.B int \fIoptions\fP, int *\fIovector\fP, int \fIovecsize\fP, +.ti +5n +.B int *\fIworkspace\fP, int \fIwscount\fP); +.P +The function \fBpcre_dfa_exec()\fP is called to match a subject string against +a compiled pattern, using a "DFA" matching algorithm. This has different +characteristics to the normal algorithm, and is not compatible with Perl. Some +of the features of PCRE patterns are not supported. Nevertheless, there are +times when this kind of matching can be useful. For a discussion of the two +matching algorithms, see the +.\" HREF +\fBpcrematching\fP +.\" +documentation. +.P +The arguments for the \fBpcre_dfa_exec()\fP function are the same as for +\fBpcre_exec()\fP, plus two extras. The \fIovector\fP argument is used in a +different way, and this is described below. The other common arguments are used +in the same way as for \fBpcre_exec()\fP, so their description is not repeated +here. +.P +The two additional arguments provide workspace for the function. The workspace +vector should contain at least 20 elements. It is used for keeping track of +multiple paths through the pattern tree. More workspace will be needed for +patterns and subjects where there are a lot of possible matches. +.P +Here is an example of a simple call to \fBpcre_exec()\fP: +.sp + int rc; + int ovector[10]; + int wspace[20]; + rc = pcre_exec( + re, /* result of pcre_compile() */ + NULL, /* we didn't study the pattern */ + "some string", /* the subject string */ + 11, /* the length of the subject string */ + 0, /* start at offset 0 in the subject */ + 0, /* default options */ + ovector, /* vector of integers for substring information */ + 10, /* number of elements (NOT size in bytes) */ + wspace, /* working space vector */ + 20); /* number of elements (NOT size in bytes) */ +. +.SS "Option bits for \fBpcre_dfa_exec()\fP" +.rs +.sp +The unused bits of the \fIoptions\fP argument for \fBpcre_dfa_exec()\fP must be +zero. The only bits that may be set are PCRE_ANCHORED, PCRE_NOTBOL, +PCRE_NOTEOL, PCRE_NOTEMPTY, PCRE_NO_UTF8_CHECK, PCRE_PARTIAL, +PCRE_DFA_SHORTEST, and PCRE_DFA_RESTART. All but the last three of these are +the same as for \fBpcre_exec()\fP, so their description is not repeated here. +.sp + PCRE_PARTIAL +.sp +This has the same general effect as it does for \fBpcre_exec()\fP, but the +details are slightly different. When PCRE_PARTIAL is set for +\fBpcre_dfa_exec()\fP, the return code PCRE_ERROR_NOMATCH is converted into +PCRE_ERROR_PARTIAL if the end of the subject is reached, there have been no +complete matches, but there is still at least one matching possibility. The +portion of the string that provided the partial match is set as the first +matching string. +.sp + PCRE_DFA_SHORTEST +.sp +Setting the PCRE_DFA_SHORTEST option causes the matching algorithm to stop as +soon as it has found one match. Because of the way the DFA algorithm works, +this is necessarily the shortest possible match at the first possible matching +point in the subject string. +.sp + PCRE_DFA_RESTART +.sp +When \fBpcre_dfa_exec()\fP is called with the PCRE_PARTIAL option, and returns +a partial match, it is possible to call it again, with additional subject +characters, and have it continue with the same match. The PCRE_DFA_RESTART +option requests this action; when it is set, the \fIworkspace\fP and +\fIwscount\fP options must reference the same vector as before because data +about the match so far is left in them after a partial match. There is more +discussion of this facility in the +.\" HREF +\fBpcrepartial\fP +.\" +documentation. +. +.SS "Successful returns from \fBpcre_dfa_exec()\fP" +.rs +.sp +When \fBpcre_dfa_exec()\fP succeeds, it may have matched more than one +substring in the subject. Note, however, that all the matches from one run of +the function start at the same point in the subject. The shorter matches are +all initial substrings of the longer matches. For example, if the pattern +.sp + <.*> +.sp +is matched against the string +.sp + This is <something> <something else> <something further> no more +.sp +the three matched strings are +.sp + <something> + <something> <something else> + <something> <something else> <something further> +.sp +On success, the yield of the function is a number greater than zero, which is +the number of matched substrings. The substrings themselves are returned in +\fIovector\fP. Each string uses two elements; the first is the offset to the +start, and the second is the offset to the end. All the strings have the same +start offset. (Space could have been saved by giving this only once, but it was +decided to retain some compatibility with the way \fBpcre_exec()\fP returns +data, even though the meaning of the strings is different.) +.P +The strings are returned in reverse order of length; that is, the longest +matching string is given first. If there were too many matches to fit into +\fIovector\fP, the yield of the function is zero, and the vector is filled with +the longest matches. +. +.SS "Error returns from \fBpcre_dfa_exec()\fP" +.rs +.sp +The \fBpcre_dfa_exec()\fP function returns a negative number when it fails. +Many of the errors are the same as for \fBpcre_exec()\fP, and these are +described +.\" HTML <a href="#errorlist"> +.\" </a> +above. +.\" +There are in addition the following errors that are specific to +\fBpcre_dfa_exec()\fP: +.sp + PCRE_ERROR_DFA_UITEM (-16) +.sp +This return is given if \fBpcre_dfa_exec()\fP encounters an item in the pattern +that it does not support, for instance, the use of \eC or a back reference. +.sp + PCRE_ERROR_DFA_UCOND (-17) +.sp +This return is given if \fBpcre_dfa_exec()\fP encounters a condition item in a +pattern that uses a back reference for the condition. This is not supported. +.sp + PCRE_ERROR_DFA_UMLIMIT (-18) +.sp +This return is given if \fBpcre_dfa_exec()\fP is called with an \fIextra\fP +block that contains a setting of the \fImatch_limit\fP field. This is not +supported (it is meaningless). +.sp + PCRE_ERROR_DFA_WSSIZE (-19) +.sp +This return is given if \fBpcre_dfa_exec()\fP runs out of space in the +\fIworkspace\fP vector. +.sp + PCRE_ERROR_DFA_RECURSE (-20) +.sp +When a recursive subpattern is processed, the matching function calls itself +recursively, using private vectors for \fIovector\fP and \fIworkspace\fP. This +error is given if the output vector is not large enough. This should be +extremely rare, as a vector of size 1000 is used. .P .in 0 -Last updated: 09 September 2004 +Last updated: 16 May 2005 .br -Copyright (c) 1997-2004 University of Cambridge. +Copyright (c) 1997-2005 University of Cambridge. diff --git a/doc/pcrebuild.3 b/doc/pcrebuild.3 index 8ac5882..62c4ea2 100644 --- a/doc/pcrebuild.3 +++ b/doc/pcrebuild.3 @@ -100,10 +100,11 @@ to the \fBconfigure\fP command. .rs .sp Internally, PCRE has a function called \fBmatch()\fP, which it calls repeatedly -(possibly recursively) when matching a pattern. By controlling the maximum -number of times this function may be called during a single matching operation, -a limit can be placed on the resources used by a single call to -\fBpcre_exec()\fP. The limit can be changed at run time, as described in the +(possibly recursively) when matching a pattern with the \fBpcre_exec()\fP +function. By controlling the maximum number of times this function may be +called during a single matching operation, a limit can be placed on the +resources used by a single call to \fBpcre_exec()\fP. The limit can be changed +at run time, as described in the .\" HREF \fBpcreapi\fP .\" @@ -112,7 +113,8 @@ setting such as .sp --with-match-limit=500000 .sp -to the \fBconfigure\fP command. +to the \fBconfigure\fP command. This setting has no effect on the +\fBpcre_dfa_exec()\fP matching function. . .SH "HANDLING VERY LARGE PATTERNS" .rs @@ -138,13 +140,14 @@ of the compiled pattern, and this changes with the link size. .SH "AVOIDING EXCESSIVE STACK USAGE" .rs .sp -PCRE implements backtracking while matching by making recursive calls to an -internal function called \fBmatch()\fP. In environments where the size of the -stack is limited, this can severely limit PCRE's operation. (The Unix -environment does not usually suffer from this problem.) An alternative approach -that uses memory from the heap to remember data, instead of using recursive -function calls, has been implemented to work round this problem. If you want to -build a version of PCRE that works this way, add +When matching with the \fBpcre_exec()\fP function, PCRE implements backtracking +by making recursive calls to an internal function called \fBmatch()\fP. In +environments where the size of the stack is limited, this can severely limit +PCRE's operation. (The Unix environment does not usually suffer from this +problem.) An alternative approach that uses memory from the heap to remember +data, instead of using recursive function calls, has been implemented to work +round this problem. If you want to build a version of PCRE that works this way, +add .sp --disable-stack-for-recursion .sp @@ -155,7 +158,8 @@ predictable: the block sizes requested are always the same, and the blocks are always freed in reverse order. A calling program might be able to implement optimized functions that perform better than the standard \fBmalloc()\fP and \fBfree()\fP functions. PCRE runs noticeably more slowly when built in this -way. +way. This option affects only the \fBpcre_exec()\fP function; it is not +relevant for the the \fBpcre_dfa_exec()\fP function. . .SH "USING EBCDIC CODE" .rs @@ -169,6 +173,6 @@ compiled to run in an EBCDIC environment by adding to the \fBconfigure\fP command. .P .in 0 -Last updated: 09 September 2004 +Last updated: 28 February 2005 .br -Copyright (c) 1997-2004 University of Cambridge. +Copyright (c) 1997-2005 University of Cambridge. diff --git a/doc/pcrecallout.3 b/doc/pcrecallout.3 index 5fd8ff8..6155d43 100644 --- a/doc/pcrecallout.3 +++ b/doc/pcrecallout.3 @@ -58,9 +58,10 @@ no match, the callout is obeyed. .rs .sp During matching, when PCRE reaches a callout point, the external function -defined by \fIpcre_callout\fP is called (if it is set). The only argument is a -pointer to a \fBpcre_callout\fP block. This structure contains the following -fields: +defined by \fIpcre_callout\fP is called (if it is set). This applies to both +the \fBpcre_exec()\fP and the \fBpcre_dfa_exec()\fP matching functions. The +only argument to the callout function is a pointer to a \fBpcre_callout\fP +block. This structure contains the following fields: .sp int \fIversion\fP; int \fIcallout_number\fP; @@ -85,9 +86,11 @@ into the pattern (that is, the number after ?C for manual callouts, and 255 for automatically generated callouts). .P The \fIoffset_vector\fP field is a pointer to the vector of offsets that was -passed by the caller to \fBpcre_exec()\fP. The contents can be inspected in -order to extract substrings that have been matched so far, in the same way as -for extracting substrings after a match has completed. +passed by the caller to \fBpcre_exec()\fP or \fBpcre_dfa_exec()\fP. When +\fBpcre_exec()\fP is used, the contents can be inspected in order to extract +substrings that have been matched so far, in the same way as for extracting +substrings after a match has completed. For \fBpcre_dfa_exec()\fP this field is +not useful. .P The \fIsubject\fP and \fIsubject_length\fP fields contain copies of the values that were passed to \fBpcre_exec()\fP. @@ -100,19 +103,22 @@ different starting points in the subject. The \fIcurrent_position\fP field contains the offset within the subject of the current match pointer. .P -The \fIcapture_top\fP field contains one more than the number of the highest -numbered captured substring so far. If no substrings have been captured, -the value of \fIcapture_top\fP is one. +When the \fBpcre_exec()\fP function is used, the \fIcapture_top\fP field +contains one more than the number of the highest numbered captured substring so +far. If no substrings have been captured, the value of \fIcapture_top\fP is +one. This is always the case when \fBpcre_dfa_exec()\fP is used, because it +does not support captured substrings. .P The \fIcapture_last\fP field contains the number of the most recently captured -substring. If no substrings have been captured, its value is -1. +substring. If no substrings have been captured, its value is -1. This is always +the case when \fBpcre_dfa_exec()\fP is used. .P The \fIcallout_data\fP field contains a value that is passed to -\fBpcre_exec()\fP by the caller specifically so that it can be passed back in -callouts. It is passed in the \fIpcre_callout\fP field of the \fBpcre_extra\fP -data structure. If no such data was passed, the value of \fIcallout_data\fP in -a \fBpcre_callout\fP block is NULL. There is a description of the -\fBpcre_extra\fP structure in the +\fBpcre_exec()\fP or \fBpcre_dfa_exec()\fP specifically so that it can be +passed back in callouts. It is passed in the \fIpcre_callout\fP field of the +\fBpcre_extra\fP data structure. If no such data was passed, the value of +\fIcallout_data\fP in a \fBpcre_callout\fP block is NULL. There is a +description of the \fBpcre_extra\fP structure in the .\" HREF \fBpcreapi\fP .\" @@ -139,10 +145,10 @@ same callout number. However, they are set for all callouts. .sp The external callout function returns an integer to PCRE. If the value is zero, matching proceeds as normal. If the value is greater than zero, matching fails -at the current point, but backtracking to test other matching possibilities -goes ahead, just as if a lookahead assertion had failed. If the value is less -than zero, the match is abandoned, and \fBpcre_exec()\fP returns the negative -value. +at the current point, but the testing of other matching possibilities goes +ahead, just as if a lookahead assertion had failed. If the value is less than +zero, the match is abandoned, and \fBpcre_exec()\fP (or \fBpcre_dfa_exec()\fP) +returns the negative value. .P Negative values should normally be chosen from the set of PCRE_ERROR_xxx values. In particular, PCRE_ERROR_NOMATCH forces a standard "no match" failure. @@ -150,6 +156,6 @@ The error number PCRE_ERROR_CALLOUT is reserved for use by callout functions; it will never be used by PCRE itself. .P .in 0 -Last updated: 09 September 2004 +Last updated: 28 February 2005 .br -Copyright (c) 1997-2004 University of Cambridge. +Copyright (c) 1997-2005 University of Cambridge. diff --git a/doc/pcrecompat.3 b/doc/pcrecompat.3 index 6a853e0..05ed3cb 100644 --- a/doc/pcrecompat.3 +++ b/doc/pcrecompat.3 @@ -114,8 +114,11 @@ package. .sp (m) Patterns compiled by PCRE can be saved and re-used at a later time, even on different hosts that have the other endianness. +.sp +(n) The alternative matching function (\fBpcre_dfa_exec()\fP) matches in a +different way and is not Perl-compatible. .P .in 0 -Last updated: 09 September 2004 +Last updated: 28 February 2005 .br -Copyright (c) 1997-2004 University of Cambridge. +Copyright (c) 1997-2005 University of Cambridge. diff --git a/doc/pcrecpp.3 b/doc/pcrecpp.3 new file mode 100644 index 0000000..01eb028 --- /dev/null +++ b/doc/pcrecpp.3 @@ -0,0 +1,220 @@ +.TH PCRE 3 +.SH NAME +PCRE - Perl-compatible regular expressions. +.SH "SYNOPSIS OF C++ WRAPPER" +.rs +.sp +.B #include <pcrecpp.h> +.PP +.SM +.br +.SH DESCRIPTION +.rs +.sp +The C++ wrapper for PCRE was provided by Google Inc. This brief man page was +constructed from the notes in the \fIpcrecpp.h\fP file, which should be +consulted for further details. +. +. +.SH "MATCHING INTERFACE" +.rs +.sp +The "FullMatch" operation checks that supplied text matches a supplied pattern +exactly. If pointer arguments are supplied, it copies matched sub-strings that +match sub-patterns into them. +.sp + Example: successful match + pcrecpp::RE re("h.*o"); + re.FullMatch("hello"); +.sp + Example: unsuccessful match (requires full match): + pcrecpp::RE re("e"); + !re.FullMatch("hello"); +.sp + Example: creating a temporary RE object: + pcrecpp::RE("h.*o").FullMatch("hello"); +.sp +You can pass in a "const char*" or a "string" for "text". The examples below +tend to use a const char*. You can, as in the different examples above, store +the RE object explicitly in a variable or use a temporary RE object. The +examples below use one mode or the other arbitrarily. Either could correctly be +used for any of these examples. +.P +You must supply extra pointer arguments to extract matched subpieces. +.sp + Example: extracts "ruby" into "s" and 1234 into "i" + int i; + string s; + pcrecpp::RE re("(\e\ew+):(\e\ed+)"); + re.FullMatch("ruby:1234", &s, &i); +.sp + Example: does not try to extract any extra sub-patterns + re.FullMatch("ruby:1234", &s); +.sp + Example: does not try to extract into NULL + re.FullMatch("ruby:1234", NULL, &i); +.sp + Example: integer overflow causes failure + !re.FullMatch("ruby:1234567891234", NULL, &i); +.sp + Example: fails because there aren't enough sub-patterns: + !pcrecpp::RE("\e\ew+:\e\ed+").FullMatch("ruby:1234", &s); +.sp + Example: fails because string cannot be stored in integer + !pcrecpp::RE("(.*)").FullMatch("ruby", &i); +.sp +The provided pointer arguments can be pointers to any scalar numeric +type, or one of: +.sp + string (matched piece is copied to string) + StringPiece (StringPiece is mutated to point to matched piece) + T (where "bool T::ParseFrom(const char*, int)" exists) + NULL (the corresponding matched sub-pattern is not copied) +.sp +The function returns true iff all of the following conditions are satisfied: +.sp + a. "text" matches "pattern" exactly; +.sp + b. The number of matched sub-patterns is >= number of supplied + pointers; +.sp + c. The "i"th argument has a suitable type for holding the + string captured as the "i"th sub-pattern. If you pass in + NULL for the "i"th argument, or pass fewer arguments than + number of sub-patterns, "i"th captured sub-pattern is + ignored. +.sp +The matching interface supports at most 16 arguments per call. +If you need more, consider using the more general interface +\fBpcrecpp::RE::DoMatch\fP. See \fBpcrecpp.h\fP for the signature for +\fBDoMatch\fP. +. +.SH "PARTIAL MATCHES" +.rs +.sp +You can use the "PartialMatch" operation when you want the pattern +to match any substring of the text. +.sp + Example: simple search for a string: + pcrecpp::RE("ell").PartialMatch("hello"); +.sp + Example: find first number in a string: + int number; + pcrecpp::RE re("(\e\ed+)"); + re.PartialMatch("x*100 + 20", &number); + assert(number == 100); +. +. +.SH "UTF-8 AND THE MATCHING INTERFACE" +.rs +.sp +By default, pattern and text are plain text, one byte per character. The UTF8 +flag, passed to the constructor, causes both pattern and string to be treated +as UTF-8 text, still a byte stream but potentially multiple bytes per +character. In practice, the text is likelier to be UTF-8 than the pattern, but +the match returned may depend on the UTF8 flag, so always use it when matching +UTF8 text. For example, "." will match one byte normally but with UTF8 set may +match up to three bytes of a multi-byte character. +.sp + Example: + pcrecpp::RE_Options options; + options.set_utf8(); + pcrecpp::RE re(utf8_pattern, options); + re.FullMatch(utf8_string); +.sp + Example: using the convenience function UTF8(): + pcrecpp::RE re(utf8_pattern, pcrecpp::UTF8()); + re.FullMatch(utf8_string); +.sp +NOTE: The UTF8 flag is ignored if pcre was not configured with the + --enable-utf8 flag. +. +. +.SH "SCANNING TEXT INCREMENTALLY" +.rs +.sp +The "Consume" operation may be useful if you want to repeatedly +match regular expressions at the front of a string and skip over +them as they match. This requires use of the "StringPiece" type, +which represents a sub-range of a real string. Like RE, StringPiece +is defined in the pcrecpp namespace. +.sp + Example: read lines of the form "var = value" from a string. + string contents = ...; // Fill string somehow + pcrecpp::StringPiece input(contents); // Wrap in a StringPiece + + string var; + int value; + pcrecpp::RE re("(\e\ew+) = (\e\ed+)\en"); + while (re.Consume(&input, &var, &value)) { + ...; + } +.sp +Each successful call to "Consume" will set "var/value", and also +advance "input" so it points past the matched text. +.P +The "FindAndConsume" operation is similar to "Consume" but does not +anchor your match at the beginning of the string. For example, you +could extract all words from a string by repeatedly calling +.sp + pcrecpp::RE("(\e\ew+)").FindAndConsume(&input, &word) +. +. +.SH "PARSING HEX/OCTAL/C-RADIX NUMBERS" +.rs +.sp +By default, if you pass a pointer to a numeric value, the +corresponding text is interpreted as a base-10 number. You can +instead wrap the pointer with a call to one of the operators Hex(), +Octal(), or CRadix() to interpret the text in another base. The +CRadix operator interprets C-style "0" (base-8) and "0x" (base-16) +prefixes, but defaults to base-10. +.sp + Example: + int a, b, c, d; + pcrecpp::RE re("(.*) (.*) (.*) (.*)"); + re.FullMatch("100 40 0100 0x40", + pcrecpp::Octal(&a), pcrecpp::Hex(&b), + pcrecpp::CRadix(&c), pcrecpp::CRadix(&d)); +.sp +will leave 64 in a, b, c, and d. +. +. +.SH "REPLACING PARTS OF STRINGS" +.rs +.sp +You can replace the first match of "pattern" in "str" with "rewrite". +Within "rewrite", backslash-escaped digits (\e1 to \e9) can be +used to insert text matching corresponding parenthesized group +from the pattern. \e0 in "rewrite" refers to the entire matching +text. For example: +.sp + string s = "yabba dabba doo"; + pcrecpp::RE("b+").Replace("d", &s); +.sp +will leave "s" containing "yada dabba doo". The result is true if the pattern +matches and a replacement occurs, false otherwise. +.P +\fBGlobalReplace\fP is like \fBReplace\fP except that it replaces all +occurrences of the pattern in the string with the rewrite. Replacements are +not subject to re-matching. For example: +.sp + string s = "yabba dabba doo"; + pcrecpp::RE("b+").GlobalReplace("d", &s); +.sp +will leave "s" containing "yada dada doo". It returns the number of +replacements made. +.P +\fBExtract\fP is like \fBReplace\fP, except that if the pattern matches, +"rewrite" is copied into "out" (an additional argument) with substitutions. +The non-matching portions of "text" are ignored. Returns true iff a match +occurred and the extraction happened successfully; if no match occurs, the +string is left unaffected. +. +. +.SH AUTHOR +.rs +.sp +The C++ wrapper was contributed by Google Inc. +.br +Copyright (c) 2005 Google Inc. diff --git a/doc/pcregrep.1 b/doc/pcregrep.1 index 56c37d8..f1244e4 100644 --- a/doc/pcregrep.1 +++ b/doc/pcregrep.1 @@ -2,7 +2,7 @@ .SH NAME pcregrep - a grep with Perl-compatible regular expressions. .SH SYNOPSIS -.B pcregrep [-Vcfhilnrsuvx] [long options] [pattern] [file1 file2 ...] +.B pcregrep [options] [long options] [pattern] [file1 file2 ...] . .SH DESCRIPTION .rs @@ -19,28 +19,60 @@ PCRE supports. A pattern must be specified on the command line unless the \fB-f\fP option is used (see below). .P -If no files are specified, \fBpcregrep\fP reads the standard input. By default, -each line that matches the pattern is copied to the standard output, and if -there is more than one file, the file name is printed before each line of -output. However, there are options that can change how \fBpcregrep\fP behaves. +If no files are specified, \fBpcregrep\fP reads the standard input. The +standard input can also be referenced by a name consisting of a single hyphen. +For example: +.sp + pcregrep some-pattern /file1 - /file3 +.sp +By default, each line that matches the pattern is copied to the standard +output, and if there is more than one file, the file name is printed before +each line of output. However, there are options that can change how +\fBpcregrep\fP behaves. In particular, the \fB-M\fP option makes it possible to +search for patterns that span line boundaries. .P -Lines are limited to BUFSIZ characters. BUFSIZ is defined in \fB<stdio.h>\fP. -The newline character is removed from the end of each line before it is matched -against the pattern. +Patterns are limited to 8K or BUFSIZ characters, whichever is the greater. +BUFSIZ is defined in \fB<stdio.h>\fP. . .SH OPTIONS .rs -.sp .TP 10 -\fB-V\fP -Write the version number of the PCRE library being used to the standard error -stream. +\fB--\fP +This terminate the list of options. It is useful if the next item on the +command line starts with a hyphen, but is not an option. +.TP +\fB-A\fP \fInumber\fP +Print \fInumber\fP lines of context after each matching line. If file names +and/or line numbers are being printed, a hyphen separator is used instead of a +colon for the context lines. A line containing "--" is printed between each +group of lines, unless they are in fact contiguous in the input file. The value +of \fInumber\fP is expected to be relatively small. However, \fBpcregrep\fP +guarantees to have up to 8K of following text available for context printing. +.TP +\fB-B\fP \fInumber\fP +Print \fInumber\fP lines of context before each matching line. If file names +and/or line numbers are being printed, a hyphen separator is used instead of a +colon for the context lines. A line containing "--" is printed between each +group of lines, unless they are in fact contiguous in the input file. The value +of \fInumber\fP is expected to be relatively small. However, \fBpcregrep\fP +guarantees to have up to 8K of preceding text available for context printing. +.TP +\fB-C\fP \fInumber\fP +Print \fInumber\fP lines of context both before and after each matching line. +This is equivalent to setting both \fB-A\fP and \fB-B\fP to the same value. .TP \fB-c\fP Do not print individual lines; instead just print a count of the number of lines that would otherwise have been printed. If several files are given, a count is printed for each of them. .TP +\fB--exclude\fP=\fIpattern\fP +When \fBpcregrep\fP is searching the files in a directory as a consequence of +the \fB-r\fP (recursive search) option, any files whose names match the pattern +are excluded. The pattern is a PCRE regular expression. If a file name matches +both \fB--include\fP and \fB--exclude\fP, it is excluded. There is no short +form for this option. +.TP \fB-f\fP\fIfilename\fP Read a number of patterns from the file, one per line, and match all of them against each line of input. A line is output if any of the patterns match it. @@ -55,30 +87,73 @@ Suppress printing of filenames when searching multiple files. \fB-i\fP Ignore upper/lower case distinctions during comparisons. .TP +\fB--include\fP=\fIpattern\fP +When \fBpcregrep\fP is searching the files in a directory as a consequence of +the \fB-r\fP (recursive search) option, only files whose names match the +pattern are included. The pattern is a PCRE regular expression. If a file name +matches both \fB--include\fP and \fB--exclude\fP, it is excluded. There is no +short form for this option. +.TP +\fB-L\fP +Instead of printing lines from the files, just print the names of the files +that do not contain any lines that would have been printed. Each file name is +printed once, on a separate line. +.TP \fB-l\fP Instead of printing lines from the files, just print the names of the files containing lines that would have been printed. Each file name is printed once, on a separate line. .TP +\fB--label\fP=\fIname\fP +This option supplies a name to be used for the standard input when file names +are being printed. If not supplied, "(standard input)" is used. There is no +short form for this option. +.TP +\fB-M\fP +Allow patterns to match more than one line. When this option is given, patterns +may usefully contain literal newline characters and internal occurrences of ^ +and $ characters. The output for any one match may consist of more than one +line. When this option is set, the PCRE library is called in "multiline" mode. +There is a limit to the number of lines that can be matched, imposed by the way +that \fBpcregrep\fP buffers the input file as it scans it. However, +\fBpcregrep\fP ensures that at least 8K characters or the rest of the document +(whichever is the shorter) are available for forward matching, and similarly +the previous 8K characters (or all the previous characters, if fewer than 8K) +are guaranteed to be available for lookbehind assertions. +.TP \fB-n\fP Precede each line by its line number in the file. .TP +\fB-q\fP +Work quietly, that is, display nothing except error messages. +The exit status indicates whether or not any matches were found. +.TP \fB-r\fP -If any file is a directory, recursively scan the files it contains. Without +If any given path is a directory, recursively scan the files it contains, +taking note of any \fB--include\fP and \fB--exclude\fP settings. Without \fB-r\fP a directory is scanned as a normal file. .TP \fB-s\fP -Work silently, that is, display nothing except error messages. -The exit status indicates whether any matches were found. +Suppress error messages about non-existent or unreadable files. Such files are +quietly skipped. However, the return code is still 2, even if matches were +found in other files. .TP \fB-u\fP Operate in UTF-8 mode. This option is available only if PCRE has been compiled with UTF-8 support. Both the pattern and each subject line must be valid strings of UTF-8 characters. .TP +\fB-V\fP +Write the version numbers of \fBpcregrep\fP and the PCRE library that is being +used to the standard error stream. +.TP \fB-v\fP Invert the sense of the match, so that lines which do \fInot\fP match the -pattern are now the ones that are found. +pattern are the ones that are found. +.TP +\fB-w\fP +Force the pattern to match only whole words. This is equivalent to having \eb +at the start and end of the pattern. .TP \fB-x\fP Force the pattern to be anchored (it must start matching at the beginning of @@ -92,39 +167,66 @@ alternative branch in the regular expression. Long forms of all the options are available, as in GNU grep. They are shown in the following table: .sp + -A --after-context + -B --before-context + -C --context -c --count + --exclude (no short form) + -f --file -h --no-filename + --help (no short form) -i --ignore-case + --include (no short form) + -L --files-without-match -l --files-with-matches + --label (no short form) -n --line-number -r --recursive + -q --quiet -s --no-messages -u --utf-8 -V --version -v --invert-match -x --line-regex -x --line-regexp +. +.SH "OPTIONS WITH DATA" +.rs +.sp +There are four different ways in which an option with data can be specified. +If a short form option is used, the data may follow immediately, or in the next +command line item. For example: +.sp + -f/some/file + -f /some/file +.sp +If a long form option is used, the data may appear in the same command line +item, separated by an = character, or it may appear in the next command line +item. For example: +.sp + --file=/some/file + --file /some/file .sp -In addition, --file=\fIfilename\fP is equivalent to -f\fIfilename\fP, and ---help shows the list of options and then exits. . .SH DIAGNOSTICS .rs .sp Exit status is 0 if any matches were found, 1 if no matches were found, and 2 -for syntax errors or inacessible files (even if matches were found). +for syntax errors and non-existent or inacessible files (even if matches were +found in other files). Using the \fB-s\fP option to suppress error messages +about inaccessble files does not affect the return code. . . .SH AUTHOR .rs .sp -Philip Hazel <ph10@cam.ac.uk> +Philip Hazel .br University Computing Service .br Cambridge CB2 3QG, England. .P .in 0 -Last updated: 09 September 2004 +Last updated: 16 May 2005 .br -Copyright (c) 1997-2004 University of Cambridge. +Copyright (c) 1997-2005 University of Cambridge. diff --git a/doc/pcregrep.txt b/doc/pcregrep.txt index 1dca003..c2374e1 100644 --- a/doc/pcregrep.txt +++ b/doc/pcregrep.txt @@ -6,7 +6,7 @@ NAME pcregrep - a grep with Perl-compatible regular expressions. SYNOPSIS - pcregrep [-Vcfhilnrsuvx] [long options] [pattern] [file1 file2 ...] + pcregrep [options] [long options] [pattern] [file1 file2 ...] DESCRIPTION @@ -20,61 +20,139 @@ DESCRIPTION A pattern must be specified on the command line unless the -f option is used (see below). - If no files are specified, pcregrep reads the standard input. By - default, each line that matches the pattern is copied to the standard - output, and if there is more than one file, the file name is printed - before each line of output. However, there are options that can change - how pcregrep behaves. + If no files are specified, pcregrep reads the standard input. The stan- + dard input can also be referenced by a name consisting of a single + hyphen. For example: - Lines are limited to BUFSIZ characters. BUFSIZ is defined in <stdio.h>. - The newline character is removed from the end of each line before it is - matched against the pattern. + pcregrep some-pattern /file1 - /file3 + By default, each line that matches the pattern is copied to the stan- + dard output, and if there is more than one file, the file name is + printed before each line of output. However, there are options that can + change how pcregrep behaves. In particular, the -M option makes it pos- + sible to search for patterns that span line boundaries. + + Patterns are limited to 8K or BUFSIZ characters, whichever is the + greater. BUFSIZ is defined in <stdio.h>. -OPTIONS +OPTIONS - -V Write the version number of the PCRE library being used to - the standard error stream. + -- This terminate the list of options. It is useful if the next + item on the command line starts with a hyphen, but is not an + option. + + -A number Print number lines of context after each matching line. If + file names and/or line numbers are being printed, a hyphen + separator is used instead of a colon for the context lines. A + line containing "--" is printed between each group of lines, + unless they are in fact contiguous in the input file. The + value of number is expected to be relatively small. However, + pcregrep guarantees to have up to 8K of following text avail- + able for context printing. + + -B number Print number lines of context before each matching line. If + file names and/or line numbers are being printed, a hyphen + separator is used instead of a colon for the context lines. A + line containing "--" is printed between each group of lines, + unless they are in fact contiguous in the input file. The + value of number is expected to be relatively small. However, + pcregrep guarantees to have up to 8K of preceding text avail- + able for context printing. + + -C number Print number lines of context both before and after each + matching line. This is equivalent to setting both -A and -B + to the same value. -c Do not print individual lines; instead just print a count of the number of lines that would otherwise have been printed. If several files are given, a count is printed for each of them. + --exclude=pattern + When pcregrep is searching the files in a directory as a con- + sequence of the -r (recursive search) option, any files whose + names match the pattern are excluded. The pattern is a PCRE + regular expression. If a file name matches both --include and + --exclude, it is excluded. There is no short form for this + option. + -ffilename - Read a number of patterns from the file, one per line, and - match all of them against each line of input. A line is out- - put if any of the patterns match it. When -f is used, no - pattern is taken from the command line; all arguments are - treated as file names. There is a maximum of 100 patterns. + Read a number of patterns from the file, one per line, and + match all of them against each line of input. A line is out- + put if any of the patterns match it. When -f is used, no + pattern is taken from the command line; all arguments are + treated as file names. There is a maximum of 100 patterns. Trailing white space is removed, and blank lines are ignored. - An empty file contains no patterns and therefore matches + An empty file contains no patterns and therefore matches nothing. -h Suppress printing of filenames when searching multiple files. -i Ignore upper/lower case distinctions during comparisons. - -l Instead of printing lines from the files, just print the - names of the files containing lines that would have been - printed. Each file name is printed once, on a separate line. + --include=pattern + When pcregrep is searching the files in a directory as a con- + sequence of the -r (recursive search) option, only files + whose names match the pattern are included. The pattern is a + PCRE regular expression. If a file name matches both + --include and --exclude, it is excluded. There is no short + form for this option. + + -L Instead of printing lines from the files, just print the + names of the files that do not contain any lines that would + have been printed. Each file name is printed once, on a sepa- + rate line. + + -l Instead of printing lines from the files, just print the + names of the files containing lines that would have been + printed. Each file name is printed once, on a separate line. + + --label=name + This option supplies a name to be used for the standard input + when file names are being printed. If not supplied, "(stan- + dard input)" is used. There is no short form for this option. + + -M Allow patterns to match more than one line. When this option + is given, patterns may usefully contain literal newline char- + acters and internal occurrences of ^ and $ characters. The + output for any one match may consist of more than one line. + When this option is set, the PCRE library is called in "mul- + tiline" mode. There is a limit to the number of lines that + can be matched, imposed by the way that pcregrep buffers the + input file as it scans it. However, pcregrep ensures that at + least 8K characters or the rest of the document (whichever is + the shorter) are available for forward matching, and simi- + larly the previous 8K characters (or all the previous charac- + ters, if fewer than 8K) are guaranteed to be available for + lookbehind assertions. -n Precede each line by its line number in the file. - -r If any file is a directory, recursively scan the files it - contains. Without -r a directory is scanned as a normal file. - - -s Work silently, that is, display nothing except error mes- - sages. The exit status indicates whether any matches were + -q Work quietly, that is, display nothing except error messages. + The exit status indicates whether or not any matches were found. + -r If any given path is a directory, recursively scan the files + it contains, taking note of any --include and --exclude set- + tings. Without -r a directory is scanned as a normal file. + + -s Suppress error messages about non-existent or unreadable + files. Such files are quietly skipped. However, the return + code is still 2, even if matches were found in other files. + -u Operate in UTF-8 mode. This option is available only if PCRE has been compiled with UTF-8 support. Both the pattern and each subject line must be valid strings of UTF-8 characters. - -v Invert the sense of the match, so that lines which do not - match the pattern are now the ones that are found. + -V Write the version numbers of pcregrep and the PCRE library + that is being used to the standard error stream. + + -v Invert the sense of the match, so that lines which do not + match the pattern are the ones that are found. + + -w Force the pattern to match only whole words. This is equiva- + lent to having \b at the start and end of the pattern. -x Force the pattern to be anchored (it must start matching at the beginning of the line) and in addition, require it to @@ -88,12 +166,22 @@ LONG OPTIONS Long forms of all the options are available, as in GNU grep. They are shown in the following table: + -A --after-context + -B --before-context + -C --context -c --count + --exclude (no short form) + -f --file -h --no-filename + --help (no short form) -i --ignore-case + --include (no short form) + -L --files-without-match -l --files-with-matches + --label (no short form) -n --line-number -r --recursive + -q --quiet -s --no-messages -u --utf-8 -V --version @@ -101,22 +189,37 @@ LONG OPTIONS -x --line-regex -x --line-regexp - In addition, --file=filename is equivalent to -ffilename, and --help - shows the list of options and then exits. + +OPTIONS WITH DATA + + There are four different ways in which an option with data can be spec- + ified. If a short form option is used, the data may follow immedi- + ately, or in the next command line item. For example: + + -f/some/file + -f /some/file + + If a long form option is used, the data may appear in the same command + line item, separated by an = character, or it may appear in the next + command line item. For example: + + --file=/some/file + --file /some/file DIAGNOSTICS Exit status is 0 if any matches were found, 1 if no matches were found, - and 2 for syntax errors or inacessible files (even if matches were - found). + and 2 for syntax errors and non-existent or inacessible files (even if + matches were found in other files). Using the -s option to suppress + error messages about inaccessble files does not affect the return code. AUTHOR - Philip Hazel <ph10@cam.ac.uk> + Philip Hazel University Computing Service Cambridge CB2 3QG, England. -Last updated: 09 September 2004 -Copyright (c) 1997-2004 University of Cambridge. +Last updated: 16 May 2005 +Copyright (c) 1997-2005 University of Cambridge. diff --git a/doc/pcrematching.3 b/doc/pcrematching.3 new file mode 100644 index 0000000..4524701 --- /dev/null +++ b/doc/pcrematching.3 @@ -0,0 +1,157 @@ +.TH PCRE 3 +.SH NAME +PCRE - Perl-compatible regular expressions +.SH "PCRE MATCHING ALGORITHMS" +.rs +.sp +This document describes the two different algorithms that are available in PCRE +for matching a compiled regular expression against a given subject string. The +"standard" algorithm is the one provided by the \fBpcre_exec()\fP function. +This works in the same was as Perl's matching function, and provides a +Perl-compatible matching operation. +.P +An alternative algorithm is provided by the \fBpcre_dfa_exec()\fP function; +this operates in a different way, and is not Perl-compatible. It has advantages +and disadvantages compared with the standard algorithm, and these are described +below. +.P +When there is only one possible way in which a given subject string can match a +pattern, the two algorithms give the same answer. A difference arises, however, +when there are multiple possibilities. For example, if the pattern +.sp + ^<.*> +.sp +is matched against the string +.sp + <something> <something else> <something further> +.sp +there are three possible answers. The standard algorithm finds only one of +them, whereas the DFA algorithm finds all three. +. +.SH "REGULAR EXPRESSIONS AS TREES" +.rs +.sp +The set of strings that are matched by a regular expression can be represented +as a tree structure. An unlimited repetition in the pattern makes the tree of +infinite size, but it is still a tree. Matching the pattern to a given subject +string (from a given starting point) can be thought of as a search of the tree. +There are two standard ways to search a tree: depth-first and breadth-first, +and these correspond to the two matching algorithms provided by PCRE. +. +.SH "THE STANDARD MATCHING ALGORITHM" +.rs +.sp +In the terminology of Jeffrey Friedl's book \fIMastering Regular +Expressions\fP, the standard algorithm is an "NFA algorithm". It conducts a +depth-first search of the pattern tree. That is, it proceeds along a single +path through the tree, checking that the subject matches what is required. When +there is a mismatch, the algorithm tries any alternatives at the current point, +and if they all fail, it backs up to the previous branch point in the tree, and +tries the next alternative branch at that level. This often involves backing up +(moving to the left) in the subject string as well. The order in which +repetition branches are tried is controlled by the greedy or ungreedy nature of +the quantifier. +.P +If a leaf node is reached, a matching string has been found, and at that point +the algorithm stops. Thus, if there is more than one possible match, this +algorithm returns the first one that it finds. Whether this is the shortest, +the longest, or some intermediate length depends on the way the greedy and +ungreedy repetition quantifiers are specified in the pattern. +.P +Because it ends up with a single path through the tree, it is relatively +straightforward for this algorithm to keep track of the substrings that are +matched by portions of the pattern in parentheses. This provides support for +capturing parentheses and back references. +. +.SH "THE DFA MATCHING ALGORITHM" +.rs +.sp +DFA stands for "deterministic finite automaton", but you do not need to +understand the origins of that name. This algorithm conducts a breadth-first +search of the tree. Starting from the first matching point in the subject, it +scans the subject string from left to right, once, character by character, and +as it does this, it remembers all the paths through the tree that represent +valid matches. +.P +The scan continues until either the end of the subject is reached, or there are +no more unterminated paths. At this point, terminated paths represent the +different matching possibilities (if there are none, the match has failed). +Thus, if there is more than one possible match, this algorithm finds all of +them, and in particular, it finds the longest. In PCRE, there is an option to +stop the algorithm after the first match (which is necessarily the shortest) +has been found. +.P +Note that all the matches that are found start at the same point in the +subject. If the pattern +.sp + cat(er(pillar)?) +.sp +is matched against the string "the caterpillar catchment", the result will be +the three strings "cat", "cater", and "caterpillar" that start at the fourth +character of the subject. The algorithm does not automatically move on to find +matches that start at later positions. +.P +There are a number of features of PCRE regular expressions that are not +supported by the DFA matching algorithm. They are as follows: +.P +1. Because the algorithm finds all possible matches, the greedy or ungreedy +nature of repetition quantifiers is not relevant. Greedy and ungreedy +quantifiers are treated in exactly the same way. +.P +2. When dealing with multiple paths through the tree simultaneously, it is not +straightforward to keep track of captured substrings for the different matching +possibilities, and PCRE's implementation of this algorithm does not attempt to +do this. This means that no captured substrings are available. +.P +3. Because no substrings are captured, back references within the pattern are +not supported, and cause errors if encountered. +.P +4. For the same reason, conditional expressions that use a backreference as the +condition are not supported. +.P +5. Callouts are supported, but the value of the \fIcapture_top\fP field is +always 1, and the value of the \fIcapture_last\fP field is always -1. +.P +6. +The \eC escape sequence, which (in the standard algorithm) matches a single +byte, even in UTF-8 mode, is not supported because the DFA algorithm moves +through the subject string one character at a time, for all active paths +through the tree. +. +.SH "ADVANTAGES OF THE DFA ALGORITHM" +.rs +.sp +Using the DFA matching algorithm provides the following advantages: +.P +1. All possible matches (at a single point in the subject) are automatically +found, and in particular, the longest match is found. To find more than one +match using the standard algorithm, you have to do kludgy things with +callouts. +.P +2. There is much better support for partial matching. The restrictions on the +content of the pattern that apply when using the standard algorithm for partial +matching do not apply to the DFA algorithm. For non-anchored patterns, the +starting position of a partial match is available. +.P +3. Because the DFA algorithm scans the subject string just once, and never +needs to backtrack, it is possible to pass very long subject strings to the +matching function in several pieces, checking for partial matching each time. +. +.SH "DISADVANTAGES OF THE DFA ALGORITHM" +.rs +.sp +The DFA algorithm suffers from a number of disadvantages: +.P +1. It is substantially slower than the standard algorithm. This is partly +because it has to search for all possible matches, but is also because it is +less susceptible to optimization. +.P +2. Capturing parentheses and back references are not supported. +.P +3. The "atomic group" feature of PCRE regular expressions is supported, but +does not provide the advantage that it does for the standard algorithm. +.P +.in 0 +Last updated: 28 February 2005 +.br +Copyright (c) 1997-2005 University of Cambridge. diff --git a/doc/pcrepartial.3 b/doc/pcrepartial.3 index 3489c18..ffc0c6e 100644 --- a/doc/pcrepartial.3 +++ b/doc/pcrepartial.3 @@ -5,10 +5,10 @@ PCRE - Perl-compatible regular expressions .rs .sp In normal use of PCRE, if the subject string that is passed to -\fBpcre_exec()\fP matches as far as it goes, but is too short to match the -entire pattern, PCRE_ERROR_NOMATCH is returned. There are circumstances where -it might be helpful to distinguish this case from other cases in which there is -no match. +\fBpcre_exec()\fP or \fBpcre_dfa_exec()\fP matches as far as it goes, but is +too short to match the entire pattern, PCRE_ERROR_NOMATCH is returned. There +are circumstances where it might be helpful to distinguish this case from other +cases in which there is no match. .P Consider, for example, an application where a human is required to type in data for a field with specific formatting requirements. An example might be a date @@ -24,10 +24,19 @@ user interface than a check that is delayed until the entire string has been entered. .P PCRE supports the concept of partial matching by means of the PCRE_PARTIAL -option, which can be set when calling \fBpcre_exec()\fP. When this is done, the -return code PCRE_ERROR_NOMATCH is converted into PCRE_ERROR_PARTIAL if at any -time during the matching process the entire subject string matched part of the -pattern. No captured data is set when this occurs. +option, which can be set when calling \fBpcre_exec()\fP or +\fBpcre_dfa_exec()\fP. When this flag is set for \fBpcre_exec()\fP, the return +code PCRE_ERROR_NOMATCH is converted into PCRE_ERROR_PARTIAL if at any time +during the matching process the last part of the subject string matched part of +the pattern. Unfortunately, for non-anchored matching, it is not possible to +obtain the position of the start of the partial match. No captured data is set +when PCRE_ERROR_PARTIAL is returned. +.P +When PCRE_PARTIAL is set for \fBpcre_dfa_exec()\fP, the return code +PCRE_ERROR_NOMATCH is converted into PCRE_ERROR_PARTIAL if the end of the +subject is reached, there have been no complete matches, but there is still at +least one matching possibility. The portion of the string that provided the +partial match is set as the first matching string. .P Using PCRE_PARTIAL disables one of PCRE's optimizations. PCRE remembers the last literal byte in a pattern, and abandons matching immediately if such a @@ -38,9 +47,10 @@ for a subject string that might match only partially. .SH "RESTRICTED PATTERNS FOR PCRE_PARTIAL" .rs .sp -Because of the way certain internal optimizations are implemented in PCRE, the -PCRE_PARTIAL option cannot be used with all patterns. Repeated single -characters such as +Because of the way certain internal optimizations are implemented in the +\fBpcre_exec()\fP function, the PCRE_PARTIAL option cannot be used with all +patterns. These restrictions do not apply when \fBpcre_dfa_exec()\fP is used. +For \fBpcre_exec()\fP, repeated single characters such as .sp a{2,4} .sp @@ -85,11 +95,90 @@ uses the date example quoted above: .sp The first data string is matched completely, so \fBpcretest\fP shows the matched substrings. The remaining four strings do not match the complete -pattern, but the first two are partial matches. +pattern, but the first two are partial matches. The same test, using DFA +matching (by means of the \eD escape sequence), produces the following output: +.sp + re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/ + data> 25jun04\eP\eD + 0: 25jun04 + data> 23dec3\eP\eD + Partial match: 23dec3 + data> 3ju\eP\eD + Partial match: 3ju + data> 3juj\eP\eD + No match + data> j\eP\eD + No match +.sp +Notice that in this case the portion of the string that was matched is made +available. +. +. +.SH "MULTI-SEGMENT MATCHING WITH pcre_dfa_exec()" +.rs +.sp +When a partial match has been found using \fBpcre_dfa_exec()\fP, it is possible +to continue the match by providing additional subject data and calling +\fBpcre_dfa_exec()\fP again with the PCRE_DFA_RESTART option and the same +working space (where details of the previous partial match are stored). Here is +an example using \fBpcretest\fP, where the \eR escape sequence sets the +PCRE_DFA_RESTART option and the \eD escape sequence requests the use of +\fBpcre_dfa_exec()\fP: +.sp + re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/ + data> 23ja\eP\eD + Partial match: 23ja + data> n05\eR\eD + 0: n05 +.sp +The first call has "23ja" as the subject, and requests partial matching; the +second call has "n05" as the subject for the continued (restarted) match. +Notice that when the match is complete, only the last part is shown; PCRE does +not retain the previously partially-matched string. It is up to the calling +program to do that if it needs to. +.P +This facility can be used to pass very long subject strings to +\fBpcre_dfa_exec()\fP. However, some care is needed for certain types of +pattern. +.P +1. If the pattern contains tests for the beginning or end of a line, you need +to pass the PCRE_NOTBOL or PCRE_NOTEOL options, as appropriate, when the +subject string for any call does not contain the beginning or end of a line. +.P +2. If the pattern contains backward assertions (including \eb or \eB), you need +to arrange for some overlap in the subject strings to allow for this. For +example, you could pass the subject in chunks that were 500 bytes long, but in +a buffer of 700 bytes, with the starting offset set to 200 and the previous 200 +bytes at the start of the buffer. +.P +3. Matching a subject string that is split into multiple segments does not +always produce exactly the same result as matching over one single long string. +The difference arises when there are multiple matching possibilities, because a +partial match result is given only when there are no completed matches in a +call to fBpcre_dfa_exec()\fP. This means that as soon as the shortest match has +been found, continuation to a new subject segment is no longer possible. +Consider this \fBpcretest\fP example: +.sp + re> /dog(sbody)?/ + data> do\eP\eD + Partial match: do + data> gsb\eR\eP\eD + 0: g + data> dogsbody\eD + 0: dogsbody + 1: dog +.sp +The pattern matches the words "dog" or "dogsbody". When the subject is +presented in several parts ("do" and "gsb" being the first two) the match stops +when "dog" has been found, and it is not possible to continue. On the other +hand, if "dogsbody" is presented as a single string, both matches are found. +.P +Because of this phenomenon, it does not usually make sense to end a pattern +that is going to be matched in this way with a variable repeat. . . .P .in 0 -Last updated: 08 September 2004 +Last updated: 28 February 2005 .br -Copyright (c) 1997-2004 University of Cambridge. +Copyright (c) 1997-2005 University of Cambridge. diff --git a/doc/pcrepattern.3 b/doc/pcrepattern.3 index 6f6a21a..fa39e53 100644 --- a/doc/pcrepattern.3 +++ b/doc/pcrepattern.3 @@ -26,15 +26,35 @@ in the main .\" page. .P +The remainder of this document discusses the patterns that are supported by +PCRE when its main matching function, \fBpcre_exec()\fP, is used. +From release 6.0, PCRE offers a second matching function, +\fBpcre_dfa_exec()\fP, which matches using a different algorithm that is not +Perl-compatible. The advantages and disadvantages of the alternative function, +and how it differs from the normal function, are discussed in the +.\" HREF +\fBpcrematching\fP +.\" +page. +.P A regular expression is a pattern that is matched against a subject string from left to right. Most characters stand for themselves in a pattern, and match the corresponding characters in the subject. As a trivial example, the pattern .sp The quick brown fox .sp -matches a portion of a subject string that is identical to itself. The power of -regular expressions comes from the ability to include alternatives and -repetitions in the pattern. These are encoded in the pattern by the use of +matches a portion of a subject string that is identical to itself. When +caseless matching is specified (the PCRE_CASELESS option), letters are matched +independently of case. In UTF-8 mode, PCRE always understands the concept of +case for characters whose values are less than 128, so caseless matching is +always possible. For characters with higher values, the concept of case is +supported if PCRE is compiled with Unicode property support, but not otherwise. +If you want to use caseless matching for characters 128 and above, you must +ensure that PCRE is compiled with Unicode property support as well as with +UTF-8 support. +.P +The power of regular expressions comes from the ability to include alternatives +and repetitions in the pattern. These are encoded in the pattern by the use of \fImetacharacters\fP, which do not stand for themselves but instead are interpreted in some special way. .P @@ -527,9 +547,13 @@ class as a literal string of bytes, or by using the \ex{ escaping mechanism. When caseless matching is set, any letters in a class represent both their upper case and lower case versions, so for example, a caseless [aeiou] matches "A" as well as "a", and a caseless [^aeiou] does not match "A", whereas a -caseful version would. When running in UTF-8 mode, PCRE supports the concept of -case for characters with values greater than 128 only when it is compiled with -Unicode property support. +caseful version would. In UTF-8 mode, PCRE always understands the concept of +case for characters whose values are less than 128, so caseless matching is +always possible. For characters with higher values, the concept of case is +supported if PCRE is compiled with Unicode property support, but not otherwise. +If you want to use caseless matching for characters 128 and above, you must +ensure that PCRE is compiled with Unicode property support as well as with +UTF-8 support. .P The newline character is never treated in any special way in character classes, whatever the setting of the PCRE_DOTALL or PCRE_MULTILINE options is. A class @@ -1451,6 +1475,6 @@ description of the interface to the callout function is given in the documentation. .P .in 0 -Last updated: 09 September 2004 +Last updated: 28 February 2005 .br -Copyright (c) 1997-2004 University of Cambridge. +Copyright (c) 1997-2005 University of Cambridge. diff --git a/doc/pcreperform.3 b/doc/pcreperform.3 index 999268e..82e454c 100644 --- a/doc/pcreperform.3 +++ b/doc/pcreperform.3 @@ -35,8 +35,8 @@ this, PCRE has to retry the match starting after every newline in the subject. .P If you are using such a pattern with subject strings that do not contain newlines, the best performance is obtained by setting PCRE_DOTALL, or starting -the pattern with ^.* to indicate explicit anchoring. That saves PCRE from -having to scan along the subject looking for a newline to restart at. +the pattern with ^.* or ^.*? to indicate explicit anchoring. That saves PCRE +from having to scan along the subject looking for a newline to restart at. .P Beware of patterns that contain nested indefinite repeats. These can take a long time to run when applied to a string that does not match. Consider the @@ -71,6 +71,6 @@ In many cases, the solution to this kind of performance issue is to use an atomic group or a possessive quantifier. .P .in 0 -Last updated: 09 September 2004 +Last updated: 28 February 2005 .br -Copyright (c) 1997-2004 University of Cambridge. +Copyright (c) 1997-2005 University of Cambridge. diff --git a/doc/pcreposix.3 b/doc/pcreposix.3 index 321dcd7..b67d6ff 100644 --- a/doc/pcreposix.3 +++ b/doc/pcreposix.3 @@ -33,8 +33,8 @@ package. See the .\" HREF \fBpcreapi\fP .\" -documentation for a description of PCRE's native API, which contains additional -functionality. +documentation for a description of PCRE's native API, which contains much +additional functionality. .P The functions described here are just wrapper functions that ultimately call the PCRE native API. Their prototypes are defined in the \fBpcreposix.h\fP @@ -76,6 +76,11 @@ about the compiled expression. The argument \fIcflags\fP is either zero, or contains one or more of the bits defined by the following macros: .sp + REG_DOTALL +.sp +The PCRE_DOTALL option is set when the expression is passed for compilation to +the native function. Note that REG_DOTALL is not part of the POSIX standard. +.sp REG_ICASE .sp The PCRE_CASELESS option is set when the expression is passed for compilation @@ -189,13 +194,13 @@ memory, after which \fIpreg\fP may no longer be used as a compiled expression. .SH AUTHOR .rs .sp -Philip Hazel <ph10@cam.ac.uk> +Philip Hazel .br University Computing Service, .br Cambridge CB2 3QG, England. .P .in 0 -Last updated: 07 September 2004 +Last updated: 28 February 2005 .br -Copyright (c) 1997-2004 University of Cambridge. +Copyright (c) 1997-2005 University of Cambridge. diff --git a/doc/pcreprecompile.3 b/doc/pcreprecompile.3 index f08939b..f359b96 100644 --- a/doc/pcreprecompile.3 +++ b/doc/pcreprecompile.3 @@ -79,15 +79,16 @@ return a non-NULL value before trying to save the study data. .rs .sp Re-using a precompiled pattern is straightforward. Having reloaded it into main -memory, you pass its pointer to \fBpcre_exec()\fP in the usual way. This should -work even on another host, and even if that host has the opposite endianness to -the one where the pattern was compiled. +memory, you pass its pointer to \fBpcre_exec()\fP or \fBpcre_dfa_exec()\fP in +the usual way. This should work even on another host, and even if that host has +the opposite endianness to the one where the pattern was compiled. .P However, if you passed a pointer to custom character tables when the pattern was compiled (the \fItableptr\fP argument of \fBpcre_compile()\fP), you must -now pass a similar pointer to \fBpcre_exec()\fP, because the value saved with -the compiled pattern will obviously be nonsense. A field in a -\fBpcre_extra()\fP block is used to pass this data, as described in the +now pass a similar pointer to \fBpcre_exec()\fP or \fBpcre_dfa_exec()\fP, +because the value saved with the compiled pattern will obviously be nonsense. A +field in a \fBpcre_extra()\fP block is used to pass this data, as described in +the .\" HTML <a href="pcreapi.html#extradata"> .\" </a> section on matching a pattern @@ -107,7 +108,8 @@ If you saved study data with the compiled pattern, you need to create your own \fBpcre_extra\fP data block and set the \fIstudy_data\fP field to point to the reloaded study data. You must also set the PCRE_EXTRA_STUDY_DATA bit in the \fIflags\fP field to indicate that study data is present. Then pass the -\fBpcre_extra\fP block to \fBpcre_exec()\fP in the usual way. +\fBpcre_extra\fP block to \fBpcre_exec()\fP or \fBpcre_dfa_exec()\fP in the +usual way. . . .SH "COMPATIBILITY WITH DIFFERENT PCRE RELEASES" @@ -120,6 +122,6 @@ advertised), you will have to recompile them for release 5.0. However, from now on, it should be possible to make changes in a compabible manner. .P .in 0 -Last updated: 10 September 2004 +Last updated: 28 February 2005 .br -Copyright (c) 1997-2004 University of Cambridge. +Copyright (c) 1997-2005 University of Cambridge. diff --git a/doc/pcretest.1 b/doc/pcretest.1 index 0c06cb7..336abcf 100644 --- a/doc/pcretest.1 +++ b/doc/pcretest.1 @@ -4,7 +4,7 @@ pcretest - a program for testing Perl-compatible regular expressions. .SH SYNOPSIS .rs .sp -.B pcretest "[-C] [-d] [-i] [-m] [-o osize] [-p] [-t] [source]" +.B pcretest "[-C] [-d] [-dfa] [-i] [-m] [-o osize] [-p] [-t] [source]" .ti +5n .B "[destination]" .P @@ -31,11 +31,16 @@ Output the version number of the PCRE library, and all available information about the optional features that are included, and then exit. .TP 10 \fB-d\fP -Behave as if each regex had the \fB/D\fP (debug) modifier; the internal +Behave as if each regex has the \fB/D\fP (debug) modifier; the internal form is output after compilation. .TP 10 +\fB-dfa\fP +Behave as if each data line contains the \eD escape sequence; this causes the +alternative matching function, \fBpcre_dfa_exec()\fP, to be used instead of the +standard \fBpcre_exec()\fP function (more detail is given below). +.TP 10 \fB-i\fP -Behave as if each regex had the \fB/I\fP modifier; information about the +Behave as if each regex has the \fB/I\fP modifier; information about the compiled pattern is given after compilation. .TP 10 \fB-m\fP @@ -50,8 +55,9 @@ for 14 capturing subexpressions. The vector size can be changed for individual matching calls by including \eO in the data line (see below). .TP 10 \fB-p\fP -Behave as if each regex has \fB/P\fP modifier; the POSIX wrapper API is used -to call PCRE. None of the other options has any effect when \fB-p\fP is set. +Behave as if each regex has the \fB/P\fP modifier; the POSIX wrapper API is +used to call PCRE. None of the other options has any effect when \fB-p\fP is +set. .TP 10 \fB-t\fP Run each compile, study, and match many times with a timer, and output @@ -131,6 +137,7 @@ not correspond to anything in Perl: \fB/A\fP PCRE_ANCHORED \fB/C\fP PCRE_AUTO_CALLOUT \fB/E\fP PCRE_DOLLAR_ENDONLY + \fB/f\fP PCRE_FIRSTLINE \fB/N\fP PCRE_NO_AUTO_CAPTURE \fB/U\fP PCRE_UNGREEDY \fB/X\fP PCRE_EXTRA @@ -257,6 +264,8 @@ recognized: .\" JOIN \eC*n pass the number n (may be negative) as callout data; this is used as the callout return value + \eD use the \fBpcre_dfa_exec()\fP match function + \eF only shortest match for \fBpcre_dfa_exec()\fP .\" JOIN \eGdd call pcre_get_substring() for substring dd after a successful match (number less than 32) @@ -272,7 +281,10 @@ recognized: .\" JOIN \eOdd set the size of the output vector passed to \fBpcre_exec()\fP to dd (any number of digits) +.\" JOIN \eP pass the PCRE_PARTIAL option to \fBpcre_exec()\fP + or \fBpcre_dfa_exec()\fP + \eR pass the PCRE_DFA_RESTART option to \fBpcre_dfa_exec()\fP \eS output details of memory get/free calls during matching \eZ pass the PCRE_NOTEOL option to \fBpcre_exec()\fP .\" JOIN @@ -308,15 +320,38 @@ any number of hexadecimal digits inside the braces. The result is from one to six bytes, encoded according to the UTF-8 rules. . . -.SH "OUTPUT FROM PCRETEST" +.SH "THE ALTERNATIVE MATCHING FUNCTION" .rs .sp +By default, \fBpcretest\fP uses the standard PCRE matching function, +\fBpcre_exec()\fP to match each data line. From release 6.0, PCRE supports an +alternative matching function, \fBpcre_dfa_test()\fP, which operates in a +different way, and has some restrictions. The differences between the two +functions are described in the +.\" HREF +\fBpcrematching\fP +.\" +documentation. +.P +If a data line contains the \eD escape sequence, or if the command line +contains the \fB-dfa\fP option, the alternative matching function is called. +This function finds all possible matches at a given point. If, however, the \eF +escape sequence is present in the data line, it stops after the first match is +found. This is always the shortest possible match. +. +. +.SH "DEFAULT OUTPUT FROM PCRETEST" +.rs +.sp +This section describes the output when the normal matching function, +\fBpcre_exec()\fP, is being used. +.P When a match succeeds, pcretest outputs the list of captured substrings that \fBpcre_exec()\fP returns, starting with number 0 for the string that matched the whole pattern. Otherwise, it outputs "No match" or "Partial match" when \fBpcre_exec()\fP returns PCRE_ERROR_NOMATCH or PCRE_ERROR_PARTIAL, respectively, and otherwise the PCRE negative error number. Here is an example -of an interactive pcretest run. +of an interactive \fBpcretest\fP run. .sp $ pcretest PCRE version 5.00 07-Sep-2004 @@ -365,13 +400,68 @@ prompt is used for continuations), data lines may not. However newlines can be included in data by means of the \en escape. . . +.SH "OUTPUT FROM THE ALTERNATIVE MATCHING FUNCTION" +.rs +.sp +When the alternative matching function, \fBpcre_dfa_exec()\fP, is used (by +means of the \eD escape sequence or the \fB-dfa\fP command line option), the +output consists of a list of all the matches that start at the first point in +the subject where there is at least one match. For example: +.sp + re> /(tang|tangerine|tan)/ + data> yellow tangerine\eD + 0: tangerine + 1: tang + 2: tan +.sp +(Using the normal matching function on this data finds only "tang".) The +longest matching string is always given first (and numbered zero). +.P +If \fB/g\P is present on the pattern, the search for further matches resumes +at the end of the longest match. For example: +.sp + re> /(tang|tangerine|tan)/g + data> yellow tangerine and tangy sultana\eD + 0: tangerine + 1: tang + 2: tan + 0: tang + 1: tan + 0: tan +.sp +Since the matching function does not support substring capture, the escape +sequences that are concerned with captured substrings are not relevant. +. +. +.SH "RESTARTING AFTER A PARTIAL MATCH" +.rs +.sp +When the alternative matching function has given the PCRE_ERROR_PARTIAL return, +indicating that the subject partially matched the pattern, you can restart the +match with additional subject data by means of the \eR escape sequence. For +example: +.sp + re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/ + data> 23ja\eP\eD + Partial match: 23ja + data> n05\eR\eD + 0: n05 +.sp +For further information about partial matching, see the +.\" HREF +\fBpcrepartial\fP +.\" +documentation. +. +. .SH CALLOUTS .rs .sp If the pattern contains any callout requests, \fBpcretest\fP's callout function -is called during matching. By default, it displays the callout number, the -start and current positions in the text at the callout time, and the next -pattern item to be tested. For example, the output +is called during matching. This works with both matching functions. By default, +the called function displays the callout number, the start and current +positions in the text at the callout time, and the next pattern item to be +tested. For example, the output .sp --->pqrabcdef 0 ^ ^ \ed @@ -396,7 +486,7 @@ example: 0: E* .sp The callout function in \fBpcretest\fP returns zero (carry on matching) by -default, but you can use an \eC item in a data line (as described above) to +default, but you can use a \eC item in a data line (as described above) to change this. .P Inserting callouts can be helpful when using \fBpcretest\fP to check @@ -471,13 +561,13 @@ result is undefined. .SH AUTHOR .rs .sp -Philip Hazel <ph10@cam.ac.uk> +Philip Hazel .br University Computing Service, .br Cambridge CB2 3QG, England. .P .in 0 -Last updated: 10 September 2004 +Last updated: 28 February 2005 .br -Copyright (c) 1997-2004 University of Cambridge. +Copyright (c) 1997-2005 University of Cambridge. diff --git a/doc/pcretest.txt b/doc/pcretest.txt index 7da6889..2badffa 100644 --- a/doc/pcretest.txt +++ b/doc/pcretest.txt @@ -7,7 +7,7 @@ NAME SYNOPSIS - pcretest [-C] [-d] [-i] [-m] [-o osize] [-p] [-t] [source] + pcretest [-C] [-d] [-dfa] [-i] [-m] [-o osize] [-p] [-t] [source] [destination] pcretest was written as a test program for the PCRE regular expression @@ -24,95 +24,100 @@ OPTIONS able information about the optional features that are included, and then exit. - -d Behave as if each regex had the /D (debug) modifier; the + -d Behave as if each regex has the /D (debug) modifier; the internal form is output after compilation. - -i Behave as if each regex had the /I modifier; information + -dfa Behave as if each data line contains the \D escape sequence; + this causes the alternative matching function, + pcre_dfa_exec(), to be used instead of the standard + pcre_exec() function (more detail is given below). + + -i Behave as if each regex has the /I modifier; information about the compiled pattern is given after compilation. - -m Output the size of each compiled pattern after it has been - compiled. This is equivalent to adding /M to each regular - expression. For compatibility with earlier versions of + -m Output the size of each compiled pattern after it has been + compiled. This is equivalent to adding /M to each regular + expression. For compatibility with earlier versions of pcretest, -s is a synonym for -m. - -o osize Set the number of elements in the output vector that is used - when calling pcre_exec() to be osize. The default value is + -o osize Set the number of elements in the output vector that is used + when calling pcre_exec() to be osize. The default value is 45, which is enough for 14 capturing subexpressions. The vec- - tor size can be changed for individual matching calls by + tor size can be changed for individual matching calls by including \O in the data line (see below). - -p Behave as if each regex has /P modifier; the POSIX wrapper - API is used to call PCRE. None of the other options has any - effect when -p is set. + -p Behave as if each regex has the /P modifier; the POSIX wrap- + per API is used to call PCRE. None of the other options has + any effect when -p is set. - -t Run each compile, study, and match many times with a timer, - and output resulting time per compile or match (in millisec- - onds). Do not set -m with -t, because you will then get the - size output a zillion times, and the timing will be dis- + -t Run each compile, study, and match many times with a timer, + and output resulting time per compile or match (in millisec- + onds). Do not set -m with -t, because you will then get the + size output a zillion times, and the timing will be dis- torted. DESCRIPTION - If pcretest is given two filename arguments, it reads from the first + If pcretest is given two filename arguments, it reads from the first and writes to the second. If it is given only one filename argument, it - reads from that file and writes to stdout. Otherwise, it reads from - stdin and writes to stdout, and prompts for each line of input, using + reads from that file and writes to stdout. Otherwise, it reads from + stdin and writes to stdout, and prompts for each line of input, using "re>" to prompt for regular expressions, and "data>" to prompt for data lines. The program handles any number of sets of input on a single input file. - Each set starts with a regular expression, and continues with any num- + Each set starts with a regular expression, and continues with any num- ber of data lines to be matched against the pattern. - Each data line is matched separately and independently. If you want to - do multiple-line matches, you have to use the \n escape sequence in a - single line of input to encode the newline characters. The maximum + Each data line is matched separately and independently. If you want to + do multiple-line matches, you have to use the \n escape sequence in a + single line of input to encode the newline characters. The maximum length of data line is 30,000 characters. - An empty line signals the end of the data lines, at which point a new - regular expression is read. The regular expressions are given enclosed + An empty line signals the end of the data lines, at which point a new + regular expression is read. The regular expressions are given enclosed in any non-alphanumeric delimiters other than backslash, for example /(a|bc)x+yz/ - White space before the initial delimiter is ignored. A regular expres- - sion may be continued over several input lines, in which case the new- - line characters are included within it. It is possible to include the + White space before the initial delimiter is ignored. A regular expres- + sion may be continued over several input lines, in which case the new- + line characters are included within it. It is possible to include the delimiter within the pattern by escaping it, for example /abc\/def/ - If you do so, the escape and the delimiter form part of the pattern, - but since delimiters are always non-alphanumeric, this does not affect - its interpretation. If the terminating delimiter is immediately fol- + If you do so, the escape and the delimiter form part of the pattern, + but since delimiters are always non-alphanumeric, this does not affect + its interpretation. If the terminating delimiter is immediately fol- lowed by a backslash, for example, /abc/\ - then a backslash is added to the end of the pattern. This is done to - provide a way of testing the error condition that arises if a pattern + then a backslash is added to the end of the pattern. This is done to + provide a way of testing the error condition that arises if a pattern finishes with a backslash, because /abc\/ - is interpreted as the first line of a pattern that starts with "abc/", + is interpreted as the first line of a pattern that starts with "abc/", causing pcretest to read the next line as a continuation of the regular expression. PATTERN MODIFIERS - A pattern may be followed by any number of modifiers, which are mostly - single characters. Following Perl usage, these are referred to below - as, for example, "the /i modifier", even though the delimiter of the - pattern need not always be a slash, and no slash is used when writing - modifiers. Whitespace may appear between the final pattern delimiter + A pattern may be followed by any number of modifiers, which are mostly + single characters. Following Perl usage, these are referred to below + as, for example, "the /i modifier", even though the delimiter of the + pattern need not always be a slash, and no slash is used when writing + modifiers. Whitespace may appear between the final pattern delimiter and the first modifier, and between the modifiers themselves. The /i, /m, /s, and /x modifiers set the PCRE_CASELESS, PCRE_MULTILINE, - PCRE_DOTALL, or PCRE_EXTENDED options, respectively, when pcre_com- - pile() is called. These four modifier letters have the same effect as + PCRE_DOTALL, or PCRE_EXTENDED options, respectively, when pcre_com- + pile() is called. These four modifier letters have the same effect as they do in Perl. For example: /caseless/i @@ -123,95 +128,96 @@ PATTERN MODIFIERS /A PCRE_ANCHORED /C PCRE_AUTO_CALLOUT /E PCRE_DOLLAR_ENDONLY + /f PCRE_FIRSTLINE /N PCRE_NO_AUTO_CAPTURE /U PCRE_UNGREEDY /X PCRE_EXTRA - Searching for all possible matches within each subject string can be - requested by the /g or /G modifier. After finding a match, PCRE is + Searching for all possible matches within each subject string can be + requested by the /g or /G modifier. After finding a match, PCRE is called again to search the remainder of the subject string. The differ- ence between /g and /G is that the former uses the startoffset argument - to pcre_exec() to start searching at a new point within the entire - string (which is in effect what Perl does), whereas the latter passes - over a shortened substring. This makes a difference to the matching + to pcre_exec() to start searching at a new point within the entire + string (which is in effect what Perl does), whereas the latter passes + over a shortened substring. This makes a difference to the matching process if the pattern begins with a lookbehind assertion (including \b or \B). - If any call to pcre_exec() in a /g or /G sequence matches an empty - string, the next call is done with the PCRE_NOTEMPTY and PCRE_ANCHORED - flags set in order to search for another, non-empty, match at the same - point. If this second match fails, the start offset is advanced by - one, and the normal match is retried. This imitates the way Perl han- + If any call to pcre_exec() in a /g or /G sequence matches an empty + string, the next call is done with the PCRE_NOTEMPTY and PCRE_ANCHORED + flags set in order to search for another, non-empty, match at the same + point. If this second match fails, the start offset is advanced by + one, and the normal match is retried. This imitates the way Perl han- dles such cases when using the /g modifier or the split() function. There are yet more modifiers for controlling the way pcretest operates. - The /+ modifier requests that as well as outputting the substring that - matched the entire pattern, pcretest should in addition output the - remainder of the subject string. This is useful for tests where the + The /+ modifier requests that as well as outputting the substring that + matched the entire pattern, pcretest should in addition output the + remainder of the subject string. This is useful for tests where the subject contains multiple copies of the same substring. - The /L modifier must be followed directly by the name of a locale, for + The /L modifier must be followed directly by the name of a locale, for example, /pattern/Lfr_FR For this reason, it must be the last modifier. The given locale is set, - pcre_maketables() is called to build a set of character tables for the - locale, and this is then passed to pcre_compile() when compiling the - regular expression. Without an /L modifier, NULL is passed as the - tables pointer; that is, /L applies only to the expression on which it + pcre_maketables() is called to build a set of character tables for the + locale, and this is then passed to pcre_compile() when compiling the + regular expression. Without an /L modifier, NULL is passed as the + tables pointer; that is, /L applies only to the expression on which it appears. - The /I modifier requests that pcretest output information about the - compiled pattern (whether it is anchored, has a fixed first character, - and so on). It does this by calling pcre_fullinfo() after compiling a - pattern. If the pattern is studied, the results of that are also out- + The /I modifier requests that pcretest output information about the + compiled pattern (whether it is anchored, has a fixed first character, + and so on). It does this by calling pcre_fullinfo() after compiling a + pattern. If the pattern is studied, the results of that are also out- put. The /D modifier is a PCRE debugging feature, which also assumes /I. It - causes the internal form of compiled regular expressions to be output + causes the internal form of compiled regular expressions to be output after compilation. If the pattern was studied, the information returned is also output. The /F modifier causes pcretest to flip the byte order of the fields in - the compiled pattern that contain 2-byte and 4-byte numbers. This - facility is for testing the feature in PCRE that allows it to execute + the compiled pattern that contain 2-byte and 4-byte numbers. This + facility is for testing the feature in PCRE that allows it to execute patterns that were compiled on a host with a different endianness. This - feature is not available when the POSIX interface to PCRE is being - used, that is, when the /P pattern modifier is specified. See also the + feature is not available when the POSIX interface to PCRE is being + used, that is, when the /P pattern modifier is specified. See also the section about saving and reloading compiled patterns below. - The /S modifier causes pcre_study() to be called after the expression + The /S modifier causes pcre_study() to be called after the expression has been compiled, and the results used when the expression is matched. - The /M modifier causes the size of memory block used to hold the com- + The /M modifier causes the size of memory block used to hold the com- piled pattern to be output. - The /P modifier causes pcretest to call PCRE via the POSIX wrapper API - rather than its native API. When this is done, all other modifiers - except /i, /m, and /+ are ignored. REG_ICASE is set if /i is present, - and REG_NEWLINE is set if /m is present. The wrapper functions force - PCRE_DOLLAR_ENDONLY always, and PCRE_DOTALL unless REG_NEWLINE is set. + The /P modifier causes pcretest to call PCRE via the POSIX wrapper API + rather than its native API. When this is done, all other modifiers + except /i, /m, and /+ are ignored. REG_ICASE is set if /i is present, + and REG_NEWLINE is set if /m is present. The wrapper functions force + PCRE_DOLLAR_ENDONLY always, and PCRE_DOTALL unless REG_NEWLINE is set. - The /8 modifier causes pcretest to call PCRE with the PCRE_UTF8 option - set. This turns on support for UTF-8 character handling in PCRE, pro- - vided that it was compiled with this support enabled. This modifier + The /8 modifier causes pcretest to call PCRE with the PCRE_UTF8 option + set. This turns on support for UTF-8 character handling in PCRE, pro- + vided that it was compiled with this support enabled. This modifier also causes any non-printing characters in output strings to be printed using the \x{hh...} notation if they are valid UTF-8 sequences. - If the /? modifier is used with /8, it causes pcretest to call - pcre_compile() with the PCRE_NO_UTF8_CHECK option, to suppress the + If the /? modifier is used with /8, it causes pcretest to call + pcre_compile() with the PCRE_NO_UTF8_CHECK option, to suppress the checking of the string for UTF-8 validity. DATA LINES - Before each data line is passed to pcre_exec(), leading and trailing - whitespace is removed, and it is then scanned for \ escapes. Some of - these are pretty esoteric features, intended for checking out some of - the more complicated features of PCRE. If you are just testing "ordi- - nary" regular expressions, you probably don't need any of these. The + Before each data line is passed to pcre_exec(), leading and trailing + whitespace is removed, and it is then scanned for \ escapes. Some of + these are pretty esoteric features, intended for checking out some of + the more complicated features of PCRE. If you are just testing "ordi- + nary" regular expressions, you probably don't need any of these. The following escapes are recognized: \a alarm (= BEL) @@ -242,6 +248,8 @@ DATA LINES reached for the nth time \C*n pass the number n (may be negative) as callout data; this is used as the callout return value + \D use the pcre_dfa_exec() match function + \F only shortest match for pcre_dfa_exec() \Gdd call pcre_get_substring() for substring dd after a successful match (number less than 32) \Gname call pcre_get_named_substring() for substring @@ -254,6 +262,8 @@ DATA LINES \Odd set the size of the output vector passed to pcre_exec() to dd (any number of digits) \P pass the PCRE_PARTIAL option to pcre_exec() + or pcre_dfa_exec() + \R pass the PCRE_DFA_RESTART option to pcre_dfa_exec() \S output details of memory get/free calls during matching \Z pass the PCRE_NOTEOL option to pcre_exec() \? pass the PCRE_NO_UTF8_CHECK option to @@ -261,35 +271,53 @@ DATA LINES \>dd start the match at offset dd (any number of digits); this sets the startoffset argument for pcre_exec() - A backslash followed by anything else just escapes the anything else. - If the very last character is a backslash, it is ignored. This gives a - way of passing an empty line as data, since a real empty line termi- + A backslash followed by anything else just escapes the anything else. + If the very last character is a backslash, it is ignored. This gives a + way of passing an empty line as data, since a real empty line termi- nates the data input. - If \M is present, pcretest calls pcre_exec() several times, with dif- - ferent values in the match_limit field of the pcre_extra data struc- - ture, until it finds the minimum number that is needed for pcre_exec() - to complete. This number is a measure of the amount of recursion and - backtracking that takes place, and checking it out can be instructive. - For most simple matches, the number is quite small, but for patterns - with very large numbers of matching possibilities, it can become large + If \M is present, pcretest calls pcre_exec() several times, with dif- + ferent values in the match_limit field of the pcre_extra data struc- + ture, until it finds the minimum number that is needed for pcre_exec() + to complete. This number is a measure of the amount of recursion and + backtracking that takes place, and checking it out can be instructive. + For most simple matches, the number is quite small, but for patterns + with very large numbers of matching possibilities, it can become large very quickly with increasing length of subject string. - When \O is used, the value specified may be higher or lower than the + When \O is used, the value specified may be higher or lower than the size set by the -O command line option (or defaulted to 45); \O applies only to the call of pcre_exec() for the line in which it appears. - If the /P modifier was present on the pattern, causing the POSIX wrap- - per API to be used, only \B and \Z have any effect, causing REG_NOTBOL + If the /P modifier was present on the pattern, causing the POSIX wrap- + per API to be used, only \B and \Z have any effect, causing REG_NOTBOL and REG_NOTEOL to be passed to regexec() respectively. - The use of \x{hh...} to represent UTF-8 characters is not dependent on - the use of the /8 modifier on the pattern. It is recognized always. - There may be any number of hexadecimal digits inside the braces. The - result is from one to six bytes, encoded according to the UTF-8 rules. + The use of \x{hh...} to represent UTF-8 characters is not dependent on + the use of the /8 modifier on the pattern. It is recognized always. + There may be any number of hexadecimal digits inside the braces. The + result is from one to six bytes, encoded according to the UTF-8 rules. + + +THE ALTERNATIVE MATCHING FUNCTION + + By default, pcretest uses the standard PCRE matching function, + pcre_exec() to match each data line. From release 6.0, PCRE supports an + alternative matching function, pcre_dfa_test(), which operates in a + different way, and has some restrictions. The differences between the + two functions are described in the pcrematching documentation. + + If a data line contains the \D escape sequence, or if the command line + contains the -dfa option, the alternative matching function is called. + This function finds all possible matches at a given point. If, however, + the \F escape sequence is present in the data line, it stops after the + first match is found. This is always the shortest possible match. -OUTPUT FROM PCRETEST +DEFAULT OUTPUT FROM PCRETEST + + This section describes the output when the normal matching function, + pcre_exec(), is being used. When a match succeeds, pcretest outputs the list of captured substrings that pcre_exec() returns, starting with number 0 for the string that @@ -345,25 +373,76 @@ OUTPUT FROM PCRETEST lines can be included in data by means of the \n escape. +OUTPUT FROM THE ALTERNATIVE MATCHING FUNCTION + + When the alternative matching function, pcre_dfa_exec(), is used (by + means of the \D escape sequence or the -dfa command line option), the + output consists of a list of all the matches that start at the first + point in the subject where there is at least one match. For example: + + re> /(tang|tangerine|tan)/ + data> yellow tangerine\D + 0: tangerine + 1: tang + 2: tan + + (Using the normal matching function on this data finds only "tang".) + The longest matching string is always given first (and numbered zero). + + If /gP is present on the pattern, the search for further matches + resumes at the end of the longest match. For example: + + re> /(tang|tangerine|tan)/g + data> yellow tangerine and tangy sultana\D + 0: tangerine + 1: tang + 2: tan + 0: tang + 1: tan + 0: tan + + Since the matching function does not support substring capture, the + escape sequences that are concerned with captured substrings are not + relevant. + + +RESTARTING AFTER A PARTIAL MATCH + + When the alternative matching function has given the PCRE_ERROR_PARTIAL + return, indicating that the subject partially matched the pattern, you + can restart the match with additional subject data by means of the \R + escape sequence. For example: + + re> /^?(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)$/ + data> 23ja\P\D + Partial match: 23ja + data> n05\R\D + 0: n05 + + For further information about partial matching, see the pcrepartial + documentation. + + CALLOUTS If the pattern contains any callout requests, pcretest's callout func- - tion is called during matching. By default, it displays the callout - number, the start and current positions in the text at the callout - time, and the next pattern item to be tested. For example, the output + tion is called during matching. This works with both matching func- + tions. By default, the called function displays the callout number, the + start and current positions in the text at the callout time, and the + next pattern item to be tested. For example, the output --->pqrabcdef 0 ^ ^ \d - indicates that callout number 0 occurred for a match attempt starting - at the fourth character of the subject string, when the pointer was at - the seventh character of the data, and when the next pattern item was - \d. Just one circumflex is output if the start and current positions + indicates that callout number 0 occurred for a match attempt starting + at the fourth character of the subject string, when the pointer was at + the seventh character of the data, and when the next pattern item was + \d. Just one circumflex is output if the start and current positions are the same. Callouts numbered 255 are assumed to be automatic callouts, inserted as - a result of the /C pattern modifier. In this case, instead of showing - the callout number, the offset in the pattern, preceded by a plus, is + a result of the /C pattern modifier. In this case, instead of showing + the callout number, the offset in the pattern, preceded by a plus, is output. For example: re> /\d?[A-E]\*/C @@ -375,76 +454,76 @@ CALLOUTS +10 ^ ^ 0: E* - The callout function in pcretest returns zero (carry on matching) by - default, but you can use an \C item in a data line (as described above) + The callout function in pcretest returns zero (carry on matching) by + default, but you can use a \C item in a data line (as described above) to change this. - Inserting callouts can be helpful when using pcretest to check compli- - cated regular expressions. For further information about callouts, see + Inserting callouts can be helpful when using pcretest to check compli- + cated regular expressions. For further information about callouts, see the pcrecallout documentation. SAVING AND RELOADING COMPILED PATTERNS - The facilities described in this section are not available when the + The facilities described in this section are not available when the POSIX inteface to PCRE is being used, that is, when the /P pattern mod- ifier is specified. When the POSIX interface is not in use, you can cause pcretest to write - a compiled pattern to a file, by following the modifiers with > and a + a compiled pattern to a file, by following the modifiers with > and a file name. For example: /pattern/im >/some/file - See the pcreprecompile documentation for a discussion about saving and + See the pcreprecompile documentation for a discussion about saving and re-using compiled patterns. - The data that is written is binary. The first eight bytes are the - length of the compiled pattern data followed by the length of the - optional study data, each written as four bytes in big-endian order - (most significant byte first). If there is no study data (either the + The data that is written is binary. The first eight bytes are the + length of the compiled pattern data followed by the length of the + optional study data, each written as four bytes in big-endian order + (most significant byte first). If there is no study data (either the pattern was not studied, or studying did not return any data), the sec- - ond length is zero. The lengths are followed by an exact copy of the + ond length is zero. The lengths are followed by an exact copy of the compiled pattern. If there is additional study data, this follows imme- - diately after the compiled pattern. After writing the file, pcretest + diately after the compiled pattern. After writing the file, pcretest expects to read a new pattern. A saved pattern can be reloaded into pcretest by specifing < and a file - name instead of a pattern. The name of the file must not contain a < - character, as otherwise pcretest will interpret the line as a pattern + name instead of a pattern. The name of the file must not contain a < + character, as otherwise pcretest will interpret the line as a pattern delimited by < characters. For example: re> </some/file Compiled regex loaded from /some/file No study data - When the pattern has been loaded, pcretest proceeds to read data lines + When the pattern has been loaded, pcretest proceeds to read data lines in the usual way. - You can copy a file written by pcretest to a different host and reload - it there, even if the new host has opposite endianness to the one on - which the pattern was compiled. For example, you can compile on an i86 + You can copy a file written by pcretest to a different host and reload + it there, even if the new host has opposite endianness to the one on + which the pattern was compiled. For example, you can compile on an i86 machine and run on a SPARC machine. - File names for saving and reloading can be absolute or relative, but - note that the shell facility of expanding a file name that starts with + File names for saving and reloading can be absolute or relative, but + note that the shell facility of expanding a file name that starts with a tilde (~) is not available. - The ability to save and reload files in pcretest is intended for test- - ing and experimentation. It is not intended for production use because - only a single pattern can be written to a file. Furthermore, there is - no facility for supplying custom character tables for use with a - reloaded pattern. If the original pattern was compiled with custom - tables, an attempt to match a subject string using a reloaded pattern - is likely to cause pcretest to crash. Finally, if you attempt to load + The ability to save and reload files in pcretest is intended for test- + ing and experimentation. It is not intended for production use because + only a single pattern can be written to a file. Furthermore, there is + no facility for supplying custom character tables for use with a + reloaded pattern. If the original pattern was compiled with custom + tables, an attempt to match a subject string using a reloaded pattern + is likely to cause pcretest to crash. Finally, if you attempt to load a file that is not in the correct format, the result is undefined. AUTHOR - Philip Hazel <ph10@cam.ac.uk> + Philip Hazel University Computing Service, Cambridge CB2 3QG, England. -Last updated: 10 September 2004 -Copyright (c) 1997-2004 University of Cambridge. +Last updated: 28 February 2005 +Copyright (c) 1997-2005 University of Cambridge. diff --git a/doc/perltest.txt b/doc/perltest.txt index f1d2c15..ca02690 100644 --- a/doc/perltest.txt +++ b/doc/perltest.txt @@ -29,5 +29,5 @@ make use of the special upper case modifiers and escapes that pcretest uses to test some features of PCRE. Some of these files also contains malformed regular expressions, in order to check that PCRE diagnoses them correctly. -Philip Hazel <ph10@cam.ac.uk> +Philip Hazel September 2004 |