diff options
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/Tech.Notes | 9 | ||||
| -rw-r--r-- | doc/pcre.3 | 130 | ||||
| -rw-r--r-- | doc/pcre.html | 162 | ||||
| -rw-r--r-- | doc/pcre.txt | 216 | ||||
| -rw-r--r-- | doc/pcregrep.1 (renamed from doc/pgrep.1) | 14 | ||||
| -rw-r--r-- | doc/pcregrep.html (renamed from doc/pgrep.html) | 16 | ||||
| -rw-r--r-- | doc/pcregrep.txt (renamed from doc/pgrep.txt) | 21 | ||||
| -rw-r--r-- | doc/pcreposix.3 | 10 | ||||
| -rw-r--r-- | doc/pcreposix.html | 11 | ||||
| -rw-r--r-- | doc/pcreposix.txt | 11 | ||||
| -rw-r--r-- | doc/pcretest.txt | 91 | ||||
| -rw-r--r-- | doc/perltest.txt | 10 |
12 files changed, 578 insertions, 123 deletions
diff --git a/doc/Tech.Notes b/doc/Tech.Notes index 03904db..7b96e5b 100644 --- a/doc/Tech.Notes +++ b/doc/Tech.Notes @@ -202,9 +202,10 @@ Forward assertions are just like other subpatterns, but starting with one of the opcodes OP_ASSERT or OP_ASSERT_NOT. Backward assertions use the opcodes OP_ASSERTBACK and OP_ASSERTBACK_NOT, and the first opcode inside the assertion is OP_REVERSE, followed by a two byte count of the number of characters to move -back the pointer in the subject string. A separate count is present in each -alternative of a lookbehind assertion, allowing them to have different fixed -lengths. +back the pointer in the subject string. When operating in UTF-8 mode, the count +is a character count rather than a byte count. A separate count is present in +each alternative of a lookbehind assertion, allowing them to have different +fixed lengths. Once-only subpatterns @@ -239,4 +240,4 @@ the compiled data. Philip Hazel -February 2000 +August 2000 @@ -44,6 +44,12 @@ pcre - Perl-compatible regular expressions. .B int *\fIovector\fR, int \fIstringcount\fR, "const char ***\fIlistptr\fR);" .PP .br +.B void pcre_free_substring(const char *\fIstringptr\fR); +.PP +.br +.B void pcre_free_substring_list(const char **\fIstringptr\fR); +.PP +.br .B const unsigned char *pcre_maketables(void); .PP .br @@ -70,7 +76,9 @@ pcre - Perl-compatible regular expressions. The PCRE library is a set of functions that implement regular expression pattern matching using the same syntax and semantics as Perl 5, with just a few differences (see below). The current implementation corresponds to Perl 5.005, -with some additional features from the Perl development release. +with some additional features from later versions. This includes some +experimental, incomplete support for UTF-8 encoded strings. Details of exactly +what is and what is not supported are given below. PCRE has its own native API, which is described in this document. There is also a set of wrapper functions that correspond to the POSIX regular expression API. @@ -84,12 +92,16 @@ contain the major and minor release numbers for the library. Applications can use these to include support for different releases. The functions \fBpcre_compile()\fR, \fBpcre_study()\fR, and \fBpcre_exec()\fR -are used for compiling and matching regular expressions, while -\fBpcre_copy_substring()\fR, \fBpcre_get_substring()\fR, and +are used for compiling and matching regular expressions. + +The functions \fBpcre_copy_substring()\fR, \fBpcre_get_substring()\fR, and \fBpcre_get_substring_list()\fR are convenience functions for extracting -captured substrings from a matched subject string. The function -\fBpcre_maketables()\fR is used (optionally) to build a set of character tables -in the current locale for passing to \fBpcre_compile()\fR. +captured substrings from a matched subject string; \fBpcre_free_substring()\fR +and \fBpcre_free_substring_list()\fR are also provided, to free the memory used +for extracted strings. + +The function \fBpcre_maketables()\fR is used (optionally) to build a set of +character tables in the current locale for passing to \fBpcre_compile()\fR. The function \fBpcre_fullinfo()\fR is used to find out information about a compiled pattern; \fBpcre_info()\fR is an obsolete version which returns only @@ -223,6 +235,14 @@ This option inverts the "greediness" of the quantifiers so that they are not greedy by default, but become greedy if followed by "?". It is not compatible with Perl. It can also be set by a (?U) option setting within the pattern. + PCRE_UTF8 + +This option causes PCRE to regard both the pattern and the subject as strings +of UTF-8 characters instead of just byte strings. However, it is available only +if PCRE has been built to include UTF-8 support. If not, the use of this option +provokes an error. Support for UTF-8 is new, experimental, and incomplete. +Details of exactly what it entails are given below. + .SH STUDYING A PATTERN When a pattern is going to be used several times, it is worth spending more @@ -558,7 +578,7 @@ extract a single substring, whose number is given as \fIstringnumber\fR. A value of zero extracts the substring that matched the entire pattern, while higher values extract the captured substrings. For \fBpcre_copy_substring()\fR, the string is placed in \fIbuffer\fR, whose length is given by -\fIbuffersize\fR, while for \fBpcre_get_substring()\fR a new block of store is +\fIbuffersize\fR, while for \fBpcre_get_substring()\fR a new block of memory is obtained via \fBpcre_malloc\fR, and its address is returned via \fIstringptr\fR. The yield of the function is the length of the string, not including the terminating zero, or one of @@ -590,6 +610,15 @@ string. This can be distinguished from a genuine zero-length substring by inspecting the appropriate offset in \fIovector\fR, which is negative for unset substrings. +The two convenience functions \fBpcre_free_substring()\fR and +\fBpcre_free_substring_list()\fR can be used to free the memory returned by +a previous call of \fBpcre_get_substring()\fR or +\fBpcre_get_substring_list()\fR, respectively. They do nothing more than call +the function pointed to by \fBpcre_free\fR, which of course could be called +directly from a C program. However, PCRE is used in some situations where it is +linked via a special interface to another programming language which cannot use +\fBpcre_free\fR directly; it is for these cases that the functions are +provided. .SH LIMITATIONS @@ -691,8 +720,14 @@ The syntax and semantics of the regular expressions supported by PCRE are described below. Regular expressions are also described in the Perl documentation and in a number of other books, some of which have copious examples. Jeffrey Friedl's "Mastering Regular Expressions", published by -O'Reilly (ISBN 1-56592-257), covers them in great detail. The description -here is intended as reference documentation. +O'Reilly (ISBN 1-56592-257), covers them in great detail. + +The description here is intended as reference documentation. The basic +operation of PCRE is on strings of bytes. However, there is the beginnings of +some support for UTF-8 character strings. To use this support you must +configure PCRE to include it, and then call \fBpcre_compile()\fR with the +PCRE_UTF8 option. How this affects the pattern matching is described in the +final section of this document. A regular expression is a pattern that is matched against a subject string from left to right. Most characters stand for themselves in a pattern, and match the @@ -1311,7 +1346,7 @@ example, the pattern (a|b\\1)+ -matches any number of "a"s and also "aba", "ababaa" etc. At each iteration of +matches any number of "a"s and also "aba", "ababbaa" etc. At each iteration of the subpattern, the back reference matches the character string corresponding to the previous iteration. In order for this to work, the pattern must be such that the first iteration does not need to match the back reference. This can be @@ -1685,6 +1720,77 @@ with the pattern above. The former gives a failure almost instantly when applied to a whole line of "a" characters, whereas the latter takes an appreciable time with strings longer than about 20 characters. + +.SH UTF-8 SUPPORT +Starting at release 3.3, PCRE has some support for character strings encoded +in the UTF-8 format. This is incomplete, and is regarded as experimental. In +order to use it, you must configure PCRE to include UTF-8 support in the code, +and, in addition, you must call \fBpcre_compile()\fR with the PCRE_UTF8 option +flag. When you do this, both the pattern and any subject strings that are +matched against it are treated as UTF-8 strings instead of just strings of +bytes, but only in the cases that are mentioned below. + +If you compile PCRE with UTF-8 support, but do not use it at run time, the +library will be a bit bigger, but the additional run time overhead is limited +to testing the PCRE_UTF8 flag in several places, so should not be very large. + +PCRE assumes that the strings it is given contain valid UTF-8 codes. It does +not diagnose invalid UTF-8 strings. If you pass invalid UTF-8 strings to PCRE, +the results are undefined. + +Running with PCRE_UTF8 set causes these changes in the way PCRE works: + +1. In a pattern, the escape sequence \\x{...}, where the contents of the braces +is a string of hexadecimal digits, is interpreted as a UTF-8 character whose +code number is the given hexadecimal number, for example: \\x{1234}. This +inserts from one to six literal bytes into the pattern, using the UTF-8 +encoding. If a non-hexadecimal digit appears between the braces, the item is +not recognized. + +2. The original hexadecimal escape sequence, \\xhh, generates a two-byte UTF-8 +character if its value is greater than 127. + +3. Repeat quantifiers are NOT correctly handled if they follow a multibyte +character. For example, \\x{100}* and \\xc3+ do not work. If you want to +repeat such characters, you must enclose them in non-capturing parentheses, +for example (?:\\x{100}), at present. + +4. The dot metacharacter matches one UTF-8 character instead of a single byte. + +5. Unlike literal UTF-8 characters, the dot metacharacter followed by a +repeat quantifier does operate correctly on UTF-8 characters instead of +single bytes. + +4. Although the \\x{...} escape is permitted in a character class, characters +whose values are greater than 255 cannot be included in a class. + +5. A class is matched against a UTF-8 character instead of just a single byte, +but it can match only characters whose values are less than 256. Characters +with greater values always fail to match a class. + +6. Repeated classes work correctly on multiple characters. + +7. Classes containing just a single character whose value is greater than 127 +(but less than 256), for example, [\\x80] or [^\\x{93}], do not work because +these are optimized into single byte matches. In the first case, of course, +the class brackets are just redundant. + +8. Lookbehind assertions move backwards in the subject by a fixed number of +characters instead of a fixed number of bytes. Simple cases have been tested +to work correctly, but there may be hidden gotchas herein. + +9. The character types such as \\d and \\w do not work correctly with UTF-8 +characters. They continue to test a single byte. + +10. Anything not explicitly mentioned here continues to work in bytes rather +than in characters. + +The following UTF-8 features of Perl 5.6 are not implemented: + +1. The escape sequence \\C to match a single byte. + +2. The use of Unicode tables and properties and escapes \\p, \\P, and \\X. + .SH AUTHOR Philip Hazel <ph10@cam.ac.uk> .br @@ -1696,6 +1802,8 @@ Cambridge CB2 3QG, England. .br Phone: +44 1223 334714 -Last updated: 27 January 2000 +Last updated: 28 August 2000, +.br + the 250th anniversary of the death of J.S. Bach. .br Copyright (c) 1997-2000 University of Cambridge. diff --git a/doc/pcre.html b/doc/pcre.html index 7eba9c3..4598698 100644 --- a/doc/pcre.html +++ b/doc/pcre.html @@ -37,7 +37,8 @@ conversion went wrong. <LI><A NAME="TOC27" HREF="#SEC27">COMMENTS</A> <LI><A NAME="TOC28" HREF="#SEC28">RECURSIVE PATTERNS</A> <LI><A NAME="TOC29" HREF="#SEC29">PERFORMANCE</A> -<LI><A NAME="TOC30" HREF="#SEC30">AUTHOR</A> +<LI><A NAME="TOC30" HREF="#SEC30">UTF-8 SUPPORT</A> +<LI><A NAME="TOC31" HREF="#SEC31">AUTHOR</A> </UL> <LI><A NAME="SEC1" HREF="#TOC1">NAME</A> <P> @@ -76,6 +77,12 @@ pcre - Perl-compatible regular expressions. <B>int *<I>ovector</I>, int <I>stringcount</I>, const char ***<I>listptr</I>);</B> </P> <P> +<B>void pcre_free_substring(const char *<I>stringptr</I>);</B> +</P> +<P> +<B>void pcre_free_substring_list(const char **<I>stringptr</I>);</B> +</P> +<P> <B>const unsigned char *pcre_maketables(void);</B> </P> <P> @@ -100,7 +107,9 @@ pcre - Perl-compatible regular expressions. The PCRE library is a set of functions that implement regular expression pattern matching using the same syntax and semantics as Perl 5, with just a few differences (see below). The current implementation corresponds to Perl 5.005, -with some additional features from the Perl development release. +with some additional features from later versions. This includes some +experimental, incomplete support for UTF-8 encoded strings. Details of exactly +what is and what is not supported are given below. </P> <P> PCRE has its own native API, which is described in this document. There is also @@ -117,12 +126,18 @@ use these to include support for different releases. </P> <P> The functions <B>pcre_compile()</B>, <B>pcre_study()</B>, and <B>pcre_exec()</B> -are used for compiling and matching regular expressions, while -<B>pcre_copy_substring()</B>, <B>pcre_get_substring()</B>, and +are used for compiling and matching regular expressions. +</P> +<P> +The functions <B>pcre_copy_substring()</B>, <B>pcre_get_substring()</B>, and <B>pcre_get_substring_list()</B> are convenience functions for extracting -captured substrings from a matched subject string. The function -<B>pcre_maketables()</B> is used (optionally) to build a set of character tables -in the current locale for passing to <B>pcre_compile()</B>. +captured substrings from a matched subject string; <B>pcre_free_substring()</B> +and <B>pcre_free_substring_list()</B> are also provided, to free the memory used +for extracted strings. +</P> +<P> +The function <B>pcre_maketables()</B> is used (optionally) to build a set of +character tables in the current locale for passing to <B>pcre_compile()</B>. </P> <P> The function <B>pcre_fullinfo()</B> is used to find out information about a @@ -297,6 +312,18 @@ This option inverts the "greediness" of the quantifiers so that they are not greedy by default, but become greedy if followed by "?". It is not compatible with Perl. It can also be set by a (?U) option setting within the pattern. </P> +<P> +<PRE> + PCRE_UTF8 +</PRE> +</P> +<P> +This option causes PCRE to regard both the pattern and the subject as strings +of UTF-8 characters instead of just byte strings. However, it is available only +if PCRE has been built to include UTF-8 support. If not, the use of this option +provokes an error. Support for UTF-8 is new, experimental, and incomplete. +Details of exactly what it entails are given below. +</P> <LI><A NAME="SEC6" HREF="#TOC1">STUDYING A PATTERN</A> <P> When a pattern is going to be used several times, it is worth spending more @@ -743,7 +770,7 @@ extract a single substring, whose number is given as <I>stringnumber</I>. A value of zero extracts the substring that matched the entire pattern, while higher values extract the captured substrings. For <B>pcre_copy_substring()</B>, the string is placed in <I>buffer</I>, whose length is given by -<I>buffersize</I>, while for <B>pcre_get_substring()</B> a new block of store is +<I>buffersize</I>, while for <B>pcre_get_substring()</B> a new block of memory is obtained via <B>pcre_malloc</B>, and its address is returned via <I>stringptr</I>. The yield of the function is the length of the string, not including the terminating zero, or one of @@ -789,6 +816,17 @@ string. This can be distinguished from a genuine zero-length substring by inspecting the appropriate offset in <I>ovector</I>, which is negative for unset substrings. </P> +<P> +The two convenience functions <B>pcre_free_substring()</B> and +<B>pcre_free_substring_list()</B> can be used to free the memory returned by +a previous call of <B>pcre_get_substring()</B> or +<B>pcre_get_substring_list()</B>, respectively. They do nothing more than call +the function pointed to by <B>pcre_free</B>, which of course could be called +directly from a C program. However, PCRE is used in some situations where it is +linked via a special interface to another programming language which cannot use +<B>pcre_free</B> directly; it is for these cases that the functions are +provided. +</P> <LI><A NAME="SEC11" HREF="#TOC1">LIMITATIONS</A> <P> There are some size limitations in PCRE but it is hoped that they will never in @@ -908,8 +946,15 @@ The syntax and semantics of the regular expressions supported by PCRE are described below. Regular expressions are also described in the Perl documentation and in a number of other books, some of which have copious examples. Jeffrey Friedl's "Mastering Regular Expressions", published by -O'Reilly (ISBN 1-56592-257), covers them in great detail. The description -here is intended as reference documentation. +O'Reilly (ISBN 1-56592-257), covers them in great detail. +</P> +<P> +The description here is intended as reference documentation. The basic +operation of PCRE is on strings of bytes. However, there is the beginnings of +some support for UTF-8 character strings. To use this support you must +configure PCRE to include it, and then call <B>pcre_compile()</B> with the +PCRE_UTF8 option. How this affects the pattern matching is described in the +final section of this document. </P> <P> A regular expression is a pattern that is matched against a subject string from @@ -1718,7 +1763,7 @@ example, the pattern </PRE> </P> <P> -matches any number of "a"s and also "aba", "ababaa" etc. At each iteration of +matches any number of "a"s and also "aba", "ababbaa" etc. At each iteration of the subpattern, the back reference matches the character string corresponding to the previous iteration. In order for this to work, the pattern must be such that the first iteration does not need to match the back reference. This can be @@ -2240,7 +2285,96 @@ with the pattern above. The former gives a failure almost instantly when applied to a whole line of "a" characters, whereas the latter takes an appreciable time with strings longer than about 20 characters. </P> -<LI><A NAME="SEC30" HREF="#TOC1">AUTHOR</A> +<LI><A NAME="SEC30" HREF="#TOC1">UTF-8 SUPPORT</A> +<P> +Starting at release 3.3, PCRE has some support for character strings encoded +in the UTF-8 format. This is incomplete, and is regarded as experimental. In +order to use it, you must configure PCRE to include UTF-8 support in the code, +and, in addition, you must call <B>pcre_compile()</B> with the PCRE_UTF8 option +flag. When you do this, both the pattern and any subject strings that are +matched against it are treated as UTF-8 strings instead of just strings of +bytes, but only in the cases that are mentioned below. +</P> +<P> +If you compile PCRE with UTF-8 support, but do not use it at run time, the +library will be a bit bigger, but the additional run time overhead is limited +to testing the PCRE_UTF8 flag in several places, so should not be very large. +</P> +<P> +PCRE assumes that the strings it is given contain valid UTF-8 codes. It does +not diagnose invalid UTF-8 strings. If you pass invalid UTF-8 strings to PCRE, +the results are undefined. +</P> +<P> +Running with PCRE_UTF8 set causes these changes in the way PCRE works: +</P> +<P> +1. In a pattern, the escape sequence \x{...}, where the contents of the braces +is a string of hexadecimal digits, is interpreted as a UTF-8 character whose +code number is the given hexadecimal number, for example: \x{1234}. This +inserts from one to six literal bytes into the pattern, using the UTF-8 +encoding. If a non-hexadecimal digit appears between the braces, the item is +not recognized. +</P> +<P> +2. The original hexadecimal escape sequence, \xhh, generates a two-byte UTF-8 +character if its value is greater than 127. +</P> +<P> +3. Repeat quantifiers are NOT correctly handled if they follow a multibyte +character. For example, \x{100}* and \xc3+ do not work. If you want to +repeat such characters, you must enclose them in non-capturing parentheses, +for example (?:\x{100}), at present. +</P> +<P> +4. The dot metacharacter matches one UTF-8 character instead of a single byte. +</P> +<P> +5. Unlike literal UTF-8 characters, the dot metacharacter followed by a +repeat quantifier does operate correctly on UTF-8 characters instead of +single bytes. +</P> +<P> +4. Although the \x{...} escape is permitted in a character class, characters +whose values are greater than 255 cannot be included in a class. +</P> +<P> +5. A class is matched against a UTF-8 character instead of just a single byte, +but it can match only characters whose values are less than 256. Characters +with greater values always fail to match a class. +</P> +<P> +6. Repeated classes work correctly on multiple characters. +</P> +<P> +7. Classes containing just a single character whose value is greater than 127 +(but less than 256), for example, [\x80] or [^\x{93}], do not work because +these are optimized into single byte matches. In the first case, of course, +the class brackets are just redundant. +</P> +<P> +8. Lookbehind assertions move backwards in the subject by a fixed number of +characters instead of a fixed number of bytes. Simple cases have been tested +to work correctly, but there may be hidden gotchas herein. +</P> +<P> +9. The character types such as \d and \w do not work correctly with UTF-8 +characters. They continue to test a single byte. +</P> +<P> +10. Anything not explicitly mentioned here continues to work in bytes rather +than in characters. +</P> +<P> +The following UTF-8 features of Perl 5.6 are not implemented: +</P> +<P> +1. The escape sequence \C to match a single byte. +</P> +<P> +2. The use of Unicode tables and properties and escapes \p, \P, and \X. +</P> +<LI><A NAME="SEC31" HREF="#TOC1">AUTHOR</A> <P> Philip Hazel <ph10@cam.ac.uk> <BR> @@ -2253,6 +2387,8 @@ Cambridge CB2 3QG, England. Phone: +44 1223 334714 </P> <P> -Last updated: 27 January 2000 +Last updated: 28 August 2000, +<BR> + the 250th anniversary of the death of J.S. Bach. <BR> Copyright (c) 1997-2000 University of Cambridge. diff --git a/doc/pcre.txt b/doc/pcre.txt index b8106e4..29cc490 100644 --- a/doc/pcre.txt +++ b/doc/pcre.txt @@ -28,6 +28,10 @@ SYNOPSIS int pcre_get_substring_list(const char *subject, int *ovector, int stringcount, const char ***listptr); + void pcre_free_substring(const char *stringptr); + + void pcre_free_substring_list(const char **stringptr); + const unsigned char *pcre_maketables(void); int pcre_fullinfo(const pcre *code, const pcre_extra *extra, @@ -48,9 +52,12 @@ DESCRIPTION The PCRE library is a set of functions that implement regu- lar expression pattern matching using the same syntax and semantics as Perl 5, with just a few differences (see + below). The current implementation corresponds to Perl - 5.005, with some additional features from the Perl develop- - ment release. + 5.005, with some additional features from later versions. + This includes some experimental, incomplete support for + UTF-8 encoded strings. Details of exactly what is and what + is not supported are given below. PCRE has its own native API, which is described in this document. There is also a set of wrapper functions that @@ -67,13 +74,18 @@ DESCRIPTION releases. The functions pcre_compile(), pcre_study(), and pcre_exec() - are used for compiling and matching regular expressions, - while pcre_copy_substring(), pcre_get_substring(), and - pcre_get_substring_list() are convenience functions for + are used for compiling and matching regular expressions. + + The functions pcre_copy_substring(), pcre_get_substring(), + and pcre_get_substring_list() are convenience functions for extracting captured substrings from a matched subject - string. The function pcre_maketables() is used (optionally) - to build a set of character tables in the current locale for - passing to pcre_compile(). + string; pcre_free_substring() and pcre_free_substring_list() + are also provided, to free the memory used for extracted + strings. + + The function pcre_maketables() is used (optionally) to build + a set of character tables in the current locale for passing + to pcre_compile(). The function pcre_fullinfo() is used to find out information about a compiled pattern; pcre_info() is an obsolete version @@ -92,10 +104,19 @@ DESCRIPTION MULTI-THREADING - The PCRE functions can be used in multi-threading applica- - tions, with the proviso that the memory management functions - pointed to by pcre_malloc and pcre_free are shared by all - threads. + The PCRE functions can be used in multi-threading + + + + + +SunOS 5.8 Last change: 2 + + + + applications, with the proviso that the memory management + functions pointed to by pcre_malloc and pcre_free are shared + by all threads. The compiled form of a regular expression is not altered during matching, so the same compiled pattern can safely be @@ -103,7 +124,6 @@ MULTI-THREADING - COMPILING A PATTERN The function pcre_compile() is called to compile a pattern into an internal form. The pattern is a C string terminated @@ -235,12 +255,23 @@ COMPILING A PATTERN followed by "?". It is not compatible with Perl. It can also be set by a (?U) option setting within the pattern. + PCRE_UTF8 + + This option causes PCRE to regard both the pattern and the + subject as strings of UTF-8 characters instead of just byte + strings. However, it is available only if PCRE has been + built to include UTF-8 support. If not, the use of this + option provokes an error. Support for UTF-8 is new, experi- + mental, and incomplete. Details of exactly what it entails + are given below. + STUDYING A PATTERN When a pattern is going to be used several times, it is worth spending more time analyzing it in order to speed up the time taken for matching. The function pcre_study() takes + a pointer to a compiled pattern as its first argument, and returns a pointer to a pcre_extra block (another void typedef) containing additional information about the pat- @@ -344,9 +375,9 @@ INFORMATION ABOUT A PATTERN PCRE_INFO_BACKREFMAX - Return the number of the highest back reference in the pat- - tern. The fourth argument should point to an int variable. - Zero is returned if there are no back references. + Return the number of the highest back reference in the + pattern. The fourth argument should point to an int vari- + able. Zero is returned if there are no back references. PCRE_INFO_FIRSTCHAR @@ -605,6 +636,15 @@ MATCHING A PATTERN EXTRACTING CAPTURED SUBSTRINGS Captured substrings can be accessed directly by using the + + + + + +SunOS 5.8 Last change: 12 + + + offsets returned by pcre_exec() in ovector. For convenience, the functions pcre_copy_substring(), pcre_get_substring(), and pcre_get_substring_list() are provided for extracting @@ -631,7 +671,7 @@ EXTRACTING CAPTURED SUBSTRINGS the entire pattern, while higher values extract the captured substrings. For pcre_copy_substring(), the string is placed in buffer, whose length is given by buffersize, while for - pcre_get_substring() a new block of store is obtained via + pcre_get_substring() a new block of memory is obtained via pcre_malloc, and its address is returned via stringptr. The yield of the function is the length of the string, not including the terminating zero, or one of @@ -665,6 +705,16 @@ EXTRACTING CAPTURED SUBSTRINGS inspecting the appropriate offset in ovector, which is nega- tive for unset substrings. + The two convenience functions pcre_free_substring() and + pcre_free_substring_list() can be used to free the memory + returned by a previous call of pcre_get_substring() or + pcre_get_substring_list(), respectively. They do nothing + more than call the function pointed to by pcre_free, which + of course could be called directly from a C program. How- + ever, PCRE is used in some situations where it is linked via + a special interface to another programming language which + cannot use pcre_free directly; it is for these cases that + the functions are provided. @@ -733,6 +783,7 @@ DIFFERENCES FROM PERL (?p{code}) constructions. However, there is some experimen- tal support for recursive patterns using the non-Perl item (?R). + 8. There are at the time of writing some oddities in Perl 5.005_02 concerned with the settings of captured strings when part of a pattern is repeated. For example, matching @@ -785,11 +836,17 @@ REGULAR EXPRESSION DETAILS The syntax and semantics of the regular expressions sup- ported by PCRE are described below. Regular expressions are also described in the Perl documentation and in a number of - other books, some of which have copious examples. Jeffrey Friedl's "Mastering Regular Expressions", published by - O'Reilly (ISBN 1-56592-257), covers them in great detail. + O'Reilly (ISBN 1-56592-257), covers them in great detail. + The description here is intended as reference documentation. + The basic operation of PCRE is on strings of bytes. However, + there is the beginnings of some support for UTF-8 character + strings. To use this support you must configure PCRE to + include it, and then call pcre_compile() with the PCRE_UTF8 + option. How this affects the pattern matching is described + in the final section of this document. A regular expression is a pattern that is matched against a subject string from left to right. Most characters stand for @@ -1004,6 +1061,7 @@ CIRCUMFLEX AND DOLLAR Outside a character class, in the default matching mode, the circumflex character is an assertion which is true only if the current matching point is at the start of the subject + string. If the startoffset argument of pcre_exec() is non- zero, circumflex can never match. Inside a character class, circumflex has an entirely different meaning (see below). @@ -1056,6 +1114,7 @@ FULL STOP (PERIOD, DOT) Outside a character class, a dot in the pattern matches any one character in the subject, including a non-printing char- acter, but not (by default) newline. If the PCRE_DOTALL + option is set, dots match newlines as well. The handling of dot is entirely independent of the handling of circumflex and dollar, the only relationship being that they both @@ -1517,18 +1576,19 @@ BACK REFERENCES A back reference that occurs inside the parentheses to which it refers fails when the subpattern is first used, so, for example, (a\1) never matches. However, such references can - be useful inside repeated subpatterns. For example, the - pattern + be useful inside repeated subpatterns. For example, the pat- + tern (a|b\1)+ - matches any number of "a"s and also "aba", "ababaa" etc. At + matches any number of "a"s and also "aba", "ababbaa" etc. At each iteration of the subpattern, the back reference matches - the character string corresponding to the previous itera- - tion. In order for this to work, the pattern must be such - that the first iteration does not need to match the back - reference. This can be done using alternation, as in the - example above, or by a quantifier with a minimum of zero. + the character string corresponding to the previous + iteration. In order for this to work, the pattern must be + such that the first iteration does not need to match the + back reference. This can be done using alternation, as in + the example above, or by a quantifier with a minimum of + zero. @@ -1681,9 +1741,9 @@ ONCE-ONLY SUBPATTERNS This kind of parenthesis "locks up" the part of the pattern it contains once it has matched, and a failure further into - the pattern is prevented from backtracking into it. Back- - tracking past it to previous items, however, works as nor- - mal. + the pattern is prevented from backtracking into it. + Backtracking past it to previous items, however, works as + normal. An alternative description is that a subpattern of this type matches the string of characters that an identical stan- @@ -1941,9 +2001,9 @@ PERFORMANCE repeat can match 0, 1, 2, 3, or 4 times, and for each of those cases other than 0, the + repeats can match different numbers of times.) When the remainder of the pattern is such - that the entire match is going to fail, PCRE has in princi- - ple to try every possible variation, and this can take an - extremely long time. + that the entire match is going to fail, PCRE has in + principle to try every possible variation, and this can take + an extremely long time. An optimization catches some of the more simple cases such as @@ -1966,6 +2026,93 @@ PERFORMANCE +UTF-8 SUPPORT + Starting at release 3.3, PCRE has some support for character + strings encoded in the UTF-8 format. This is incomplete, and + is regarded as experimental. In order to use it, you must + configure PCRE to include UTF-8 support in the code, and, in + addition, you must call pcre_compile() with the PCRE_UTF8 + option flag. When you do this, both the pattern and any sub- + ject strings that are matched against it are treated as + UTF-8 strings instead of just strings of bytes, but only in + the cases that are mentioned below. + + If you compile PCRE with UTF-8 support, but do not use it at + run time, the library will be a bit bigger, but the addi- + tional run time overhead is limited to testing the PCRE_UTF8 + flag in several places, so should not be very large. + + PCRE assumes that the strings it is given contain valid + UTF-8 codes. It does not diagnose invalid UTF-8 strings. If + you pass invalid UTF-8 strings to PCRE, the results are + undefined. + + Running with PCRE_UTF8 set causes these changes in the way + PCRE works: + + 1. In a pattern, the escape sequence \x{...}, where the con- + tents of the braces is a string of hexadecimal digits, is + interpreted as a UTF-8 character whose code number is the + given hexadecimal number, for example: \x{1234}. This + inserts from one to six literal bytes into the pattern, + using the UTF-8 encoding. If a non-hexadecimal digit appears + between the braces, the item is not recognized. + + 2. The original hexadecimal escape sequence, \xhh, generates + a two-byte UTF-8 character if its value is greater than 127. + + 3. Repeat quantifiers are NOT correctly handled if they fol- + low a multibyte character. For example, \x{100}* and \xc3+ + do not work. If you want to repeat such characters, you must + enclose them in non-capturing parentheses, for example + (?:\x{100}), at present. + + 4. The dot metacharacter matches one UTF-8 character instead + of a single byte. + + 5. Unlike literal UTF-8 characters, the dot metacharacter + followed by a repeat quantifier does operate correctly on + UTF-8 characters instead of single bytes. + + 4. Although the \x{...} escape is permitted in a character + class, characters whose values are greater than 255 cannot + be included in a class. + + 5. A class is matched against a UTF-8 character instead of + just a single byte, but it can match only characters whose + values are less than 256. Characters with greater values + always fail to match a class. + + 6. Repeated classes work correctly on multiple characters. + + 7. Classes containing just a single character whose value is + greater than 127 (but less than 256), for example, [\x80] or + [^\x{93}], do not work because these are optimized into sin- + gle byte matches. In the first case, of course, the class + brackets are just redundant. + + 8. Lookbehind assertions move backwards in the subject by a + fixed number of characters instead of a fixed number of + bytes. Simple cases have been tested to work correctly, but + there may be hidden gotchas herein. + + 9. The character types such as \d and \w do not work + correctly with UTF-8 characters. They continue to test a + single byte. + + 10. Anything not explicitly mentioned here continues to work + in bytes rather than in characters. + + The following UTF-8 features of Perl 5.6 are not imple- + mented: + + 1. The escape sequence \C to match a single byte. + + 2. The use of Unicode tables and properties and escapes \p, + \P, and \X. + + + AUTHOR Philip Hazel <ph10@cam.ac.uk> University Computing Service, @@ -1973,5 +2120,6 @@ AUTHOR Cambridge CB2 3QG, England. Phone: +44 1223 334714 - Last updated: 27 January 2000 + Last updated: 28 August 2000, + the 250th anniversary of the death of J.S. Bach. Copyright (c) 1997-2000 University of Cambridge. diff --git a/doc/pgrep.1 b/doc/pcregrep.1 index d9e9b57..41b9051 100644 --- a/doc/pgrep.1 +++ b/doc/pcregrep.1 @@ -1,20 +1,20 @@ -.TH PGREP 1 +.TH PCREGREP 1 .SH NAME -pgrep - a grep with Perl-compatible regular expressions. +pcregrep - a grep with Perl-compatible regular expressions. .SH SYNOPSIS -.B pgrep [-Vchilnsvx] pattern [file] ... +.B pcregrep [-Vchilnsvx] pattern [file] ... .SH DESCRIPTION -\fBpgrep\fR searches files for character patterns, in the same way as other +\fBpcregrep\fR searches files for character patterns, in the same way as other grep commands do, but it uses the PCRE regular expression library to support patterns that are compatible with the regular expressions of Perl 5. See \fBpcre(3)\fR for a full description of syntax and semantics. -If no files are specified, \fBpgrep\fR reads the standard input. By default, +If no files are specified, \fBpcregrep\fR reads the standard input. By default, each line that matches the pattern is copied to the standard output, and if there is more than one file, the file name is printed before each line of -output. However, there are options that can change how \fBpgrep\fR behaves. +output. However, there are options that can change how \fBpcregrep\fR behaves. Lines are limited to BUFSIZ characters. BUFSIZ is defined in \fB<stdio.h>\fR. The newline character is removed from the end of each line before it is matched @@ -73,4 +73,4 @@ for syntax errors or inacessible files (even if matches were found). .SH AUTHOR Philip Hazel <ph10@cam.ac.uk> .br -Copyright (c) 1997-1999 University of Cambridge. +Copyright (c) 1997-2000 University of Cambridge. diff --git a/doc/pgrep.html b/doc/pcregrep.html index 54efed6..77da7c4 100644 --- a/doc/pgrep.html +++ b/doc/pcregrep.html @@ -1,9 +1,9 @@ <HTML> <HEAD> -<TITLE>pgrep specification</TITLE> +<TITLE>pcregrep specification</TITLE> </HEAD> <body bgcolor="#FFFFFF" text="#00005A"> -<H1>pgrep specification</H1> +<H1>pcregrep specification</H1> This HTML document has been generated automatically from the original man page. If there is any nonsense in it, please consult the man page in case the conversion went wrong. @@ -18,24 +18,24 @@ conversion went wrong. </UL> <LI><A NAME="SEC1" HREF="#TOC1">NAME</A> <P> -pgrep - a grep with Perl-compatible regular expressions. +pcregrep - a grep with Perl-compatible regular expressions. </P> <LI><A NAME="SEC2" HREF="#TOC1">SYNOPSIS</A> <P> -<B>pgrep [-Vchilnsvx] pattern [file] ...</B> +<B>pcregrep [-Vchilnsvx] pattern [file] ...</B> </P> <LI><A NAME="SEC3" HREF="#TOC1">DESCRIPTION</A> <P> -<B>pgrep</B> searches files for character patterns, in the same way as other +<B>pcregrep</B> searches files for character patterns, in the same way as other grep commands do, but it uses the PCRE regular expression library to support patterns that are compatible with the regular expressions of Perl 5. See <B>pcre(3)</B> for a full description of syntax and semantics. </P> <P> -If no files are specified, <B>pgrep</B> reads the standard input. By default, +If no files are specified, <B>pcregrep</B> reads the standard input. By default, each line that matches the pattern is copied to the standard output, and if there is more than one file, the file name is printed before each line of -output. However, there are options that can change how <B>pgrep</B> behaves. +output. However, there are options that can change how <B>pcregrep</B> behaves. </P> <P> Lines are limited to BUFSIZ characters. BUFSIZ is defined in <B><stdio.h></B>. @@ -102,4 +102,4 @@ for syntax errors or inacessible files (even if matches were found). <P> Philip Hazel <ph10@cam.ac.uk> <BR> -Copyright (c) 1997-1999 University of Cambridge. +Copyright (c) 1997-2000 University of Cambridge. diff --git a/doc/pgrep.txt b/doc/pcregrep.txt index bcd08c0..3483f9e 100644 --- a/doc/pgrep.txt +++ b/doc/pcregrep.txt @@ -1,25 +1,26 @@ NAME - pgrep - a grep with Perl-compatible regular expressions. + pcregrep - a grep with Perl-compatible regular expressions. SYNOPSIS - pgrep [-Vchilnsvx] pattern [file] ... + pcregrep [-Vchilnsvx] pattern [file] ... DESCRIPTION - pgrep searches files for character patterns, in the same way - as other grep commands do, but it uses the PCRE regular + pcregrep searches files for character patterns, in the same + way as other grep commands do, but it uses the PCRE regular expression library to support patterns that are compatible with the regular expressions of Perl 5. See pcre(3) for a full description of syntax and semantics. - If no files are specified, pgrep reads the standard input. - By default, each line that matches the pattern is copied to - the standard output, and if there is more than one file, the - file name is printed before each line of output. However, - there are options that can change how pgrep behaves. + If no files are specified, pcregrep reads the standard + input. By default, each line that matches the pattern is + copied to the standard output, and if there is more than one + file, the file name is printed before each line of output. + However, there are options that can change how pcregrep + behaves. Lines are limited to BUFSIZ characters. BUFSIZ is defined in <stdio.h>. The newline character is removed from the end of @@ -82,5 +83,5 @@ DIAGNOSTICS AUTHOR Philip Hazel <ph10@cam.ac.uk> - Copyright (c) 1997-1999 University of Cambridge. + Copyright (c) 1997-2000 University of Cambridge. diff --git a/doc/pcreposix.3 b/doc/pcreposix.3 index 1be5d9a..41716ea 100644 --- a/doc/pcreposix.3 +++ b/doc/pcreposix.3 @@ -77,6 +77,14 @@ to the native function. The PCRE_MULTILINE option is set when the expression is passed for compilation to the native function. +In the absence of these flags, no options are passed to the native function. +This means the the regex is compiled with PCRE default semantics. In +particular, the way it handles newline characters in the subject string is the +Perl way, not the POSIX way. Note that setting PCRE_MULTILINE has only +\fIsome\fR of the effects specified for REG_NEWLINE. It does not affect the way +newlines are matched by . (they aren't) or a negative class such as [^a] (they +are). + The yield of \fBregcomp()\fR is zero on success, and non-zero otherwise. The \fIpreg\fR structure is filled in on success, and one member of the structure is publicized: \fIre_nsub\fR contains the number of capturing subpatterns in @@ -138,4 +146,4 @@ Cambridge CB2 3QG, England. .br Phone: +44 1223 334714 -Copyright (c) 1997-1999 University of Cambridge. +Copyright (c) 1997-2000 University of Cambridge. diff --git a/doc/pcreposix.html b/doc/pcreposix.html index 121d90f..9c89478 100644 --- a/doc/pcreposix.html +++ b/doc/pcreposix.html @@ -107,6 +107,15 @@ The PCRE_MULTILINE option is set when the expression is passed for compilation to the native function. </P> <P> +In the absence of these flags, no options are passed to the native function. +This means the the regex is compiled with PCRE default semantics. In +particular, the way it handles newline characters in the subject string is the +Perl way, not the POSIX way. Note that setting PCRE_MULTILINE has only +<I>some</I> of the effects specified for REG_NEWLINE. It does not affect the way +newlines are matched by . (they aren't) or a negative class such as [^a] (they +are). +</P> +<P> The yield of <B>regcomp()</B> is zero on success, and non-zero otherwise. The <I>preg</I> structure is filled in on success, and one member of the structure is publicized: <I>re_nsub</I> contains the number of capturing subpatterns in @@ -179,4 +188,4 @@ Cambridge CB2 3QG, England. Phone: +44 1223 334714 </P> <P> -Copyright (c) 1997-1999 University of Cambridge. +Copyright (c) 1997-2000 University of Cambridge. diff --git a/doc/pcreposix.txt b/doc/pcreposix.txt index 4a7036f..2d76f7c 100644 --- a/doc/pcreposix.txt +++ b/doc/pcreposix.txt @@ -80,6 +80,15 @@ COMPILING A PATTERN The PCRE_MULTILINE option is set when the expression is passed for compilation to the native function. + In the absence of these flags, no options are passed to the + native function. This means the the regex is compiled with + PCRE default semantics. In particular, the way it handles + newline characters in the subject string is the Perl way, + not the POSIX way. Note that setting PCRE_MULTILINE has only + some of the effects specified for REG_NEWLINE. It does not + affect the way newlines are matched by . (they aren't) or a + negative class such as [^a] (they are). + The yield of regcomp() is zero on success, and non-zero oth- erwise. The preg structure is filled in on success, and one member of the structure is publicized: re_nsub contains the @@ -147,4 +156,4 @@ AUTHOR Cambridge CB2 3QG, England. Phone: +44 1223 334714 - Copyright (c) 1997-1999 University of Cambridge. + Copyright (c) 1997-2000 University of Cambridge. diff --git a/doc/pcretest.txt b/doc/pcretest.txt index 0e6783a..add2979 100644 --- a/doc/pcretest.txt +++ b/doc/pcretest.txt @@ -43,6 +43,10 @@ backslash, because is interpreted as the first line of a pattern that starts with "abc/", causing pcretest to read the next line as a continuation of the regular expression. + +PATTERN MODIFIERS +----------------- + The pattern may be followed by i, m, s, or x to set the PCRE_CASELESS, PCRE_MULTILINE, PCRE_DOTALL, or PCRE_EXTENDED options, respectively. For example: @@ -103,37 +107,48 @@ compiled, and the results used when the expression is matched. The /M modifier causes the size of memory block used to hold the compiled pattern to be output. -Finally, the /P modifier causes pcretest to call PCRE via the POSIX wrapper API -rather than its native API. When this is done, all other modifiers except /i, -/m, and /+ are ignored. REG_ICASE is set if /i is present, and REG_NEWLINE is -set if /m is present. The wrapper functions force PCRE_DOLLAR_ENDONLY always, -and PCRE_DOTALL unless REG_NEWLINE is set. +The /P modifier causes pcretest to call PCRE via the POSIX wrapper API rather +than its native API. When this is done, all other modifiers except /i, /m, and +/+ are ignored. REG_ICASE is set if /i is present, and REG_NEWLINE is set if /m +is present. The wrapper functions force PCRE_DOLLAR_ENDONLY always, and +PCRE_DOTALL unless REG_NEWLINE is set. + +The /8 modifier causes pcretest to call PCRE with the PCRE_UTF8 option set. +This turns on the (currently incomplete) support for UTF-8 character handling +in PCRE, provided that it was compiled with this support enabled. This modifier +also causes any non-printing characters in output strings to be printed using +the \x{hh...} notation if they are valid UTF-8 sequences. + + +DATA LINES +---------- Before each data line is passed to pcre_exec(), leading and trailing whitespace is removed, and it is then scanned for \ escapes. The following are recognized: - \a alarm (= BEL) - \b backspace - \e escape - \f formfeed - \n newline - \r carriage return - \t tab - \v vertical tab - \nnn octal character (up to 3 octal digits) - \xhh hexadecimal character (up to 2 hex digits) - - \A pass the PCRE_ANCHORED option to pcre_exec() - \B pass the PCRE_NOTBOL option to pcre_exec() - \Cdd call pcre_copy_substring() for substring dd after a successful match - (any decimal number less than 32) - \Gdd call pcre_get_substring() for substring dd after a successful match - (any decimal number less than 32) - \L call pcre_get_substringlist() after a successful match - \N pass the PCRE_NOTEMPTY option to pcre_exec() - \Odd set the size of the output vector passed to pcre_exec() to dd - (any number of decimal digits) - \Z pass the PCRE_NOTEOL option to pcre_exec() + \a alarm (= BEL) + \b backspace + \e escape + \f formfeed + \n newline + \r carriage return + \t tab + \v vertical tab + \nnn octal character (up to 3 octal digits) + \xhh hexadecimal character (up to 2 hex digits) + \x{hh...} hexadecimal UTF-8 character + + \A pass the PCRE_ANCHORED option to pcre_exec() + \B pass the PCRE_NOTBOL option to pcre_exec() + \Cdd call pcre_copy_substring() for substring dd after a successful + match (any decimal number less than 32) + \Gdd call pcre_get_substring() for substring dd after a successful + match (any decimal number less than 32) + \L call pcre_get_substringlist() after a successful match + \N pass the PCRE_NOTEMPTY option to pcre_exec() + \Odd set the size of the output vector passed to pcre_exec() to dd + (any number of decimal digits) + \Z pass the PCRE_NOTEOL option to pcre_exec() A backslash followed by anything else just escapes the anything else. If the very last character is a backslash, it is ignored. This gives a way of passing @@ -143,6 +158,15 @@ If /P was present on the regex, causing the POSIX wrapper API to be used, only \B, and \Z have any effect, causing REG_NOTBOL and REG_NOTEOL to be passed to regexec() respectively. +The use of \x{hh...} to represent UTF-8 characters is not dependent on the use +of the /8 modifier on the pattern. It is recognized always. There may be any +number of hexadecimal digits inside the braces. The result is from one to six +bytes, encoded according to the UTF-8 rules. + + +OUTPUT FROM PCRETEST +-------------------- + When a match succeeds, pcretest outputs the list of captured substrings that pcre_exec() returns, starting with number 0 for the string that matched the whole pattern. Here is an example of an interactive pcretest run. @@ -158,8 +182,9 @@ whole pattern. Here is an example of an interactive pcretest run. No match If the strings contain any non-printing characters, they are output as \0x -escapes. If the pattern has the /+ modifier, then the output for substring 0 is -followed by the the rest of the subject string, identified by "0+" like this: +escapes, or as \x{...} escapes if the /8 modifier was present on the pattern. +If the pattern has the /+ modifier, then the output for substring 0 is followed +by the the rest of the subject string, identified by "0+" like this: re> /cat/+ data> cataract @@ -190,6 +215,10 @@ Note that while patterns can be continued over several lines (a plain ">" prompt is used for continuations), data lines may not. However newlines can be included in data by means of the \n escape. + +COMMAND LINE OPTIONS +-------------------- + If the -p option is given to pcretest, it is equivalent to adding /P to each regular expression: the POSIX wrapper API is used to call PCRE. None of the following flags has any effect in this case. @@ -208,10 +237,10 @@ a synonym for -m. If the -t option is given, each compile, study, and match is run 20000 times while being timed, and the resulting time per compile or match is output in -milliseconds. Do not set -t with -s, because you will then get the size output +milliseconds. Do not set -t with -m, because you will then get the size output 20000 times and the timing will be distorted. If you want to change the number of repetitions used for timing, edit the definition of LOOPREPEAT at the top of pcretest.c Philip Hazel <ph10@cam.ac.uk> -January 2000 +August 2000 diff --git a/doc/perltest.txt b/doc/perltest.txt index 6c38ebe..5a40401 100644 --- a/doc/perltest.txt +++ b/doc/perltest.txt @@ -13,11 +13,17 @@ for perltest as well as for pcretest, and the special upper case modifiers such as /A that pcretest recognizes are not used in these files. The output should be identical, apart from the initial identifying banner. +For testing UTF-8 features, an alternative form of perltest, called perltest8, +is supplied. This requires Perl 5.6 or higher. It recognizes the special +modifier /8 that pcretest uses to invoke UTF-8 functionality. The testinput5 +file can be fed to perltest8. + The testinput2 and testinput4 files are not suitable for feeding to perltest, since they do make use of the special upper case modifiers and escapes that pcretest uses to test some features of PCRE. The first of these files also contains malformed regular expressions, in order to check that PCRE diagnoses -them correctly. +them correctly. Similarly, testinput6 tests UTF-8 features that do not relate +to Perl. Philip Hazel <ph10@cam.ac.uk> -January 2000 +August 2000 |