Documentation updates.

git-svn-id: svn://vcs.exim.org/pcre/code/trunk@683 2f5784b3-3f2a-0410-8824-cb99058d5e15

7 files changed, 128 insertions, 77 deletions

diff --git a/PrepareRelease b/PrepareRelease
index 6b2841b..784f0d4 100755
--- a/PrepareRelease
+++ b/PrepareRelease
@@ -202,6 +202,8 @@ files="\
   pcre_get.c \
   pcre_globals.c \
   pcre_info.c \
+  pcre_jit_compile.c \
+  pcre_jit_test.c \  
   pcre_maketables.c \
   pcre_newline.c \
   pcre_ord2utf8.c \
diff --git a/doc/pcre_assign_jit_stack.3 b/doc/pcre_assign_jit_stack.3
index 55a348a..2045a7f 100644
--- a/doc/pcre_assign_jit_stack.3
+++ b/doc/pcre_assign_jit_stack.3
@@ -36,7 +36,11 @@ stack, the result of calling \fBpcre_jit_stack_alloc()\fP.
 .P
 You may safely assign the same JIT stack to multiple patterns, as long as they 
 are all matched in the same thread. In a multithread application, each thread 
-must use its own JIT stack.
+must use its own JIT stack. For more details, see the
+.\" HREF
+\fBpcrejit\fP
+.\"
+page.
 .P
 There is a complete description of the PCRE native API in the
 .\" HREF
diff --git a/doc/pcre_jit_stack_alloc.3 b/doc/pcre_jit_stack_alloc.3
index 656f546..b2b1ac8 100644
--- a/doc/pcre_jit_stack_alloc.3
+++ b/doc/pcre_jit_stack_alloc.3
@@ -18,7 +18,13 @@ This function is used to create a stack for use by the code compiled by the JIT
 optimization of \fBpcre_study()\fP. The arguments are a starting size for the
 stack, and a maximum size to which it is allowed to grow. The result can be
 passed to the JIT runtime code by \fBpcre_assign_jit_stack()\fP, or that
-function can set up a callback for obtaining a stack.
+function can set up a callback for obtaining a stack. A maximum stack size of
+512K to 1M should be more than enough for any pattern. For more details, see 
+the
+.\" HREF
+\fBpcrejit\fP
+.\"
+page.
 .P
 There is a complete description of the PCRE native API in the
 .\" HREF
diff --git a/doc/pcre_jit_stack_free.3 b/doc/pcre_jit_stack_free.3
index f3e6fb2..f03c86b 100644
--- a/doc/pcre_jit_stack_free.3
+++ b/doc/pcre_jit_stack_free.3
@@ -12,8 +12,13 @@ PCRE - Perl-compatible regular expressions
 .SH DESCRIPTION
 .rs
 .sp
-This function is used to free a JIT stack that was created by 
-\fBpcre_jit_stack_alloc()\fP when it is no longer needed.
+This function is used to free a JIT stack that was created by
+\fBpcre_jit_stack_alloc()\fP when it is no longer needed. For more details, see
+the
+.\" HREF
+\fBpcrejit\fP
+.\"
+page.
 .P
 There is a complete description of the PCRE native API in the
 .\" HREF
diff --git a/doc/pcreapi.3 b/doc/pcreapi.3
index eac7190..e368e8c 100644
--- a/doc/pcreapi.3
+++ b/doc/pcreapi.3
@@ -1823,11 +1823,11 @@ return from the function is 2, because the highest used capturing subpattern
 number is 1, and the offsets for for the second and third capturing subpatterns
 (assuming the vector is large enough, of course) are set to -1.
 .P
-\fBNote\fP: Elements of \fIovector\fP that do not correspond to capturing
-parentheses in the pattern are never changed. That is, if a pattern contains
-\fIn\fP capturing parentheses, no more than \fIovector[0]\fP to
-\fIovector[2n+1]\fP are set by \fBpcre_exec()\fP. The other elements retain
-whatever values they previously had.
+\fBNote\fP: Elements in the first two-thids of \fIovector\fP that do not
+correspond to capturing parentheses in the pattern are never changed. That is,
+if a pattern contains \fIn\fP capturing parentheses, no more than
+\fIovector[0]\fP to \fIovector[2n+1]\fP are set by \fBpcre_exec()\fP. The other
+elements (in the first two-thirds) retain whatever values they previously had.
 .P
 Some convenience functions are provided for extracting the captured substrings
 as separate strings. These are described below.
@@ -2428,7 +2428,8 @@ returns data, even though the meaning of the strings is different.)
 The strings are returned in reverse order of length; that is, the longest
 matching string is given first. If there were too many matches to fit into
 \fIovector\fP, the yield of the function is zero, and the vector is filled with
-the longest matches.
+the longest matches. Unlike \fBpcre_exec()\fP, \fBpcre_dfa_exec()\fP can use 
+the entire \fIovector\fP for returning matched strings.
 .
 .
 .SS "Error returns from \fBpcre_dfa_exec()\fP"
@@ -2497,6 +2498,6 @@ Cambridge CB2 3QH, England.
 .rs
 .sp
 .nf
-Last updated: 27 August 2011
+Last updated: 29 August 2011
 Copyright (c) 1997-2011 University of Cambridge.
 .fi
diff --git a/doc/pcrejit.3 b/doc/pcrejit.3
index da14ca9..78d0513 100644
--- a/doc/pcrejit.3
+++ b/doc/pcrejit.3
@@ -5,15 +5,15 @@ PCRE - Perl-compatible regular expressions
 .rs
 .sp
 Just-in-time compiling is a heavyweight optimization that can greatly speed up
-pattern matching. However, it comes at the cost of extra processing before the 
-match is performed. Therefore, it is of most benefit when the same pattern is 
-going to be matched many times. This does not necessarily mean many calls of 
+pattern matching. However, it comes at the cost of extra processing before the
+match is performed. Therefore, it is of most benefit when the same pattern is
+going to be matched many times. This does not necessarily mean many calls of
 \fPpcre_exec()\fP; if the pattern is not anchored, matching attempts may take
 place many times at various positions in the subject, even for a single call to
-\fBpcre_exec()\fP. If the subject string is very long, it may still pay to use 
+\fBpcre_exec()\fP. If the subject string is very long, it may still pay to use
 JIT for one-off matches.
 .P
-JIT support applies only to the traditional matching function, 
+JIT support applies only to the traditional matching function,
 \fBpcre_exec()\fP. It does not apply when \fBpcre_dfa_exec()\fP is being used.
 The code for this support was written by Zoltan Herczeg.
 .
@@ -26,14 +26,14 @@ JIT support is an optional feature of PCRE. The "configure" option --enable-jit
 JIT. The support is limited to the following hardware platforms:
 .sp
   ARM v5, v7, and Thumb2
+  Intel x86 32-bit and 64-bit
   MIPS 32-bit
   Power PC 32-bit and 64-bit
-  Intel x86 32-bit and 64-bit
-.sp      
+.sp
 If --enable-jit is set on an unsupported platform, compilation fails.
 .P
-A program can tell if JIT support is available by calling \fBpcre_config()\fP 
-with the PCRE_CONFIG_JIT option. The result is 1 when JIT is available, and 0 
+A program can tell if JIT support is available by calling \fBpcre_config()\fP
+with the PCRE_CONFIG_JIT option. The result is 1 when JIT is available, and 0
 otherwise. However, a simple program does not need to check this in order to
 use JIT. The API is implemented in a way that falls back to the ordinary PCRE
 code if JIT is not available.
@@ -47,12 +47,12 @@ You have to do two things to make use of the JIT support in the simplest way:
   (1) Call \fBpcre_study()\fP with the PCRE_STUDY_JIT_COMPILE option for
       each compiled pattern, and pass the resulting \fBpcre_extra\fP block to
       \fBpcre_exec()\fP.
-      
+
   (2) Use \fBpcre_free_study()\fP to free the \fBpcre_extra\fP block when it is
-      no longer needed instead of just freeing it yourself. This ensures that
-      any JIT data is also freed.
+      no longer needed instead of just freeing it yourself. This 
+      ensures that any JIT data is also freed.
 .sp
-In some circumstances you may need to call additional functions. These are 
+In some circumstances you may need to call additional functions. These are
 described in the section entitled
 .\" HTML <a href="#stackcontrol">
 .\" </a>
@@ -60,16 +60,16 @@ described in the section entitled
 .\"
 below.
 .P
-If JIT support is not available, PCRE_STUDY_JIT_COMPILE is ignored, and no JIT 
-data is set up. Otherwise, the compiled pattern is passed to the JIT compiler, 
-which turns it into machine code that executes much faster than the normal 
-interpretive code. When \fBpcre_exec()\fP is passed a \fBpcre_extra\fP block 
-containing a pointer to JIT code, it obeys that instead of the normal code. The 
-result is identical, but the code runs much faster. 
+If JIT support is not available, PCRE_STUDY_JIT_COMPILE is ignored, and no JIT
+data is set up. Otherwise, the compiled pattern is passed to the JIT compiler,
+which turns it into machine code that executes much faster than the normal
+interpretive code. When \fBpcre_exec()\fP is passed a \fBpcre_extra\fP block
+containing a pointer to JIT code, it obeys that instead of the normal code. The
+result is identical, but the code runs much faster.
 .P
 There are some \fBpcre_exec()\fP options that are not supported for JIT
-execution. There are also some pattern items that JIT cannot handle. Details 
-are given below. In both cases, execution automatically falls back to the 
+execution. There are also some pattern items that JIT cannot handle. Details
+are given below. In both cases, execution automatically falls back to the
 interpretive code.
 .P
 If the JIT compiler finds an unsupported item, no JIT data is generated. You
@@ -84,8 +84,8 @@ JIT compiler was not able to handle the pattern.
 .rs
 .sp
 The only \fBpcre_exec()\fP options that are supported for JIT execution are
-PCRE_NO_UTF8_CHECK, PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, and 
-PCRE_NOTEMPTY_ATSTART. Note in particular that partial matching is not 
+PCRE_NO_UTF8_CHECK, PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, and
+PCRE_NOTEMPTY_ATSTART. Note in particular that partial matching is not
 supported.
 .P
 The unsupported pattern items are:
@@ -93,7 +93,7 @@ The unsupported pattern items are:
   \eC            match a single byte, even in UTF-8 mode
   (?Cn)          callouts
   (?(<name>)...  conditional test on setting of a named subpattern
-  (?(R)...       conditional test on whole pattern recursion 
+  (?(R)...       conditional test on whole pattern recursion
   (?(Rn)...      conditional test on recursion, by number
   (?(R&name)...  conditional test on recursion, by name
   (*COMMIT)      )
@@ -101,22 +101,24 @@ The unsupported pattern items are:
   (*PRUNE)       ) the backtracking control verbs
   (*SKIP)        )
   (*THEN)        )
-.sp       
+.sp
 Support for some of these may be added in future.
 .
 .
 .SH "RETURN VALUES FROM JIT EXECUTION"
 .rs
 .sp
-When a pattern is matched using JIT execution, the return values are the same 
-as those given by the interpretive \fBpcre_exec()\fP code, with the addition of 
-one new error code: PCRE_ERROR_JIT_STACKLIMIT. This means that the memory used 
+When a pattern is matched using JIT execution, the return values are the same
+as those given by the interpretive \fBpcre_exec()\fP code, with the addition of
+one new error code: PCRE_ERROR_JIT_STACKLIMIT. This means that the memory used
 for the JIT stack was insufficient. See
 .\" HTML <a href="#stackcontrol">
 .\" </a>
 "Controlling the JIT stack"
 .\"
-below for a discussion of JIT stack usage. 
+below for a discussion of JIT stack usage. For compatibility with the
+interpretive \fBpcre_exec()\fP code, no more than two-thirds of the
+\fIovector\fP argument is used for passing back captured substrings.
 .P
 The error code PCRE_ERROR_MATCHLIMIT is returned by the JIT code if searching a
 very large pattern tree goes on for too long, as it is in the same circumstance
@@ -128,39 +130,44 @@ execution.
 .SH "SAVING AND RESTORING COMPILED PATTERNS"
 .rs
 .sp
-The code that is generated by the JIT compiler is architecture-specific, and is 
-also position dependent. For those reasons it cannot be saved and restored like 
+The code that is generated by the JIT compiler is architecture-specific, and is
+also position dependent. For those reasons it cannot be saved and restored like
 the bytecode and other data of a compiled pattern. You should be able run
 \fBpcre_study()\fP on a saved and restored pattern, and thereby recreate the
 JIT data, but because JIT compilation uses significant resources, it is
-probably not worth doing.
+probably not worth doing this.
 .
 .
 .\" HTML <a name="stackcontrol"></a>
 .SH "CONTROLLING THE JIT STACK"
 .rs
 .sp
-When the compiled JIT code runs, it needs a block of memory to use as a stack. 
-By default, it uses 32K on the machine stack. However, some large or 
-complicated patterns need more than this. The error PCRE_ERROR_JIT_STACKLIMIT 
-is given when there is not enough stack. Three functions are provided for 
-setting up alternative blocks of memory for use as JIT stacks.
+When the compiled JIT code runs, it needs a block of memory to use as a stack.
+By default, it uses 32K on the machine stack. However, some large or
+complicated patterns need more than this. The error PCRE_ERROR_JIT_STACKLIMIT
+is given when there is not enough stack. Three functions are provided for
+managing blocks of memory for use as JIT stacks.
 .P
-The \fBpcre_jit_stack_alloc()\fP function creates a JIT stack. Its arguments 
-are a starting size and a maximum size, and it returns an opaque value 
-of type \fBpcre_jit_stack\fP that represents a JIT stack, or NULL if there is 
+The \fBpcre_jit_stack_alloc()\fP function creates a JIT stack. Its arguments
+are a starting size and a maximum size, and it returns an opaque value
+of type \fBpcre_jit_stack\fP that represents a JIT stack, or NULL if there is
 an error. The \fBpcre_jit_stack_free()\fP function can be used to free a stack
-that is no longer needed.
+that is no longer needed. (For the technically minded: the address space is
+allocated by mmap or VirtualAlloc.)
+.P
+JIT uses far less memory for recursion than the interpretive code, 
+and a maximum stack size of 512K to 1M should be more than enough for any
+pattern.
 .P
-The \fBpcre_assign_jit_stack()\fP function specifies which stack JIT code 
+The \fBpcre_assign_jit_stack()\fP function specifies which stack JIT code
 should use. Its arguments are as follows:
 .sp
   pcre_extra         *extra
   pcre_jit_callback  callback
   void               *data
-.sp    
-The \fIextra\fP argument must be the result of studying a pattern with 
-PCRE_STUDY_JIT_COMPILE. There are three cases for the values of the other two 
+.sp
+The \fIextra\fP argument must be the result of studying a pattern with
+PCRE_STUDY_JIT_COMPILE. There are three cases for the values of the other two
 options:
 .sp
   (1) If \fIcallback\fP is NULL and \fIdata\fP is NULL, an internal 32K block
@@ -170,42 +177,61 @@ options:
       a valid JIT stack, the result of calling \fBpcre_jit_stack_alloc()\fP.
 .sp
   (3) If \fIcallback\fP not NULL, it must point to a function that is called
-      with \fIdata\fP as an argument at the start of matching, in order to 
-      set up a JIT stack. If the result is NULL, the internal 32K stack 
-      is used; otherwise the return value must be a valid JIT stack, 
+      with \fIdata\fP as an argument at the start of matching, in order to
+      set up a JIT stack. If the result is NULL, the internal 32K stack
+      is used; otherwise the return value must be a valid JIT stack,
       the result of calling \fBpcre_jit_stack_alloc()\fP.
 .sp
 You may safely assign the same JIT stack to more than one pattern, as long as
 they are all matched sequentially in the same thread. In a multithread
 application, each thread must use its own JIT stack.
 .P
+Strictly speaking, even more is allowed. You can assign the same stack to any
+number of patterns as long as they are not used for matching by multiple
+threads at the same time. For example, you can assign the same stack to all
+compiled patterns, and use a global mutex in the callback to wait until the
+stack is available for use. However, this is an inefficient solution, and
+not recommended.
+.P
+This is a suggestion for how a typical multithreaded program might operate:
+.sp
+  During thread initalization
+    thread_local_var = pcre_jit_stack_alloc(...)
+  
+  During thread exit
+    pcre_jit_stack_free(thread_local_var)
+  
+  Use a one-line callback function
+    return thread_local_var
+.sp
 All the functions described in this section do nothing if JIT is not available,
-and \fBpcre_assign_jit_stack()\fP does nothing unless the \fBextra\fP argument 
-is non-NULL and points to a \fBpcre_extra\fP block that is the result of a 
+and \fBpcre_assign_jit_stack()\fP does nothing unless the \fBextra\fP argument
+is non-NULL and points to a \fBpcre_extra\fP block that is the result of a
 successful study with PCRE_STUDY_JIT_COMPILE.
 .
 .
 .SH "EXAMPLE CODE"
 .rs
 .sp
-This is a single-threaded example that specifies a JIT stack without using a 
-callback.
+This is a single-threaded example that specifies a JIT stack without using a
+callback. 
 .sp
   int rc;
+  int ovector[30];
   pcre *re;
-  pcre_extra *extra; 
-  pcre_jit_stack *jit_stack; 
-.sp   
+  pcre_extra *extra;
+  pcre_jit_stack *jit_stack;
+.sp
   re = pcre_compile(pattern, 0, &error, &erroffset, NULL);
   /* Check for errors */
   extra = pcre_study(re, PCRE_STUDY_JIT_COMPILE, &error);
-  jit_stack = pcre_jit_stack_alloc(1, 512 * 1024);
+  jit_stack = pcre_jit_stack_alloc(32*1024, 512*1024);
   /* Check for error (NULL) */
   pcre_assign_jit_stack(extra, NULL, jit_stack);
-  rc = pcre_exec(re, extra, subject, length, 0, 0, ovector, ovecsize);
+  rc = pcre_exec(re, extra, subject, length, 0, 0, ovector, 30);
   /* Check results */
   pcre_free(re);
-  pcre_free_study(extra);    
+  pcre_free_study(extra);
 .sp
 .
 .
@@ -229,6 +255,6 @@ Cambridge CB2 3QH, England.
 .rs
 .sp
 .nf
-Last updated: 28 August 2011
+Last updated: 06 September 2011
 Copyright (c) 1997-2011 University of Cambridge.
 .fi
diff --git a/doc/pcreunicode.3 b/doc/pcreunicode.3
index f37ae6e..d2de091 100644
--- a/doc/pcreunicode.3
+++ b/doc/pcreunicode.3
@@ -70,15 +70,19 @@ codes. In this case, it does not diagnose an invalid UTF-8 string.
 If you pass an invalid UTF-8 string when PCRE_NO_UTF8_CHECK is set, what
 happens depends on why the string is invalid. If the string conforms to the
 "old" definition of UTF-8 (RFC 2279), it is processed as a string of characters
-in the range 0 to 0x7FFFFFFF. In other words, apart from the initial validity
-test, PCRE (when in UTF-8 mode) handles strings according to the more liberal
-rules of RFC 2279. However, if the string does not even conform to RFC 2279,
-the result is undefined. Your program may crash.
+in the range 0 to 0x7FFFFFFF by \fBpcre_dfa_exec()\fP and the interpreted
+version of \fBpcre_exec()\fP. In other words, apart from the initial validity
+test, these functions (when in UTF-8 mode) handle strings according to the more
+liberal rules of RFC 2279. However, the just-in-time (JIT) optimization for
+\fBpcre_exec()\fP supports only RFC 3629. If you are using JIT optimization, or
+if the string does not even conform to RFC 2279, the result is undefined. Your
+program may crash.
 .P
 If you want to process strings of values in the full range 0 to 0x7FFFFFFF,
 encoded in a UTF-8-like manner as per the old RFC, you can set
 PCRE_NO_UTF8_CHECK to bypass the more restrictive test. However, in this
-situation, you will have to apply your own validity check.
+situation, you will have to apply your own validity check, and avoid the use of 
+JIT optimization.
 .
 .
 .SS "General comments about UTF-8 mode"
@@ -97,7 +101,10 @@ bytes, for example: \ex{100}{3}.
 .P
 5. The escape sequence \eC can be used to match a single byte in UTF-8 mode,
 but its use can lead to some strange effects. This facility is not available in
-the alternative matching function, \fBpcre_dfa_exec()\fP.
+the alternative matching function, \fBpcre_dfa_exec()\fP, nor is it supported 
+by the JIT optimization of \fBpcre_exec()\fP. If JIT optimization is requested 
+for a pattern that contains \eC, it will not succeed, and so the matching will 
+be carried out by the normal interpretive function.
 .P
 6. The character escapes \eb, \eB, \ed, \eD, \es, \eS, \ew, and \eW correctly
 test characters of any code value, but, by default, the characters that PCRE
@@ -151,6 +158,6 @@ Cambridge CB2 3QH, England.
 .rs
 .sp
 .nf
-Last updated: 24 August 2011
+Last updated: 06 September 2011
 Copyright (c) 1997-2011 University of Cambridge.
 .fi