Pass back the bumpalong value for partial matches.

git-svn-id: svn://vcs.exim.org/pcre/code/trunk@1251 2f5784b3-3f2a-0410-8824-cb99058d5e15

10 files changed, 130 insertions, 65 deletions

diff --git a/ChangeLog b/ChangeLog
index bce4862..14fa79e 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -60,6 +60,10 @@ Version 8.33 xx-xxxx-201x
 
 15. JIT compiler now supports 32 bit Macs thanks to Lawrence Velazquez.
 
+16. Partial matches now set offsets[2] to the "bumpalong" value, that is, the
+    offset of the starting point of the matching process, provided the offsets 
+    vector is large enough.
+
 
 Version 8.32 30-November-2012
 -----------------------------
diff --git a/README b/README
index a2c3d9b..a07b024 100644
--- a/README
+++ b/README
@@ -633,11 +633,15 @@ run forces pcre_study() to be called for all patterns except for a few in some
 tests that are marked "never study" (see the pcretest program for how this is
 done). If JIT support is available, the non-DFA tests are run a third time,
 this time with a forced pcre_study() with the PCRE_STUDY_JIT_COMPILE option.
+This testing can be suppressed by putting "nojit" on the RunTest command line.
 
 The entire set of tests is run once for each of the 8-bit, 16-bit and 32-bit
 libraries that are enabled. If you want to run just one set of tests, call
 RunTest with either the -8, -16 or -32 option.
 
+If valgrind is installed, you can run the tests under it by putting "valgrind" 
+on the RunTest command line.
+
 RunTest uses a file called testtry to hold the main output from pcretest.
 Other files whose names begin with "test" are used as working files in some
 tests. To run pcretest on just one or more specific test files, give their
@@ -932,4 +936,4 @@ pcre_xxx, one with the name pcre16_xx, and a third with the name pcre32_xxx.
 Philip Hazel
 Email local part: ph10
 Email domain: cam.ac.uk
-Last updated: 27 October 2012
+Last updated: 20 February 2013
diff --git a/RunTest b/RunTest
index 546e6e5..aa91cbf 100755
--- a/RunTest
+++ b/RunTest
@@ -14,9 +14,10 @@
 # disabled with /SS.
 #
 # When JIT support is available, all the tests are also run with -s+ to test
-# (again, almost) everything with studying and the JIT option. There are also
-# two tests for JIT-specific features, one to be run when JIT support is
-# available, and one when it is not.
+# (again, almost) everything with studying and the JIT option, unless "nojit"
+# is given on the command line. There are also two tests for JIT-specific 
+# features, one to be run when JIT support is available (unless "nojit" is
+# specified), and one when it is not.
 #
 # Whichever of the 8-, 16- and 32-bit libraries exist are tested. It is also
 # possible to select which to test by the arguments -8, -16 or -32.
@@ -151,11 +152,12 @@ fi
 
 # Default values
 
-valgrind=
-sim=
 arg8=
 arg16=
 arg32=
+nojit=
+sim=
+valgrind=
 
 # This is in case the caller has set aliases (as I do - PH)
 unset cp ls mv rm
@@ -221,9 +223,10 @@ while [ $# -gt 0 ] ; do
    -8) arg8=yes;;
   -16) arg16=yes;;
   -32) arg32=yes;;
+   nojit) nojit=yes;; 
+   sim) shift; sim=$1;;
    valgrind) valgrind="valgrind --tool=memcheck -q --smc-check=all";;
    valgrind-log) valgrind="valgrind --tool=memcheck --num-callers=30 --leak-check=no --error-limit=no --smc-check=all --log-file=report.%p ";;
-   sim) shift; sim=$1;;
     *) echo "Unknown test number '$1'"; exit 1;;
   esac
   shift
@@ -309,7 +312,7 @@ ucp=$?
 jitopt=
 $sim ./pcretest -C jit >/dev/null
 jit=$?
-if [ $jit -ne 0 ] ; then
+if [ $jit -ne 0 -a "$nojit" != "yes" ] ; then
   jitopt=-s+
 fi
 
@@ -700,7 +703,7 @@ fi
 
 if [ $do12 = yes ] ; then
   echo $title12
-  if [ $jit -eq 0 ] ; then
+  if [ $jit -eq 0 -o "$nojit" = "yes" ] ; then
     echo "  Skipped because JIT is not available or not usable"
   else
     $sim $valgrind ./pcretest -q $bmode $testdata/testinput12 testtry
diff --git a/doc/pcrepartial.3 b/doc/pcrepartial.3
index d5cd74e..a860825 100644
--- a/doc/pcrepartial.3
+++ b/doc/pcrepartial.3
@@ -1,4 +1,4 @@
-.TH PCREPARTIAL 3 "24 June 2012" "PCRE 8.31"
+.TH PCREPARTIAL 3 "20 February 2013" "PCRE 8.33"
 .SH NAME
 PCRE - Perl-compatible regular expressions
 .SH "PARTIAL MATCHING IN PCRE"
@@ -56,31 +56,34 @@ strings. This optimization is also disabled for partial matching.
 .rs
 .sp
 A partial match occurs during a call to \fBpcre_exec()\fP or
-\fBpcre[16|32]_exec()\fP when the end of the subject string is reached successfully,
-but matching cannot continue because more characters are needed. However, at
-least one character in the subject must have been inspected. This character
-need not form part of the final matched string; lookbehind assertions and the
-\eK escape sequence provide ways of inspecting characters before the start of a
-matched substring. The requirement for inspecting at least one character exists
-because an empty string can always be matched; without such a restriction there
-would always be a partial match of an empty string at the end of the subject.
+\fBpcre[16|32]_exec()\fP when the end of the subject string is reached
+successfully, but matching cannot continue because more characters are needed.
+However, at least one character in the subject must have been inspected. This
+character need not form part of the final matched string; lookbehind assertions
+and the \eK escape sequence provide ways of inspecting characters before the
+start of a matched substring. The requirement for inspecting at least one
+character exists because an empty string can always be matched; without such a
+restriction there would always be a partial match of an empty string at the end
+of the subject.
 .P
 If there are at least two slots in the offsets vector when a partial match is
 returned, the first slot is set to the offset of the earliest character that
 was inspected. For convenience, the second offset points to the end of the
-subject so that a substring can easily be identified.
+subject so that a substring can easily be identified. If there are at least 
+three slots in the offsets vector, the third slot is set to the offset of the 
+character where matching started.
 .P
-For the majority of patterns, the first offset identifies the start of the
-partially matched string. However, for patterns that contain lookbehind
-assertions, or \eK, or begin with \eb or \eB, earlier characters have been
-inspected while carrying out the match. For example:
+For the majority of patterns, the contents of the first and third slots will be
+the same. However, for patterns that contain lookbehind assertions, or begin
+with \eb or \eB, characters before the one where matching started may have been
+inspected while carrying out the match. For example, consider this pattern:
 .sp
   /(?<=abc)123/
 .sp
 This pattern matches "123", but only if it is preceded by "abc". If the subject
-string is "xyzabc12", the offsets after a partial match are for the substring
-"abc12", because all these characters are needed if another match is tried
-with extra characters added to the subject.
+string is "xyzabc12", the first two offsets after a partial match are for the
+substring "abc12", because all these characters were inspected. However, the
+third offset is set to 6, because that is the offset where matching began. 
 .P
 What happens when a partial match is identified depends on which of the two
 partial matching options are set.
@@ -308,10 +311,9 @@ processing time is needed.
 .P
 \fBNote:\fP If the pattern contains lookbehind assertions, or \eK, or starts
 with \eb or \eB, the string that is returned for a partial match includes
-characters that precede the partially matched string itself, because these must
-be retained when adding on more characters for a subsequent matching attempt.
-However, in some cases you may need to retain even earlier characters, as
-discussed in the next section.
+characters that precede the start of what would be returned for a complete 
+match, because it contains all the characters that were inspected during the 
+partial match.
 .
 .
 .SH "ISSUES WITH MULTI-SEGMENT MATCHING"
@@ -330,12 +332,32 @@ includes the effect of PCRE_NOTEOL.
 offsets that are returned for a partial match. However a lookbehind assertion
 later in the pattern could require even earlier characters to be inspected. You
 can handle this case by using the PCRE_INFO_MAXLOOKBEHIND option of the
-\fBpcre_fullinfo()\fP or \fBpcre[16|32]_fullinfo()\fP functions to obtain the length
-of the largest lookbehind in the pattern. This length is given in characters,
-not bytes. If you always retain at least that many characters before the
-partially matched string, all should be well. (Of course, near the start of the
-subject, fewer characters may be present; in that case all characters should be
-retained.)
+\fBpcre_fullinfo()\fP or \fBpcre[16|32]_fullinfo()\fP functions to obtain the
+length of the longest lookbehind in the pattern. This length is given in
+characters, not bytes. If you always retain at least that many characters
+before the partially matched string, all should be well. (Of course, near the
+start of the subject, fewer characters may be present; in that case all
+characters should be retained.)
+.P
+From release 8.33, there is a more accurate way of deciding which characters to
+retain. Instead of subtracting the length of the longest lookbehind from the
+earliest inspected character (\fIoffsets[0]\fP), the match start position
+(\fIoffsets[2]\fP) should be used, and the next match attempt started at the 
+\fIoffsets[2]\fP character by setting the \fIstartoffset\fP argument of 
+\fBpcre_exec()\fP or \fBpcre_dfa_exec()\fP.
+.P
+For example, if the pattern "(?<=123)abc" is partially
+matched against the string "xx123a", the three offset values returned are 2, 6,
+and 5. This indicates that the matching process that gave a partial match
+started at offset 5, but the characters "123a" were all inspected. The maximum
+lookbehind for that pattern is 3, so taking that away from 5 shows that we need
+only keep "123a", and the next match attempt can be started at offset 3 (that
+is, at "a") when further characters have been added. When the match start is 
+not the earliest inspected character, \fBpcretest\fP shows it explicitly:
+.sp
+    re> "(?<=123)abc"
+  data> xx123a\eP\eP
+  Partial match at offset 5: 123a
 .P
 3. Because a partial match must always contain at least one character, what
 might be considered a partial match of an empty string actually gives a "no
@@ -440,6 +462,6 @@ Cambridge CB2 3QH, England.
 .rs
 .sp
 .nf
-Last updated: 24 June 2012
-Copyright (c) 1997-2012 University of Cambridge.
+Last updated: 20 February 2013
+Copyright (c) 1997-2013 University of Cambridge.
 .fi
diff --git a/pcre_dfa_exec.c b/pcre_dfa_exec.c
index d126a2b..79b7b9f 100644
--- a/pcre_dfa_exec.c
+++ b/pcre_dfa_exec.c
@@ -7,7 +7,7 @@ and semantics are as close as possible to those of the Perl 5 language (but see
 below for why this module is different).
 
                        Written by Philip Hazel
-           Copyright (c) 1997-2012 University of Cambridge
+           Copyright (c) 1997-2013 University of Cambridge
 
 -----------------------------------------------------------------------------
 Redistribution and use in source and binary forms, with or without
@@ -3023,15 +3023,7 @@ for (;;)
           ptr > md->start_used_ptr)            /* Inspected non-empty string */
           )
         )
-      {
-      if (offsetcount >= 2)
-        {
-        offsets[0] = (int)(md->start_used_ptr - start_subject);
-        offsets[1] = (int)(end_subject - start_subject);
-        }
       match_count = PCRE_ERROR_PARTIAL;
-      }
-
     DPRINTF(("%.*sEnd of internal_dfa_exec %d: returning %d\n"
       "%.*s---------------------\n\n", rlevel*2-2, SP, rlevel, match_count,
       rlevel*2-2, SP));
@@ -3545,7 +3537,17 @@ for (;;)
   /* Anything other than "no match" means we are done, always; otherwise, carry
   on only if not anchored. */
 
-  if (rc != PCRE_ERROR_NOMATCH || anchored) return rc;
+  if (rc != PCRE_ERROR_NOMATCH || anchored) 
+    {
+    if (rc == PCRE_ERROR_PARTIAL && offsetcount >= 2)
+      {
+      offsets[0] = (int)(md->start_used_ptr - (PCRE_PUCHAR)subject);
+      offsets[1] = (int)(end_subject - (PCRE_PUCHAR)subject);
+      if (offsetcount > 2) 
+        offsets[2] = (int)(current_subject - (PCRE_PUCHAR)subject);
+      }
+    return rc;
+    } 
 
   /* Advance to the next subject character unless we are at the end of a line
   and firstline is set. */
diff --git a/pcre_exec.c b/pcre_exec.c
index cd5475f..238a7ab 100644
--- a/pcre_exec.c
+++ b/pcre_exec.c
@@ -6,7 +6,7 @@
 and semantics are as close as possible to those of the Perl 5 language.
 
                        Written by Philip Hazel
-           Copyright (c) 1997-2012 University of Cambridge
+           Copyright (c) 1997-2013 University of Cambridge
 
 -----------------------------------------------------------------------------
 Redistribution and use in source and binary forms, with or without
@@ -6286,6 +6286,7 @@ const pcre_uint8 *start_bits = NULL;
 PCRE_PUCHAR start_match = (PCRE_PUCHAR)subject + start_offset;
 PCRE_PUCHAR end_subject;
 PCRE_PUCHAR start_partial = NULL;
+PCRE_PUCHAR match_partial;
 PCRE_PUCHAR req_char_ptr = start_match - 1;
 
 const pcre_study_data *study;
@@ -6837,7 +6838,11 @@ for(;;)
   md->match_function_type = 0;
   md->end_offset_top = 0;
   rc = match(start_match, md->start_code, start_match, 2, md, NULL, 0);
-  if (md->hitend && start_partial == NULL) start_partial = md->start_used_ptr;
+  if (md->hitend && start_partial == NULL) 
+    {
+    start_partial = md->start_used_ptr;
+    match_partial = start_match;
+    }  
 
   switch(rc)
     {
@@ -7045,6 +7050,8 @@ if (start_partial != NULL)
     {
     offsets[0] = (int)(start_partial - (PCRE_PUCHAR)subject);
     offsets[1] = (int)(end_subject - (PCRE_PUCHAR)subject);
+    if (offsetcount > 2) 
+      offsets[2] = (int)(match_partial - (PCRE_PUCHAR)subject);
     }
   rc = PCRE_ERROR_PARTIAL;
   }
diff --git a/pcretest.c b/pcretest.c
index fcf1c25..c3d2896 100644
--- a/pcretest.c
+++ b/pcretest.c
@@ -5263,14 +5263,17 @@ while (!done)
           }
         }
 
-      /* There was a partial match */
+      /* There was a partial match. If the bumpalong point is not the same as 
+      the first inspected character, show the offset explicitly. */
 
       else if (count == PCRE_ERROR_PARTIAL)
         {
-        if (markptr == NULL) fprintf(outfile, "Partial match");
-        else
+        fprintf(outfile, "Partial match");
+        if (use_size_offsets > 2 && use_offsets[0] != use_offsets[2])
+          fprintf(outfile, " at offset %d", use_offsets[2]);  
+        if (markptr != NULL) 
           {
-          fprintf(outfile, "Partial match, mark=");
+          fprintf(outfile, ", mark=");
           PCHARSV(markptr, 0, -1, outfile);
           }
         if (use_size_offsets > 1)
diff --git a/testdata/testinput2 b/testdata/testinput2
index dd57c59..f16444a 100644
--- a/testdata/testinput2
+++ b/testdata/testinput2
@@ -3836,5 +3836,13 @@ settings of the anchored and startline bits. --/
 /(ab)/
     ab\O3
     ab\O2 
+    
+/(?<=123)(*MARK:xx)abc/K
+    xxxx123a\P\P
+    xxxx123a\P
+    
+/123\Kabc/
+    xxxx123a\P\P
+    xxxx123a\P
 
 /-- End of testinput2 --/
diff --git a/testdata/testoutput2 b/testdata/testoutput2
index 709be61..9fc539a 100644
--- a/testdata/testoutput2
+++ b/testdata/testoutput2
@@ -9719,17 +9719,17 @@ Partial match: abc12
     xyzabc123pqr 
  0: 123
     xyzabc12\P
-Partial match: abc12
+Partial match at offset 6: abc12
     xyzabc12\P\P
-Partial match: abc12
+Partial match at offset 6: abc12
 
 /\babc\b/
     +++abc+++
  0: abc
     +++ab\P
-Partial match: +ab
+Partial match at offset 3: +ab
     +++ab\P\P  
-Partial match: +ab
+Partial match at offset 3: +ab
 
 /(?&word)(?&element)(?(DEFINE)(?<element><[^m][^>]>[^<])(?<word>\w*+))/BZ
 ------------------------------------------------------------------
@@ -10840,7 +10840,7 @@ No match
 
 /(?<=abc)def/
     abc\P\P
-Partial match: abc
+Partial match at offset 3: abc
 
 /abc$/
     abc
@@ -12599,5 +12599,17 @@ Matched, but too many substrings
     ab\O2 
 Matched, but too many substrings
  0: ab
+    
+/(?<=123)(*MARK:xx)abc/K
+    xxxx123a\P\P
+Partial match at offset 7, mark=xx: 123a
+    xxxx123a\P
+Partial match at offset 7, mark=xx: 123a
+    
+/123\Kabc/
+    xxxx123a\P\P
+Partial match: 123a
+    xxxx123a\P
+Partial match: 123a
 
 /-- End of testinput2 --/
diff --git a/testdata/testoutput8 b/testdata/testoutput8
index 73e0eae..527ba4d 100644
--- a/testdata/testoutput8
+++ b/testdata/testoutput8
@@ -985,7 +985,7 @@ Partial match: abc
    xyzfo\P 
 No match
    foob\P\>2 
-Partial match: foob
+Partial match at offset 3: foob
    foobar...\R\P\>4 
  0: ar
    xyzfo\P
@@ -7466,17 +7466,17 @@ Error -16 (item unsupported for DFA matching)
     xyzabc123pqr 
  0: 123
     xyzabc12\P
-Partial match: abc12
+Partial match at offset 6: abc12
     xyzabc12\P\P
-Partial match: abc12
+Partial match at offset 6: abc12
 
 /\babc\b/
     +++abc+++
  0: abc
     +++ab\P
-Partial match: +ab
+Partial match at offset 3: +ab
     +++ab\P\P  
-Partial match: +ab
+Partial match at offset 3: +ab
 
 /(?=C)/g+
     ABCDECBA
@@ -7625,7 +7625,7 @@ Error -16 (item unsupported for DFA matching)
 
 /(?<=abc)def/
     abc\P\P
-Partial match: abc
+Partial match at offset 3: abc
 
 /abc$/
     abc