Add pcre2_substitute checks to enforce pattern, subject, offset and options haven't changed (#807)

* Check for pattern/subject/offset/option changes when using PCRE2_SUBSTITUTE_MATCHED.
* Return PCRE2_ERROR_DFA_UFUNC if using PCRE2_SUBSTITUTE_MATCHED after a call to
  pcre2_dfa_match().
* Add new error codes to pcre2_substitute when using PCRE2_SUBSTITUTE_MATCHED.
* Change the behaviour of the matching methods so that the match_data fields are populated
  on all matches with "(rc >= 0 || rc==NO_MATCH || rc==PARTIAL)". We previously ensured that
  every call to a match method guarantees to set the rc field on the match_data.
* Add modifiers to pcre2test to better exercise these pcre2_substitute conditions

---------

Co-authored-by: Isaac Oscar Gariano <[email protected]>

Author: NWilson

doc/html/pcre2_substitute.html

@@ -91,8 +91,9 @@ <h2>
 </p>
 <p>
 If PCRE2_SUBSTITUTE_MATCHED is set, <i>match_data</i> must be non-NULL; its
-contents must be the result of a call to <b>pcre2_match()</b> using the same
-pattern and subject.
+contents must be the result of a call to <b>pcre2_match()</b> (or
+<b>pcre2_jit_match()</b>) using the same pattern, subject pointer, effective
+subject length, start offset, and match options.
 </p>
 <p>
 The function returns the number of substitutions, which may be zero if there

doc/html/pcre2api.html

@@ -3863,11 +3863,15 @@ <h2><a name="SEC38" href="#TOC1">CREATING A NEW STRING WITH SUBSTITUTIONS</a></h
 options can be set in the <i>options</i> argument of <b>pcre2_substitute()</b>.
 One such option is PCRE2_SUBSTITUTE_MATCHED. When this is set, an external
 <i>match_data</i> block must be provided, and it must have already been used for
-an external call to <b>pcre2_match()</b> with the same pattern and subject
-arguments. The data in the <i>match_data</i> block (return code, offset vector)
-is then used for the first substitution instead of calling <b>pcre2_match()</b>
-from within <b>pcre2_substitute()</b>. This allows an application to check for a
-match before choosing to substitute, without having to repeat the match.
+an external call to <b>pcre2_match()</b> (or <b>pcre2_jit_match()</b>) with the
+same pattern, subject, effective subject length, start offset, and match option
+arguments (substitute-specific options can be added to the <i>options</i>
+argument). If any of these parameters is changed, <b>pcre2_substitute()</b>
+returns an error. The data in the <i>match_data</i> block (return code, offset
+vector) is used for the first substitution instead of calling
+<b>pcre2_match()</b> from within <b>pcre2_substitute()</b>. This allows an
+application to check for a match before choosing to substitute, without having
+to repeat the match.
 </p>
 <p>
 The contents of the externally supplied match data block are not changed when
@@ -3883,6 +3887,18 @@ <h2><a name="SEC38" href="#TOC1">CREATING A NEW STRING WITH SUBSTITUTIONS</a></h
 UTF setting and the number of capturing parentheses in the pattern.
 </p>
 <p>
+When using PCRE2_SUBSTITUTE_MATCHED, you should not modify the subject string
+in between the prior call to <b>pcre2_match()</b> and <b>pcre2_substitute()</b>,
+as the substitution assumes that the passed-in ovector is compatible with the
+subject string. Although PCRE2 does verify that the subject is a pointer to the
+same buffer, it cannot in general verify whether the contents of the buffer have
+changed. For example, if the subject buffer is mutated from one valid UTF-8
+string to another valid string, of the same length in code units, the ovector
+offsets are no longer guaranteed to point to the start of a character. Beware
+that with PCRE2_SUBSTITUTE_MATCHED in UTF mode, the subject string is not
+re-scanned for UTF validity when <b>pcre2_substitute()</b> first uses it.
+</p>
+<p>
 The default action of <b>pcre2_substitute()</b> is to return a copy of the
 subject string with matched substrings replaced. However, if
 PCRE2_SUBSTITUTE_REPLACEMENT_ONLY is set, only the replacement substrings are

doc/html/pcre2test.html

@@ -1194,7 +1194,8 @@ <h3>
       heapframes_size             show match data heapframes size
       jitstack=&#60;n&#62;                set size of JIT stack
       mark                        show mark values
-      replace=&#60;string&#62;            specify a replacement string
+      null_substitute_match_data  substitute with NULL match data
+      replace=&#60;str&#62;               specify a replacement string
       startchar                   show starting character when relevant
       substitute_callout          use substitution callouts
       substitute_case_callout     use substitution case callouts
@@ -1369,11 +1370,12 @@ <h3>
       null_context               match with a NULL context
       null_replacement           substitute with NULL replacement
       null_subject               match with NULL subject
+      null_substitute_match_data substitute with NULL match data
       offset=&#60;n&#62;                 set starting offset
       offset_limit=&#60;n&#62;           set offset limit
       ovector=&#60;n&#62;                set size of output vector
       recursion_limit=&#60;n&#62;        obsolete synonym for depth_limit
-      replace=&#60;string&#62;           specify a replacement string
+      replace=&#60;str&#62;              specify a replacement string
       startchar                  show startchar when relevant
       startoffset=&#60;n&#62;            same as offset=&#60;n&#62;
       substitute_callout         use substitution callouts
@@ -1385,6 +1387,7 @@ <h3>
       substitute_replacement_only use PCRE2_SUBSTITUTE_REPLACEMENT_ONLY
       substitute_skip=&#60;n&#62;        skip substitution number n
       substitute_stop=&#60;n&#62;        skip substitution number n and greater
+      substitute_subject=&#60;str&#62;   specify a different subject for substitution
       substitute_unknown_unset   use PCRE2_SUBSTITUTE_UNKNOWN_UNSET
       substitute_unset_empty     use PCRE2_SUBSTITUTE_UNSET_EMPTY
       zero_terminate             pass the subject as zero-terminated
@@ -1615,6 +1618,13 @@ <h3>
 A replacement string is ignored with POSIX and DFA matching. Specifying partial
 matching provokes an error return ("bad option value") from
 <b>pcre2_substitute()</b>.
+<br>
+<br>
+The <b>substitute_subject</b> modifier may be used to test the use of the PCRE2
+API, in which a client calls <b>pcre2_match()</b> followed by <b>pcre2_substitute()</b>
+with PCRE2_SUBSTITUTE_MATCHED, but the client performs an unexpected and
+unsupported modification of the subject buffer in-place, in between the match
+and substitution.
 </p>
 <h3>
 Testing substitute callouts