Partial DFA match semantics

[sjshuck]

I apologize for the vague title. I opened an issue in 2021, bug 2761 in the old exim issue tracker, and I don't have access to that. I wanted to keep tracking it here as I've had that in my list of things to poke at since then.

I have emails from exim with Philip's and others' contributions to that thread, but not the whole thread (and especially not my own OP).

[NWilson]

I'm afraid I don't have access to that either.

Are you able to reproduce or reconstruct an outline of the problem?

[sjshuck]

I can't reproduce it right now but I think I was trying to match over a chunk boundary using PCRE2_PARTIAL_HARD and PCRE2_PARTIAL_SOFT and neither was fulfilling my expectation "of achieving the same results that would happen if the entire string was available for searching all the time" (quoting the docs). Here's a quote from Philip in the bug thread:

Consider the pattern /1234|3789/. If the first segment is "ABC123" there is a partial match for "1234". There is no partial match for the second alternative, because such a match does not start at the same point in the subject string. Attempting to continue with the string "7890" does not yield a match because only those alternatives that match at one point in the subject are remembered.

In short, even assuming that the user held onto all chunks comprising a prospective match, it's unclear which offset to jump back to to retry the match at an advanced starting position. Non-partial matching takes care of retrying automatically. The partial matching documentation doesn't indicate this extra responsibility of the user or how best to accomplish it.

I'm not sure what the issue is here, whether just docs or an actual technical impossibility given the current implementation. I'd have to sit down and attempt a PoC to know. For now I'll just leave it with these comments in the thread from Philip and Petr Pisar, respectively, which seem to contradict the above quoted docs:

For multi-segment matching it is probably best not to make use of the "restart"
feature at all, but instead to retain the partially matched string and do a
normal match when some more text has been joined on. You can do this with DFA
or non-DFA matching.

An inability to remember an arbitrary long string
is an inherent limitation of any deterministic finite automaton. If a users
needs to remember the strings, he needs to use a different matching algorithm
than DFA.

[PhilipHazel]

Restarting a DFA partial match seemed like a good idea at the time, but I had not thought it through properly, leading to issues like the 1234|3789 problem mentioned above. In retrospect, it should probably not have been implemented. I stand by comment quoted above, advising against the use of the restart facility. Perhaps the documentation could be made stronger and/or the facility could be deprecated.