Regression against v0.3.77, found by scoring .md against pymupdf4llm at the release tip. Narrow — one document, five cells — but it is word fusion, and nothing else in the battery caught it.
What
pdfjs/test/pdfs/TAMReview.pdf, .md:
| v0.3.77 |
v0.3.78 |
| and Capabilities | |
| andCapabilities | |
| Perceived Facilitation | |
| PerceivedFacilitation | |
| System Features | |
| SystemFeatures | |
| Use System | |
| UseSystem | |
The words are still present as substrings, which is why a byte-level or occurrence-count check does not see this: they simply stop being separate tokens.
How it surfaced
The .md surface had never been compared against an external engine. pymupdf4llm aborts the process on some documents (munmap_chunk(): invalid pointer), which killed every previous attempt and left markdown checked only arm-to-arm and against our own .text. Running the engine in a child process per document makes a crash cost one document instead of the run — 16 documents were skipped that way, and 207 scored.
Against a two-of-two gate (a token that BOTH pymupdf4llm and our own .text report is one the page draws, so .md must carry it):
.md vs pymupdf4llm: 207 docs, 120,239 agreed token types
missed by v0.3.77: 1377 missed by HEAD: 996 -> BETTER by 381
This document is the largest of six regressions, at 4 tokens: Beliefs, Capabilities, Facilitation, Features.
Extent
Scanning every markdown-changed document in the 2008-document corpus for a table cell that had an internal space in v0.3.77 and lost it: 1 document, 5 cells. It is not systemic.
Not to be confused with
The same page has BehavioralBeliefs and NormativeBeliefs fused in both arms — that is pre-existing and a different cause. Only the cells listed above changed in this release.
Note also that Use System -> UseSystem shows the fusion is not confined to the cell whose token the panel flagged, so a fix should be judged on the cell text rather than on the four token names.
Regression against v0.3.77, found by scoring
.mdagainst pymupdf4llm at the release tip. Narrow — one document, five cells — but it is word fusion, and nothing else in the battery caught it.What
pdfjs/test/pdfs/TAMReview.pdf,.md:| and Capabilities || andCapabilities || Perceived Facilitation || PerceivedFacilitation || System Features || SystemFeatures || Use System || UseSystem |The words are still present as substrings, which is why a byte-level or occurrence-count check does not see this: they simply stop being separate tokens.
How it surfaced
The
.mdsurface had never been compared against an external engine. pymupdf4llm aborts the process on some documents (munmap_chunk(): invalid pointer), which killed every previous attempt and left markdown checked only arm-to-arm and against our own.text. Running the engine in a child process per document makes a crash cost one document instead of the run — 16 documents were skipped that way, and 207 scored.Against a two-of-two gate (a token that BOTH pymupdf4llm and our own
.textreport is one the page draws, so.mdmust carry it):This document is the largest of six regressions, at 4 tokens:
Beliefs,Capabilities,Facilitation,Features.Extent
Scanning every markdown-changed document in the 2008-document corpus for a table cell that had an internal space in v0.3.77 and lost it: 1 document, 5 cells. It is not systemic.
Not to be confused with
The same page has
BehavioralBeliefsandNormativeBeliefsfused in both arms — that is pre-existing and a different cause. Only the cells listed above changed in this release.Note also that
Use System->UseSystemshows the fusion is not confined to the cell whose token the panel flagged, so a fix should be judged on the cell text rather than on the four token names.