fix: page # should be completely removed with remove_headers

[ultrasaurus]

Description

"page #" pattern should not be lumped in with first occurrence logic -- all occurrences should be removed.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Performance improvement
• Code refactoring
• Tests
• CI/CD changes

Changes Made

added two tests
- remove_headers_removes_simple_page_number - passes
- remove_headers_removes_word_plus_page_number - passes
- remove_headers_with_text_and_pagenum_keeps_top_line_of_text - passes

I expected the second bug to pass (was just adding some test coverage).

Testing

• I have added tests that prove my fix is effective or that my feature works
• All existing tests pass locally
• I have run cargo test --all-features
• I have run cargo clippy -- -D warnings
• I have run cargo fmt

Python Bindings (if applicable)

• Python bindings updated (if needed)
• Python tests pass
• Python code formatted with ruff format
• Python code linted with ruff check

Documentation

• I have updated the documentation (README, docs/, code comments)
• I have added/updated examples (if applicable)
• I have updated CHANGELOG.md

Checklist

• My code follows the project's coding guidelines (see CONTRIBUTING.md)
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• My changes generate no new warnings
• I have checked my code and corrected any misspellings
• The PR title follows conventional commits format (e.g., feat:, fix:, docs:)

Screenshots (if applicable)

Additional Notes

I didn't change the name of the branch now that it has a fix, let me know if you want me to

[yfedoseev]

Thanks for the focused repro, @ultrasaurus — and nice work building the synthetic PDF helper; that's exactly the right way to pin this down. Your instinct is correct: "page #" should not be governed by the first-occurrence logic, and it lines up with the spec.

Why remove_headers_removes_word_plus_page_number fails (page 0 leaks): in mark_running_artifact_spans the guard

if let Some(&first_seen_on) = signatures.get(&sig) {
    if page_index == first_seen_on { continue; }   // exempts page 0's "page 1"
    ...
}

is a cover-page-title protection — it keeps the first appearance of a running line in case that first line is actually the document title rather than chrome. The problem is it fires unconditionally, so for a varying-literal signature like "page #" it exempts page 0's "page 1" while stripping "page 2".."page 5". Pass 2 then can't catch it either, because it dedups by exact text and "page 1".."page 5" are five distinct one-off keys below the occurrence threshold. So page 0 leaks.

The fix is exactly your decoupling, and it's spec-aligned. ensure_running_artifact_signatures already computes literal_variants per signature but then discards it — the signatures map only carries first_seen_on. Carry an is_varying flag alongside it (or the variants count) and gate the exemption on !is_varying:

• constant-literal signature (variants < 2): first-occurrence exemption stays — it may genuinely be a title that only looks like chrome because later pages repeat it.
• varying-literal signature (variants >= 2): the digits change per page, so it's a folio/running-head by definition — its first occurrence ("page 1", "Chapter 3 42") is chrome exactly like the rest and must not be exempted.

ISO 32000-1 §14.8.2.2 backs this up: it groups "running heads and folios (page numbers)" as a single pagination-artifact class "typically used on every page" — first page included — so the varying page-number component is intrinsic to the artifact and shouldn't defeat detection of the constant part. Go ahead with the focused fix.

Two notes before it's a merge candidate (same as on #795, so no surprises):

1. This branch also carries the #794 behavior change (band-matching Other/PageNumber, parity detection), so it's not test-only — it'll need the full ~400-PDF corpus sweep before merge, same as #795. Might be cleanest to split the pure test-coverage commit from the behavior change so the tests can land independently.
2. Please drop data/1965-nelson.pdf — your in-code synthetic builders are the right pattern; the example can take a path argument instead of a committed third-party PDF.

And note this connects to the line-grouping idea from your #795 comment: once signatures are computed at the line level, "word + page number" headers fall out naturally (the whole line "Chapter 3 #" is the signature, first page included), which addresses both this and the "the" false positive with one mechanism.

[ultrasaurus]

apologies -- I was testing on both branches. PR now rebased on main, which shows the same failing test:

cargo test --test test_remove_headers
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.44s
     Running tests/test_remove_headers.rs (target/debug/deps/test_remove_headers-73cf75edfcc76c33)

running 2 tests
test remove_headers_removes_simple_page_number ... ok
test remove_headers_removes_word_plus_page_number ... FAILED

failures:

---- remove_headers_removes_word_plus_page_number stdout ----

thread 'remove_headers_removes_word_plus_page_number' (14531101) panicked at tests/test_remove_headers.rs:183:9:
page 0: marker "page 1" should have been removed: "page 1\n\n\nBody text placeholder"
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    remove_headers_removes_word_plus_page_number

test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s

error: test failed, to rerun pass `--test test_remove_headers`

[yfedoseev]

Thanks for the careful test coverage here, and for asking before diving into a fix — this is a good catch.

I agree with your instinct: "page #"-style (word + varying number) headers shouldn't be lumped in with the first-occurrence exemption. Here's the reasoning, plus what I found checking how other tools in this space handle the same question.

Why it's wrong today: is_bare_page_number_text (isolated bare digits like "1", "2") has no first-occurrence exemption and removes correctly on every page. But "page 1" / "page 2" falls through to the signature-matching path instead (normalize_artifact_signature collapses it to "page #"), and that path has a blanket "keep the first appearance" rule — meant for a genuinely different case: a repeated running header whose text is constant (e.g. a document title that also appears once as a real heading on the cover page). The exemption is currently applied to every signature that reaches that code path, including ones where the underlying digits actually vary from page to page — but a pattern whose text changes per page can't simultaneously be a legitimate one-off cover-page heading. The fix is to only apply the first-occurrence exemption when the signature has no varying digits (i.e. skip it when literal_variants shows the text actually changed across pages) — everything else about the detection logic stays as-is.

How other tools handle this: I checked PyMuPDF, Docling, unstructured.io, Adobe's own header/footer tool, and the closest academic precedent (Xiaofan Lin's HP Labs paper on header/footer extraction by page-association — which uses essentially the same "normalize digits to a placeholder, detect cross-page recurrence" approach this codebase already uses). None of them exempt the first occurrence of a digit-varying pattern — once something is classified as running/varying page furniture, every matching occurrence is stripped, including page 1's. The exemption is really only a "real repeated title" special case, not a general first-page carve-out.

Go ahead and put together a focused fix along those lines — happy to review it. And since you're already deep in this area: I noticed while looking at #800 that the digit-detection here (is_ascii_digit()) and the citation-keyword matching in looks_like_stable_pagination are ASCII/English-only — non-Latin page numbers (Arabic-Indic, Devanagari, full-width CJK digits) wouldn't be recognized at all. Not something to fold into this PR, just flagging it as a separate follow-up since we're trying to make this feature work well across languages, not just English.

[ultrasaurus]

thank you for the detailed advice

I added another failing test, an example and a fix that causes both tests to pass

[yfedoseev]

Thanks for pushing this through, @ultrasaurus — the core decoupling is exactly right and spec-aligned: carrying is_varying alongside first_seen_on and gating the first-occurrence exemption on !is_varying is the correct shape, and the three tests pin the intended behavior cleanly.

Before merging I ran the corpus sweep I flagged earlier (this branch changes real detection behavior, so it needs it). Methodology, so it's reproducible: to_markdown with strip_running_headers_footers = true — the path that consumes the mark_running_artifact_spans marking your change touches — comparing this branch against its merge-base with main, so the diff isolates only your change. word-level diff on the stripped markdown across the corpus.

The good news: the fix does what it's meant to. Confirmed-correct removals include genuine running footers that previously leaked their first-page occurrence — e.g. a "<date> 2 of 8" footer and a trailing "2 of 3" folio, both now stripped on page 0 as they should be.

The concern — it also deletes real content on page 0. The sweep surfaced two clear false positives, both the "number embedded in substantive text" case:

• IRS_Form_1120_2024.pdf p0: **1a** Consolidated return (attach Form 851) → the 1a is stripped, leaving Consolidated return (attach Form 851). The tell is that its siblings b, 2, 3, 4 on the same form are all kept — only 1a goes, because page 0 was the first occurrence the old exemption happened to protect. 1a is a form line-item label, not a folio.
• section heading p0: **4.** **Discussion** → **Discussion** — the section number is stripped off the heading.

(Two more are borderline: (99) / (2010) form markers on a 1040, and an OCR-garbled number in an 1845 scan.)

Root cause: the first-occurrence exemption was doing double duty. Besides shielding a cover-page title, it was incidentally protecting legitimate first-page numbered content that shares the varying-digit signature shape — form line labels (1a …), numbered section headings (4. Discussion). Un-exempting varying signatures wholesale fixes the "page 1" folio leak but trades it for deleting that content on page 0. This is the "miss-rather-than-drop" line — a false positive that removes real text is worse than a leaked folio.

Suggested direction: keep the decoupling, but narrow the un-exemption so it only fires when the varying signature is a pure folio — bare "page #", "# of #", an isolated "#" — and not when the digit is embedded in substantive text ("1a Consolidated return…", "4. Discussion"). In practice that's a shape check on the normalized signature before flipping is_varying's first-occurrence behavior: a pure-folio signature is short and is essentially all placeholder/pagination-word, whereas "#a Consolidated return (attach Form 851)" carries a substantive literal tail. Gate on that and the two false positives above go away while the "page 1" fix stays.

Coverage note: the sweep completed ~390/419 docs (I OOM'd my own run on a large book by running two dumps at once); I'll finish the remaining ~29 sequentially and post the full counts, but the two cases above are already conclusive on the direction.

Separately — I'm opening a follow-up issue for the RTL/CJK universality gap we discussed (is_ascii_digit() in normalize_artifact_signature / is_bare_page_number_text / looks_like_stable_pagination means non-Latin folios aren't recognized at all). That's out of scope here — flagging so it's tracked, not something to fold into this PR.

[ultrasaurus]

thanks for running the big corpus.. will have to give that some thought, and maybe we need some more tests!

Meanwhile, I'll pull out the example and submit in a separate PR, since it would be helpful for me in my interactive testing (and others who are learning)

[ultrasaurus]

Recent change uses the number itself as a stronger signal. Monotonically increasing numbers are detected as page numbers (not simply varying), and so are no longer exempted from removal on their first occurrence. The code currently sets two as the floor; for longer documents the existing 50%-of-pages recurrence threshold already requires more.

This addresses my motivating case (1965 Nelson paper, with "ACM • 84" as the folio).

I'll wait to submit as a PR till #795 has been reviewed, since they are both in the same area of the code. I welcome feedback in the meantime @yfedoseev

[yfedoseev]

Ran a correctness review plus our native corpus-signature sweep (corpus_sig: text hash + word-length + page-rotation per PDF) against a 419-PDF regression set, isolated at this branch's own fork point so it isn't muddied by unrelated changes that have landed on main since.

Corpus result: 5/419 files changed (all small char-count drops, word-max-length and page rotation identical everywhere — no word-fusion, no crashes). Consistent with the intended scope.

One of those five is worth a second look: pdfjs/test/pdfs/prefilled_f1040.pdf (an IRS Form 1040). The diff there strips the (99) OMB-clearance code next to the title, and one of two duplicate (2010) revision-year stamps in the page-2 footer (the other instance, in "...Page 2", is kept). That's boilerplate/versioning chrome rather than taxpayer data — no dollar amounts, line numbers, or line labels are touched — so it's not a clear regression, but the asymmetric within-page behavior (stripping one of two duplicate fragments and not the other) isn't obviously explained by the fix as written and is worth your own gut-check.

Code-level gap: the fix lifts the first-occurrence exemption for signatures that are both varying and sequential, but for the motivating counter-case (varying_line_item_label_kept_on_first_page, numbered_section_heading_kept_on_first_page — substantive content whose digit varies non-monotonically, e.g. a form line-item label or section heading) it only protects the first occurrence. A later, verbatim repeat of that exact same text still falls through to the old, unchanged else branch and gets unconditionally stripped — and the new tests only assert page 0, so this isn't exercised. Might be intentional/out-of-scope, but flagging since the title reads as a more complete guarantee than the diff delivers.

Process: no Signed-off-by: on the commit (DCO — CONTRIBUTING.md requires it on every commit); no corpus-regression sweep documented in the PR body despite this touching a layout/extraction heuristic (rule 2) — the above now covers that gap for this 419-PDF set. Also currently showing a merge conflict against main, so it'll need a rebase.

[yfedoseev]

Draft, so just a status note from a pass over it today.

• Both test files pass on the branch: 3 in test_remove_headers.rs, 3 in test_running_header_varying_content_kept.rs.
• The branch conflicts with current main (mergeStateStatus: DIRTY), so I couldn't sweep it merged onto main the way I did for fix(794): partial fix for remove_artifacts / remove_footer #795. A rebase would let CI and the corpus sweep give you a real answer.

Also flagging overlap so it doesn't bite later: this, #795 and #967 all touch header/footer/artifact removal, and #795 and #967 additionally collide on a test filename. The sequential-page-number requirement you added here looks like it interacts with the parity work in #795 in particular — landing them in a deliberate order will save you re-doing the heuristics twice.

No changes requested from me at this stage; happy to run the full sweep on it once it's rebased and out of draft.

[yfedoseev]

@ultrasaurus — picking this back up, and starting with a correction I owe you.

Back in July I flagged is_ascii_digit() and told you it was out of scope here, to be a separate follow-up. That follow-up has since landed: normalize_artifact_signature now uses is_folio_digit / folio_digit_value, covering Arabic-Indic, Extended Arabic-Indic, Devanagari and full-width digits.

Which makes the new digit_run_values the one that is now out of step — through no fault of yours, since it was consistent with the codebase when you wrote it. It parses with is_ascii_digit() while the signatures it keys to come from the now-Unicode-aware normaliser. For a folio in any of those scripts the normaliser masks the digits, digit_run_values returns empty, is_sequential stays false, the exemption holds, and page 1 keeps its folio — the fix silently not applying rather than failing loudly. folio_digit_value returns the value rather than a bool, so accumulating v * 10 + d across a run drops both the ASCII assumption and the intermediate String. One Arabic-Indic case in test_digit_run_values would pin it; that test is otherwise thorough — start, end, empty, multiple runs.

That is the only code change I would ask for. The wider tangle — a second header/footer detector carrying its own ASCII-only normaliser, and several inconsistent notions of "digit" across the crate — is ours, and is tracked as #1120.

Two limits worth a line in the doc comment, since both silently keep the exemption: roman-numeral front matter produces no digit runs at all, and section-restarting pagination (1,2,3,1,2,3) fails the strict-increase test. Both are defensible; neither is obvious from the code.

The sequential-folio discriminator itself is the right shape, for what it is worth — "the digit tracks page order" is what separates a folio from a renumbered form label, and it keeps the cover-page case working.

On the rebase: the branch is still DIRTY against current main. One thing not to lose when you do it — main and this branch now carry two different fixtures under numbered_section_heading_kept_on_first_page: yours uses a constant 1a label with non-monotonic attached numbers, main uses changing leading digits. They test different things and both are valid; keeping only one loses coverage. Once it is rebased I will run the sweep merged onto main and post the result.