Header/footer detection: recognize non-Latin folio digits (Arabic-Indic, Persian, Devanagari, full-width CJK)

[yfedoseev]

Summary

The running-header / folio (page-number) detection is ASCII/English-only: it recognizes only ASCII digits 0-9. For any document paginated in a non-Latin script — Arabic-Indic (٠١٢…), Extended Arabic-Indic / Persian–Urdu (۰۱۲…), Devanagari (०१२…), full-width CJK (０１２…), or CJK numerals (一二三…) — the entire varying-folio mechanism silently no-ops, so page numbers and running heads leak into extracted body text.

This is the standing "make text heuristics work across languages, not just English" lens, and was flagged as an explicit out-of-scope follow-up on #798.

Where

All in src/document.rs:

• normalize_artifact_signature — collapses digit runs to # using c.is_ascii_digit(). This is the root cause. Non-ASCII digits don't collapse, so "صفحه ۱", "صفحه ۲"… stay distinct signatures → variants never reaches 2 → the varying-literal detection path (variants >= 2) never fires. The feature is inert for these scripts.
• is_bare_page_number_text — c.is_ascii_digit() + parse::<u32>(); bare "۳" / "３" / "三" not detected, and parse::<u32>() is ASCII-only.
• looks_like_stable_pagination — is_ascii_digit() plus English-only keywords (volume, vol., article, issue, doi, www.).

Proposed approach

1. A shared Unicode-aware decimal-digit predicate. is_ascii_digit() is too narrow; char::is_numeric() is too broad (matches ½, ①, superscripts). Target Unicode Nd (decimal digit) via a small curated predicate over the blocks that actually appear as folios:

   fn is_folio_digit(c: char) -> bool {
       c.is_ascii_digit()
           || matches!(c as u32,
               0x0660..=0x0669 |  // Arabic-Indic
               0x06F0..=0x06F9 |  // Extended Arabic-Indic (Persian/Urdu)
               0x0966..=0x096F |  // Devanagari
               0xFF10..=0xFF19)   // Full-width
   }

   Use it in normalize_artifact_signature and both predicates.
2. Value extraction without parse::<u32>(). ⚠️ char::to_digit(10) is also ASCII-only (returns None for '٥'), so is_bare_page_number_text's numeric range check needs an offset-map helper (digit_value(c) = c - block_start) rather than parse.
3. CJK ideographic numerals (一二三…十百千) are the hard tail — they're not Nd, and collapsing them risks over-normalizing real headings (第一章 "Chapter One" → 第#章). Suggest excluding them from a first pass; the variants >= 2 gate limits false positives, but ideographic-numeral handling deserves its own treatment.
4. Keyword universality (looks_like_stable_pagination) is a softer, separate effort — 頁/页/ページ, 巻/卷, صفحة, 第… — but the digit gate above is the priority.

Verification

Cover with unit tests on constructed spans (no third-party fixtures): a synthetic doc paginated in Arabic-Indic and one in full-width digits should have their folios collapse to a shared "… #" signature and get stripped, matching ASCII behavior. An RTL span test is worthwhile since digit runs are LTR-embedded within RTL text — confirm logical-order char walking still collapses correctly.

Scope

Out of scope for #798 (which is the ASCII first-occurrence fix). This is the universality follow-up.