spec-processing

Docling Spec Processing Runbook

Date: 2026-07-25 Scope: canonical source PDFs under spec-processing/source-material/ and finalized cleaned outputs under spec-processing/source-material/parsed-markdown/docling-cleaned/.

Purpose

Provide a reproducible, deterministic workflow for:

1. Parsing selected PDF waves with Docling.
2. Promoting final cleaned markdown artifacts into canonical repo storage.
3. Capturing provenance metadata.
4. Running a lightweight quality gate.

Prerequisites (fish shell)

cd <repo-root>
source .venv/bin/activate.fish

Parsing requires Python 3.10 or newer, Fish, and the exact Docling version pinned in spec-processing/requirements-docling.txt. Provenance generation and strict verification require Node 22 but do not install or run Docling, so they work from a clean checkout.

Quick bootstrap (first-time setup):

cd <repo-root>/spec-processing
./setup-spec-parser-env.fish
source ../.venv/bin/activate.fish

Make targets (spec-specific)

All spec-processing tasks are available through:

make -C spec-processing -f Makefile.spec <target>

Networking-focused tasks can also be invoked via:

make -C spec-processing -f Makefile.networking <networking-target>

Or via spec-processing/Makefile compatibility wrapper:

cd spec-processing && make <target>

Common targets:

• setup
  creates/refreshes the parser venv
• parse-new
  parse new-source-material PDFs
• finalize-new
  verify + move PDFs from new-source-material into source-material
• parse-base
  base corpus rerun
• parse-remaining
  parse remaining wave set
• parse-all
  run base + new + remaining parses
• promote
  copy cleaned outputs into canonical corpus
• networking-ingest
  parse new-source queue and finalize canonical move (parse-new + finalize-new)
• provenance / provenance-check / provenance-test generate the one current provenance snapshot, verify it without writes, or test failure modes
• quality / quality-strict
  run docling cleaned checks

Networking helper targets (via Makefile.networking):

make -C spec-processing -f Makefile.networking networking-parse-only
make -C spec-processing -f Makefile.networking networking-verify
make -C spec-processing -f Makefile.networking networking-promote-only
make -C spec-processing -f Makefile.networking networking-ingest
make -C spec-processing -f Makefile.networking networking-dryrun

Deterministic profile

Shared profile source:

• spec-processing/scripts/docling-profile.fish

Profile flags:

• --from pdf --to md
• --image-export-mode placeholder
• --no-ocr --no-force-ocr
• --pdf-backend dlparse_v4

Parse waves

Base wave:

./spec-processing/parse-pdf.fish

New source-folder wave:

./spec-processing/parse-new-source-material.fish

This parser writes into tmp/docling-new-source-material/core and emits tmp/docling-new-source-material/cleanup-report.txt in the same legacy-compatible format.

Finalize and ingest new materials:

./spec-processing/finalize-new-source-material.fish

This will:

• verify each PDF in spec-processing/new-source-material/ has a matching cleaned output in either:
  • tmp/docling-new-source-material/core/*.cleaned.md, or
  • spec-processing/source-material/parsed-markdown/docling-cleaned/*.cleaned.md
• move each source PDF into spec-processing/source-material/ once verification passes
• keep duplicate names case-insensitively (skip identical file contents; fail on conflict by default)

Options:

./spec-processing/finalize-new-source-material.fish [--dry-run] [--skip-verify] [--skip-promote] [--force] [--copy]

Use --copy for non-destructive queue audits, and --force when you only want a soft-approval mode for conflicts.

Remaining wave:

./spec-processing/parse-pdf-remaining.fish

Outputs are written to (legacy-compatible):

• tmp/docling-rerun/{core,ext}
• tmp/docling-rerun-remaining/{core,ext}
• tmp/docling-new-source-material/core

Promote final cleaned artifacts

./spec-processing/scripts/promote-docling-cleaned.fish

Canonical destination:

• spec-processing/source-material/parsed-markdown/docling-cleaned/

Generate provenance snapshot

./spec-processing/scripts/generate-docling-provenance.sh --write

Generated artifacts:

• docs/waves/SOURCE_CLEAN_PROVENANCE_MANIFEST.md (generated current-snapshot summary)
• docs/waves/provenance/docling-provenance-current.json (complete per-file metadata)

The generator discovers every canonical *.cleaned.md file, requires one unique root-level source PDF with the same stem, and records input/output SHA-256 hashes, sizes, cleaned line counts, the required pinned Docling version, and the exact profile hash. It never appends. A rerun with the same snapshot ID is an exact no-op; if any generated bytes would change under that ID, generation fails and requires an explicit snapshotId and recordedOn update in spec-processing/docling-provenance-policy.json.

Strict verification is write-free and requires only Node 22:

./spec-processing/scripts/generate-docling-provenance.sh --check
node --test spec-processing/scripts/tests/docling-provenance.test.mjs

CI runs both commands plus the strict quality gate. Missing, duplicate, stale, extra, or mismatched records fail with the affected path; there is no supported manual synchronization path for the generated JSON or Markdown manifest.

Run quality gate

Advisory mode:

./spec-processing/scripts/check-docling-cleaned-quality.sh

Strict mode (non-zero exit on warnings):

./spec-processing/scripts/check-docling-cleaned-quality.sh --strict

Notes

• The Wiley DTD-token findings have a reviewed, hash- and count-bound disposition in spec-processing/docling-provenance-policy.json; new or changed findings fail strict checking.
• Canonical planning/compliance docs should reference canonical cleaned corpus and provenance artifacts, not temporary tmp/ paths.
• External/supplemental networking source classification is tracked separately in:
  • spec-processing/external-source-index.json
  • docs/waves/NETWORKING_EXTERNAL_SOURCE_INDEX.md
• Future Kannel/Wireshark external implementation snapshots should land under:
  • spec-processing/new-source-material/external-networking/