feat: bootstrap ingestion quality gates and isolate scheduler state

[Paulo Lacerda (placerda)]

Ingestion implementation checkpoint - adoption remains incomplete

Implements ingestion scope of Azure/GPT-RAG#681 and accepted ADR-0005.
Coordination: Azure/GPT-RAG#689. Published-doc coordination: Azure/GPT-RAG#688.

Head: 26358cb8f57c982e3c80171bf8dc6b154491388d
Base/current develop: 38a395586ee1d440a8e1ca8233413f8c25b3fdc2 (no drift).
Branch feature/ingestion-quality-gates-681; draft, develop target, unmerged.
The isolated component worktree is clean; no parent files were edited.

Delivered implementation

• jobs/runtime.py, jobs/__init__.py, main.py, api/admin.py: remove direct
  and panel-mediated API-to-main cycles while preserving one scheduler,
  registry, cron map, running state/lock, lifecycle and run-now reservation.
  No factory or ingestion src restructure.
• .quality/*, requirements-quality.txt, pyproject.toml,
  .github/scripts/*, .github/workflows/tests.yml, .github/CODEOWNERS:
  development-only lint/typing, closed records, monotonic scope and exact
  individual debt; static flat/package/late/type-only imports and private
  boundaries; exact handler dispositions including logged/rethrown handlers.
  Existing pytest runner and actual same-workflow aggregate are retained.
  Protected evaluation uses initial/child interpreter isolation, neutral cwd,
  nonexecuting module metadata and restricted plugin/config loading.
• tests/test_quality_policy.py: 168 regression cases, including all five
  independently closed original findings, candidate source/startup shadowing,
  malformed tool diagnostics, duplicate JSON keys and actual aggregate CLI
  missing/skipped/error/stale/wrong-SHA/self-approval cases. The independent
  protected-tool follow-up found no significant new issues.
• tools/aisearch.py, jobs/nl2sql_purger.py, api/retrieval.py: require matching
  positive SDK confirmations, including the actual direct-upload endpoint;
  no positive audit/index count for missing/unrelated/duplicate results.
  Purger run-level late-page/count/delete failures cannot claim completion.
• tools/appconfig.py, dependencies.py, tools/aoai.py,
  tools/doc_intelligence.py, tools/content_understanding.py and adapter
  tests: real pinned provider/Search evidence, selected-provider failure
  propagation, SDK-specific retry/failure handling, safe diagnostics, actual
  HTTP status enforcement and real RSA/PyJWT authorization failure controls.
• Latest integrated runtime batch: jobs/nl2sql_indexer.py,
  chunking/document_chunking.py, three focused chunkers,
  tools/figure_extraction.py, utils/file_utils.py, plus
  test_nl2sql_indexer_failures.py and test_chunking_failure_boundaries.py.
  Preserve cancellation, stable keys, explicit per-document errors, confirmed
  partial images and independent cleanup. Narrow optional SDK/parser failures;
  programming errors cannot silently become successful/missing-state defaults.
• AGENTS.md, README, changelog and docs/python-quality.md: current commands,
  task milestones, compatibility, inactive proposals and recovery limits.

Compatibility

PUT /api/config still returns 200/applied after confirmed durable write and
failed optional local refresh; no false failed entry. Genuine apply failure
500, worker/provider failure propagation and confirmed Search accounting are
observable contract-restoring corrections, not identical failure paths.
Reload/apply are not covered by optional refresh recovery.

Provider 2.5.0 selector order and unlabelled-last-wins remain unchanged, as do
constructor source fallback, credentials, public auth/status outcomes, audit
schema bytes and VERSION. /ingest-documents uses its existing API-key
authorization and per-record response envelope; direct route tests now exist.
Audit export failure cannot alter confirmed primary upload/delete results.

FR-008 allows necessary existing public translations/cleanup/best-effort
boundaries, not blanket recovery. 16 exact proposals, zero active approvals:
four audit, post-write refresh, three purger cleanup, direct-upload translation,
two optional auth diagnostics, NL2SQL document translation and three indexer
cleanup, and root document-chunking translation. Every proposal references
passing boundary tests. No automatic debt, whitelist or human approval.

Current evidence

Exact-head CI:
unit job 101559478595 666 passed, 96 warnings, 24.77s on Python 3.12.14.
Typing and architecture passed; lint 89, exceptions 157, policy 10
and the always-evaluated aggregate failed with explicit violations.
Agent-assets workflow
passed.

Local evidence on the integrated source	Result
New chunking/NL2SQL focused subset	77 passed, 5 warnings
Full maintained Python suite	666 passed, 100 warnings, 190.36s
Checker regression subset	168 passed
Agent assets / Windows-aware whitespace	Passed
Blocking typing / individual baseline	3 of 52 inventoried modules; 0 in-scope findings; 0 baseline entries
Bound full quality report	Exit 1, 256 findings
Frontend	npm test: Vitest missing; npm ci: existing React 19 / React DOM 18 type-peer conflict; lint/build not run

The 256 findings are 89 BLE001 + 141 unapproved-handler + 16
exception-review-pending + 1 policy-bootstrap + 9 protected-policy-change.
The 141 broad sites already include the 16 inactive proposals: 125 sites
still have no proposal, not 141 additional sites. These are remaining code
dispositions, not solely administrative blockers.

Regression-first batches include Q6 14 failures/18 controls, NL2SQL
20 failures/6 controls, and returned chunking 29 failures/6 controls.
The latter excludes three corrected fixture mistakes. Current focused and full
suites pass after the repairs.

python -m pip install -r requirements.txt
python -m pip install -r requirements-quality.txt
python -m pip install pytest pytest-asyncio
python -m pytest tests -q --junitxml=.artifacts\pytest.xml -o junit_family=legacy
python -I .github\scripts\quality-evidence.py --junit .artifacts\pytest.xml --base-ref 38a395586ee1d440a8e1ca8233413f8c25b3fdc2 --report .artifacts\test-evidence.json
python -I .github\scripts\check-quality.py --check all --base-ref 38a395586ee1d440a8e1ca8233413f8c25b3fdc2 --test-evidence .artifacts\test-evidence.json --report .artifacts\quality.json
python .github\scripts\validate-agentic-assets.py
git -c core.whitespace=cr-at-eol diff --check

Initial python -I is required. Aggregate entry point: quality-gate.py.
Checker exits: 0 pass, 1 findings, 2 invalid/incomplete execution.
Reports and behavior evidence bind base/head/run/attempt and integrity hashes.
Local reports bind this head; CI independently reran the full committed source.

Exact quality pins: Ruff 0.16.5, mypy 2.3.1, Import Linter 2.14, Grimp 3.16.
Research candidates Ruff 0.16.6 / Import Linter 2.15 / Grimp 3.17 were unavailable
from the configured index. Local Python 3.12.9; frontend Node 20.14.0,
npm 10.7.0. Runtime pins remain provider 2.5.0, Search 11.5.2.
Earlier ambient provider 2.1/Search 11.7-beta claims are superseded, not shipped
evidence. Local SDK overlay inherits other installed packages; CI restores the
full declared requirements. No runtime/frontend dependency pins changed.

Task accounting and remaining acceptance

Task	Implemented scope / remaining acceptance
T003	Inventory and current-develop reconciliation delivered
T007	Pinned tools and closed four-record parsing delivered
T010	Q6 mutation/tool/receipt cases delivered, 168 checker tests
T013	Flat lint/type enforcement and monotonic individual ratchets delivered; inherited handlers remain red
T016	Protected execution, verified owner proposal and same-workflow aggregate delivered; human bootstrap/required-check activation separate
T019	Full static boundary/cycle fixtures and ownership-move protections delivered
T022	Scheduler ownership extraction and lifecycle/run-now evidence complete
T025	Enforcement delivered; remaining 125 unproposed sites require surgical disposition/evidence, then exact human review
T028	Named scheduler/index/delete/direct-upload/retrieval/config/strict-auth and new chunking/NL2SQL failure evidence delivered; no blanket legacy/live certification
T040	Contributor commands, evidence, compatibility and recovery guidance delivered
Component T045	Python/CI/assets delivered; frontend restore conflict, remaining adoption debt and unavailable live integration remain explicit

Blocking typing seeds are telemetry.audit_contract,
telemetry.audit_sanitizer, jobs.runtime; new runtime modules automatically
enter blocking scope. Stable IDs, exact diagnostics/multiplicity and protected
scope changes prevent count baselines or blanket missing-import suppression.

Human policy/exception/bootstrap review and required-check activation remain
separate. The base has no evaluator: candidate self-approval cannot activate
policy. No settings or bypasses were changed. No live deployment/Search test
was authorized, so cloud topology/credential/recovery evidence is not claimed.

Shipped peers remain orchestrator v4.1.1 and UI v2.6.2; no unmerged peer is
required. Recovery is a reviewed code revert or separately authorized artifact
restoration. Artifact rollback does not undo persisted configuration or
restore deleted documents; source-backed reingestion/config restoration
requires its own authorization and evidence.

No merge, tag, release, image/package publication or deployment was performed.

[Paulo Lacerda (placerda)]

Post-commit checkpoint evidence for bbe52923dbaf2b8ce4f6f371e492ad32ae7ffe45: full maintained pytest rerun passed 295 tests / 87 deprecation warnings in 58.53s, and same-SHA test evidence was produced. The exact-SHA quality command with --test-evidence .artifacts/test-evidence.json completed with exit 1 / 358 findings, unchanged: 218 unapproved handlers, 131 Ruff findings, 9 bootstrap/protected-policy findings. Architecture and in-scope typing remain green; this is not a green aggregate or completed acceptance claim.

Parent and docs handoff posted to Azure/GPT-RAG#689 and Azure/GPT-RAG#688. Checkpoint work is intentionally paused for review as requested; the remaining original acceptance scope is retained in this draft.

[Paulo Lacerda (placerda)]

Current follow-up 46d08d3 is pushed, with a clean component worktree and draft/develop PR unchanged. Exact-head local suite: 348 passed, evidence and full-quality reports bind that SHA. Full quality remains exit 1 / 346 findings, not a completed acceptance claim.

Current-head GitHub run https://github.com/Azure/gpt-rag-ingestion/actions/runs/34041647661 is now complete: unit-tests, typing and architecture passed; lint failed with 131 findings, exceptions with 206 (202 handlers + 4 pending proposals), policy with 9. The actual same-workflow quality-gate failed. These jobs returned explicit violations, not tool-execution errors. The separate agent-assets workflow also passed.

Complete files/commands/tool versions/task and residual-acceptance accounting are in the updated PR body. Parent Azure/GPT-RAG#689 and documentation Azure/GPT-RAG#688 received the exact follow-up SHA and failure-semantics handoff. T022/T040 are complete; inherited-handler/remaining failure-contract disposition, tooling/surface review, administrative activation and live integration/recovery remain open. Four exact proposals remain unapproved, zero active exceptions; no blanket whitelist or success claim is substituted for those requirements. No merge, publication, deployment, settings changes or parent-file edits.

[Paulo Lacerda (placerda)]

Final frozen ingestion handoff: #296 is draft/develop at 3a46472, clean worktree. Fetched develop remains 38a3955. The PR body is fully consolidated and no longer advertises the superseded PUT-refresh 207 behavior or old SDKs as pinned. All five original tooling findings are independently closed, including the implicit builtins at 0fdfc46.

Exact final evidence: local SDK-aligned suite 422 passed / 95 warnings / 78.25s; CI 422 passed / 96 warnings / 15.24s (Python 3.12.14). Checker subset 116 passed; bounded provider/PUT/purge/SDK subset 60 passed. Final CI https://github.com/Azure/gpt-rag-ingestion/actions/runs/34046006654: unit-tests, typing, architecture pass; lint 131, exceptions 210, policy 9 return violations; actual aggregate exits 1. Asset workflow34046006433 passes. The earlier CI fixture failure is fixed, not reclassified as intentional red.

Full local gate exit1 / 350 findings: 202 unapproved handlers, 131 BLE001, eight pending proposals, nine policy findings. Eight proposed / zero active exceptions; zero typing baseline entries. Three blocking modules have zero findings; 344 imported local diagnostics remain visible. Graph 53 nodes / 145 edges / 41 Grimp package nodes, no reported violations. Receipt/report both bind the exact final head/base.

CLI:

python -m pytest tests -q --junitxml=.artifacts\pytest-final.xml -o junit_family=legacy
python .github\scripts\quality-evidence.py --junit .artifacts\pytest-final.xml --base-ref 38a395586ee1d440a8e1ca8233413f8c25b3fdc2 --report .artifacts\test-evidence-final.json
python .github\scripts\check-quality.py --check all --base-ref 38a395586ee1d440a8e1ca8233413f8c25b3fdc2 --test-evidence .artifacts\test-evidence-final.json --report .artifacts\quality-final.json

Tool pins: Ruff0.16.5 / mypy2.3.1 / ImportLinter2.14 / Grimp3.16. Final SDK evidence uses unchanged appconfiguration1.8.1 / provider2.5.0 / Search11.5.2 / tenacity8.5.0. Local Python3.12.9 SDK overlay inherited other existing packages; CI installs the full runtime requirements. Earlier ambient2.1/11.7 SDK claims are expressly superseded.

PUT successful durable write + failed best-effort refresh retains 200/applied, with safe warning and no false failed key. Genuine /apply failures return500; worker/provider propagation and confirmed Search/purge outcomes are observable contract-restoring changes. The last-selector-wins proof uses real2.5 paging/dedup/merge without reordering labels. Cleanup attempts preserve the primary exception or confirmed result; direct /ingest-documents is NOT covered by wrapper proofs. Code/artifact rollback cannot recreate deleted data or undo persisted configuration.

T022/T040 complete; other implementation/task evidence is tabulated in the PR. Full T025/T028/adoption/live/activation/recovery acceptance remains explicitly incomplete. No broad census, active approvals, settings change, merge, publication, deployment or parent-file edit. No more implementation is planned for this frozen checkpoint.

[Paulo Lacerda (placerda)]

Immutable ingestion tool-isolation/Q6 slice pushed for independent review: f47c98f, existing draft #296 (develop). Runtime follow-up continues separately; do not review moving HEAD as this snapshot.

Regression-first: candidate Ruff/mypy/Grimp shadows and PYTHONPATH sitecustomize produced 7 failing marker-execution fixtures before isolation; qualified/imported/assigned no_type_check decorators produced 3 failing policy fixtures before the AST suppression repair.

Commands/results on this slice: python -m pytest -q tests\test_quality_policy.py --tb=short => 136 passed (133.22s); python -I .github\scripts\check-quality.py --check architecture --base-ref 38a395586ee1d440a8e1ca8233413f8c25b3fdc2 --report .artifacts\isolation-architecture.json => 0 findings, 53 modules/145 edges/41 Grimp modules; corresponding --check typing => 0 findings; python .github\scripts\validate-agentic-assets.py => valid 4 agents/6 skills; whitespace check clean. Local Python3.12.9, Ruff0.16.5, mypy2.3.1, ImportLinter2.14, Grimp3.16. Real graph forbidden-import positive/negative controls pass, not merely absence of execution.

Protected CLI refinement for docs Azure/GPT-RAG#688: use python -I for check-quality.py, quality-evidence.py and quality-gate.py (arguments unchanged). Children use isolated interpreter/neutral cwd; static source specs reject execution; Import Linter application API avoids its cwd sys.path insertion and emits a checked completion receipt. Executable plugin/external config rejected; Git resolves outside candidate and disables fsmonitor/inherited overrides. CI static jobs install selected protected requirements in neutral cwd; behavioral job remains separate. Installed tooling/environment is trusted, not an OS sandbox; bootstrap remains explicitly blocked. Runtime pins untouched (provider2.5.0/Search11.5.2). 0 active approvals. No merge/settings/deploy/publication. Worktree clean at commit; continuing assigned runtime/lint/evidence without waiting for review.

[Paulo Lacerda (placerda)]

Runtime follow-up committed/pushed: 932ae7583d3c597f6536999d72748e4d167027a4 on draft #296 (develop). The independent isolation slice remains immutable at f47c98f5ca9762a539e7279e681e4a0d75e967f7.

Implemented regression-first: SDK-only Blob retry/original failure, explicit Key Vault/Cosmos/analysis failure contracts, parser narrowing, SharePoint provider propagation, removed redundant credential wrappers, and actual /ingest-documents direct-SDK confirmation (including missing/duplicate/unrelated/malformed/transport results and real audit-export failure). API-key auth, payload/ACL identity, per-record 200, and PUT durable-write/refresh-only-failure 200/applied remain. No runtime pin or audit bytes changed.

Exact commands/results on this source:

• SDK-overlay Python 3.12.9 (azure-appconfiguration-provider==2.5.0, azure-search-documents==11.5.2): python -m pytest tests -q --junitxml=.artifacts/pytest.xml -o junit_family=legacy: 500 passed, 95 warnings.
• python .github/scripts/validate-agentic-assets.py: passed (4 agents, 6 skills/scoped instructions).
• python -I .github/scripts/quality-evidence.py --junit .artifacts/pytest.xml --base-ref 38a395586ee1d440a8e1ca8233413f8c25b3fdc2 --report .artifacts/test-evidence.json, then python -I .github/scripts/check-quality.py --check all --base-ref 38a395586ee1d440a8e1ca8233413f8c25b3fdc2 --test-evidence .artifacts/test-evidence.json --report .artifacts/quality.json: exit1, 304 findings =111 BLE001 +174 unapproved handlers +9 proposed records +1 bootstrap +9 protected-policy changes; architecture/typing have no findings. Precommit local evidence is not a new-head CI receipt; CI is pending.

Nine individually authored proposals / zero active. Ninth is only the established direct-upload per-record failure translation, with actual route evidence. CLI/tool pins unchanged: Ruff0.16.5, mypy2.3.1, Import Linter2.14, Grimp3.16. The failed attempt to pass --junit to the checker exited2; corrected to the existing evidence-binder/--test-evidence interface above.

Not final: remaining handler/lint dispositions, Q6 follow-up and full final accounting continue. Docs should replace the wrapper-only direct-upload disclaimer; repository docs now do so. Artifact rollback still cannot undo persisted configuration or restore deleted documents. No approvals/settings/merge/deployment/publication occurred.

[Paulo Lacerda (placerda)]

Independent Q6 checkpoint now pushed: d7a71f38624266e0ed4ae28b8443545c99ad6d89 (draft ingestion#296/develop). Only checker/quality fixtures/docs in this commit; my underway runtime and your delegated chunking edits are not staged.

Regression-first: 14 failures/18 passing controls exposed malformed mypy/Ruff diagnostic acceptance (including boolean positions, null rule, non-string message) and an aggregate CLI accepting status=passed with execution-error findings. Repair validates diagnostic types/positions/duplicate keys and status consistency; aggregate requires empty findings for a passing report. The additional CLI matrix executes missing/skipped/error jobs, missing report, wrong base/head/policy/source/run/attempt, skipped/stale test evidence and duplicate test keys plus clean control. Existing pinned tools remain usable.

python -m pytest -q tests/test_quality_policy.py --tb=short: 168 passed. Actual isolated mypy typing: passed/0; actual Ruff lint against current in-progress working source: 100 findings, not a final committed gate count. The latter includes my unstaged SDK/auth slice and must not be attributed to d7a71f3 as an immutable runtime result.

Confirmed prior runtime 932ae7583d3c597f6536999d72748e4d167027a4 CI run34050288735: unit job101532564728 500 passed/96 warnings, typing/architecture success; lint/policy/exceptions/aggregate fail closed. This supersedes pending-CI wording in its checkpoint, not final acceptance. Your seven-file delegation remains untouched by me.

[Paulo Lacerda (placerda)]

Resumed implementation checkpoint 9022245 supersedes the 26358cb counts in the prior body: 703 local tests pass; 35 exact proposed/0 active; 95 broad sites still lack proposals; 81 BLE001. Full gate remains 256 findings (81 lint +130 unapproved +35 review-pending +10 policy). Operator/startup failure repairs and nineteen independently described public-boundary proposals are committed. Implementation continues; this is not a completion/freeze or required-check activation claim.