ci: release CI follow-ups: tighter coverage

[t1mato]

Summary

Implements all remaining sub-items from #747, covering four areas:

Files & Scripts

• Rename examples/deep-researcher/utils.py to deep_researcher_utils.py to avoid a RAT basename collision with four other ASF-owned utils.py files
• Thread --skip-signing through cmd_verify so CI can verify artifacts without GPG keys
• Extend Apache RAT scanning to .whl artifacts in addition to source and sdist tarballs

Smoke Test

• Replace the hardcoded time.sleep(2) with a polling loop on /api/v0/projects that fails fast if the server exits
• Launch the server in its own process group and send SIGTERM to the whole group on teardown to prevent orphaned uvicorn processes
• Add a GET / check to verify the UI is served
• Add --cleanup/--no-cleanup flag (auto-disabled in GITHUB_ACTIONS so workspaces survive for artifact upload)

CI Coverage Gaps

• Add a bare-install job that installs the wheel without optional extras and imports core symbols, catching accidental leakage of optional dependencies
• Add an sdist-wheel-equivalence job that rebuilds the wheel from the sdist tarball and compares file content hashes against the CI-built wheel, catching missing sdist files
• Pin the Apache RAT JAR download with a SHA256 checksum

Hygiene

• Add scripts/check_asf_headers.py and wire it as a pre-commit hook that checks Python, YAML, and shell files for the ASF license header, reading .rat-excludes at runtime to stay in sync with RAT automatically
• Add a weekly cron (Mondays 09:00 UTC) to the release validation workflow to catch dependency drift between releases

Files

• .rat-excludes, examples/deep-researcher/ — rename and exclusion update
• scripts/apache_release.py — --skip-signing in cmd_verify
• scripts/verify_apache_artifacts.py — RAT on wheels, compare-wheels subcommand
• scripts/ci_smoke_server.py — polling, process group, cleanup flag, GET /
• scripts/check_asf_headers.py — new ASF header checker
• .pre-commit-config.yaml — new check-asf-headers hook
• .github/workflows/release-validation.yml — SHA256 pin, bare-install, sdist-wheel-equivalence, weekly cron
• tests/test_apache_release.py, tests/test_verify_apache_artifacts.py, tests/test_ci_smoke_server.py, tests/test_check_asf_headers.py — test coverage for all of the above

Test Plan

• Run pytest tests/test_apache_release.py tests/test_verify_apache_artifacts.py tests/test_ci_smoke_server.py tests/test_check_asf_headers.py — all 54 tests pass
• Release Validation workflow runs clean on this PR, including the new bare-install and sdist-wheel-equivalence jobs
• Pre-commit hook fires on a staged Python file missing the ASF header

Closes #747

[skrawcz]

Nice, tightly-scoped follow-up to #747 — this materially strengthens the release pipeline. Highlights: the bare-install job that guards against optional-dep leakage into core (directly reinforces the project's vendor-neutrality invariant), the shift from whole-file SHA512 to per-file content hashing for wheel equivalence (correctly ignoring zip timestamps and RECORD), replacing the flaky time.sleep(2) with a real poll that fails fast on server death, and process-group teardown to kill orphaned uvicorn. Test coverage is excellent — 54 focused tests, all passing locally, covering happy paths, timeouts, early-exit, and glob-matching edge cases.
Backwards compatibility is clean: no burr/ library surface, defaults, return types, or serialization change; --skip-signing and compare-wheels are additive with safe defaults, and the example rename updates every import consistently.
Two blocking items before merge, both mechanical: (1) black fails on four files under the repo-pinned black 23.11.0 — the pre-commit hook will block CI; (2) two unused import pytest statements trip flake8 F401. There's also one small doc gap: the deep-researcher README still references the old utils.py filename. Fix those three and this is ready.

---

Inline notes (line refs approximate — folded into body since this is a body-only draft):

• tests/test_check_asf_headers.py:22 [BLOCKING] — import pytest is unused; flake8 (F401) will fail. Remove it.
• tests/test_ci_smoke_server.py:22 [BLOCKING] — import pytest is unused; flake8 (F401) will fail. Remove it.
• tests/test_ci_smoke_server.py:110 [BLOCKING] — black wants a blank line after the inline import json inside read(), and this file has other black diffs. Run black --line-length 100 on the file. (Confirmed with pinned black 23.11.0.)
• tests/test_ci_smoke_server.py:147 [BLOCKING] — the monkeypatch.setattr(..., lambda *a, **kw: (_ for _ in ()).throw(...)) line exceeds 100 cols; black will wrap it across multiple lines.
• scripts/ci_smoke_server.py:245 [BLOCKING] — _fail(f"GET / returned HTTP {resp.status}, ...") exceeds 100 cols; black will wrap the call. Reformat the file.
• scripts/verify_apache_artifacts.py:757 [BLOCKING] — the print(f"Found {len(rat_artifacts)} artifact(s) ...") line exceeds 100 cols; black will wrap it. Reformat the file.
• examples/deep-researcher/README.md:28 [NIT] — still references utils.py; update to deep_researcher_utils.py to match the rename.
• scripts/check_asf_headers.py:56 [NIT] — prefer parameterized list[str] over bare list for these annotations to match repo conventions (would require from __future__ import annotations for 3.9 runtime safety). Non-blocking.
• scripts/verify_apache_artifacts.py:625 [SUGGESTION] — the new .whl → zipfile.ZipFile(...).extractall(extract_dir) branch in _check_licenses_with_rat isn't directly unit-tested (only reached via a mocked _check_licenses_with_rat in test_verify_licenses_runs_rat_on_wheel_...). Consider a small test that actually extracts a fake wheel to lock the branch in. Low risk.
• .github/workflows/release-validation.yml:125 [SUGGESTION] — the RAT JAR SHA256 pin is a good hardening step; consider a brief comment noting how to regenerate the checksum when bumping the RAT version (0.18 → future), so the next upgrader knows the provenance.

[skrawcz]

why this change?

[elijahbenizzy]

Thanks! @t1mato can we fix it up so the tests pass? Some simple formatting + the release validation (i.e. the stuff we're testing here.)

[elijahbenizzy]

Since `maintainerCanModify` is on, I pushed a small fix commit on top of your branch to unblock CI. Feel free to pull it back locally.

• black on the four new/modified files (`scripts/{ci_smoke_server,verify_apache_artifacts}.py`, `tests/test_{ci_smoke_server,verify_apache_artifacts}.py`). This unblocks the `pre-commit` check.
• `.rat-excludes` — added `/METADATA`, `/WHEEL`, `**/RECORD` for the wheel dist-info metadata. Sdist and src RAT scans were already passing; only the newly-added wheel scan was failing, and it was failing on those three auto-generated files that have no ASF header. This unblocks `Release Validation / build-artifacts`.

If CI still fails after this, my next guesses are (in order): the built wheel bundles additional non-source content from `telemetry/ui/build/` that also needs excluding, or the licenses-inside-dist-info path structure confuses RAT. Happy to dig deeper if it recurs.

Thanks for the substantive PR!

[elijahbenizzy]

+1, CI green after the fixes for pre-commit / RAT / sdist-wheel-equivalence. Nice PR — the sdist↔wheel equivalence check flushed out a real latent bug in `apache_release.py wheel`'s `--output-dir` handling, which is exactly the kind of thing that job is meant to catch. Thanks for the work.