tests

tests/

Automated tests for the Agent Kaizen harness. Standard-library unittest only—no extra dependencies beyond the pinned runtime in requirements-kaizen.txt. Some tests skip unless optional PDF backends from requirements-docs.txt, sentence-transformers from requirements-pytorch.txt, or explicitly enabled live backends and servers are available.

Isolation

The outer runner exclusively allocates one AI/work/<run-id>/ root, exports a complete matching KAIZEN_TEST_RUN_ID and absolute KAIZEN_TEST_TEMP_ROOT, and supplies child-only environment overlays for temporary files, caches, bytecode, package managers, fixtures, sockets, databases, journals, installers, benchmarks, and logs. Nested runners validate and inherit that pair. They do not allocate another run or set KAIZEN_REPO_ROOT globally.

Protocol, serialization, ownership, and vector-data tests append immutable rows to regular quoted "tests" tables. Product operations never see those rows. Production DDL, API, transaction, and native F32/F64 behavior uses throwaway tables in one run-owned scratch database; a separate database file is used only when journaling, corruption, locking, recovery, swap, or file ownership is the behavior under test.

Only the outer runner removes the run root. Cleanup is strict: a remaining child, lock, journal, cache, or deletion failure fails the run. Tests invoke sys.executable, so run the canonical wrapper with the shared Kaizen venv that has pyturso installed.

Run

The no-argument default is the fast core lane. It contains deterministic unit, static, schema, and contract tests; it never includes the slow, live, or extension lanes. The current supported release scope is genuine Windows core, platform, and slow; slow remains an explicit manual selection.

# Windows
& "$env:DEVROOT\Python\venvs\kaizen\Scripts\python.exe" tests/run_tests.py

# Linux / macOS development evaluation only
"$DEVROOT/Python/venvs/kaizen/bin/python" tests/run_tests.py

List lane ownership without creating test scratch:

& "$env:DEVROOT\Python\venvs\kaizen\Scripts\python.exe" tests/run_tests.py --list-lanes

Run the bounded host-contract lane when changing filesystem, process, installer, or loopback behavior:

& "$env:DEVROOT\Python\venvs\kaizen\Scripts\python.exe" tests/run_tests.py --lane platform

Subprocess-heavy, concurrency, timeout, benchmark, and integration tests are explicit because they dominate suite runtime. Run the affected module during development, and run the whole slow lane only when the change spans those surfaces or before a release candidate:

& "$env:DEVROOT\Python\venvs\kaizen\Scripts\python.exe" tests/run_tests.py test_session_drive.WriterLeaseConcurrencyTest
& "$env:DEVROOT\Python\venvs\kaizen\Scripts\python.exe" tests/run_tests.py --lane slow

Provider/live tests retain their own environment gates and never run implicitly. The canonical runner removes an ambient KAIZEN_RUN_LIVE unless --lane live is selected, including for raw unittest selectors. The extension lane also requires an explicit selection; its Node runner discovers every compiled test instead of maintaining a manual allowlist:

& "$env:DEVROOT\Python\venvs\kaizen\Scripts\python.exe" tests/run_tests.py --lane live
& "$env:DEVROOT\Python\venvs\kaizen\Scripts\python.exe" tests/run_tests.py --lane extension

The release driver lists and preflights its deterministic gates without running them, executes the current Windows release scope plus explicitly selected development scopes, validates an existing report, and admits live systems only through explicit provider selections:

& "$env:DEVROOT\Python\venvs\kaizen\Scripts\python.exe" tests/release_validation.py --list
& "$env:DEVROOT\Python\venvs\kaizen\Scripts\python.exe" tests/release_validation.py --preflight
& "$env:DEVROOT\Python\venvs\kaizen\Scripts\python.exe" tests/release_validation.py --all-local
& "$env:DEVROOT\Python\venvs\kaizen\Scripts\python.exe" tests/release_validation.py --preflight --include-extension
& "$env:DEVROOT\Python\venvs\kaizen\Scripts\python.exe" tests/release_validation.py --preflight --include-linux
& "$env:DEVROOT\Python\venvs\kaizen\Scripts\python.exe" tests/release_validation.py --check <report.json>

Genuine Linux and the extension are IN DEVELOPMENT, not default release blockers. An unselected development scope is INAPPLICABLE; after explicit selection, an unavailable host or missing prerequisite is NOT_RUN with a disposition. --include-linux requires a genuine Linux host; WSL execution is not genuine Linux release evidence.

Network dependency lookup, authentication, downloads, live providers, WSL registration, installer machine changes, and Windows security or Sandbox actions require separate owner approval. An unselected selection-gated external check is INAPPLICABLE; a selected check without its approval or prerequisite is NOT_RUN, not a pass.

GitHub pull requests and pushes automatically run the current Windows core and platform scope. workflow_dispatch exposes run_slow for Windows slow, run_linux for genuine Linux development core/platform (and Linux slow when combined with run_slow), and run_extension for the Windows extension development job. Linux and extension jobs are manual, non-required development evidence until the owner promotes them. The Ubuntu dependency-audit and DCO governance hosts do not establish Linux runtime support. Remote CI exists only after the owner pushes and is not a failed or missing local release gate before that boundary.

Dependency audit (CI parity)

CI runs a second job, audit, that scans every pinned dependency set for known advisories; mirror it locally before pushing with pipx run pip-audit --progress-spinner off -r requirements-kaizen.txt -r requirements-docs.txt -r requirements-pytorch.txt — a clean audit run means the job will pass.

Representative coverage map

This curated map highlights major suites; it is not exhaustive. lane_manifest.py is the authoritative lane inventory; every root test_*.py module must have exactly one owner, and the runner fails closed on an unclassified, overlapping, or missing module.

• test_op_coverage.py — the conformance matrix: AST-scans the whole suite and fails if any CLI operation is not exercised by at least one test (ALLOWED_UNTESTED is empty by design).
• test_doc_examples.py — executes every kaizen.py command example in README.md, setup/SETUP.md, and evals/README.md against a scratch database (placeholder IDs threaded between commands), and asserts CLAUDE.md/AGENTS.md stay byte-identical as whole files.
• test_bench_smoke.py — tests/bench_kaizen.py --quick runs green into a temp output dir (valid results JSON, perfect retrieval hit rate, semantic skipped, no personal paths) and never rewrites the repo's own README.md/docs/BENCHMARKS.md.
• test_schema.py — K1 is idempotent, K2 reports a healthy schema, K3/K6 run, plus ARC-067 schema-difference classification (identity-only drift reconciles at K1 and never denies a write; K1 --restamp-manifest remains the repair for what K1 cannot reconcile), the typed four-domain K1 --integrity report, and the deterministic half of K4 restore planning.
• test_input_hardening.py — the unified repo-only path policy (E1/A2 deny outside-repo paths; --allow-external stores a sanitized external: origin), the UTF-8 decode denial on --*-file inputs, LIKE wildcard escaping in evidence search, and the PDF ingestion guards (size/pages/encrypted/no-text; pypdf-dependent cases skip when the docs backend is absent).
• test_search_escaping.py — the shared text_search.like_pattern helper plus record-plane escaping: a literal _/% in a query matches literally (not as a wildcard) across G3/L5/X3/S2/Q9, the surfaces that route through the same LIKE ? ESCAPE '\' path as evidence search.
• test_db_retry.py — retry-vocabulary/backoff unit tests plus a 4-process concurrent K1 race regression against one shared data plane.
• test_learning_ops.py — the full L* lifecycle: adds, L2/L3 promotion linkage (source_gotcha_id/source_learning_id), promoted-status transitions, G5 updates, L10 lineage narratives, denials.
• test_policy_ops.py — X1–X5 round-trips, priority-over-recency ordering (an old critical rule survives the LIMIT window), trigger filtering, retired-rule exclusion.
• test_quality_ops.py — Q1–Q7, Q9 filters, artifact ops, *-file JSON fallbacks, Q7 routing to eval runs/cases.
• test_report_ops.py — R0 session digest (sections, counts, drift signals, limits), R1–R6 report files, R4 severity/actionability filters and enriched columns, R9/R10 windows.
• test_plan_packet_ops.py — W2–W8: plan revisions, packet round-trips, object-only payload enforcement, revision counting via K6.
• test_migration_ops.py — M1–M5: allowlists, dry-run purity (hash-snapshot), stub content, backups + manifest, verify/report.
• test_irl_ops.py — I1–I5: review/prediction/correction/outcome lifecycle and report files.
• test_lab_misc_ops.py — O1–O3 improvement lab, S4 export, E5 inspect, and the newer redaction pattern classes.
• test_integration_chains.py — end-to-end chains across families (GOTCHA→LEARNED→report, task→artifact→verification→digest, trace→score→query, artifact path containment).
• test_cli_wiring.py — --help/--version, unknown/missing-op denials, ALIASES ↔ README command-table parity, and that every operation reaches a real dispatch branch (OK or a structured DENIED, never ERROR_UNEXPECTED).
• test_records.py — create/inspect/query round-trips (GOTCHA, source locks, artifacts, traces), structured denials on missing required fields, and a clean DENIED_FILE_NOT_FOUND on a missing --*-file.
• test_reports.py — R7 --query regression (no missing-body column error), R8–R10 time-windowed ledger reports (incl. a recent-event inclusion check), and R11 topic requiring --query.
• test_redaction.py — the secret/personal-path/email scanner, plus the trace-write gate denying secrets in environment/tags.
• test_output_validate.py — Q8 lists schemas, accepts valid payloads, rejects invalid ones.
• test_aux.py — the prompt-builder imports cleanly with its external-checker read guarded, plus the repository doc-link contract (every repository-relative path written in tracked prose must resolve).
• test_vectors.py — Turso-native vector storage + cosine-distance nearest-neighbour search (the engine claim), exercised in a throwaway temp database.
• test_comfyui.py — the ComfyUI generative-run ops (Y*): Y1 --dry-run records a queued run with a deterministic workflow hash + extracted seed/models (no network), Y2/Y3 inspect/list, and graceful denials (DENIED_BACKEND_UNAVAILABLE with no server, missing --path, bad workflow type).
• test_backends.py — the model-backend ops (B*) + embedding seam, with no live server: B1 reports unconfigured/unreachable backends, B2/B3/E4 --semantic deny cleanly when unconfigured, and E1→E3→E4 still chunk + lexically search with no embeddings (graceful degradation). Also unit-tests the HTTP retry classifier (transient vs permanent, Retry-After honoring) and the bounded embed_batched helper (ordered batches, count-mismatch denial).
• test_pytorch.py — the sentence-transformers selection + semantic chunker without the heavy extra installed: the in-process embedder is recognized, the absent extra denies cleanly, neural is reserved, and semantic chunking requires a configured backend (absent-extra tests skip if installed).
• test_comfyui_live.py — the ComfyUI live path against an in-process mock server: Y5 reachable, Y1 submit→wait→fetch→save→register→completed (output saved + hashed), and Y4 replay.
• test_backends_live.py — the model-backend live path against an in-process mock OpenAI server: B1 reachable + dimension, E3 stores embeddings, the semantic chunker splits by similarity, E4 --semantic ranks by cosine, B2 records a model_call trace, B3 backfills embeddings.