perf(vectors): lifespan-cached LayeredIgnore + is_ignored memo (PR-P3)

[HumanBean17]

Scope

This PR implements PR-P3 from plans/active/PLAN-INIT-INCREMENT-PERF.md. It hoists LayeredIgnore to a cocoindex ContextKey (constructed once per flow run) and memoizes is_ignored's _mega computation by directory.

Independent of PR-P1 and PR-P2 — touches different files.

Changes

• java_index_flow_lancedb.py:

  • Define IGNORE = coco.ContextKey[LayeredIgnore] alongside PROJECT_ROOT/EMBEDDER/LANCE_DB using the same _ck_params version-detection pattern
  • Add builder.provide(IGNORE, LayeredIgnore(root)) in coco_lifespan — built once per flow run
  • Convert process_java_file, process_sql_file, process_yaml_file to use ignore = coco.use_context(IGNORE) instead of LayeredIgnore(project_root).is_ignored(...)
  • Sites :182 (in _approximate_vectors_total) and :578 (in app_main pre-walk) left untouched — they call cocoindex_excluded_patterns() once per run, not per-file

• path_filtering.py:

  • Add self._mega_cache: dict[str, tuple[...]] in LayeredIgnore.__init__
  • Memoize _mega(rel) keyed by Path(rel_project).parent.as_posix() — files in the same directory share the same _mega computation

• tests/test_path_filtering.py:

  • Add test_is_ignored_mega_caches_by_directory — asserts _mega computed once per directory
  • Add test_layered_ignore_memo_preserves_decisions — asserts cached decisions match uncached for nested ignore + gitignore negations

• tests/test_lancedb_e2e.py:

  • Add test_layered_ignore_provided_once_per_flow (HEAVY) — asserts single LayeredIgnore instance per flow run

Manual Evidence

Single id(ignore) per flow

# Micro-benchmark showing cache hits for same-directory files
$ .venv/bin/python /tmp/test_ignore_cache_bench.py
is_ignored calls: 100, files in same directory
Cache entries: 1
Elapsed time: 0.003065s
Time per call: 0.031ms
✓ Cache hit: all 100 files shared 1 _mega computation

Sentinel Checks

All sentinel greps from the PR prompt pass:

• ✅ grep -nE "LayeredIgnore\(project_root\)\.is_ignored" java_index_flow_lancedb.py → empty (3 process sites converted)
• ✅ grep -n "coco.use_context(IGNORE)" java_index_flow_lancedb.py → 3 sites (:357, :430, :479)
• ✅ grep -n "_mega_cache" path_filtering.py → 4 hits (cache present)
• ✅ Sites :182 and :578 unchanged (use bare constructor for once-per-run calls)

Test Results

$ .venv/bin/python -m pytest tests/test_path_filtering.py tests/test_lancedb_e2e.py -q
.................sssss                                                   [100%]
17 passed, 5 skipped in 0.06s

$ .venv/bin/python -m pytest tests -q -k "ignore or path_filter or vectors_progress"
.............ss.......................ss                                 [100%]
36 passed, 4 skipped, 815 deselected, 2 warnings in 23.18s

$ .venv/bin/ruff check .
All checks passed!

HEAVY test: The test_layered_ignore_provided_once_per_flow test is gated behind JAVA_CODEBASE_RAG_RUN_HEAVY=1 and requires cocoindex e2e to run locally. Not executed in this environment, but the test is present for CI/validation.

Plan Reference

Implements § PR-P3 from plans/active/PLAN-INIT-INCREMENT-PERF.md and plans/AGENT-PROMPTS-INIT-INCREMENT-PERF.md.

Co-Authored-By: Claude noreply@anthropic.com