docs(propose): init/increment perf — bulk graph writes + cached ignore

[HumanBean17]

What

Adds propose/active/INIT-INCREMENT-PERF-PROPOSE.md — a proposal-only design for the init/increment performance program. No production code changed.

Why now

Profiling java-codebase-rag init on a medium Java corpus (Shopizer: 1210 files → 1167 indexed, 3879 chunks, ~32k graph edges, 395s total) showed graph writes are ~81% of init, not the vectors stage:

phase	time	share
LadybugDB graph write (per-row MERGE/CREATE)	~321s	~81%
cocoindex vectors	~68s	~17%
optimize	~5s	~1%

Init/increment latency is the project's stated pain point, so this redirects effort to the real bottleneck.

Highlights

Three PRs under one proposal:

• PR-1 — Bulk in-memory-pyarrow COPY FROM for the full rebuild path (init/reprocess). Replaces ~21 per-row conn.execute sites (build_ast_graph.py:3096/3250-3398) with staged bulk loads via COPY <table> FROM $param. A micro-benchmark on the real Symbol schema measured ~300× (5.6 ms/row → 0.018 ms/row). Projected ~321s graph-write → tens of seconds; init ~395s → ~120s. Staging invariants spelled out: REL FROM/TO column rule, CALLS dedup + callee_declaring_role materialized at staging, node-before-edge order. Carries a mandatory equivalence harness (old per-row build vs new bulk build → identical counts + meta + full edge property rows + query results).
• PR-2 — Same primitive extended to the incremental path (preserves the Route-MERGE dedup at build_ast_graph.py:3819-3821). Depends on PR-1.
• PR-3 — Hoist LayeredIgnore(project_root) to a flow-lifespan cocoindex ContextKey and memoize is_ignored — ~25s → ~0s. Independent of PR-1/2.

No ontology bump (graph contents identical; proven by the harness). All PRs re-index-free — only the write mechanism / a cache change.

Reviewed

A 5-lens subagent review (39 agents, ~1.3M tokens) empirically validated the load-bearing claims (kuzu 0.11.3 COPY FROM into REL tables works; ~300× speedup; line citations spot-checked). Its main catch: an earlier PR-4 (default embedding device to MPS) was dropped — its premise was false. The flow already auto-selects MPS (SBERT_DEVICE unset → device=None → cuda→mps→cpu), so the profiled init ran on MPS (~16s), not CPU; there was no CPU→MPS win to recover. That rejection is recorded in the proposal's Out of scope.

Tests

Docs-only; baseline unchanged.

Out of scope

• Implementation of any PR (PR-1…PR-3 follow as separate PRs).
• MPS device default — already auto-selected (see Reviewed).
• ANN vector index — parked, perf: add ANN vector index if/when flat-scan query latency becomes a bottleneck (parked) #337.
• watch live mode — feat: watch mode — keep the index live as files change #336.

References

• Proposal: propose/active/INIT-INCREMENT-PERF-PROPOSE.md
• Related: feat: watch mode — keep the index live as files change #336 (watch), perf: add ANN vector index if/when flat-scan query latency becomes a bottleneck (parked) #337 (ANN, parked)