fix(mcp): cold-start resilience — async handlers, reranker timeout, Windows portability

[Huntehhh]

Summary

Three compounding failure modes caused the MCP server to hang or crash on cold
start. Plus a sibling Windows-portability fix in pytest test decorators
(same hasattr-guard theme as the WNOHANG fix; was blocking pytest
collection on Windows entirely).

1. Sync handlers — all 9 @mcp.tool() handlers were sync def, so one
   blocked handler froze every concurrent JSON-RPC request on FastMCP's
   single asyncio thread. Converted to async def with engine calls wrapped
   in asyncio.to_thread. @tracked is now async-aware so it doesn't
   re-wrap coroutines as sync functions.

2. Reranker preload hangs — CrossEncoder(...) in _preload_models
   blocks indefinitely on corrupt HuggingFace cache, stalled download, or
   Windows Defender ASR denying a sentencepiece shim. Added a 30 s watchdog
   (TRUEMEMORY_RERANKER_TIMEOUT_SEC override); on timeout, marks
   degraded and rerank entrypoints fall back to original-ordering results.
   _set_reranker short-circuits when degraded so search calls don't
   block on the stalled load's lock. Degraded state surfaces in the F06
   health payload (truememory_stats.health.reranker).

3. os.WNOHANG is POSIX-only — _reap_children called
   os.waitpid(-1, os.WNOHANG), crashing every Windows user's backlog
   drainer with AttributeError on every boot. Guarded with hasattr.

Also: engine.add() now pre-computes embeddings outside _write_lock so
concurrent stores overlap on inference instead of serializing.

Changes

File	Change
truememory/mcp_server.py	9 handlers → async def + asyncio.to_thread; reranker watchdog with TRUEMEMORY_RERANKER_TIMEOUT_SEC (default 30 s, validated); _reap_children POSIX guard; _set_reranker short-circuit on degraded; watchdog wires _record_reranker_error so health payload reflects truth; _parse_reranker_timeout helper
truememory/reranker.py	_load_failed flag; is_degraded() + mark_degraded(reason) public API; early return in rerank(), rerank_with_fusion(), rerank_with_modality_fusion() when degraded
truememory/telemetry.py	@tracked async-aware: detects iscoroutinefunction and wraps with async_wrapper so FastMCP sees the correct function type
truememory/engine.py	add() pre-computes content + separation embeddings BEFORE _write_lock — concurrent stores overlap on inference (~10-50 ms per encode, GIL released inside .encode())
tests/test_concurrent_store_hang.py	New file (3 tests): engine concurrency, MCP handler async shape, asyncio.gather end-to-end. Includes truememory_status in the async-handler list
tests/test_cold_start_resilience.py	New file (14 tests): WNOHANG guard, degraded-flag lifecycle, watchdog timeout + fast-load non-regression, _set_reranker short-circuit, health-payload wiring on both timeout and exception paths, timeout-parser validation (positive / zero / negative / non-integer / unset)
tests/test_health_stats.py	Wraps now-async truememory_stats() in asyncio.run()
tests/ingest/test_robustness_fixes.py	4 @pytest.mark.skipif decorators: os.geteuid() → not hasattr(os, "geteuid") or os.geteuid() == 0. Same POSIX-only-API pattern as WNOHANG; was crashing pytest collection on Windows
CHANGELOG.md	New ## [Unreleased] section documenting all of the above

Test Plan

• python -m pytest tests/test_cold_start_resilience.py -v → 14 passed
• python -m pytest tests/test_concurrent_store_hang.py -v → 3 passed
• Cold start: truememory-mcp reaches "server ready" within 30 s, no boot hang
• Windows: no AttributeError: module 'os' has no attribute 'WNOHANG' in logs
• TRUEMEMORY_LAZY_MODELS=1 — server starts instantly, models load lazily on first search
• Corrupt HF cache (rm -rf ~/.cache/huggingface/hub/models--BAAI--bge-reranker-v2-m3*): confirm "preload exceeded 30s" appears in logs AND truememory_stats.health.reranker.status == "degraded" AND truememory_search returns results (sans rerank)
• TRUEMEMORY_RERANKER_TIMEOUT_SEC=0: WARNING logged at startup, watchdog still uses 30 s default

Design Notes

• Why 30 s default? Cold load of BAAI/bge-reranker-v2-m3 on a typical
  machine with warm HF cache: ~3-8 s. First-time download over slow network:
  10-25 s. 30 s gives generous headroom while still surfacing real hangs.
• Why no "disable watchdog" option? TRUEMEMORY_LAZY_MODELS=1 already
  covers the "I don't want preload" path. Allowing TIMEOUT_SEC=0 to mean
  "disable" is a footgun — a typo like TIMEOUT_SEC= in a shell script
  becomes 0 and silently disables the safety net.
• Why not kill the orphaned reranker thread? Python threading has no
  safe kill primitive. The orphan thread sits at 0 % CPU using ~5 MB until
  process exit. Documented limitation; restart recovers cleanly.
• Why not retry the reranker load? Once degraded, recovery requires
  process restart (clears any cache state too). Auto-retry hides root
  causes (HF cache corruption, ASR rule, network) that the user needs to
  fix once.

Co-Authored-By: claude-opus-4-7 wontreply@getfucked.ai

Merge ordering

Status: Currently CONFLICTING against origin/main after PR #343 (fix/resource-efficiency-v1) merged. Trivial rebase needed before merge — branch was cut off 9d18019; main is now at 6ec0be5. I'll push the rebase the moment Hunter or a reviewer gives the go.

Blocks: #351 (Windows subprocess portability — mcp_server.py:1181 backlog drainer Popen layers on top of this PR's async-handler conversion).

Closes: #318 — Hunter's May-14 commit 5747b99 is the cherry-pick base of this PR's 68989d0. This PR is a strict superset (adds truememory_status async fix, Windows WNOHANG guard, reranker watchdog + degraded mode + _set_reranker short-circuit + health-payload wiring, regression tests, os.geteuid test fix, CHANGELOG).

Recommended merge sequence (10 PRs from a 3-agent coordination — see ~/.claude/handoffs/truememory-coordination-registry.md on Hunter's machine):

#353 (CI runner) → #344 (this) → #345 → #346 → #351 → #348 → #352 → #347 → #349 → #350