feat(caching): add seed_cache_dir for cross-run cache reuse

[gchlebus]

Problem

When the same evaluation is run on different clusters (e.g., CW-PDX → CW-DFW), each gets a separate output_dir and therefore a separate cache_dir. The cache keys are identical (SHA-256 of the request JSON body), but the caches are physically isolated on separate Lustre filesystems. If a run times out on one cluster and is restarted on another, all cached results are lost — even though the exact same requests would produce the exact same cache keys.

Solution

Add a seed_cache_dir parameter to CachingInterceptor that acts as a read-only fallback cache. On a cache miss in the primary cache, the interceptor checks the seed cache. Seed cache hits are automatically promoted into the primary cache, so the output cache is always self-contained after a run. The seed cache itself is never modified.

Usage

Legacy config:

adapter_config:
  seed_cache_dir: /path/to/previous/run/cache

Interceptor config:

interceptors:
  - name: caching
    config:
      seed_cache_dir: /path/to/previous/run/cache

Workflow

1. Run eval on Cluster A → cache populated at {output_dir}/cache/
2. Copy cache directory to Cluster B: rsync -a clusterA:/path/cache/ clusterB:/path/seed-cache/
3. Run eval on Cluster B with seed_cache_dir: /path/seed-cache
4. Cluster B gets instant cache hits for all previously completed items
5. Output cache on Cluster B is self-contained — includes both promoted seed entries and newly generated responses

Changes

• caching_interceptor.py: Added seed_cache_dir param to Params, seed Cache initialization with directory existence validation (warns when subdirs are missing), fallback logic in _get_from_cache, automatic promotion of seed hits into primary cache via direct cache writes (bypasses response counter to avoid inflating _cached_responses_count)
• adapter_config.py: Added seed_cache_dir to LegacyAdapterConfig, wired through to caching interceptor in from_legacy_config()
• test_seed_cache.py: 11 tests covering fallback, promotion to primary, primary precedence, both-miss, no-seed, nonexistent dir, partial seed dir, write isolation, seed immutability, and legacy config passthrough
• docs/libraries/nemo-evaluator/interceptors/caching.md: Added "Seed Cache" section documenting both legacy and interceptor configuration formats, promotion behavior, and cross-cluster usage guide

Testing

Unit tests

• 11 tests, all passing
• Existing tests pass with zero regressions

Real cluster validation

1. End-to-end test (AA_math_test_500, CW-PDX → CW-DFW):

• Copied cache (2,500 entries, 56MB) from CW-PDX to CW-DFW
• Ran eval with pre_cmd installing this branch
• 2,500 / 2,500 LLM responses served from seed cache — 100% hit rate
• Zero GPU inference on DFW — all responses from cache

2. Production use (HLE benchmark, CW-DFW → CW-PDX):

• DFW multi-node run timed out at 63.5% (1,677/2,158 items)
• Copied 151MB cache from DFW to PDX
• Ran HLE eval on PDX with seed_cache_dir pointing to copied cache
• 1,677 cached items served in ~11 seconds, then generated remaining 481 items in ~2h
• GPT-4o judging completed in ~30 min
• Result: 23.17% judge_correct — complete end-to-end evaluation with cross-cluster cache reuse
• Primary cache on PDX ended up self-contained (all 2,158 entries) thanks to seed promotion

Test details

• Invocations: 84774da8d9154adc (AA_math test), d8f3b1db276dd6e2 (HLE production)
• Clusters: CW-DFW, CW-PDX
• Model: Qwen3.5-122B-A10B (vLLM, 8×GPU per node, 8 nodes)
• Branch install: pip install nemo-evaluator @ git+...@feature/cache-seed-dir via pre_cmd

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[gchlebus]

/ok to test 4a15ea8

[gchlebus]

/ok to test 8ddbf16

[piojanu]

Please seed also the requests cache. Now it's confusing that we omit only that one.