term cache files should be free of spurious diffs

[dragon-ai-agent]

Currently it looks like term files are written fresh eat time, updates should be incremental and diff friendly

Copying content of issue from: monarch-initiative/dismech#40

Problem

When adding new disorder files or updating existing ones, the cache files (e.g., cache/hp/terms.csv, cache/mondo/terms.csv) show large diffs where all timestamps get updated, not just the newly added terms.

Example from PR #38

In PR #38, adding Frontotemporal Dementia caused:

• cache/hp/terms.csv: 324 additions, 322 deletions (646 lines changed)
• cache/mondo/terms.csv: 65 additions, 64 deletions (129 lines changed)

However, only 1-2 new terms were actually added. The bulk of the diff is timestamp updates for existing terms:

-HP:0040282,Frequent,2025-12-16T10:24:02.844891
+HP:0040282,Frequent,2025-12-23T11:16:01.374193

Expected Behavior

The cache should be incremental:

• Only newly added terms should get timestamps
• Existing cached terms should retain their original timestamps
• Diffs should only show actual new content

Current Behavior

Every validation run appears to re-write the entire cache with fresh timestamps, making code review difficult:

• Hard to identify which terms are actually new
• Large diffs obscure the meaningful changes
• Git history becomes noisy

Impact

• Code review friction: Reviewers must manually verify which terms are genuinely new
• Merge conflicts: Higher likelihood of cache file conflicts in concurrent PRs
• Git history noise: Harder to track when specific terms were first introduced

Potential Root Cause

This may be an upstream issue with linkml-term-validator or the caching mechanism in the validation stack. The cache re-generation logic may be:

1. Reading all existing terms
2. Re-fetching/re-validating them
3. Writing them back with updated timestamps

Instead of:

1. Loading existing cache with original timestamps
2. Only fetching new terms
3. Appending new terms without touching existing ones

Reproduction

# Add a single new HPO term to any disorder file
# Run validation
just validate kb/disorders/YourDisorder.yaml

# Observe: entire cache gets timestamp refresh
git diff cache/hp/terms.csv

Possible Solutions

1. Preserve timestamps on cache reads: When loading cache, preserve the retrieved_at field
2. Append-only mode: Add flag to only append new terms without re-writing existing ones
3. Explicit re-cache command: Make full cache refresh opt-in via just recache-all or similar
4. Upstream fix: If this is in linkml-reference-validator, contribute a fix there

Questions to Investigate

• Is this behavior in linkml-term-validator or linkml-reference-validator?
• Does the cache loader preserve timestamps when reading?
• Is there an existing flag/config to enable incremental mode?
• Should timestamps represent "first cached" vs "last validated"?

Related

• PR #38 (example where this occurs)
• linkml-reference-validator repository (potential upstream source)

---

Workaround: For now, users can explicitly choose to re-cache all terms, but the default should be incremental updates.

[github-actions]

Analysis Summary

I've analyzed the caching implementation in linkml-term-validator and confirmed the root cause of the spurious timestamp diffs.

---

Root Cause Identified

The issue is in src/linkml_term_validator/plugins/base.py lines 164-189. The _save_to_cache() method has a critical flaw:

Current Implementation (Problematic)

def _save_to_cache(self, prefix: str, curie: str, label: str) -> None:
    """Save a label to the cache."""
    cache_file = self._get_cache_file(prefix)
    
    # Load existing cache
    existing = self._load_cache(prefix)  # ← Loads ONLY curie:label mapping
    existing[curie] = label
    
    # Write back
    with open(cache_file, "w") as f:
        writer = csv.DictWriter(f, fieldnames=["curie", "label", "retrieved_at"])
        writer.writeheader()
        for curie, label in existing.items():
            writer.writerow({
                "curie": curie,
                "label": label,
                "retrieved_at": datetime.now().isoformat(),  # ← ALWAYS fresh timestamp
            })

The Problem

1. _load_cache() discards timestamps (line 161): Only reads curie and label, ignoring retrieved_at
2. _save_to_cache() regenerates timestamps (line 187): Writes datetime.now() for ALL entries, even existing ones
3. Result: Every cache write rewrites the entire file with fresh timestamps for all terms

This is exactly what's causing the 646-line diffs in monarch-initiative/dismech#40 when adding just 1-2 new terms.

---

Codebase Architecture

Cache Files Structure

Location: cache/{prefix_lowercase}/terms.csv

Format:

curie,label,retrieved_at
HP:0040282,Frequent,2025-12-16T10:24:02.844891
MONDO:0004975,Alzheimer disease,2025-12-16T10:24:03.123456

Caching Layers

The project implements two-level caching:

1. In-memory cache: self._label_cache dict (per-plugin instance)
2. File cache: CSV files in cache/ directory (persistent across runs)

Cache Strategies

Configured via ValidationConfig.cache_strategy:

• CacheStrategy.PROGRESSIVE (default): Cache terms on-demand as encountered
• CacheStrategy.GREEDY: Pre-expand and cache entire dynamic enums upfront

---

Impact Assessment

Affected Workflows

This issue impacts any workflow that:

• Commits cache files to git (like dismech)
• Adds new terms incrementally
• Reviews PRs with cache updates

Severity

High for repositories tracking cache in version control:

• Makes code review nearly impossible (signal lost in noise)
• Increases merge conflict probability
• Pollutes git history
• Defeats the purpose of caching for review purposes

Low for projects with .gitignored cache (like this repo):

• Cache works correctly for performance
• No version control issues

---

Proposed Solution

Option 1: Preserve Original Timestamps (Recommended)

Modify _load_cache() to preserve timestamps:

def _load_cache(self, prefix: str) -> dict[str, tuple[str, str]]:
    """Load cached labels with timestamps for a prefix.
    
    Returns:
        Dict mapping CURIEs to (label, retrieved_at) tuples
    """
    cache_file = self._get_cache_file(prefix)
    if not cache_file.exists():
        return {}
    
    cached = {}
    with open(cache_file) as f:
        reader = csv.DictReader(f)
        for row in reader:
            cached[row["curie"]] = (row["label"], row.get("retrieved_at", ""))
    return cached

def _save_to_cache(self, prefix: str, curie: str, label: str) -> None:
    """Save a label to the cache, preserving existing timestamps."""
    cache_file = self._get_cache_file(prefix)
    
    # Load existing cache with timestamps
    existing = self._load_cache(prefix)
    existing[curie] = (label, datetime.now().isoformat())  # Only new term gets new timestamp
    
    # Write back with preserved timestamps
    with open(cache_file, "w") as f:
        writer = csv.DictWriter(f, fieldnames=["curie", "label", "retrieved_at"])
        writer.writeheader()
        for curie, (label, timestamp) in existing.items():
            writer.writerow({
                "curie": curie,
                "label": label,
                "retrieved_at": timestamp,  # Preserved for existing, fresh for new
            })

Benefits:

• Minimal code changes
• Timestamps represent "first cached" rather than "last validated"
• Diffs show only genuinely new terms
• Backward compatible (works with existing cache files)

Option 2: Append-Only Mode

Use append-only writes for new terms:

def _save_to_cache(self, prefix: str, curie: str, label: str) -> None:
    """Append a new label to the cache."""
    cache_file = self._get_cache_file(prefix)
    
    # Check if already cached
    existing = self._load_cache(prefix)
    if curie in existing:
        return  # Already cached, skip write
    
    # Append new entry
    with open(cache_file, "a") as f:
        writer = csv.DictWriter(f, fieldnames=["curie", "label", "retrieved_at"])
        if not cache_file.exists() or cache_file.stat().st_size == 0:
            writer.writeheader()
        writer.writerow({
            "curie": curie,
            "label": label,
            "retrieved_at": datetime.now().isoformat(),
        })

Benefits:

• Maximum efficiency (no rewriting)
• Truly incremental updates
• Simple implementation

Drawbacks:

• Can't update existing labels if they change
• Can't remove invalid entries without manual cache clearing

Option 3: Separate Timestamp File

Decouple timestamps from label cache:

• cache/{prefix}/terms.csv: Just curie,label (no timestamps)
• cache/{prefix}/timestamps.csv: Optional curie,retrieved_at for metadata

Benefits:

• Clean diffs (label changes separate from timestamp noise)
• Timestamps become optional metadata
• Easier cache management

Drawbacks:

• More complex implementation
• Two files to manage per prefix

---

Recommended Action

I recommend Option 1 (preserve timestamps) as it:

• Requires minimal changes to existing code
• Maintains backward compatibility
• Solves the git diff problem
• Keeps timestamp semantics meaningful (first cached, not last validated)

Implementation Checklist

• Update _load_cache() to return dict[str, tuple[str, str]]
• Update _save_to_cache() to preserve timestamps from loaded cache
• Update get_ontology_label() to handle tuple unpacking
• Add similar fix to src/linkml_term_validator/validator.py (legacy validator)
• Add test case verifying timestamp preservation
• Update documentation explaining timestamp semantics

---

Related Files

Primary implementations:

• src/linkml_term_validator/plugins/base.py:164-189 (main issue)
• src/linkml_term_validator/validator.py:167 (legacy, similar issue)

Tests to update:

• tests/test_dynamic_enums.py::test_dynamic_enum_plugin_with_cache
• Add new test for timestamp preservation

Documentation:

• Document that retrieved_at means "first cached" not "last validated"

---

Next Steps

If you'd like me to implement the fix, I can create a PR with Option 1 (timestamp preservation). Please confirm the approach or suggest an alternative.

Alternatively, I can investigate whether there are any downstream dependencies on the current timestamp behavior before making changes.