Empty records should be hidden in the browser

[caufieldjh]

The kb/disorders directory contains some YAML entries without content. I suspect these were causing the HTML rendering workflow to fail previously, but now they become "Unknown" pages in the browser:

In this case, the YAML link points to https://github.com/monarch-initiative/dismech/blob/main/kb/disorders/ALK_Rearranged_NSCLC.history.yaml

Because this entry doesn't have a name, the page link is to https://monarch-initiative.github.io/dismech/pages/disorders/Unknown.html (which is a 404).

Records without content should not appear in the browser.

[github-actions]

Issue Analysis: Empty Records in Browser

Summary

This issue reports that .history.yaml files in kb/disorders/ are being rendered as "Unknown" pages in the browser, creating broken links and cluttering the UI. The history files are metadata-only files that track edit provenance but don't contain actual disorder content.

Root Cause

The rendering pipeline processes all *.yaml files in kb/disorders/ without filtering out .history.yaml files. Specifically:

Files Affected

All 150+ .history.yaml files (one per disorder) like:

• ALK_Rearranged_NSCLC.history.yaml
• Asthma.history.yaml
• Cystic_Fibrosis.history.yaml

What History Files Contain

source_file: Asthma.yaml
edit_history:
- timestamp: 2025-12-01T05:00:11.391Z
  model: claude-opus-4-5-20251101
  action: EDITED
  agent_tool: claude-code
  agent_version: 2.0.55

These files have no name field, causing the renderer to use fallback logic that generates "Unknown" pages.

Current State of Codebase

Files That CORRECTLY Filter History Files ✅

src/dismech/embed.py:100-103

for yaml_path in sorted(kb_dir.glob("*.yaml")):
    # Skip history files
    if yaml_path.name.endswith(".history.yaml"):
        continue

Files That DO NOT Filter History Files ❌

1. src/dismech/render.py:630 (render_all_disorders)

   yaml_files = sorted(input_dir.glob('*.yaml'))

2. src/dismech/export/browser_export.py:331 (main function)

   disorder_files = sorted(input_dir.glob("*.yaml"))

3. src/dismech/render.py:357 (render_all_comorbidities)

4. src/dismech/render.py:536 (render_classification_pages)

5. src/dismech/graph.py:221 (build graphs from disorders)

Impact

User-Facing Issues

• Browser displays 150+ "Unknown" entries
• Clicking these entries leads to 404 errors
• Clutters the UI and makes navigation confusing
• Reduces trust in the knowledge base

Related Historical Issues

• Issue Building pages fails when disorder name is not present #274 (closed): "Building pages fails when disorder name is not present" - Same underlying cause, partially fixed
• Issue Building pages fails when encountering empty field #102 (closed): "Building pages fails when encountering empty field" - Related null handling issue in browser_export.py
• Issue Epic: plan for agent provenance on yamls #68 (open): "Epic: plan for agent provenance on yamls" - Context on why history files exist

Recommended Solution

Add filtering logic to all rendering/export functions to skip .history.yaml files:

for yaml_path in sorted(input_dir.glob('*.yaml')):
    # Skip history files
    if yaml_path.name.endswith(".history.yaml"):
        continue
    # ... process disorder ...

Files to Update

1. src/dismech/render.py (3 locations: lines 357, 536, 630)
2. src/dismech/export/browser_export.py (line 331)
3. src/dismech/graph.py (line 221)

Alternative Solution

Use a more specific glob pattern:

# Only match disorder files, not history files
yaml_files = [p for p in sorted(input_dir.glob('*.yaml')) 
              if not p.name.endswith('.history.yaml')]

Testing Plan

1. Verify that history files are excluded from rendering
2. Confirm that the browser no longer shows "Unknown" entries
3. Check that all actual disorder pages still render correctly
4. Validate that graph building and exports work correctly

Priority

High - This affects user experience and data quality perception in the production browser.

---

Note: History files serve an important purpose (tracking AI agent provenance per issue #68) and should not be deleted. They just need to be filtered from rendering/export pipelines.

[caufieldjh]

The history files should be omitted from the QC dashboard, too.

[caufieldjh]

History pages are still producing Unknown entries because they're still present in the pages directory, though they are not built anew anymore.

[caufieldjh]

I think #329 will fix this as a side-effect of rebuilding the main index pages

[caufieldjh]

Should be fixed by #345 - the remaining issue was that the just gen-browser-data target was still working with history files.