Merge pull request #11 from monarch-initiative/dashboard

Add QC dashboard and fix fabricated evidence references

Author: dragon-ai-agent

CLAUDE.md

@@ -136,7 +136,92 @@ uv run runoak -i sqlite:obo:maxo search "physical therapy"
 ## Testing
 
 Tests are in `tests/test_data.py`:
-- Schema validation for all 55 disorder files
+- Schema validation for all 56 disorder files
 - Required field checks
 - Evidence reference validation
 - Unique name verification
+
+## Standard Operating Procedure: Adding/Editing Evidence
+
+When adding or editing evidence items in disorder files, follow this SOP to prevent hallucinations:
+
+### 1. Never Fabricate Snippets
+
+Evidence snippets MUST be exact quotes from the cited paper's abstract. Do not paraphrase.
+
+**Wrong:**
+```yaml
+evidence:
+  - reference: PMID:12345678
+    snippet: The study showed that X causes Y through Z mechanism.  # Paraphrase - will fail validation
+```
+
+**Correct:**
+```yaml
+evidence:
+  - reference: PMID:12345678
+    snippet: "X causes Y through the Z mechanism, as demonstrated by..."  # Exact quote from abstract
+```
+
+### 2. Verify PMIDs Before Use
+
+Always check that a PMID actually corresponds to the paper you think it does:
+
+```bash
+# Check cached abstract (if previously fetched)
+cat references_cache/pmid_12345678.md
+
+# Or fetch fresh and validate
+just validate-references kb/disorders/MyDisease.yaml
+```
+
+### 3. Validation Workflow
+
+Before committing changes to any disorder file:
+
+```bash
+# 1. Schema validation (structure correct)
+just validate kb/disorders/MyDisease.yaml
+
+# 2. Reference validation (snippets match abstracts)
+just validate-references kb/disorders/MyDisease.yaml
+
+# 3. Term validation (ontology IDs/labels correct)
+just validate-terms-file kb/disorders/MyDisease.yaml
+```
+
+### 4. When Evidence Cannot Be Verified
+
+If a claim is well-established but you cannot find a quotable snippet:
+
+- **Option A**: Move the claim to the `notes` field (no evidence required)
+- **Option B**: Find a different paper with a quotable abstract
+- **Option C**: Remove the evidence block entirely, keep the description
+
+**Do NOT** fabricate quotes or use incorrect PMIDs.
+
+### 5. Common Validation Errors
+
+| Error | Cause | Fix |
+|-------|-------|-----|
+| "Text part not found as substring" | Snippet is paraphrased | Use exact quote from abstract |
+| "Reference not found" | PMID doesn't exist | Verify PMID on PubMed |
+| Low similarity score | Wrong PMID for the paper | Check abstract matches topic |
+
+### 6. Running Full QC
+
+```bash
+# All validation checks
+just qc
+
+# Compliance analysis (recommended field coverage)
+just compliance-all
+
+# With weighted scoring and threshold checks
+just compliance-weighted
+
+# Generate visual dashboard (dashboard/index.html)
+just gen-dashboard
+```
+
+The dashboard shows priority curation targets - the 10 files with lowest compliance scores.

NOTES.md

@@ -0,0 +1,80 @@
+# Development Notes
+
+## 2025-12-08
+
+### linkml-data-qc Now on PyPI
+
+Updated project to use `linkml-data-qc` from PyPI instead of local development install:
+- Removed local path override from `pyproject.toml`
+- Added `[viz]` extras for dashboard visualization features
+- Version 0.1.0 installed with matplotlib, seaborn, pillow dependencies
+
+### QC Dashboard
+
+Added new `just gen-dashboard` target that generates a visual HTML dashboard:
+- Uses `linkml-data-qc --dashboard-dir dashboard/`
+- Creates `dashboard/index.html` with charts and tables
+- Shows slot compliance comparison across all 56 disorder files
+- Highlights the 10 lowest-compliance files as priority curation targets
+- Includes detailed per-file charts for priority files
+
+Dashboard contents:
+- `index.html` - Main dashboard page
+- `comparison.png` - Slot compliance bar chart
+- `detail_*.png` - Per-file heatmaps for priority files
+- `reports.json` - Raw report data
+
+### Reference Validation Findings
+
+Ran comprehensive reference validation (`just validate-references-all`) and discovered significant issues with fabricated evidence snippets in several Mendelian disease files.
+
+#### Key Issues Found
+
+1. **Fabricated snippets**: Evidence snippets were AI-generated paraphrases rather than actual quotes from cited papers. The reference validator correctly flagged these with low similarity scores (0-37%).
+
+2. **Wrong PMIDs**: Several PMIDs pointed to completely unrelated papers:
+   - `PMID:30084541` in Dravet_syndrome.yaml was about "Black Phosphorus Nanosheets Passivation Using a Tripeptide" - not Dravet syndrome
+   - `PMID:22267103` was about "How to use insulin-like growth factor 1 (IGF1)" - not Dravet syndrome
+   - `PMID:34812478` was about "catastrophic natural disasters impact on arts nonprofits" - not Dravet syndrome
+   - `PMID:31428203` in Fanconi_Anemia.yaml was about "insulin-glucose metabolism in diabetic mice" - not Fanconi anemia
+
+#### Files Fixed
+
+**Fanconi_Anemia.yaml:**
+- Replaced 5 fabricated snippets with real quotes from PMID:35596788 (Peake & Noguchi 2022 review) and PMID:20301575 (GeneReviews)
+- Removed 8 unverifiable evidence items, converted claims to `notes` fields
+- Quotes now use exact text from abstracts (with proper YAML quoting for colons)
+
+**Dravet_syndrome.yaml:**
+- Removed all evidence citing wrong PMIDs (30084541, 22267103, 34812478)
+- Used only PMID:21463282 (Oakley et al. 2011 "Insights into pathophysiology and therapy from a mouse model of Dravet syndrome")
+- Added 4 verified quotes from that paper
+- Moved unverifiable claims to `notes` fields
+
+#### Lessons Learned
+
+1. **Always validate references**: The reference validator is essential for catching AI hallucinations. Run `just validate-references file` before committing evidence items.
+
+2. **Use actual quotes**: Snippets must be exact quotes from abstracts, not paraphrases. The validator checks substring matching.
+
+3. **Verify PMIDs independently**: Don't trust that a PMID is correct - check the cached abstract in `references_cache/pmid_*.md` or fetch it fresh.
+
+4. **When in doubt, use notes**: If a claim is well-established but you can't find a quotable snippet, put it in `notes` rather than fabricating evidence.
+
+### Compliance Analysis
+
+Ran `just compliance-weighted` with the QC config:
+
+- **Global compliance**: 56.1%
+- **Weighted compliance**: 75.3%
+- **Term coverage**: 93.0%
+- **Evidence coverage**: 77.7%
+- **Description coverage**: 26.4%
+
+Critical paths are meeting thresholds:
+- `phenotypes[].phenotype_term.term`: 99.5% (threshold: 90%)
+- `disease_term.term`: 98.2% (threshold: 95%)
+- `pathophysiology[].cell_types[].term`: 100% (threshold: 85%)
+- `treatments[].treatment_term.term`: 100% (threshold: 80%)
+
+Violations are in sparse data paths (locations, chemical_entities, pathways) indicating areas for future data enrichment, not config issues.

README.md

@@ -4,7 +4,7 @@ A curated knowledge base of disease pathophysiology, with structured evidence fr
 
 ## Browse the Knowledge Base
 
-**[View all disorders online](https://monarch-initiative.github.io/dismech/disorders/)**
+**[View all disorders online](https://monarch-initiative.github.io/dismech/disorders/)** | **[QC Dashboard](https://monarch-initiative.github.io/dismech/dashboard/)**
 
 Each disorder page includes:
 - Disease mechanisms and pathophysiology
@@ -68,15 +68,56 @@ All claims must cite PubMed references with exact quotes. This prevents misinfor
 
 ### Validation Pipeline
 
-Multiple layers of automated validation:
-1. **Schema validation**: Ensures correct YAML structure
-2. **Ontology term validation**: Verifies term IDs exist and labels match authoritative sources
-3. **Reference validation**: Confirms quoted snippets appear in cited abstracts
+Multiple layers of automated validation ensure data quality and prevent AI hallucinations:
+
+1. **Schema validation**: Ensures correct YAML structure against the LinkML schema
+2. **Ontology term validation**: Verifies term IDs exist and labels match authoritative sources (HPO, MONDO, GO, etc.)
+3. **Reference validation**: Confirms that quoted snippets actually appear in cited PubMed abstracts
+4. **Compliance analysis**: Measures coverage of recommended fields (descriptions, evidence, ontology terms)
+
+```bash
+# Run schema + term validation
+just qc
+
+# Validate a single file
+just validate kb/disorders/Asthma.yaml
+
+# Validate references against PubMed abstracts
+just validate-references kb/disorders/Asthma.yaml
+
+# Analyze compliance with recommended field coverage
+just compliance-all
+
+# Compliance with weighted scoring and threshold checks
+just compliance-weighted
+```
+
+### Why Reference Validation Matters
+
+All evidence snippets must be **exact quotes** from paper abstracts, not paraphrases. The reference validator fetches abstracts from PubMed and checks that the quoted text appears verbatim. This catches:
+
+- AI-generated paraphrases that don't match the actual paper
+- Wrong PMIDs (e.g., a PMID that points to an unrelated paper)
+- Fabricated citations
+
+When validation fails, either fix the snippet to match the actual abstract or remove the evidence item.
+
+### QC Dashboard
+
+Generate a visual dashboard showing compliance metrics across all disorder files:
 
 ```bash
-just qc  # Run all quality checks
+just gen-dashboard
 ```
 
+This creates `dashboard/index.html` with:
+- Summary metrics (files analyzed, average compliance, violations)
+- Slot compliance comparison chart
+- Detailed views of the 10 lowest-compliance files (priority curation targets)
+- Full table of all files sorted by compliance
+
+View online: [QC Dashboard](https://monarch-initiative.github.io/dismech/dashboard/) or locally: `open dashboard/index.html`
+
 ### HTML Generation
 
 YAML files are rendered to browsable HTML pages with clickable ontology term links.