Skip to content

Add a clinical-domain label-map coverage eval suite over the synthetic fixtures #866

Description

@maziyarpanahi

Summary

As clinical sub-domain label maps and synthetic fixtures multiply, there is no single eval that verifies every shipped domain has a matching fixture, every fixture label resolves to a catalog label, and per-label span coverage is non-empty. Without this gate, domain maps and fixtures drift apart silently.

Scope

  • Add eval/domain_coverage.py that enumerates label_maps/defaults.json domains, loads each tests/fixtures/clinical/.jsonl, and reports per-domain/per-label span coverage
  • Cross-check every fixture label against a catalog label in core/labels.py and flag orphans
  • Emit a leakage-first-style coverage report (offsets/labels only, no raw PHI) and a machine-readable summary
  • Wire the suite into the existing benchmark CLI and add a CI gate that fails on missing-fixture or orphan-label

Acceptance criteria

  • Running the suite over all current clinical domains passes offline with zero orphan labels
  • Adding a domain without a fixture, or a fixture label not in the catalog, fails the gate with a clear message
  • Report contains only labels/offsets/counts, never raw fixture text in logs
  • Suite registered in the eval harness and discoverable via the eval CLI

Out of scope

  • No F1/accuracy scoring against a gold model (coverage and consistency only)
  • No restricted datasets; operates on committed synthetic fixtures
  • No new model training

Files

  • openmed/eval/domain_coverage.py
  • tests/unit/eval/test_domain_coverage.py
  • .github/workflows/eval.yml
  • docs/clinical-domains.md

Task: OM-508 · Milestone: v2.2 · Priority: P1 · Size: M
Depends on: OM-018, OM-176, OM-412 · Blocks: —
Roadmap: v2.2 trustworthy clinical data exchange follow-on wave
Spec: PLANS/V2/EXECUTION/tasks/OM-508.md

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1HighfeatureNew capabilityroadmap-v2OpenMed V2 roadmap backlog

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions