Summary
When multiple annotators label the same corpus, OpenMed cannot quantify agreement, so the quality of golden fixtures is unverified. Inter-annotator agreement (kappa) is the standard reliability check for span-annotation tasks.
Scope
Acceptance criteria
Out of scope
- Adjudication workflow (separate active-learning concern)
- Token-level alignment heuristics beyond exact and overlap matching
Files
- openmed/eval/annotation/agreement.py
- tests/unit/eval/test_agreement.py
Task: OM-577 · Milestone: v2.2 · Priority: P2 · Size: M
Depends on: OM-556 · Blocks: —
Roadmap: v2.2 trustworthy clinical data exchange follow-on wave
Spec: PLANS/V2/EXECUTION/tasks/OM-577.md
Summary
When multiple annotators label the same corpus, OpenMed cannot quantify agreement, so the quality of golden fixtures is unverified. Inter-annotator agreement (kappa) is the standard reliability check for span-annotation tasks.
Scope
Acceptance criteria
Out of scope
Files
Task: OM-577 · Milestone: v2.2 · Priority: P2 · Size: M
Depends on: OM-556 · Blocks: —
Roadmap: v2.2 trustworthy clinical data exchange follow-on wave
Spec: PLANS/V2/EXECUTION/tasks/OM-577.md