Skip to content

Add an inter-annotator-agreement scorer with Cohen and Fleiss kappa #934

Description

@maziyarpanahi

Summary

When multiple annotators label the same corpus, OpenMed cannot quantify agreement, so the quality of golden fixtures is unverified. Inter-annotator agreement (kappa) is the standard reliability check for span-annotation tasks.

Scope

  • Compute span-level Cohen kappa (two annotators) and Fleiss kappa (multiple annotators) over aligned annotations
  • Add label-level and overall agreement breakdowns with disagreement examples by offset
  • Emit an agreement report consumable by the active-learning queue

Acceptance criteria

  • Kappa values match hand-computed reference results on a synthetic two- and three-annotator fixture
  • Disagreement listings reference offsets and labels only, no raw PHI
  • Perfect-agreement and total-disagreement edge cases return 1.0 and the expected low values

Out of scope

  • Adjudication workflow (separate active-learning concern)
  • Token-level alignment heuristics beyond exact and overlap matching

Files

  • openmed/eval/annotation/agreement.py
  • tests/unit/eval/test_agreement.py

Task: OM-577 · Milestone: v2.2 · Priority: P2 · Size: M
Depends on: OM-556 · Blocks: —
Roadmap: v2.2 trustworthy clinical data exchange follow-on wave
Spec: PLANS/V2/EXECUTION/tasks/OM-577.md

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2MediumfeatureNew capabilityroadmap-v2OpenMed V2 roadmap backlog

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions