concordia-iaa

Reproducible inter-annotator agreement (IAA) toolkit for human + machine cultural-heritage annotation. Computes Cohen's/Fleiss' κ and Krippendorff's α with bootstrap 95% CIs, CI-based acceptance gates, sensitivity analysis, a self-contained reproducibility bundle, and swappable domain profiles. See paper/paper.md (SoftwareX manuscript).

Release status (updated 2026-06-27). This repository is public as lbiceice/concordia-iaa, with release v1.0.1 as the current SoftwareX submission release. The Zenodo software concept DOI is 10.5281/zenodo.20550918; it is distinct from the companion dataset DOI 10.5281/zenodo.19718760.

Contents

• src/yongle-iaa-toolkit-v2.9.60.html — single-file browser GUI and IAA core (deterministic; fixed fixture κ=0.768971 / α=0.769219).
• src/iaa-core.js — importable headless core (CommonJS/Node): cohenKappa, fleissKappa, krippendorffAlpha, percentAgreement, confusionMatrix, interpret, bootstrapCI. Verbatim extraction of the HTML's IAA core; kept in sync by tests/iaa_core_identity.js.
• bin/iaa-cli.js — thin CLI (concordia-iaa input.json --level ordinal --boot 2000).
• tests/v2960_smoke.js — smoke suite (module presence, sensitivity analysis, bundle/readiness wrappers, 33-version identity chain).
• tests/iaa_core_identity.js — asserts src/iaa-core.js is behaviour-identical to the shipped HTML core.
• tests/crossval_compute.js + tests/crossval_reference.py — numerical cross-validation against scikit-learn / krippendorff / simpledorff.
• tests/requirements.txt — Python reference libraries for cross-validation.
• examples/anchor-fixture.json — minimal CLI example using the locked fixture values.
• docs/index.md — installation, API, CLI and release notes.
• paper/paper.md — SoftwareX manuscript draft.
• package.json · LICENSE (MIT) · CITATION.cff · .github/workflows/ci.yml.

Programmatic use

const IAA = require('concordia-iaa');         // or require('./src/iaa-core')
IAA.cohenKappa(raterA, raterB);
IAA.krippendorffAlpha(raterA, raterB, { level: 'ordinal' });
IAA.bootstrapCI(raterA, raterB, { metric: 'krippendorffAlpha', iters: 2000, seed: 7777 });

concordia-iaa input.json --level nominal --boot 2000   # input: {"raterA":[...],"raterB":[...]}

Running the tests

npm ci                  # install jsdom (the only JS dependency)
npm test                # smoke suite + headless-core identity check (Node + jsdom)

# numerical cross-validation (requires Python >=3.12)
python3 -m pip install -r tests/requirements.txt
npm run crossvalidate   # JS core vs scikit-learn / krippendorff / simpledorff

npm test and npm run crossvalidate both run in CI (.github/workflows/ci.yml) on Node 20, 22, and 24.

How the cross-validation works

The tests exercise the exact single-file artifact users run: crossval_compute.js loads src/yongle-iaa-toolkit-v2.9.60.html in jsdom (preserving the byte-identical-core guarantee), computes Cohen's κ and Krippendorff's α (nominal + ordinal) on four deterministic fixtures, and writes tests/crossval_results.json. crossval_reference.py then recomputes the same coefficients on the same fixtures with three independent libraries and asserts agreement (κ to 1e-9, α to 1e-6). On the canonical locked fixture (n=300, seed=7777) the agreement is bit-level exact (κ=0.768971, α=0.769219).

Done (engineering)

• ✅ Tests are CI-visible (npm ci → npm test → npm run crossvalidate), with a committed package.json / package-lock.json.
• ✅ Cross-validation of κ/α (nominal + ordinal, incl. missing data) against scikit-learn, krippendorff and simpledorff on shared fixtures, committed as a CI step (exact agreement; the strongest single correctness signal).
• ✅ Importable headless core src/iaa-core.js + CLI bin/iaa-cli.js, extracted verbatim from the HTML and guarded against divergence by tests/iaa_core_identity.js (19/19 coefficients identical).
• ✅ Bootstrap method verified: bootstrapCI is unit (case) resampling — standard for κ; for α a unit bootstrap (not the Hayes–Krippendorff coincidence-matrix method). Documented honestly; no H–K claim is made.

Repository and archival identifiers

• GitHub repository: https://github.com/lbiceice/concordia-iaa
• Public release tag: v1.0.1
• Companion dataset DOI, not the software DOI: 10.5281/zenodo.19718760
• Zenodo software concept DOI: 10.5281/zenodo.20550918

TODO — engineering

• Add an H–K coincidence-matrix bootstrap option for α and raise the default iteration count to ≥2000 (current default 500; the companion P-A CIs used B=1000 in Python).
• Expand docs/ with profile-schema and worked-example screenshots.

Release notes

• The repository name concordia-iaa is retained to avoid confusion with the broader term "Concordia"; the scoped package and manuscript title identify this project as an inter-annotator-agreement toolkit.
• The Zenodo archival software concept DOI is recorded in CITATION.cff, paper/paper.md, and the SoftwareX manuscript. .zenodo.json keeps the release metadata and companion-dataset relation used by Zenodo.