Fast computation of Krippendorff's alpha agreement measure in Python.

CrowdTruth framework for crowdsourcing ground truth for training & evaluation of AI systems

Inter-annotator agreement for Brat annotation projects

Exploring the inter-annotator agreement between ISIC Archive segmentation masks

This repository holds code to annotate textual data using LLMs, and calculate different measures of Inter-Annotator Agreement (IAA).

command line stuff to calculate inter annotator agreement

Official Repository of 'Leveraging Epistemic Uncertainty to Improve Tumour Segmentation in Breast MRI: An exploratory analysis' In SPIE Medical Imaging 2024

Evaluation toolkit for multi-annotator human annotation research with agreement metrics, disagreement analysis, reports, and plots.

Browser-based IAA and annotation workflow tools for Yongle Palace mural patches

Production IAA engine — Krippendorff's Alpha, SBERT semantic agreement, Fleiss' Kappa, Shannon entropy, 3-layer collusion detection. Ran behind RawEval's 9-annotator workbench.

[MICCAI ISIC 2024] Code for "Segmentation Style Discovery: Application to Skin Lesion Images"

Digita Literacy for School of Foreign Languages (NRU HSE, 2018)

Concordia: reproducible inter-annotator agreement toolkit for human and machine cultural-heritage annotation

Generate controlled synthetic annotator disagreement before real reviewer data is collected.

Python package implementing measure from my working paper "Kappa-IoU: Inter-Rater Reliability for Spatial Annotation"

Measure inter-annotator agreement with modern metrics, uncertainty, ordinal support, and missing-data handling.

Production-grade pipeline for validating annotation consistency and evaluating LLM output quality using agreement metrics, schema validation, LLM-as-judge scoring, SQLite logging, and a Streamlit dashboard.