Traceability and Auditing for Variant-level Job Results #627
Description
Goal
Enable maintainers (and users) to track all jobs run on a given score set, view the results and status of each job, understand why annotations may be missing from certain variants, and display overall pipeline progress in the UI.
Key Requirements
- Each job run is recorded with metadata.
- Jobs can be grouped by some mechanism to support a progress view in a user interface.
- Per-variant annotation status and missing reason are tracked in a dedicated table.
- Efficient understanding of the 'latest' job run, and the 'current' annotation statuses + reasons for variants
- API endpoints allow querying jobs by correlation_id, viewing job details, and listing per-variant annotation status and reasons.
- Support for polling/streaming of status into the UI for API driven updates of pipeline progress.
Benefits
- Maintainers can audit all jobs and their outcomes for any score set.
- Users can see real-time or near-real-time progress of their pipelines.
- Missing annotations are explained at the variant level, improving transparency and troubleshooting.