Skip to content

feat(pipeline): implement FHIR validation pipeline step#200

Open
trobanga wants to merge 1 commit intomainfrom
7-implement-validation-pipeline-step
Open

feat(pipeline): implement FHIR validation pipeline step#200
trobanga wants to merge 1 commit intomainfrom
7-implement-validation-pipeline-step

Conversation

@trobanga
Copy link
Collaborator

Summary

  • Adds a new validation pipeline step that validates FHIR resources against an external FHIR validation service (mii-fhir-validator) via HTTP POST to /validateResource
  • Produces per-file OperationOutcome reports (*.validation.ndjson) in jobs/<job-id>/validation/ and fails the step if any resource has error-level or fatal-level issues
  • Supports configurable concurrent validation requests (max_concurrent_requests, default: 4), resumption (skips files with existing reports), and transparent step semantics (subsequent steps look past validation to find their input data)

Configuration

services:
  validation:
    url: "http://validator:8080/fhir"
    max_concurrent_requests: 4  # optional, default: 4

pipeline:
  enabled_steps:
    - local_import
    - validation  # can go anywhere after import
    - dimp

New files

  • internal/services/validation_client.go — HTTP client for FHIR validation service
  • internal/pipeline/validation.goExecuteValidationStep with concurrent validation
  • tests/unit/validation_client_test.go — 9 client unit tests
  • tests/unit/pipeline_validation_test.go — 10 pipeline step unit tests

Modified files

  • internal/models/config.go — Added ValidationConfig struct
  • internal/services/config.go — Added validation config loading
  • internal/services/state.go — Added validation to dirs, unified GetStepInputDir into a single backwards walk
  • internal/models/validation.go — Added validation step to service URL checks
  • cmd/pipeline.go — Replaced stub with real ExecuteValidationStep call
  • Test files updated to include validation URL where needed

Closes #7

@trobanga trobanga force-pushed the 7-implement-validation-pipeline-step branch from 6f0650f to c4a93a1 Compare February 26, 2026 11:48
@codecov-commenter
Copy link

codecov-commenter commented Feb 26, 2026

Codecov Report

❌ Patch coverage is 86.06965% with 56 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.73%. Comparing base (2ac2dc4) to head (22f310e).
⚠️ Report is 6 commits behind head on main.

Files with missing lines Patch % Lines
internal/pipeline/validation.go 82.06% 32 Missing and 20 partials ⚠️
internal/services/validation_client.go 94.44% 2 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #200      +/-   ##
==========================================
+ Coverage   84.68%   84.73%   +0.04%     
==========================================
  Files          46       48       +2     
  Lines        4773     5167     +394     
==========================================
+ Hits         4042     4378     +336     
- Misses        567      603      +36     
- Partials      164      186      +22     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@trobanga trobanga force-pushed the 7-implement-validation-pipeline-step branch 2 times, most recently from 4dd8415 to 88d2098 Compare February 26, 2026 12:19
@trobanga trobanga force-pushed the 7-implement-validation-pipeline-step branch 13 times, most recently from d272df2 to d0a07f7 Compare March 2, 2026 14:47
@trobanga trobanga enabled auto-merge March 2, 2026 15:14
@trobanga trobanga force-pushed the 7-implement-validation-pipeline-step branch from d0a07f7 to 0927b63 Compare March 9, 2026 10:53
juliangruendner
juliangruendner previously approved these changes Mar 9, 2026
@trobanga trobanga force-pushed the 7-implement-validation-pipeline-step branch from 0927b63 to 22f310e Compare March 9, 2026 14:12
- Add validation step that sends FHIR resources to a validation service
- Chunk resources into Bundles with configurable size and concurrency
- Write per-file OperationOutcome reports for all validation results
- Default fail_on_error to true (stop pipeline on data quality errors)
- Support resumption by skipping files with existing reports
- Add fullUrl to inner Bundle entries for proper reference resolution
@trobanga trobanga force-pushed the 7-implement-validation-pipeline-step branch from 22f310e to 76e3899 Compare March 10, 2026 11:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement Validation Pipeline Step

3 participants