Description

Implement the validation step as part of the Aether FHIR processing pipeline. This step validates FHIR resources from the pseudonymized output against FHIR R4 schema and business rules.

Architecture

The validation step will follow the same pattern as the existing DIMP step:

• Input: FHIR NDJSON files from jobs/<job-id>/import/ or jobs/<job-id>/pseudonymized/
• Output: Validation reports in jobs/<job-id>/validation/
• Type: External HTTP service (contract to be defined)
• Position in Pipeline: After DIMP pseudonymization, before conversion steps

Acceptance Criteria

• Service contract defined (request/response format for validation service)
• HTTP client implementation for validation service communication
• Pipeline step execution logic implemented (ExecuteValidationStep())
• Support for resumption (skip already-validated files)
• Proper error handling and classification (transient vs non-transient)
• Progress tracking with output feedback
• Integration with job state management
• Comprehensive error messages for validation failures
• Unit tests with mock validation service
• Integration tests with actual validation workflow

Technical Details

• The validator can be found here: https://github.com/medizininformatik-initiative/mii-fhir-validator
• An example implementation here: https://github.com/medizininformatik-initiative/dataportal/tree/main/data-node
• The validator takes one resource at a time
• Follow existing patterns from internal/pipeline/dimp.go
• Implement in internal/pipeline/validation.go
• Update step execution logic in internal/pipeline/job.go
• Use existing error handling patterns (StepError, IsTransientHTTPStatus())
• Configuration via validation_service_url in YAML

curl --request POST \
  --url http://localhost:8080/validateResource \
  --header 'accept: application/fhir+json' \
  --header 'content-type: application/fhir+json' \
  --data '{
  "resourceType": "Condition",
  "id": "mii-condition-example-1",
  "meta": {
    "profile": [
      "https://www.medizininformatik-initiative.de/fhir/core/modul-diagnose/StructureDefinition/Diagnose"
    ]
  },
  "clinicalStatus": {
    "coding": [
      {
        "system": "http://terminology.hl7.org/CodeSystem/condition-clinical",
        "code": "active"
      }
    ]
  },
  "verificationStatus": {
    "coding": [
      {
        "system": "http://terminology.hl7.org/CodeSystem/condition-ver-status",
        "code": "confirmed"
      }
    ]
  },
  "code": {
    "coding": [
      {
        "system": "http://fhir.de/CodeSystem/bfarm/icd-10-gm",
        "version": "2024",
        "code": "I50.01"
      }
    ]
  },
  "subject": {
    "reference": "Patient/mii-patient-example-1"
  },
  "recordedDate": "2024-01-15T10:30:00+01:00",
  "extension": [
    {
      "url": "http://hl7.org/fhir/StructureDefinition/condition-assertedDate",
      "valueDateTime": "2024-01-15"
    }
  ]
}'

References

• Related to: Bundle Splitting (004-bundle-splitting) integration
• Pattern reference: internal/pipeline/dimp.go
• Job state management: internal/pipeline/job.go