Skip to content

Commit 95f9d8c

Browse files
Copilotlitlfred
Copilot
and
litlfred
committed
Implement comprehensive FML Execution Validation Test Suite
Co-authored-by: litlfred <662242+litlfred@users.noreply.github.com>
1 parent 188331d commit 95f9d8c

15 files changed

+1103
-0
lines changed

‎README.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -184,6 +184,9 @@ The project includes comprehensive test coverage across platforms:
184184
- **Execution Tests**: Transformation logic and cross-platform behavior
185185
- **Terminology Tests**: ConceptMap, ValueSet, CodeSystem operations
186186
- **Platform Tests**: JVM and JavaScript specific functionality
187+
- **FML Execution Validation Test Suite**: FHIR IG-based test plan with real-world test cases
188+
189+
### Core Testing
187190

188191
Run specific test suites:
189192
```bash
@@ -203,6 +206,29 @@ gradle test --info
203206
gradle jvmTest --tests "FmlRunnerTest"
204207
```
205208

209+
### FML Execution Validation Test Suite
210+
211+
The project includes a comprehensive FHIR IG-based validation test suite located in `input/`:
212+
213+
```bash
214+
# Import test cases from community FML projects
215+
npm run import:test-data
216+
217+
# Explore available test repositories
218+
npm run explore:test-repos
219+
```
220+
221+
The test suite includes:
222+
- **TestPlan Definition**: `input/fsh/tests/FMLExecutionValidationTestPlan.fsh`
223+
- **Test Data**: `input/testdata/` with properly attributed test cases from:
224+
- ahdis/matchbox (Apache 2.0 license)
225+
- FHIR/fhir-test-cases (HL7 copyright/CC0 license)
226+
- Local examples (MIT license)
227+
228+
**Published Test Documentation**: https://litlfred.github.io/fmlrunner/TestPlan/FMLExecutionValidationTestPlan.html
229+
230+
For detailed information, see [`input/README.md`](input/README.md).
231+
206232
## API Reference
207233

208234
### Core FmlRunner API

‎input/README.md‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
# FML Execution Validation Test Suite
2+
3+
This directory contains a comprehensive FHIR IG-based validation test suite for FML (FHIR Mapping Language) using FHIR TestPlan and TestScript resources.
4+
5+
## Overview
6+
7+
The test suite validates FML execution capabilities using real-world test cases sourced from community FML projects, with proper license compliance and attribution.
8+
9+
## Directory Structure
10+
11+
```
12+
input/
13+
├── fsh/
14+
│ └── tests/
15+
│ └── FMLExecutionValidationTestPlan.fsh # FSH TestPlan definition
16+
└── testdata/ # All test data with license attribution
17+
├── examples/ # Local example test cases
18+
│ ├── patient-transform.map # Example FML mapping
19+
│ ├── patient-input.json # Example input data
20+
│ └── patient-output.json # Expected output data
21+
├── fhir-test-cases/ # Test cases from FHIR/fhir-test-cases
22+
└── matchbox/ # Test cases from ahdis/matchbox
23+
```
24+
25+
## Test Data Sources
26+
27+
### Local Examples (`input/testdata/examples/`)
28+
- **License**: MIT (local examples)
29+
- **Purpose**: Immediate testing and development
30+
- **Files**: Basic patient transformation examples
31+
32+
### FHIR Test Cases (`input/testdata/fhir-test-cases/`)
33+
- **Source**: [FHIR/fhir-test-cases/r5/structure-mapping](https://github.com/FHIR/fhir-test-cases/tree/main/r5/structure-mapping)
34+
- **License**: HL7 FHIR License (CC0 "No Rights Reserved")
35+
- **Attribution**: HL7 FHIR trademark acknowledgment included
36+
37+
### Matchbox Test Cases (`input/testdata/matchbox/`)
38+
- **Source**: [ahdis/matchbox test resources](https://github.com/ahdis/matchbox/tree/main/matchbox-server/src/test/resources)
39+
- **License**: Apache License 2.0
40+
- **Attribution**: Apache 2.0 license header included
41+
42+
## Test File Mapping
43+
44+
Test cases follow these naming conventions:
45+
- `*-map.txt` or `*.map` — FML mapping specification
46+
- `*-input.json` or `*-input.xml` — FHIR resource to be mapped
47+
- `*-output.json` or `*-output.xml` — Expected output after applying the map
48+
49+
Files are paired by base name (e.g., `patient-transform.map`, `patient-input.json`, `patient-output.json`).
50+
51+
## Usage
52+
53+
### Import Test Data
54+
55+
```bash
56+
# Import test cases from external repositories
57+
npm run import:test-data
58+
59+
# Explore available test files in repositories
60+
npm run explore:test-repos
61+
```
62+
63+
### TestPlan Structure
64+
65+
The `FMLExecutionValidationTestPlan` defines:
66+
- **Test Cases**: Each mapping scenario with input/output validation
67+
- **Test Data**: References to map files, input data, and expected outputs
68+
- **Dependencies**: FHIR R5 StructureMap specification requirements
69+
70+
### TestScript Execution
71+
72+
The `FMLPatientTransformTestScript` provides:
73+
- **Executable Tests**: Actual test execution logic
74+
- **FHIRPath Assertions**: Output validation using FHIRPath expressions
75+
- **Fixtures**: Test data loading and reference management
76+
77+
## License Compliance
78+
79+
All imported files include proper license attribution:
80+
81+
- **FHIR Test Cases**: HL7 copyright and CC0 license
82+
- **Matchbox Test Cases**: Apache 2.0 license
83+
- **Local Examples**: MIT license
84+
85+
License headers are automatically added by the import script to ensure compliance with original contribution requirements.
86+
87+
## Development
88+
89+
### Adding New Test Cases
90+
91+
1. Place test files in appropriate subdirectory under `input/testdata/`
92+
2. Ensure proper license attribution headers
93+
3. Update `FMLExecutionValidationTestPlan.fsh` with new test case definitions
94+
4. Add corresponding TestScript assertions for validation
95+
96+
### Test Execution
97+
98+
Test execution will be available through:
99+
- FHIR IG publisher validation
100+
- FML Runner library test suites
101+
- Continuous integration workflows
102+
103+
## Published Documentation
104+
105+
When published, the test plan documentation will be available at:
106+
**https://litlfred.github.io/fmlrunner/TestPlan/FMLExecutionValidationTestPlan.html**
107+
108+
This provides comprehensive documentation of all test cases, validation requirements, and execution results for the FML execution validation test suite.

‎input/fsh/tests/FMLExecutionValidationTestPlan.fsh‎

Lines changed: 197 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,197 @@
1+
// FML Execution Validation Test Plan
2+
// Defines a comprehensive test suite for validating FML (FHIR Mapping Language) execution
3+
4+
Instance: FMLExecutionValidationTestPlan
5+
InstanceOf: TestPlan
6+
Usage: #example
7+
Title: "FML Execution Validation Test Plan"
8+
Description: "Comprehensive test plan for validating FML execution with real-world test cases from community FML projects"
9+
* meta.versionId = "1.0.0"
10+
* url = "http://litlfred.github.io/fmlrunner/TestPlan/FMLExecutionValidationTestPlan"
11+
* identifier.value = "fml-execution-validation-test-plan"
12+
* version = "1.0.0"
13+
* name = "FMLExecutionValidationTestPlan"
14+
* title = "FML Execution Validation Test Plan"
15+
* status = #active
16+
* experimental = false
17+
* publisher = "FML Runner Project"
18+
* contact.name = "FML Runner Development Team"
19+
* contact.telecom.system = #url
20+
* contact.telecom.value = "https://github.com/litlfred/fmlrunner"
21+
* description = "This test plan validates FML execution capabilities using test cases sourced from community FML projects including ahdis/matchbox and FHIR/fhir-test-cases repositories. Each test case validates the complete FML transformation pipeline from input data through map execution to expected output validation."
22+
23+
* purpose = "Ensure FML execution engine correctly transforms input data according to mapping specifications and produces expected outputs that conform to FHIR resource definitions."
24+
25+
* scope.artifact = "http://hl7.org/fhir/StructureDefinition/StructureMap"
26+
* scope.conformance.requirements = "Validate that FML execution produces outputs that match expected results for known test cases"
27+
28+
// Test Case 1: Basic Patient Transform
29+
* testCase[0].id = "patient-basic-transform"
30+
* testCase[0].sequence = 1
31+
* testCase[0].scope.artifact = "http://example.org/StructureMap/PatientTransform"
32+
* testCase[0].scope.phase = #unit-test
33+
* testCase[0].requirement.linkId = "REQ-001"
34+
* testCase[0].requirement.description = "Basic patient data transformation using FML mapping"
35+
36+
* testCase[0].testRun[0].id = "patient-basic-transform-run"
37+
* testCase[0].testRun[0].description = "Execute patient transform with valid input and validate output"
38+
39+
// Test Data References
40+
* testCase[0].testData[0].id = "patient-transform-map"
41+
* testCase[0].testData[0].description = "Patient transformation mapping file"
42+
* testCase[0].testData[0].type = #test-data
43+
* testCase[0].testData[0].content.sourceAttachment.url = "testdata/examples/patient-transform.map"
44+
* testCase[0].testData[0].content.sourceAttachment.contentType = "text/plain"
45+
46+
* testCase[0].testData[1].id = "patient-input-data"
47+
* testCase[0].testData[1].description = "Input patient data for transformation"
48+
* testCase[0].testData[1].type = #test-data
49+
* testCase[0].testData[1].content.sourceAttachment.url = "testdata/examples/patient-input.json"
50+
* testCase[0].testData[1].content.sourceAttachment.contentType = "application/fhir+json"
51+
52+
* testCase[0].testData[2].id = "patient-expected-output"
53+
* testCase[0].testData[2].description = "Expected output after patient transformation"
54+
* testCase[0].testData[2].type = #test-data
55+
* testCase[0].testData[2].content.sourceAttachment.url = "testdata/examples/patient-output.json"
56+
* testCase[0].testData[2].content.sourceAttachment.contentType = "application/fhir+json"
57+
58+
// Additional test cases can be added here for:
59+
// - Complex transformations with nested data
60+
// - Error handling scenarios
61+
// - Performance testing with large datasets
62+
// - Terminology-aware transformations using ConceptMaps
63+
// - Test cases from imported external repositories
64+
65+
// Test Case 2: Basic Observation Transform
66+
* testCase[1].id = "observation-basic-transform"
67+
* testCase[1].sequence = 2
68+
* testCase[1].scope.artifact = "http://example.org/StructureMap/ObservationTransform"
69+
* testCase[1].scope.phase = #unit-test
70+
* testCase[1].requirement.linkId = "REQ-002"
71+
* testCase[1].requirement.description = "Basic observation data transformation using FML mapping"
72+
73+
* testCase[1].testRun[0].id = "observation-basic-transform-run"
74+
* testCase[1].testRun[0].description = "Execute observation transform with valid input and validate output"
75+
76+
// Test Data References for Observation
77+
* testCase[1].testData[0].id = "observation-transform-map"
78+
* testCase[1].testData[0].description = "Observation transformation mapping file"
79+
* testCase[1].testData[0].type = #test-data
80+
* testCase[1].testData[0].content.sourceAttachment.url = "testdata/examples/observation-transform.map"
81+
* testCase[1].testData[0].content.sourceAttachment.contentType = "text/plain"
82+
83+
* testCase[1].testData[1].id = "observation-input-data"
84+
* testCase[1].testData[1].description = "Input observation data for transformation"
85+
* testCase[1].testData[1].type = #test-data
86+
* testCase[1].testData[1].content.sourceAttachment.url = "testdata/examples/observation-input.json"
87+
* testCase[1].testData[1].content.sourceAttachment.contentType = "application/fhir+json"
88+
89+
* testCase[1].testData[2].id = "observation-expected-output"
90+
* testCase[1].testData[2].description = "Expected output after observation transformation"
91+
* testCase[1].testData[2].type = #test-data
92+
* testCase[1].testData[2].content.sourceAttachment.url = "testdata/examples/observation-output.json"
93+
* testCase[1].testData[2].content.sourceAttachment.contentType = "application/fhir+json"
94+
95+
// Additional test cases can be added here for:
96+
// - Complex transformations with nested data
97+
// - Error handling scenarios
98+
// - Performance testing with large datasets
99+
// - Terminology-aware transformations using ConceptMaps
100+
// - Test cases from imported external repositories
101+
102+
// Dependency on FHIR R5 StructureMap specification
103+
* dependency[0].description = "FHIR R5 StructureMap Resource"
104+
* dependency[0].predecessor = "http://hl7.org/fhir/5.0.0/StructureDefinition/StructureMap"
105+
106+
// Test Script Reference for Execution
107+
Instance: FMLPatientTransformTestScript
108+
InstanceOf: TestScript
109+
Usage: #example
110+
Title: "FML Patient Transform Test Script"
111+
Description: "Executable test script for patient transformation validation"
112+
* url = "http://litlfred.github.io/fmlrunner/TestScript/FMLPatientTransformTestScript"
113+
* identifier.value = "fml-patient-transform-test-script"
114+
* version = "1.0.0"
115+
* name = "FMLPatientTransformTestScript"
116+
* title = "FML Patient Transform Test Script"
117+
* status = #active
118+
* experimental = false
119+
* publisher = "FML Runner Project"
120+
* description = "Test script that executes patient transformation and validates results using FHIRPath assertions"
121+
122+
* origin[0].index = 1
123+
* origin[0].profile.system = "http://terminology.hl7.org/CodeSystem/testscript-profile-origin-types"
124+
* origin[0].profile.code = #FHIR-Client
125+
126+
* destination[0].index = 1
127+
* destination[0].profile.system = "http://terminology.hl7.org/CodeSystem/testscript-profile-destination-types"
128+
* destination[0].profile.code = #FHIR-Server
129+
130+
// Variable definitions for test data
131+
* variable[0].name = "inputPatient"
132+
* variable[0].description = "Input patient data"
133+
* variable[0].sourceId = "patient-input-fixture"
134+
135+
* variable[1].name = "expectedOutput"
136+
* variable[1].description = "Expected transformation output"
137+
* variable[1].sourceId = "patient-output-fixture"
138+
139+
* variable[2].name = "transformMap"
140+
* variable[2].description = "Patient transformation map"
141+
* variable[2].sourceId = "patient-map-fixture"
142+
143+
// Fixtures for test data
144+
* fixture[0].id = "patient-input-fixture"
145+
* fixture[0].autocreate = false
146+
* fixture[0].autodelete = false
147+
* fixture[0].resource.reference = "testdata/examples/patient-input.json"
148+
149+
* fixture[1].id = "patient-output-fixture"
150+
* fixture[1].autocreate = false
151+
* fixture[1].autodelete = false
152+
* fixture[1].resource.reference = "testdata/examples/patient-output.json"
153+
154+
* fixture[2].id = "patient-map-fixture"
155+
* fixture[2].autocreate = false
156+
* fixture[2].autodelete = false
157+
* fixture[2].resource.reference = "testdata/examples/patient-transform.map"
158+
159+
// Test execution steps
160+
* test[0].id = "PatientTransformTest"
161+
* test[0].name = "Patient Transform Validation Test"
162+
* test[0].description = "Execute patient transformation and validate output"
163+
164+
// Step 1: Execute the transformation
165+
* test[0].action[0].operation.type.system = "http://terminology.hl7.org/CodeSystem/testscript-operation-codes"
166+
* test[0].action[0].operation.type.code = #create
167+
* test[0].action[0].operation.description = "Execute patient transformation using StructureMap"
168+
* test[0].action[0].operation.destination = 1
169+
* test[0].action[0].operation.encodeRequestUrl = false
170+
* test[0].action[0].operation.url = "/StructureMap/$transform"
171+
* test[0].action[0].operation.sourceId = "patient-input-fixture"
172+
173+
// Step 2: Validate transformation results
174+
* test[0].action[1].assert.description = "Validate transformation was successful"
175+
* test[0].action[1].assert.direction = #response
176+
* test[0].action[1].assert.response = #okay
177+
178+
// Step 3: Validate output structure using FHIRPath
179+
* test[0].action[2].assert.description = "Validate patient name was preserved"
180+
* test[0].action[2].assert.direction = #response
181+
* test[0].action[2].assert.expression = "Patient.name.family = 'Doe'"
182+
* test[0].action[2].assert.warningOnly = false
183+
184+
* test[0].action[3].assert.description = "Validate patient given name was preserved"
185+
* test[0].action[3].assert.direction = #response
186+
* test[0].action[3].assert.expression = "Patient.name.given = 'John'"
187+
* test[0].action[3].assert.warningOnly = false
188+
189+
* test[0].action[4].assert.description = "Validate patient active status was preserved"
190+
* test[0].action[4].assert.direction = #response
191+
* test[0].action[4].assert.expression = "Patient.active = true"
192+
* test[0].action[4].assert.warningOnly = false
193+
194+
* test[0].action[5].assert.description = "Validate patient gender was preserved"
195+
* test[0].action[5].assert.direction = #response
196+
* test[0].action[5].assert.expression = "Patient.gender = 'male'"
197+
* test[0].action[5].assert.warningOnly = false

‎input/testdata/examples/LICENSE.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# Test Data License Attribution
2+
3+
## Examples Directory (`examples/`)
4+
5+
All files in this directory are local examples created for the FML Runner project.
6+
7+
**License**: MIT License
8+
**Copyright**: 2025 Carl Leitner
9+
10+
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
13+
14+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
15+
16+
## Files Covered
17+
18+
- `patient-transform.map` - Example patient transformation mapping
19+
- `patient-input.json` - Example patient input data
20+
- `patient-output.json` - Example expected patient output
21+
- `observation-transform.map` - Example observation transformation mapping
22+
- `observation-input.json` - Example observation input data
23+
- `observation-output.json` - Example expected observation output

‎input/testdata/examples/observation-input.json‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
{
2+
"resourceType": "Observation",
3+
"status": "final",
4+
"code": {
5+
"coding": [
6+
{
7+
"system": "http://loinc.org",
8+
"code": "29463-7",
9+
"display": "Body Weight"
10+
}
11+
]
12+
},
13+
"subject": {
14+
"reference": "Patient/example"
15+
},
16+
"effectiveDateTime": "2023-01-15",
17+
"valueQuantity": {
18+
"value": 70.5,
19+
"unit": "kg",
20+
"system": "http://unitsofmeasure.org",
21+
"code": "kg"
22+
}
23+
}

0 commit comments

Comments
 (0)