Skip to content

Commit 5d90b27

Browse files
Copilotlitlfred
Copilot
and
litlfred
committed
Implement FML Execution Validation Test Suite infrastructure
Co-authored-by: litlfred <662242+litlfred@users.noreply.github.com>
1 parent abf3b9e commit 5d90b27

20 files changed

+1539
-0
lines changed

‎.gitignore‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,13 @@ lib-cov
1818
coverage/
1919
*.lcov
2020

21+
# FHIR IG generated files
22+
output/
23+
fsh-generated/
24+
temp/
25+
template/
26+
input-cache/
27+
2128
# nyc test coverage
2229
.nyc_output
2330

‎TEST_SUITE_README.md‎

Lines changed: 180 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,180 @@
1+
# FML Execution Validation Test Suite
2+
3+
A comprehensive FHIR Implementation Guide defining a test suite for validating FHIR Mapping Language (FML) execution using real-world test cases.
4+
5+
## Overview
6+
7+
This test suite provides a standardized way to validate FML implementations using:
8+
- Real-world test cases from the ahdis/matchbox project
9+
- Official FHIR test cases from FHIR/fhir-test-cases repository
10+
- FHIR TestPlan resources for structured test execution
11+
- Comprehensive licensing compliance for all imported content
12+
13+
## Quick Start
14+
15+
### Prerequisites
16+
17+
- [SUSHI](https://fshschool.org/docs/sushi/) (FHIR Shorthand IG generator)
18+
- [FHIR IG Publisher](https://confluence.hl7.org/display/FHIR/IG+Publisher+Documentation)
19+
- Node.js (for test data import scripts)
20+
21+
### Building the Implementation Guide
22+
23+
1. **Clone the repository**
24+
```bash
25+
git clone https://github.com/litlfred/fmlrunner.git
26+
cd fmlrunner
27+
```
28+
29+
2. **Generate the IG**
30+
```bash
31+
# Compile FSH files
32+
sushi
33+
34+
# Generate the full IG
35+
_genonce.sh # or .bat on Windows
36+
```
37+
38+
3. **Import additional test data** (optional)
39+
```bash
40+
node scripts/test-data-import/import-all.js
41+
```
42+
43+
## Directory Structure
44+
45+
```
46+
input/
47+
├── fsh/
48+
│ └── tests/
49+
│ └── FMLExecutionValidationTestPlan.fsh
50+
├── testdata/
51+
│ ├── matchbox/
52+
│ │ ├── qr2patgender/
53+
│ │ │ ├── qr2patgender.map
54+
│ │ │ ├── qr-input.json
55+
│ │ │ └── patient-output.json
56+
│ │ └── test-cases-metadata.json
57+
│ └── fhir-test-cases/
58+
│ ├── tutorial-step1/
59+
│ │ ├── tutorial-step1.map
60+
│ │ └── tutorial-input.json
61+
│ └── test-cases-metadata.json
62+
└── pagecontent/
63+
├── index.md
64+
├── test-suite.md
65+
├── test-data.md
66+
└── license-compliance.md
67+
```
68+
69+
## Test Data Import
70+
71+
The test suite includes scripts to import test data from external repositories:
72+
73+
### Import from ahdis/matchbox
74+
```bash
75+
node scripts/test-data-import/import-matchbox.js
76+
```
77+
78+
### Import from FHIR/fhir-test-cases
79+
```bash
80+
node scripts/test-data-import/import-fhir-test-cases.js
81+
```
82+
83+
### Import all sources
84+
```bash
85+
node scripts/test-data-import/import-all.js
86+
```
87+
88+
## Using the Test Suite
89+
90+
### For FML Engine Implementers
91+
92+
1. **Load the TestPlan**: Import the `FMLExecutionValidationTestPlan` resource
93+
2. **Execute Test Cases**: Run each test case against your FML engine
94+
3. **Validate Results**: Compare outputs against expected results
95+
4. **Check Assertions**: Evaluate FHIRPath assertions for pass/fail status
96+
97+
### For Test Suite Maintainers
98+
99+
1. **Add Test Data**: Place new test files in appropriate directories
100+
2. **Update Metadata**: Modify metadata JSON files to include new test cases
101+
3. **Regenerate TestPlan**: Run import scripts to update the FSH TestPlan
102+
4. **Rebuild IG**: Use SUSHI and IG Publisher to generate updated documentation
103+
104+
## Test Case Structure
105+
106+
Each test case includes:
107+
108+
- **Map File**: FML mapping specification (`.map` or `.fml`)
109+
- **Input File**: Source FHIR resource (JSON or XML)
110+
- **Output File**: Expected transformation result (when available)
111+
- **Assertions**: FHIRPath expressions for validation
112+
113+
### Example Test Case
114+
115+
```
116+
input/testdata/matchbox/qr2patgender/
117+
├── qr2patgender.map # FML mapping
118+
├── qr-input.json # Input QuestionnaireResponse
119+
└── patient-output.json # Expected Patient output
120+
```
121+
122+
## License Compliance
123+
124+
The test suite incorporates content from multiple sources:
125+
126+
- **ahdis/matchbox**: Apache License 2.0
127+
- **FHIR/fhir-test-cases**: HL7 FHIR License
128+
129+
All imported files include proper attribution headers. See [License Compliance](input/pagecontent/license-compliance.md) for details.
130+
131+
## Contributing
132+
133+
To contribute to the test suite:
134+
135+
1. **Add Test Cases**: Include proper licensing headers
136+
2. **Update Documentation**: Describe new test scenarios
137+
3. **Maintain Attribution**: Preserve all license information
138+
4. **Test Changes**: Verify IG builds successfully
139+
140+
## File Naming Conventions
141+
142+
Test files follow these patterns:
143+
- Map files: `*.map`, `*.fml`, `*-map.txt`
144+
- Input files: `*-input.json`, `*-input.xml`, `*source*`
145+
- Output files: `*-output.json`, `*-output.xml`, `*-expected.*`
146+
147+
## Building and Testing
148+
149+
### Build the IG
150+
```bash
151+
# Quick build
152+
sushi
153+
154+
# Full build with publisher
155+
./_genonce.sh
156+
```
157+
158+
### Validate FSH
159+
```bash
160+
sushi -s
161+
```
162+
163+
### Update test data
164+
```bash
165+
node scripts/test-data-import/import-all.js
166+
sushi
167+
```
168+
169+
## Support
170+
171+
For questions or issues:
172+
- Review the [Implementation Guide](https://litlfred.github.io/fmlrunner/) documentation
173+
- Check existing [GitHub Issues](https://github.com/litlfred/fmlrunner/issues)
174+
- Create a new issue for bugs or feature requests
175+
176+
## License
177+
178+
This project is licensed under the MIT License. See [LICENSE](LICENSE) for details.
179+
180+
Individual test cases are licensed under their original terms (Apache 2.0 or HL7 FHIR License) as documented in each file.

‎generate-test-suite.sh‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
#!/bin/bash
2+
3+
# FML Test Suite IG Generation Script
4+
# This script builds the FHIR Implementation Guide for the test suite
5+
6+
echo "FML Execution Validation Test Suite - IG Generation"
7+
echo "=================================================="
8+
9+
# Check if sushi is available
10+
if ! command -v sushi &> /dev/null; then
11+
echo "Error: SUSHI not found. Please install SUSHI:"
12+
echo "npm install -g fsh-sushi"
13+
exit 1
14+
fi
15+
16+
echo "1. Generating test plan from existing test data..."
17+
npm run test-suite:generate
18+
19+
echo "2. Compiling FSH files with SUSHI..."
20+
sushi
21+
22+
if [ $? -eq 0 ]; then
23+
echo "✓ SUSHI compilation successful"
24+
else
25+
echo "✗ SUSHI compilation failed"
26+
exit 1
27+
fi
28+
29+
echo "3. Implementation Guide generation complete!"
30+
echo ""
31+
echo "Generated files:"
32+
echo "- output/: Compiled FHIR resources"
33+
echo "- fsh-generated/: Generated FSH artifacts"
34+
echo ""
35+
echo "To import additional test data, run:"
36+
echo " npm run test-suite:import"
37+
echo ""
38+
echo "To view the generated resources:"
39+
echo " ls output/"

‎input/fsh/tests/FMLExecutionValidationTestPlan.fsh‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
// FHIR Mapping Language (FML) Execution Validation Test Plan
2+
// This TestPlan validates FML execution using real-world test cases
3+
// sourced from community FML projects with proper license compliance
4+
//
5+
// Generated on: 2025-09-29T18:07:19.558Z
6+
7+
Instance: FMLExecutionValidationTestPlan
8+
InstanceOf: TestPlan
9+
Usage: #definition
10+
* id = "fml-execution-validation"
11+
* name = "FMLExecutionValidationTestPlan"
12+
* title = "FHIR Mapping Language Execution Validation Test Plan"
13+
* status = #draft
14+
* version = "0.1.0"
15+
* publisher = "FML Runner Project"
16+
* description = "A comprehensive test suite for validating FML execution using real-world test cases sourced from ahdis/matchbox and FHIR/fhir-test-cases repositories"
17+
18+
19+
* testCase[+]
20+
* id = "matchbox-qr2patgender"
21+
* sequence = 1
22+
* scope[+]
23+
* artifact = Reference(StructureMap/qr2patgender)
24+
* testRun[+]
25+
* narrative = "Test qr2patgender mapping from ahdis/matchbox"
26+
* script
27+
* language = #application/fhir+json
28+
* sourceReference = Reference(qr2patgender-input)
29+
* testData[+]
30+
* type = #input
31+
* content = Reference(qr2patgender-input)
32+
* testData[+]
33+
* type = #output
34+
* content = Reference(qr2patgender-output)
35+
* assertion[+]
36+
* type = #response
37+
* direction = #response
38+
* expression = "Bundle.entry.exists() or Patient.exists() or QuestionnaireResponse.exists()"
39+
* description = "Verify transformation produced valid output"
40+
41+
* testCase[+]
42+
* id = "fhir-tutorial-step1"
43+
* sequence = 2
44+
* scope[+]
45+
* artifact = Reference(StructureMap/tutorial-step1)
46+
* testRun[+]
47+
* narrative = "Test tutorial-step1 mapping from FHIR/fhir-test-cases"
48+
* script
49+
* language = #application/fhir+json
50+
* sourceReference = Reference(tutorial-step1-input)
51+
* testData[+]
52+
* type = #input
53+
* content = Reference(tutorial-step1-input)
54+
55+
// Total test cases: 2
56+
// Matchbox test cases: 1
57+
// FHIR test cases: 1

‎input/pagecontent/index.md‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
# FML Execution Validation Test Suite
2+
3+
This Implementation Guide defines a comprehensive test suite for validating FHIR Mapping Language (FML) execution using real-world test cases sourced from community FML projects.
4+
5+
## Purpose
6+
7+
The FML Execution Validation Test Suite is designed to:
8+
9+
1. **Validate FML Implementation Correctness**: Ensure that FML engines correctly implement the FHIR Mapping Language specification
10+
2. **Provide Real-World Test Cases**: Use actual mapping scenarios from community projects to test practical use cases
11+
3. **Enable Regression Testing**: Support continuous validation of FML engines across different versions
12+
4. **Facilitate Compliance Testing**: Help implementers verify their FML engines meet specification requirements
13+
14+
## Test Data Sources
15+
16+
The test suite incorporates test cases from two primary sources:
17+
18+
### ahdis/matchbox Repository
19+
- **Source**: [ahdis/matchbox test resources](https://github.com/ahdis/matchbox/tree/main/matchbox-server/src/test/resources)
20+
- **License**: Apache License 2.0
21+
- **Content**: Real-world mapping scenarios and test cases used in the Matchbox FHIR server
22+
23+
### FHIR/fhir-test-cases Repository
24+
- **Source**: [FHIR structure-mapping test cases](https://github.com/FHIR/fhir-test-cases/tree/main/r5/structure-mapping)
25+
- **License**: HL7 FHIR License
26+
- **Content**: Official FHIR test cases for structure mapping functionality
27+
28+
## Test Suite Structure
29+
30+
The test suite is organized using FHIR TestPlan resources that define:
31+
32+
- **Test Cases**: Individual mapping scenarios with input/output validation
33+
- **Test Data**: Input FHIR resources and expected output resources
34+
- **Assertions**: FHIRPath expressions to validate transformation results
35+
- **Metadata**: Attribution and licensing information for all test data
36+
37+
## Implementation
38+
39+
Test implementers can use this test suite to:
40+
41+
1. Validate their FML engine implementation
42+
2. Perform regression testing during development
43+
3. Verify compliance with FHIR Mapping Language specifications
44+
4. Test edge cases and real-world scenarios
45+
46+
For more information about specific test cases and implementation guidance, see the [Test Suite Overview](test-suite.html) and [Test Data Sources](test-data.html) pages.

0 commit comments

Comments
 (0)