Skip to content

Commit 5346908

Browse files
denreadenrea
authored
Merge pull request #265123 from shellyhaverkamp/smh-fhir-import
add performance factors
2 parents b5008b6 + c1677f5 commit 5346908

File tree

1 file changed

+91
-84
lines changed

1 file changed

+91
-84
lines changed

‎articles/healthcare-apis/fhir/import-data.md

Lines changed: 91 additions & 84 deletions
Original file line numberDiff line numberDiff line change
@@ -1,56 +1,74 @@
11
---
2-
title: Executing the import by invoking $import operation on FHIR service in Azure Health Data Services
3-
description: This article describes how to import FHIR data using $import.
2+
title: Import data into the FHIR service in Azure Health Data Services
3+
description: Learn how to import data into the FHIR service for Azure Health Data Services.
44
author: expekesheth
55
ms.service: healthcare-apis
66
ms.subservice: fhir
77
ms.topic: how-to
8-
ms.date: 06/06/2022
8+
ms.date: 02/06/2024
99
ms.author: kesheth
1010
---
1111

12+
# Import FHIR data
1213

13-
# Import Operation
14-
Import operation enables loading Fast Healthcare Interoperability Resources (FHIR®) data to the FHIR server at high throughput. Import supports both initial and incremental data load into FHIR server.
14+
The `import` operation allows you to ingest FHIR® data into the FHIR server with high throughput.
1515

16+
## Import operation modes
1617

17-
There are two modes of $import supported today-
18-
* Initial mode
19-
1. Initial mode is intended to load FHIR resources into an empty FHIR server.
20-
1. Initial mode only supports CREATE operations and, when enabled, blocks API writes to the FHIR server.
21-
* Incremental mode
22-
1. Incremental mode is optimized to load data into FHIR server periodically and doesn't block writes via API.
23-
1. Incremental mode allows you to load lastUpdated and versionId from resource Meta (if present in resource JSON).
24-
Incase
25-
* Import files don't have `version` and `lastUpdated` field values specified, there's no guarantee of importing resources in FHIR service.
26-
* Import files have resources with duplicate `version` and `lastUpdated` field values, then only one resource is ingested in the FHIR service.
27-
1. Incremental mode allows you to ingest soft deleted resources. This capability is beneficial in case you would like to migrate from Azure API for FHIR to Azure Health Data Services, FHIR service.
18+
The `import` operation supports two modes: initial mode and incremental mode. Each mode has different features and use cases.
2819

20+
#### Initial mode
2921

30-
Note:
31-
* Import operation does not support conditional references in resources.
32-
* During import operation, If multiple resources share the same resource ID, then only one of those resources is imported at random. There is an error logged for the resources sharing the same resource ID.
22+
- Is intended for loading FHIR resources into an empty FHIR server.
3323

24+
- Only supports `create` operations and when enabled, blocks API writes to the FHIR server.
3425

35-
## Using $import operation
26+
#### Incremental mode
3627

37-
> [!NOTE]
38-
> You must have the **FHIR Data Importer** role on the FHIR server to use $import.
28+
- Is optimized for loading data into the FHIR server periodically and doesn't block writes through the API.
3929

40-
To use $import, you need to configure the FHIR server using the instructions in the [Configure import settings](configure-import-data.md) article.
41-
The FHIR data to be imported must be stored in resource specific files in FHIR NDJSON format on the Azure blob store.
30+
- Allows you to load `lastUpdated` and `versionId` from resource metadata if present in the resource JSON.
31+
32+
* If import files don't have the `version` and `lastUpdated` field values specified, there's no guarantee of importing resources in FHIR service.
4233

43-
For import operation, ensure
44-
* All the resources in a file must be of the same type. You may have multiple files per resource type.
45-
* The data to be imported must be in the same Tenant as of the FHIR service.
46-
* Maximum number of files to be imported per operation is 10,000.
34+
* If import files have resources with duplicate `version` and `lastUpdated` field values, then only one resource is randomly ingested in the FHIR service.
4735

36+
- Allows you to ingest soft deleted resources. This capability is beneficial when you migrate from Azure API for FHIR to the FHIR service in Azure Health Data Services.
4837

49-
### Calling $import
38+
> [!IMPORTANT]
39+
> The `import` operation doesn't support conditional references in resources.
40+
>
41+
> Also, if multiple resources share the same resource ID, then only one of those resources is imported at random. An error is logged for the resources sharing the same resource ID.
5042
51-
Make a ```POST``` call to ```<<FHIR service base URL>>/$import``` with the request header and body shown:
43+
## Performance considerations
5244

53-
#### Request Header
45+
To achieve the best performance with the `import` operation, consider these factors:
46+
47+
- **Use large files for import.** The file size of a single `import` operation should be more than 200 MB. Smaller files might result in slower import times.
48+
49+
- **Import FHIR resource files as a single batch.** For optimal performance, import all the FHIR resource files that you want to ingest in the FHIR server in one `import` operation. Importing all the files in one operation reduces the overhead of creating and managing multiple import jobs.
50+
51+
- **Limit the number of parallel import jobs.** You can run multiple `import` jobs at the same time, but running multiple jobs might affect the overall throughput of the import operation. The FHIR server can handle up to five parallel `import` jobs. If you exceed this limit, the FHIR server might throttle or reject your requests.
52+
53+
## Perform the import operation
54+
55+
### Prerequisites
56+
57+
- You need the **FHIR Data Importer** role on the FHIR server to use the `import` operation.
58+
59+
- Configure the FHIR server. The FHIR data must be stored in resource-specific files in FHIR NDJSON format on the Azure blob store. For more information, see [Configure import settings](configure-import-data.md).
60+
61+
- All the resources in a file must be the same type. You can have multiple files per resource type.
62+
63+
- The data must be in the same tenant as the FHIR service.
64+
65+
- The maximum number of files allowed per `import` operation is 10,000.
66+
67+
### Call $import
68+
69+
Make a `POST` call to `<<FHIR service base URL>>/$import` with the request header and body shown.
70+
71+
#### Request header
5472

5573
```http
5674
Prefer:respond-async
@@ -61,17 +79,15 @@ Content-Type:application/fhir+json
6179

6280
| Parameter Name | Description | Card. | Accepted values |
6381
| ----------- | ----------- | ----------- | ----------- |
64-
| inputFormat | String representing the name of the data source format. Currently only FHIR NDJSON files are supported. | 1..1 | ```application/fhir+ndjson``` |
65-
| mode | Import mode value | 1..1 | For initial mode import, use ```InitialLoad``` mode value. For incremental mode import, use ```IncrementalLoad``` mode value. If no mode value is provided, IncrementalLoad mode value is considered by default. |
66-
| input | Details of the input files. | 1..* | A JSON array with three parts described in the table below. |
82+
| inputFormat | String representing the name of the data source format. Only FHIR NDJSON files are supported. | 1..1 | ```application/fhir+ndjson``` |
83+
| mode | Import mode value | 1..1 | For an initial mode `import`, use the `InitialLoad` mode value. For incremental mode `import`, use the `IncrementalLoad` mode value. If no mode value is provided, the `IncrementalLoad`` mode value is used by default. |
84+
| input | Details of the input files. | 1..* | A JSON array with the three parts described in the table. |
6785

6886
| Input part name | Description | Card. | Accepted values |
6987
| ----------- | ----------- | ----------- | ----------- |
7088
| type | Resource type of input file | 1..1 | A valid [FHIR resource type](https://www.hl7.org/fhir/resourcelist.html) that matches the input file. |
71-
|URL | Azure storage url of input file | 1..1 | URL value of the input file that can't be modified. |
72-
| etag | Etag of the input file on Azure storage; used to verify the file content has not changed after $import registration. | 0..1 | Etag value of the input file. |
73-
74-
**Sample body for import:**
89+
|URL | Azure storage URL of the input file | 1..1 | URL value of the input file. The value can't be modified. |
90+
| etag | Etag of the input file on Azure storage; used to verify the file content isn't changed after `import` registration. | 0..1 | Etag value of the input file. |
7591

7692
```json
7793
{
@@ -119,32 +135,31 @@ Content-Type:application/fhir+json
119135
}
120136
```
121137

122-
### Checking import status
138+
### Check $import status
139+
140+
After an $import is initiated, an empty response body with a `callback` link is returned in the `Content-location` header of the response, together with an `202-Accepted` status code. Store the callback link to check the import status.
123141

124-
Once $import is initiated, an empty response body with a **callback** link is returned in the `Content-location` header of the response together with ```202-Accepted``` status code. Store this callback link to check the import status.
142+
The import operation registration is implemented as an idempotent call. The same registration payload yields the same registration, which affects ability to reprocess files with the same name. Refrain from updating files in-place, instead we suggest you use different file name for updated data, or, if update in-place with same file name is unavoidable, add e-tags in the registration payload.
125143

126-
$import operation registration is implemented as idempotent call (same registration payload yields same registration). This affects ability to re-process files with the same name. Refrain from updating files in-place, instead we suggest you use different file name for updated data, or, if update in-place with same file name is unavoidable, add e-tags in the registration payload.
144+
To check import status, make the REST call with the `GET` method to the `callback` link returned in the previous step.
127145

128-
To check import status, make the REST call with the ```GET``` method to the **callback** link returned in the previous step.
129-
You can interpret the response using the following table:
146+
Interpret the response by using the table:
130147

131148
| Response code | Response body |Description |
132149
| ----------- | ----------- |----------- |
133150
| 202 Accepted | |The operation is still running.|
134151
| 200 OK |The response body doesn't contain any error.url entry|All resources were imported successfully.|
135-
| 200 OK |The response body contains some error.url entry|Error occurred while importing some of the resources. See the files located at error.url for the details. Rest of the resources were imported successfully.|
136-
| Other||A fatal error occurred and the operation has stopped. Successfully imported resources haven't been rolled back.|
152+
| 200 OK |The response body contains some error.url entry|Error occurred while importing some of the resources. See the files located at error.url for the details. The remaining resources were imported successfully.|
153+
| Other||A fatal error occurred and the operation stopped. Successfully imported resources aren't rolled back.|
137154

138-
Table below provides some of the important fields in the response body:
155+
Table describes the important fields in the response body:
139156

140157
| Field | Description |
141158
| ----------- | ----------- |
142159
|transactionTime|Start time of the bulk-import operation.|
143-
|output.count|Count of resources that were successfully imported|
144-
|error.count|Count of resources that weren't imported due to some error|
145-
|error.url|URL of the file containing details of the error. Each error.url is unique to an input URL. |
146-
147-
**Sample response:**
160+
|output.count|Count of resources that were successfully imported.|
161+
|error.count|Count of resources that weren't imported due to an error.|
162+
|error.url|URL of the file containing details of the error. Each error.url is unique to an input URL.|
148163

149164
```json
150165
{
@@ -172,35 +187,33 @@ Table below provides some of the important fields in the response body:
172187
]
173188
}
174189
```
175-
### Ingestion of soft deleted resources
176-
Incremental mode import supports ingestion of soft deleted resources. You need to use the extension to ingest soft deleted resources in FHIR service.
190+
### Ingest soft deleted resources
191+
Incremental mode `import` supports ingestion of soft deleted resources. You need to use the extension to ingest soft deleted resources in the FHIR service.
177192

178-
**Sample import file with soft deleted resources:**
179-
Extension needs to be added to the resource to inform FHIR service, if it is soft deleted. Below is an example of the extension.
193+
Add the extension to the resource to inform the FHIR service that the resource was soft deleted.
180194

181195
```ndjson
182196
{"resourceType": "Patient", "id": "example10", "meta": { "lastUpdated": "2023-10-27T04:00:00.000Z", "versionId": 4, "extension": [ { "url": "http://azurehealthcareapis.com/data-extensions/deleted-state", "valueString": "soft-deleted" } ] } }
183197
```
184198

185-
**Validate ingestion of soft deleted resources:**
186-
After import operation is successfully completed, to validate soft deleted resources in FHIR service, you need to perform history search on the resource.
187-
If the ID of the resource that was deleted is known, use the following URL pattern:
199+
After the `import` operation completes successfully, perform history search on the resource to validate soft deleted resources. If you know the ID of the deleted resource, use the URL pattern in the example.
188200

189201
```json
190202
<FHIR_URL>/<resource-type>/<resource-id>/_history
191203
```
192204

193-
Incase the ID of the resource isn't known, do a history search on the entire resource type:
205+
If you don't know the ID of the resource, do a history search on the resource type.
194206
```json
195207
<FHIR_URL>/<resource-type>/_history
196208
```
197209

198-
## Troubleshooting
210+
## Troubleshoot the import operation
211+
212+
Here are the error messages that occur if the `import` operation fails, and recommended actions to take to resolve the issue.
199213

200-
Lets walk-through solutions to some error codes you may encounter during the import operation.
201-
### 200 OK, but there's an error with the URL in the response
214+
#### 200 OK, but there's an error with the URL in the response
202215

203-
**Behavior:** Import operation succeeds and returns ```200 OK```. However, `error.url` are present in the response body. Files present at the `error.url` location contain JSON fragments similar to below example:
216+
**Behavior:** The `import` operation succeeds and returns `200 OK`. However, `error.url` is present in the response body. Files present at the `error.url` location contain JSON fragments similar to the example:
204217

205218
```json
206219
{
@@ -217,13 +230,13 @@ Lets walk-through solutions to some error codes you may encounter during the imp
217230
}
218231
```
219232

220-
**Cause:** NDJSON files contain resources with conditional references, which are currently not supported by $import.
233+
**Cause:** NDJSON files contain resources with conditional references not supported by $import.
221234

222235
**Solution:** Replace the conditional references to normal references in the NDJSON files.
223236

224-
### 400 Bad Request
237+
#### 400 Bad Request
225238

226-
**Behavior:** Import operation failed and ```400 Bad Request``` is returned. Response body has the following content:
239+
**Behavior:** The import operation failed and `400 Bad Request` is returned. The response body includes this content:
227240

228241
```json
229242
{
@@ -239,11 +252,11 @@ Lets walk-through solutions to some error codes you may encounter during the imp
239252
}
240253
```
241254

242-
**Solution:** Verify the link to the Azure storage is correct. Check the network and firewall settings to make sure that the FHIR server is able to access the storage. If your service is in a VNet, ensure that the storage is in the same VNet or in a VNet that has peering with the FHIR service VNet.
255+
**Solution:** Verify the link to the Azure storage is correct. Check the network and firewall settings to make sure that the FHIR server is able to access the storage. If your service is in a virtual network, ensure that the storage is in the same virtual network or in a virtual network peered with the FHIR service virtual network.
243256

244-
### 403 Forbidden
257+
#### 403 Forbidden
245258

246-
**Behavior:** Import operation failed and ```403 Forbidden``` is returned. The response body has the following content:
259+
**Behavior:** The import operation failed and `403 Forbidden` is returned. The response body contains this content:
247260

248261
```json
249262
{
@@ -259,13 +272,13 @@ Lets walk-through solutions to some error codes you may encounter during the imp
259272
}
260273
```
261274

262-
**Cause:** We use managed identity for source storage auth. This error may be caused by a missing or wrong role assignment.
275+
**Cause:** The FHIR service uses managed identity for source storage authentication. This error indicates a missing or incorrect role assignment.
263276

264-
**Solution:** Assign _Storage Blob Data Contributor_ role to the FHIR server following [the RBAC guide.](../../role-based-access-control/role-assignments-portal.md?tabs=current)
277+
**Solution:** Assign the **Storage Blob Data Contributor** role to the FHIR server. For more information, see [Assign Azure roles](../../role-based-access-control/role-assignments-portal.md?tabs=current).
265278

266-
### 500 Internal Server Error
279+
#### 500 Internal Server Error
267280

268-
**Behavior:** Import operation failed and ```500 Internal Server Error``` is returned. Response body has the following content:
281+
**Behavior:** The import operation failed and `500 Internal Server Error` is returned. The response body contains this content:
269282

270283
```json
271284
{
@@ -281,22 +294,16 @@ Lets walk-through solutions to some error codes you may encounter during the imp
281294
}
282295
```
283296

284-
**Cause:** You've reached the storage limit of the FHIR service.
297+
**Cause:** You reached the storage limit of the FHIR service.
285298

286299
**Solution:** Reduce the size of your data or consider Azure API for FHIR, which has a higher storage limit.
287300

288301
## Next steps
289302

290-
In this article, you've learned about how the import operation enables importing FHIR data to the FHIR server at high throughput. For more information about converting data to FHIR, exporting settings to set up a storage account, and moving data to Azure Synapse, see
291-
292-
293-
>[!div class="nextstepaction"]
294-
>[Converting your data to FHIR](convert-data.md)
303+
[Convert your data to FHIR](convert-data.md)
295304

296-
>[!div class="nextstepaction"]
297-
>[Configure export settings and set up a storage account](configure-export-data.md)
305+
[Configure export settings and set up a storage account](configure-export-data.md)
298306

299-
>[!div class="nextstepaction"]
300-
>[Copy data from Azure API for FHIR to Azure Synapse Analytics](copy-to-synapse.md)
307+
[Copy data to Azure Synapse Analytics](copy-to-synapse.md)
301308

302-
FHIR&#174; is a registered trademark of [HL7](https://hl7.org/fhir/) and is used with the permission of HL7.
309+
[!INCLUDE [FHIR trademark statement](../includes/healthcare-apis-fhir-trademark.md)]

0 commit comments

Comments
 (0)