Skip to content

Commit 3745c98

Browse files
denreadenrea
authored
Merge pull request #300458 from wsugarman/users/wisuga/dicom-storage-indexing-preview
Add Azure Health Data Services Document for DICOM Service Storage Indexing Preview
2 parents bfbb26c + 7be2351 commit 3745c98

File tree

3 files changed

+343
-12
lines changed

3 files changed

+343
-12
lines changed

‎articles/healthcare-apis/dicom/index-dicom-storage.md

Lines changed: 329 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,329 @@
1+
---
2+
title: Automatically index Azure Data Lake Storage Changes for DICOM Files
3+
description: Learn how to configure the DICOM service to react to Data Lake Storage events
4+
author: wsugarman
5+
ms.service: azure-health-data-services
6+
ms.subservice: dicom-service
7+
ms.topic: how-to
8+
ms.date: 05/31/2025
9+
ms.author: wisuga
10+
---
11+
12+
# Azure Data Lake Storage Indexing (Preview)
13+
14+
The [DICOM® service](overview.md) automatically uploads DICOM files to Azure Data Lake Storage (ADLS) when using [STOW-RS](dicom-services-conformance-statement-v2.md#store-stow-rs). That way, users can query their data either using [DICOMweb™ APIs](dicomweb-standard-apis-with-dicom-services.md), like [WADO-RS](dicom-services-conformance-statement-v2.md#retrieve-wado-rs), or [Azure Blob/Data Lake APIs](../../storage/blobs/storage-blob-upload.md). However, with storage indexing, the DICOM service automatically indexes DICOM files after they're uploaded directly to the ADLS Gen 2 file system. Whether the files were uploaded using STOW-RS, an Azure Blob SDK, or even [AzCopy](../../storage/common/storage-use-azcopy-v10.md), they can be accessed using DICOMweb™ or ADLS Gen 2 APIs.
15+
16+
## Prerequisites
17+
18+
* An Azure Storage account configured with [Hierarchical Namespaces (HNS) enabled](../../storage/blobs/data-lake-storage-introduction.md)
19+
* An optional DICOM Service [connected to the Azure Data Lake Storage file system](deploy-dicom-services-in-azure-data-lake.md)
20+
21+
## Configuring Storage Indexing
22+
23+
The DICOM service indexes an ADLS Gen 2 file system by reacting to [Blob or Data Lake storage events](../../event-grid/event-schema-blob-storage.md). These events must be read from an [Azure Storage Queue](../../storage/queues/storage-queues-introduction.md) in the Azure Storage Account that contains the file system. Once in the queue, the DICOM service asynchronously processes each event and update the index accordingly.
24+
25+
### Create the Destination for Storage Events
26+
27+
First, create a storage queue in the same Azure Storage Account connected to the DICOM service. The DICOM service also needs access to the queue; it needs to be able to both dequeue and enqueue messages, including messages for errors and broken-down complex tasks. So, make sure the same managed identity used by the DICOM service, either user-assigned or system-assigned, has the [**Storage Queue Data Contributor**](../../role-based-access-control/built-in-roles.md#storage) role assigned.
28+
29+
### Publish Storage Events to the Queue
30+
31+
With the Storage Queue in place, events must be published from the Storage Account to an [Azure Event Grid System Topic](../../event-grid/system-topics.md) and routed to queue using an [Azure Event Grid Subscription](../../event-grid/create-view-manage-event-subscriptions.md). Before creating the event subscription, be sure to grant the role [**Storage Queue Data Message Sender**](../../role-based-access-control/built-in-roles.md#storage) to the event subscription; the event subscription needs permissions to enqueue messages. The event subscription can either use a [user-assigned or system-assigned managed identity from the system topic](../../event-grid/enable-identity-system-topics.md) to authenticate its operations.
32+
33+
> [!NOTE]
34+
> By default, event subscriptions send all of the subscribed event types to their designated output. However, while the DICOM service gracefully handles any message, it can only successfully process ones that meet the following criteria:
35+
>- The message must be a Base64 [CloudEvent](../../event-grid/event-schema-subscriptions.md)
36+
>- The event type must be one of the following event types:
37+
>- `Microsoft.Storage.BlobCreated`
38+
>- `Microsoft.Storage.BlobDeleted`
39+
>- The file system must be the same one configured for the DICOM service
40+
>- The file path must be within `AHDS/{workspace-name}/dicom/{dicom-service-name}[/{partition-name}]`
41+
>- The file must be a DICOM file as defined in Part 10 of the DICOM standard
42+
>- The operation can't be performed the DICOM service itself
43+
44+
The event subscription can be configured to filter out irrelevant data to avoid unnecessary processing and billing. Make sure to configure filter such that:
45+
- The *subject* must begin with `/blobServices/default/containers/{file-system-name}/blobs/AHDS/{workspace-name}/dicom/{dicom-service-name}/`
46+
- Optionally, the *subject* ends with `.dcm`
47+
- Under *advanced filters*, the key `data.clientRequestId` doesn't begin with `tag:{workspace-name}-{dicom-service-name}.dicom.azurehealthcareapis.com,`
48+
49+
### Enable Storage Indexing
50+
51+
Once the Event Grid subscription is configured, the DICOM service must know from where to read the storage events. While in preview, storage indexing may only be configured using an [Azure Resource Manager (ARM) template](../../azure-resource-manager/templates/overview.md) using version `2025-04-01-preview`, which introduced a new property called `storageConfiguration.storageIndexingConfiguration.storageEventQueueName`. It's currently unavailable to configure via the Azure portal.
52+
53+
The following example ARM template may be deployed using the [Azure CLI](../../azure-resource-manager/templates/deploy-cli.md). It includes all of the necessary resources for a DICOM service:
54+
55+
```json
56+
{
57+
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
58+
"contentVersion": "1.0.0.0",
59+
"parameters": {
60+
"workspaceName": {
61+
"type": "String"
62+
},
63+
"dicomServiceName": {
64+
"type": "String"
65+
},
66+
"enableDataPartitions": {
67+
"defaultValue": false,
68+
"type": "bool"
69+
},
70+
"storageAccountName": {
71+
"type": "String"
72+
},
73+
"storageAccountSku": {
74+
"defaultValue": "Standard_LRS",
75+
"type": "String"
76+
},
77+
"fileSystemName": {
78+
"type": "String"
79+
},
80+
"storageEventQueueName": {
81+
"defaultValue": "storage-events",
82+
"type": "String"
83+
},
84+
"systemTopicName": {
85+
"type": "String"
86+
},
87+
"eventSubscriptionName": {
88+
"defaultValue": "dicom-storage-events",
89+
"type": "String"
90+
}
91+
},
92+
"variables": {
93+
"storageBlobDataContributor": "[subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'ba92f5b4-2d11-453d-a403-e96b0029c9fe')]",
94+
"storageQueueDataContributor": "[subscriptionResourceId('Microsoft.Authorization/roleDefinitions', '974c5e8b-45b9-4653-ba55-5f855dd0fb88')]",
95+
"storageQueueDataMessageSender": "[subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'c6a89b2d-59bc-44d0-9896-0f6e12d7b80a')]",
96+
"dicomIdentityName": "[concat(parameters('storageAccountName'), '-', parameters('storageEventQueueName'))]"
97+
},
98+
"resources": [
99+
{
100+
"type": "Microsoft.ManagedIdentity/userAssignedIdentities",
101+
"apiVersion": "2023-01-31",
102+
"name": "[variables('dicomIdentityName')]",
103+
"location": "[resourceGroup().location]"
104+
},
105+
{
106+
"type": "Microsoft.ManagedIdentity/userAssignedIdentities",
107+
"apiVersion": "2023-01-31",
108+
"name": "[parameters('systemTopicName')]",
109+
"location": "[resourceGroup().location]"
110+
},
111+
{
112+
"type": "Microsoft.Storage/storageAccounts",
113+
"apiVersion": "2022-05-01",
114+
"name": "[parameters('storageAccountName')]",
115+
"location": "[resourceGroup().location]",
116+
"sku": {
117+
"name": "[parameters('storageAccountSku')]"
118+
},
119+
"kind": "StorageV2",
120+
"properties": {
121+
"isHnsEnabled": true,
122+
"accessTier": "Hot",
123+
"supportsHttpsTrafficOnly": true,
124+
"minimumTlsVersion": "TLS1_2",
125+
"defaultToOAuthAuthentication": true,
126+
"allowBlobPublicAccess": false,
127+
"allowSharedKeyAccess": false,
128+
"encryption": {
129+
"keySource": "Microsoft.Storage",
130+
"requireInfrastructureEncryption": true,
131+
"services": {
132+
"blob": {
133+
"enabled": true
134+
},
135+
"queue": {
136+
"enabled": true
137+
}
138+
}
139+
}
140+
}
141+
},
142+
{
143+
"type": "Microsoft.Storage/storageAccounts/blobServices/containers",
144+
"apiVersion": "2022-05-01",
145+
"name": "[format('{0}/default/{1}', parameters('storageAccountName'), parameters('fileSystemName'))]",
146+
"dependsOn": [
147+
"[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName'))]"
148+
]
149+
},
150+
{
151+
"type": "Microsoft.Storage/storageAccounts/queueServices/queues",
152+
"apiVersion": "2024-01-01",
153+
"name": "[format('{0}/default/{1}', parameters('storageAccountName'), parameters('storageEventQueueName'))]",
154+
"dependsOn": [
155+
"[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName'))]"
156+
]
157+
},
158+
{
159+
"type": "Microsoft.Storage/storageAccounts/queueServices/queues",
160+
"apiVersion": "2024-01-01",
161+
"name": "[format('{0}/default/{1}-poison', parameters('storageAccountName'), parameters('storageEventQueueName'))]",
162+
"dependsOn": [
163+
"[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName'))]"
164+
]
165+
},
166+
{
167+
"type": "Microsoft.Authorization/roleAssignments",
168+
"apiVersion": "2021-04-01-preview",
169+
"name": "[guid(resourceGroup().id, parameters('workspaceName'), parameters('dicomServiceName'))]",
170+
"location": "[resourceGroup().location]",
171+
"dependsOn": [
172+
"[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName'))]",
173+
"[resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', variables('dicomIdentityName'))]"
174+
],
175+
"properties": {
176+
"roleDefinitionId": "[variables('storageBlobDataContributor')]",
177+
"principalId": "[reference(resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', variables('dicomIdentityName'))).principalId]",
178+
"principalType": "ServicePrincipal"
179+
},
180+
"scope": "[concat('Microsoft.Storage/storageAccounts', '/', parameters('storageAccountName'))]"
181+
},
182+
{
183+
"type": "Microsoft.Authorization/roleAssignments",
184+
"apiVersion": "2021-04-01-preview",
185+
"name": "[guid(resourceGroup().id, parameters('workspaceName'), parameters('dicomServiceName'), parameters('storageEventQueueName'))]",
186+
"location": "[resourceGroup().location]",
187+
"dependsOn": [
188+
"[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName'))]",
189+
"[resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', variables('dicomIdentityName'))]"
190+
],
191+
"properties": {
192+
"roleDefinitionId": "[variables('storageQueueDataContributor')]",
193+
"principalId": "[reference(resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', variables('dicomIdentityName'))).principalId]",
194+
"principalType": "ServicePrincipal"
195+
},
196+
"scope": "[concat('Microsoft.Storage/storageAccounts', '/', parameters('storageAccountName'))]"
197+
},
198+
{
199+
"type": "Microsoft.Authorization/roleAssignments",
200+
"apiVersion": "2021-04-01-preview",
201+
"name": "[guid(resourceGroup().id, parameters('systemTopicName'), parameters('storageEventQueueName'))]",
202+
"location": "[resourceGroup().location]",
203+
"dependsOn": [
204+
"[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName'))]",
205+
"[resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', parameters('systemTopicName'))]"
206+
],
207+
"properties": {
208+
"roleDefinitionId": "[variables('storageQueueDataMessageSender')]",
209+
"principalId": "[reference(resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', parameters('systemTopicName'))).principalId]",
210+
"principalType": "ServicePrincipal"
211+
},
212+
"scope": "[concat('Microsoft.Storage/storageAccounts', '/', parameters('storageAccountName'))]"
213+
},
214+
{
215+
"type": "Microsoft.EventGrid/systemTopics",
216+
"apiVersion": "2025-02-15",
217+
"name": "[parameters('systemTopicName')]",
218+
"location": "[resourceGroup().location]",
219+
"identity": {
220+
"type": "userAssigned",
221+
"userAssignedIdentities": {
222+
"[resourceId('Microsoft.ManagedIdentity/userAssignedIdentities/', parameters('systemTopicName'))]": {}
223+
}
224+
},
225+
"dependsOn": [
226+
"[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName'))]",
227+
"[resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', parameters('systemTopicName'))]"
228+
],
229+
"properties": {
230+
"source": "[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName'))]",
231+
"topicType": "Microsoft.Storage.StorageAccounts"
232+
}
233+
},
234+
{
235+
"type": "Microsoft.EventGrid/systemTopics/eventSubscriptions",
236+
"apiVersion": "2025-02-15",
237+
"name": "[concat(parameters('systemTopicName'), '/', parameters('eventSubscriptionName'))]",
238+
"dependsOn": [
239+
"[resourceId('Microsoft.EventGrid/systemTopics', parameters('systemTopicName'))]"
240+
],
241+
"properties": {
242+
"deliveryWithResourceIdentity": {
243+
"identity": {
244+
"type": "UserAssigned",
245+
"userAssignedIdentity": "[resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', parameters('systemTopicName'))]"
246+
},
247+
"destination": {
248+
"properties": {
249+
"resourceId": "[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName'))]",
250+
"queueName": "[parameters('storageEventQueueName')]",
251+
"queueMessageTimeToLiveInSeconds": 604800
252+
},
253+
"endpointType": "StorageQueue"
254+
}
255+
},
256+
"filter": {
257+
"subjectBeginsWith": "[format('/blobServices/default/containers/{0}/blobs/AHDS/{1}/dicom/{2}/', parameters('fileSystemName'), parameters('workspaceName'), parameters('dicomServiceName'))]",
258+
"subjectEndsWith": ".dcm",
259+
"includedEventTypes": [
260+
"Microsoft.Storage.BlobCreated",
261+
"Microsoft.Storage.BlobDeleted"
262+
],
263+
"isSubjectCaseSensitive": true,
264+
"enableAdvancedFilteringOnArrays": true,
265+
"advancedFilters": [
266+
{
267+
"values": [
268+
"[format('tag:{0}-{1}.dicom.azurehealthcareapis.com,', parameters('workspaceName'), parameters('dicomServiceName'))]"
269+
],
270+
"operatorType": "StringNotBeginsWith",
271+
"key": "data.clientRequestId"
272+
}
273+
]
274+
},
275+
"labels": [],
276+
"eventDeliverySchema": "CloudEventSchemaV1_0",
277+
"retryPolicy": {
278+
"maxDeliveryAttempts": 30,
279+
"eventTimeToLiveInMinutes": 1440
280+
}
281+
}
282+
},
283+
{
284+
"type": "Microsoft.HealthcareApis/workspaces",
285+
"name": "[parameters('workspaceName')]",
286+
"apiVersion": "2025-04-01-preview",
287+
"location": "[resourceGroup().location]"
288+
},
289+
{
290+
"type": "Microsoft.HealthcareApis/workspaces/dicomservices",
291+
"apiVersion": "2025-04-01-preview",
292+
"name": "[concat(parameters('workspaceName'), '/', parameters('dicomServiceName'))]",
293+
"location": "[resourceGroup().location]",
294+
"dependsOn": [
295+
"[resourceId('Microsoft.HealthcareApis/workspaces', parameters('workspaceName'))]",
296+
"[resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', variables('dicomIdentityName'))]",
297+
"[resourceId('Microsoft.EventGrid/systemTopics/eventSubscriptions', parameters('systemTopicName'), parameters('eventSubscriptionName'))]"
298+
],
299+
"identity": {
300+
"type": "userAssigned",
301+
"userAssignedIdentities": {
302+
"[resourceId('Microsoft.ManagedIdentity/userAssignedIdentities/', variables('dicomIdentityName'))]": {}
303+
}
304+
},
305+
"properties": {
306+
"storageConfiguration": {
307+
"storageResourceId": "[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName'))]",
308+
"fileSystemName": "[parameters('fileSystemName')]",
309+
"storageIndexingConfiguration": {
310+
"storageEventQueueName": "[parameters('storageEventQueueName')]"
311+
}
312+
},
313+
"enableDataPartitions": "[parameters('enableDataPartitions')]"
314+
}
315+
}
316+
]
317+
}
318+
```
319+
320+
## Diagnosing Issues
321+
322+
:::image type="content" source="media/storage-indexing/diagnostic-logs.png" alt-text="A screenshot of the Azure portal showing a Kusto Query Language (KQL) query for the AHDSDicomAuditLogs table. The example query is filtering for all logs where OperationName is the string index-storage. A table of the query results is underneath." lightbox="media/storage-indexing/diagnostic-logs.png":::
323+
324+
If there's an error when processing an event, the problematic event is enqueued in a "poison queue" called `{queue-name}-poison` in the same storage account. Details about every processed event can be found in the `AHDSDicomAuditLogs` and `AHDSDicomDiagnosticLogs` tables by filtering for all logs where `OperationName = 'index-storage'`. The audit logs only record when the operation started and completed whereas the diagnostic table provides details about each operation including any errors, if any. Operations may be correlated across the tables using `CorrelationId`.
325+
326+
Failures are divided into two types: `User` and `Server`. User errors include any problem connecting to the storage account or with the DICOM file itself, while server errors include any unexpected error that prevents processing. Unlike server errors, the DICOM service doesn't retry user errors.
327+
328+
## Next Steps
329+
* [Interact with SOP instances using DICOMweb™](dicomweb-standard-apis-with-dicom-services.md)

0 commit comments

Comments
 (0)