You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/healthcare-apis/dicom/dicom-storage-indexing.md
+24-21Lines changed: 24 additions & 21 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,43 +11,46 @@ ms.author: wisuga
11
11
12
12
# Azure Data Lake Storage Indexing (Preview)
13
13
14
-
The [DICOM® service](overview.md) automatically uploads DICOM files to Azure Data Lake Storage when using STOW-RS. This way, users can query their data either using DICOMweb™ APIs, like WADO-RS, or Azure Blob/Data Lake APIs. However, with storage indexing DICOM files will be automatically indexed by the DICOM service after uploading the data directly to the ADLS Gen 2 file system. Whether the files were uploaded using STOW-RS, an Azure Blob SDK, or even `AzCopy`, they can be accessed using either DICOMweb™APIs or directly through the ADLS Gen 2 file system.
14
+
The [DICOM® service](overview.md) automatically uploads DICOM files to Azure Data Lake Storage (ADLS) when using [STOW-RS](dicom-services-conformance-statement-v2.md#store-stow-rs). That way, users can query their data either using [DICOMweb™ APIs](dicomweb-standard-apis-with-dicom-services.md), like [WADO-RS](dicom-services-conformance-statement-v2.md#retrieve-wado-rs), or [Azure Blob/Data Lake APIs](../../storage/blobs/storage-blob-upload.md). However, with storage indexing DICOM files are automatically indexed by the DICOM service after uploading the data directly to the ADLS Gen 2 file system. Whether the files were uploaded using STOW-RS, an Azure Blob SDK, or even [AzCopy](../../storage/common/storage-use-azcopy-v10.md), they can be accessed using DICOMweb™ or ADLS Gen 2 APIs.
15
15
16
16
## Prerequisites
17
17
18
-
* An Azure Storage account configured with Hierarchical Namespaces (HNS) enabled
18
+
* An Azure Storage account configured with [Hierarchical Namespaces (HNS) enabled](../../storage/blobs/data-lake-storage-introduction.md)
19
19
* Optionally, a DICOM Service [connected to the Azure Data Lake Storage file system](deploy-dicom-services-in-azure-data-lake.md)
20
20
21
21
## Configuring Storage Indexing
22
22
23
-
The DICOM service indexes an ADLS Gen 2 file system by reacting to Blob or Data Lake storage events. These events must be read from an Azure Storage Queue in the Azure Storage Account that contains the file system. Once in the queue, the DICOM service will asynchronously process each event and update the index accordingly.
23
+
The DICOM service indexes an ADLS Gen 2 file system by reacting to [Blob or Data Lake storage events](../../event-grid/event-schema-blob-storage.md). These events must be read from an [Azure Storage Queue](../../storage/queues/storage-queues-introduction.md) in the Azure Storage Account that contains the file system. Once in the queue, the DICOM service will asynchronously process each event and update the index accordingly.
24
24
25
25
### Create the Destination for Storage Events
26
26
27
-
First, create an [Azure Storage Queue](../../storage/queues/storage-queues-introduction.md)in the same Azure Storage Account connected to the DICOM service The DICOM service will also need access to the queue; it will need to be able to dequeue messages as well as enqueue its own messages for errors and complex tasks. So, make sure the same managed identity used by the DICOM service, either user-assigned or system-assigned, has the [**Storage Queue Data Contributor**](../../storage/queues/assign-azure-role-data-access.md) role assigned.
27
+
First, create a storage queue in the same Azure Storage Account connected to the DICOM service. The DICOM service will also need access to the queue; it will need to be able to dequeue messages as well as enqueue its own messages for errors and complex tasks. So, make sure the same managed identity used by the DICOM service, either user-assigned or system-assigned, has the [**Storage Queue Data Contributor**](../../role-based-access-control/built-in-roles.md#storage) role assigned.
28
28
29
29
### Publish Storage Events to the Queue
30
30
31
-
With the Storage Queue in place, events must be published from the Storage Account to an Azure Event Grid System Topic and routed to queue using an Azure Event Grid Subscription. Before creating the event subscription, be sure to grant the role **Storage Queue Data Message Sender** to the event subscription so that it is authorized to send messages. The event subscription can either use a user-assigned or system-assigned managed identity from the system topic to authenticate its operations.
31
+
With the Storage Queue in place, events must be published from the Storage Account to an [Azure Event Grid System Topic](../../event-grid/system-topics.md) and routed to queue using an [Azure Event Grid Subscription](../../event-grid/create-view-manage-event-subscriptions.md). Before creating the event subscription, be sure to grant the role [**Storage Queue Data Message Sender**](../../role-based-access-control/built-in-roles.md#storage) to the event subscription so that it is authorized to send messages. The event subscription can either use a [user-assigned or system-assigned managed identity from the system topic](../../event-grid/enable-identity-system-topics.md) to authenticate its operations.
32
32
33
-
By default, event subscriptions send all of the subscribed event types to their designated output. However, while the DICOM service will gracefully handle any message, it will only process ones that meet the following criteria:
34
-
- The message must be a Base64 `CloudEvent`
35
-
- The event type must be `Microsoft.Storage.BlobCreated` or `Microsoft.Storage.BlobDeleted`
36
-
- The file system must be the same one configured for the DICOM service
37
-
- The file path must be within `AHDS/<workspace-name>/dicom/<dicom-service-name>[/<partition-name>]`
38
-
- The file must be a DICOM file as defined in Part 10 of the DICOM standard
39
-
- The operation must not have been performed by the DICOM service itself
33
+
> [!NOTE]
34
+
> By default, event subscriptions send all of the subscribed event types to their designated output. However, while the DICOM service will gracefully handle any message, it will only process ones that meet the following criteria:
35
+
>- The message must be a Base64 [CloudEvent](../../event-grid/event-schema-subscriptions.md#cloud-event-schema)
36
+
>- The event type must one of the following event types:
37
+
>-`Microsoft.Storage.BlobCreated`
38
+
>-`Microsoft.Storage.BlobDeleted`
39
+
>- The file system must be the same one configured for the DICOM service
40
+
>- The file path must be within `AHDS/<workspace-name>/dicom/<dicom-service-name>[/<partition-name>]`
41
+
>- The file must be a DICOM file as defined in Part 10 of the DICOM standard
42
+
>- The operation must not have been performed by the DICOM service itself
40
43
41
-
Thankfully, the event subscription can be configured to filter out irrelevant data to avoid unnecessary processing and billing. Make sure to configure filter such that:
42
-
- The subject must begin with `/blobServices/default/containers/<file-system-name>/blobs/AHDS/<workspace-name>/dicom/<dicom-service-name>/`
43
-
- Optionally, the subject ends with `.dcm`
44
-
- Under advanced filters, the key `data.clientRequestId` does not begin with `tag:<workspace-name>-<dicom-service-name>.dicom.azurehealthcareapis.com,`
44
+
The event subscription can be configured to filter out irrelevant data to avoid unnecessary processing and billing. Make sure to configure filter such that:
45
+
- The *subject* must begin with `/blobServices/default/containers/<file-system-name>/blobs/AHDS/<workspace-name>/dicom/<dicom-service-name>/`
46
+
- Optionally, the *subject* ends with `.dcm`
47
+
- Under *advanced filters*, the key `data.clientRequestId` does not begin with `tag:<workspace-name>-<dicom-service-name>.dicom.azurehealthcareapis.com,`
45
48
46
49
### Enable Storage Indexing
47
50
48
-
Once the event grid subscription has been successfully configured, it's time to let the DICOM service know from where to read the storage events. Storage indexing is available starting in version `2025-04-01-preview` for Azure Resource Manager (ARM) which introduced a new property `storageConfiguration.storageIndexingConfiguration.storageEventQueueName`. It is currently unavailable to configure via the Azure Portal.
51
+
Once the event grid subscription has been successfully configured, the DICOM service must know from where to read the storage events. While in preview, storage indexing may only be configured using an [Azure Resource Manager (ARM) template](../../azure-resource-manager/templates/overview.md) using version `2025-04-01-preview`which introduced a new property called`storageConfiguration.storageIndexingConfiguration.storageEventQueueName`. It is currently unavailable to configure via the Azure Portal.
49
52
50
-
The example ARM template below can be deployed using the Azure CLI command [`az deployment group create`](../../azure-resource-manager/bicep/deploy-cli.md). It includes all of the necessary resources for a DICOM service:
53
+
The example ARM template below can be deployed using the [Azure CLI](../../azure-resource-manager/templates/deploy-cli.md). It includes all of the necessary resources for a DICOM service:
51
54
52
55
```json
53
56
{
@@ -316,11 +319,11 @@ The example ARM template below can be deployed using the Azure CLI command [`az
316
319
317
320
## Diagnosing Issues
318
321
319
-
:::image type="content" source="media/storage-indexing/diagnostic-logs.png" alt-text="A screenshot of the Azure Portal showing a KQL query of the AHDSDicomAuditLogs table. The example query is filtering for all logs where OperationName is the string 'index-storage'. Undernearth the KQL query is a table of results." lightbox="media/storage-indexing/diagnostic-logs.png":::
322
+
:::image type="content" source="media/storage-indexing/diagnostic-logs.png" alt-text="A screenshot of the Azure Portal showing a KQL query for the AHDSDicomAuditLogs table. The example query is filtering for all logs where OperationName is the string 'index-storage'. Undernearth the KQL query is a table of results." lightbox="media/storage-indexing/diagnostic-logs.png":::
320
323
321
-
If there is an error when processing an event, the problematic event will be enqueued in a "poison queue" called `<queue-name>-poison` in the same Storage Account. Details about every processed event can be found in the `AHDSDicomAuditLogs` and `AHDSDicomDiagnosticLogs` tables by filtering for all logs where `OperationName = 'index-storage'`. The audit logs will only record when the operation started and completed whereas the diagnostic table will provide details about each operation including any errors, if any. Operations may be correlated across the tables using `CorrelationId`.
324
+
If there is an error when processing an event, the problematic event will be enqueued in a "poison queue" called `<queue-name>-poison` in the same storage account. Details about every processed event can be found in the `AHDSDicomAuditLogs` and `AHDSDicomDiagnosticLogs` tables by filtering for all logs where `OperationName = 'index-storage'`. The audit logs will only record when the operation started and completed whereas the diagnostic table will provide details about each operation including any errors, if any. Operations may be correlated across the tables using `CorrelationId`.
322
325
323
326
Failures are divided into two types: `User` and `Server`. User errors include any problem connecting to the storage account or with the DICOM file itself, while server errors include any unexpected error that prevents processing. Unlike server errors, user errors are not retried by the DICOM service.
324
327
325
328
## Next Steps
326
-
*[Interact with data using DICOMweb™](dicomweb-standard-apis-with-dicom-services.md)
329
+
*[Interact with SOP instances using DICOMweb™](dicomweb-standard-apis-with-dicom-services.md)
0 commit comments