You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: data-explorer/ingest-data-event-grid-overview.md
+10-10Lines changed: 10 additions & 10 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,7 +3,7 @@ title: Ingest from storage using Event Grid subscription - Azure Data Explorer
3
3
description: This article describes Ingest from storage using Event Grid subscription in Azure Data Explorer.
4
4
ms.reviewer: leshalev
5
5
ms.topic: how-to
6
-
ms.date: 06/10/2025
6
+
ms.date: 08/11/2025
7
7
ms.custom: devx-track-azurepowershell
8
8
---
9
9
# Event Grid data connection
@@ -21,7 +21,7 @@ For general information about data ingestion in Azure Data Explorer, see [Azure
21
21
*[Managed Identity](managed-identities-overview.md) based data connection (recommended): Using a managed identity-based data connection is the most secure way to connect to data sources. It provides full control over the ability to fetch data from a data source.
22
22
23
23
Setup of an Event Grid data connection using managed identity requires the following steps and permissions:
24
-
1. Make sure you have [EventGrid Contributor](/azure/role-based-access-control/built-in-roles/integration#eventgrid-contributor) role assignment on the Azure subscribtion of the source data storage account.
24
+
1. Make sure you have [EventGrid Contributor](/azure/role-based-access-control/built-in-roles/integration#eventgrid-contributor) role assignment on the Azure subscription of the source data storage account.
25
25
1.[Add a managed identity to your cluster](configure-managed-identities-cluster.md).
26
26
1.[Grant permissions to the managed identity on the data source](ingest-data-managed-identity.md#grant-permissions-to-the-managed-identity). To fetch data from Azure Storage, the managed identity must have at least [Storage Blob Data Reader](/azure/role-based-access-control/built-in-roles#storage-blob-data-reader) permissions on the Azure Storage account.
27
27
1. Grant permissions to the managed identity on the event hub. To fetch blob notifications from the event hub, the managed identity must have [Azure Event Hubs Data Receiver](/azure/role-based-access-control/built-in-roles#azure-event-hubs-data-receiver) permissions on the Azure Event Hubs.
@@ -76,7 +76,7 @@ Routing data to an alternate database is off by default. To send the data to a d
76
76
To specify an alternate database, set the *Database*[ingestion property](#ingestion-properties).
77
77
78
78
> [!WARNING]
79
-
> Specifying an alternate database without setting the connection as a multi-database data connection will cause the ingestion to fail.
79
+
> Specifying an alternate database without setting the connection as a multi-database data connection causes the ingestion to fail.
80
80
81
81
### Route event data to an alternate table
82
82
@@ -122,9 +122,10 @@ You can create a blob from a local file, set ingestion properties to the blob me
122
122
>
123
123
> * We highly recommend using `BlockBlob` to generate data, as using `AppendBlob` may result in unexpected behavior.
124
124
> * Using Azure Data Lake Gen2 storage SDK requires using `CreateFile` for uploading files and `Flush` at the end with the close parameter set to `true`. For a detailed example of Data Lake Gen2 SDK correct usage, see [Use the Event Grid data connection](create-event-grid-connection.md?tabs=azure-data-lake#use-the-event-grid-data-connection).
125
-
> * Triggering ingestion following a `CopyBlob` operation is not supported for storage accounts that have the hierarchical namespace feature enabled on them.
125
+
> * Triggering ingestion following a `CopyBlob` operation isn't supported for storage accounts that have the hierarchical namespace feature enabled on them.
126
126
> * When the event hub endpoint doesn't acknowledge receipt of an event, Azure Event Grid activates a retry mechanism. If this retry delivery fails, Event Grid can deliver the undelivered events to a storage account using a process of *dead-lettering*. For more information, see [Event Grid message delivery and retry](/azure/event-grid/delivery-and-retry#retry-schedule-and-duration).
127
-
> * Using the "OpenWrite" API to write to a blob is not recommended, as it triggers a notification for an empty blob and causes an empty-blob error. Additionally, flush the stream only once to prevent duplicate notifications and multiple ingestions of the same blob.
127
+
> * Using the "OpenWrite" API to write to a blob isn't recommended, as it triggers a notification for an empty blob and causes an empty-blob error. Additionally, flush the stream only once to prevent duplicate notifications and multiple ingestions of the same blob.
128
+
> * Azure Data Explorer tries to filter out duplicate notifications for the same blob sent by upstream services like Event Grid or Storage. When it detects a duplicate event, it skips ingestion and logs the error `BlobAlreadyReceived_DuplicateEventGridNotification`, which means the blob is already processed.
128
129
129
130
## Rename blobs
130
131
@@ -137,22 +138,21 @@ When using ADLSv2, you can rename a blob to trigger blob ingestion to Azure Data
137
138
138
139
## Delete blobs using storage lifecycle
139
140
140
-
Azure Data Explorer won't delete the blobs after ingestion. Use [Azure Blob storage lifecycle](/azure/storage/blobs/storage-lifecycle-management-concepts?tabs=azure-portal) to manage your blob deletion. It's recommended to keep the blobs for three to five days.
141
+
Logic build into the Azure Data Explorer won't delete the blobs after ingestion. Use [Azure Blob storage lifecycle](/azure/storage/blobs/storage-lifecycle-management-concepts?tabs=azure-portal) to manage your blob deletion. It's recommended to keep the blobs for three to five days.
141
142
142
143
## Known Event Grid issues
143
144
144
145
### Working without local authentication
145
146
146
147
If local authentication is disabled on the Event Hubs namespace that contains the event hub used for streaming notifications, use the following steps to ensure that data flows properly from storage to the event hub using managed identities:
147
148
148
-
#### [Steps](#tab/steps)
149
+
#### [Steps](#tab/steps)
149
150
150
151
1. Assign a system-assigned managed identity to the Event Grid system topic of the storage account. For more information, see [Enable managed identity for system topics](/azure/event-grid/enable-identity-system-topics).
151
152
1. Grant the managed identity sender permissions by assigning it the *Azure Event Hubs Data Sender* role on the event hub. For more information, see [Add identity to Azure roles on destinations](/azure/event-grid/add-identity-roles).
152
153
1. Make sure that the Event Grid subscription uses managed identity for event delivery. For more information, see [Create event subscriptions that use an identity](/azure/event-grid/managed-service-identity).
153
154
154
-
155
-
#### [PowerShell script](#tab/powershell)
155
+
#### [PowerShell script](#tab/powershell)
156
156
157
157
```powershell
158
158
$eventGridSubscriptionId = "<AZURE SUBSCRIPTION ID OF EVENTGRID SYSTEM TOPIC>"
@@ -230,7 +230,7 @@ When using Azure Data Explorer to [export](/kusto/management/data-export/export-
230
230
231
231
### Emulating Storage events from custom components
232
232
233
-
When using custom components to emulate Azure Storage events, the emulated events must strictly comply with [Azure Blob Storage event schema](/azure/event-grid/event-schema-blob-storage?tabs=cloud-event-schema), as Azure Data Explorer will discard events that cannot be parsed by the Event Grid SDK.
233
+
When using custom components to emulate Azure Storage events, the emulated events must strictly comply with [Azure Blob Storage event schema](/azure/event-grid/event-schema-blob-storage?tabs=cloud-event-schema), as Azure Data Explorer will discard events that can't be parsed by the Event Grid SDK.
0 commit comments