You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: data-explorer/ingest-data-overview.md
+18-8Lines changed: 18 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,7 +3,7 @@ title: Azure Data Explorer data ingestion overview
3
3
description: Learn about the different ways you can ingest (load) data in Azure Data Explorer
4
4
ms.reviewer: akshay.dixit
5
5
ms.topic: conceptual
6
-
ms.date: 02/16/2024
6
+
ms.date: 04/07/2025
7
7
---
8
8
9
9
# Azure Data Explorer data ingestion overview
@@ -79,8 +79,7 @@ Azure Data Explorer offers the following ingestion management commands, which in
79
79
***Ingest from storage**: The [.ingest into command](/kusto/management/data-ingestion/ingest-from-storage?view=azure-data-explorer&preserve-view=true) gets the data to ingest from external storage, such as Azure Blob Storage, accessible by your cluster and pointed-to by the command.
80
80
81
81
> [!NOTE]
82
-
> In the event of a failure, ingestion is performed again and is retried for up to 48 hours using the exponential backoff method for wait time between tries.
83
-
82
+
> In the event of a failure, ingestion is performed again, and is retried for up to 48 hours using the exponential backoff method for wait time between tries.
84
83
85
84
## Compare ingestion methods
86
85
@@ -89,7 +88,7 @@ The following table compares the main ingestion methods:
89
88
| Ingestion name | Data type | Maximum file size | Streaming, queued, direct | Most common scenarios | Considerations |
90
89
|--|--|--|--|--|--|
91
90
|[Apache Spark connector](spark-connector.md)| Every format supported by the Spark environment | Unlimited | Queued | Existing pipeline, preprocessing on Spark before ingestion, fast way to create a safe (Spark) streaming pipeline from the various sources the Spark environment supports. | Consider cost of Spark cluster. For batch write, compare with Azure Data Explorer data connection for Event Grid. For Spark streaming, compare with the data connection for event hub. |
92
-
|[Azure Data Factory (ADF)](data-factory-integration.md)|[Supported data formats](/azure/data-factory/copy-activity-overview#supported-data-stores-and-formats)| Unlimited. Inherits ADF restrictions. | Queued or per ADF trigger | Supports formats that are unsupported, such as Excel and XML, and can copy large files from over 90 sources, from on perm to cloud | This method takes relatively more time until data is ingested. ADF uploads all data to memory and then begins ingestion. |
91
+
|[Azure Data Factory (ADF)](data-factory-integration.md)|[Supported data formats](/azure/data-factory/copy-activity-overview#supported-data-stores-and-formats)| Unlimited. Inherits ADF restrictions. | Queued or per ADF trigger | Supports formats that are unsupported, such as Excel and XML, and can copy large files from over 90 sources, from on-premises to cloud | This method takes relatively more time until data is ingested. ADF uploads all data to memory and then begins ingestion. |
93
92
|[Event Grid](ingest-data-event-grid-overview.md)|[Supported data formats](ingest-data-event-grid-overview.md#data-format)| 1 GB uncompressed | Queued | Continuous ingestion from Azure storage, external data in Azure storage | Ingestion can be triggered by blob renaming or blob creation actions |
|[Get data experience](get-data-file.md)|*SV, JSON | 1 GB uncompressed | Queued or direct ingestion | One-off, create table schema, definition of continuous ingestion with Event Grid, bulk ingestion with container (up to 5,000 blobs; no limit when using historical ingestion) ||
@@ -105,11 +104,22 @@ For information on other connectors, see [Connectors overview](integrate-data-ov
105
104
106
105
## Permissions
107
106
108
-
The following list describes the permissions required for various ingestion scenarios:
107
+
The following list describes the [permissions](/kusto/access-control/role-based-access-control?view=azure-data-explorer&preserve-view=true) required for various ingestion scenarios:
108
+
109
+
* To create a new table, you must have at least Database User permissions.
110
+
* To ingest data into an existing table, without changing its schema, you must have at least Table Ingestor permissions.
111
+
* To change the schema of an existing table, you must have at least Table Admin or Database Admin permissions.
109
112
110
-
* To create a new table requires at least Database User permissions.
111
-
* To ingest data into an existing table, without changing its schema, requires at least Database Ingestor permissions.
112
-
* To change the schema of an existing table requires at least Table Admin or Database Admin permissions.
113
+
The following table describes the permissions required for each ingestion method:
114
+
115
+
| Ingestion method | Permissions |
116
+
|--|--|
117
+
|[One-time ingestion](#one-time-data-ingestion)| At least Table Ingestor |
118
+
|[Continuous streaming ingestion](#continuous-data-ingestion)| At least Table Ingestor |
119
+
|[Continuous queued ingestion](#continuous-data-ingestion)| At least Table Ingestor |
120
+
|[Direct inline ingestion](#direct-ingestion-with-management-commands)| At least Table Ingestor and also Database Viewer |
121
+
|[Direct ingestion from query](#direct-ingestion-with-management-commands)| At least Table Ingestor and also Database Viewer |
122
+
|[Direct ingestion from storage](#direct-ingestion-with-management-commands)| At least Table Ingestor |
113
123
114
124
For more information, see [Kusto role-based access control](/kusto/access-control/role-based-access-control?view=azure-data-explorer&preserve-view=true).
0 commit comments