You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/data-explorer/ingest-data-streaming.md
+15-12Lines changed: 15 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,9 +11,9 @@ ms.date: 08/30/2019
11
11
12
12
# Streaming ingestion (Preview)
13
13
14
-
Streaming ingestion is targeted for scenarios that require low latency with an ingestion time of less than 10 seconds for varied volume data. It's used to optimize operational processing of many tables, in one or more databases, where the stream of data into each table is relatively small (few records per second) but overall data ingestion volume is high (thousands of records per second).
14
+
Use streaming ingestion when you require low latency with an ingestion time of less than 10 seconds for varied volume data. It's used to optimize operational processing of many tables, in one or more databases, where the stream of data into each table is relatively small (few records per second) but overall data ingestion volume is high (thousands of records per second).
15
15
16
-
Use the classic (bulk) ingestion instead of streaming ingestion when the amount of data grows to more than 1 MB per second per table. Read [Data ingestion overview](/azure/data-explorer/ingest-data-overview) to learn more about the various methods of ingestion.
16
+
Use bulk ingestion instead of streaming ingestion when the amount of data grows to more than 1 MB per second per table. Read [Data ingestion overview](/azure/data-explorer/ingest-data-overview) to learn more about the various methods of ingestion.
17
17
18
18
## Prerequisites
19
19
@@ -43,22 +43,22 @@ Use the classic (bulk) ingestion instead of streaming ingestion when the amount
43
43
There are two supported streaming ingestion types:
44
44
45
45
46
-
*[**Event Hub**](/azure/data-explorer/ingest-data-event-hub) used as a data source
46
+
*[**Event Hub**](/azure/data-explorer/ingest-data-event-hub), which is used as a data source
47
47
***Custom ingestion** requires you to write an application that uses one of the Azure Data Explorer client libraries. See [streaming ingestion sample](https://github.com/Azure/azure-kusto-samples-dotnet/tree/master/client/StreamingIngestionSample) for a sample application.
48
48
49
49
### Choose the appropriate streaming ingestion type
50
50
51
51
||Event Hub |Custom Ingestion |
52
52
|---------|---------|---------|
53
-
|Data delay between ingestion initiation and the data available for query |longer delay |shorter delay |
54
-
|Development overhead |fast and easy setup, no development overhead |high development overhead for application to handle errors and ensure data consistency |
53
+
|Data delay between ingestion initiation and the data available for query |Longer delay |Shorter delay |
54
+
|Development overhead |Fast and easy setup, no development overhead |High development overhead for application to handle errors and ensure data consistency |
55
55
56
56
## Disable streaming ingestion on your cluster
57
57
58
58
> [!WARNING]
59
-
> Disabling streaming ingestion may take a few hours.
59
+
> Disabling streaming ingestion could take a few hours.
60
60
61
-
1. Drop [streaming ingestion policy](/azure/kusto/concepts/streamingingestionpolicy) from all relevant tables and databases. The streaming ingestion policy removal triggers streaming ingestion data movement from the initial storage to the permanent storage in the column store (extents or shards). The data movement can last between a few seconds to a few hours, depending on the amount of data in the initial storage and how the CPU and memory is used by the cluster.
61
+
1. Drop [streaming ingestion policy](/azure/kusto/concepts/streamingingestionpolicy) from all relevant tables and databases. The streaming ingestion policy removal triggers streaming ingestion data movement from the initial storage to the permanent storage in the column store (extents or shards). The data movement can last between a few seconds to a few hours, depending on the amount of data in the initial storage and the CPU, and memory utilization of the cluster..
62
62
1. In the Azure portal, go to your Azure Data Explorer cluster. In **Settings**, select **Configurations**.
63
63
1. In the **Configurations** pane, select **Off** to disable **Streaming ingestion**.
64
64
1. Select **Save**.
@@ -67,16 +67,19 @@ There are two supported streaming ingestion types:
67
67
68
68
## Limitations
69
69
70
+
Streaming ingestion doesn't support the following features:
*[Data mapping](/azure/kusto/management/mappings). Only [pre-created](/azure/kusto/management/tables#create-ingestion-mapping) data mapping is supported.
74
+
75
+
Other limitations include:
76
+
70
77
* Streaming ingestion performance and capacity scales with increased VM and cluster sizes. Concurrent ingestions are limited to six ingestions per core. For example, for 16 core SKUs, such as D14 and L16, the maximal supported load is 96 concurrent ingestions. For two core SKUs, such as D11, the maximal supported load is 12 concurrent ingestions.
71
78
* The data size limitation per ingestion request is 4 MB.
72
-
* Schema updates, such as creation and modification of tables and ingestion mappings, may take up to 5 minutes for the streaming ingestion service.
79
+
* Schema updates, such as creation and modification of tables and ingestion mappings, may take up to five minutes for the streaming ingestion service.
73
80
* Enabling streaming ingestion on a cluster, even when data isn't ingested via streaming, uses part of the local SSD disk of the cluster machines for streaming ingestion data and reduces the storage available for hot cache.
74
81
*[Extent tags](/azure/kusto/management/extents-overview#extent-tagging) can't be set on the streaming ingestion data.
75
82
76
-
Streaming ingestion doesn't support the following features:
*[Data mapping](/azure/kusto/management/mappings). Only [pre-created](/azure/kusto/management/tables#create-ingestion-mapping) data mapping is supported.
79
-
80
83
## Next steps
81
84
82
85
*[Query data in Azure Data Explorer](web-query-data.md)
0 commit comments