Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
622cd80
new file from #7100
ktalmor Sep 21, 2025
305c050
format and acolinx
ktalmor Sep 21, 2025
d88d61d
add new TOC entry for HTTP ingestion endpoint
ktalmor Sep 21, 2025
25e449b
formatting
ktalmor Sep 21, 2025
c6fea0b
new PR - ms.date refresh
ktalmor Sep 21, 2025
74180d5
content refresh
ktalmor Sep 21, 2025
9e73f4e
Merge branch 'main' of https://github.com/MicrosoftDocs/dataexplorer-…
ktalmor Sep 25, 2025
dc51288
Refresh screenshots; documentor
ktalmor Sep 25, 2025
21ee446
updates screenshots
ktalmor Sep 25, 2025
09d2485
documentor and acrolinx
ktalmor Sep 25, 2025
e81c4bd
update screens, documentor, acrolinx
ktalmor Sep 25, 2025
1f388fd
Merge branch 'KUS-freshnesh' of https://github.com/ktalmor/dataexplor…
ktalmor Sep 25, 2025
b99c292
more documentor
ktalmor Sep 25, 2025
1baf6fc
typo
ktalmor Sep 25, 2025
cb4a287
added lightboxes
ktalmor Sep 25, 2025
bd5a3e0
image
ktalmor Sep 28, 2025
8e42dd9
Merge branch 'main' of https://github.com/MicrosoftDocs/dataexplorer-…
ktalmor Sep 28, 2025
c2c3634
TOC update
ktalmor Sep 30, 2025
406efb9
Comments - Change sourceId type to Guid and clarify blob limits
ozmizrahiMicrosoft Sep 30, 2025
a037237
Merge branch 'main' of https://github.com/MicrosoftDocs/dataexplorer-…
ktalmor Sep 30, 2025
b0c4589
copied from Shani's PR 6877
ktalmor Sep 30, 2025
6232b56
ms.dates
ktalmor Sep 30, 2025
1ab3d80
implement PR review
ktalmor Sep 30, 2025
480bc13
PR Review Resolutions (#2)
v-regandowner Sep 30, 2025
be5b340
Merge pull request #7121 from ktalmor/KUS--ingest-with-HTTPS
Stacyrch140 Sep 30, 2025
3277845
Merge pull request #7122 from ktalmor/KUS-freshnesh
v-regandowner Sep 30, 2025
2409bd5
Merge branch 'main' of https://github.com/MicrosoftDocs/dataexplorer-…
ktalmor Sep 30, 2025
34b6783
Merge remote-tracking branch 'upstream/main' into KUS-single-blob-que…
ktalmor Sep 30, 2025
59a1517
Acrolinx
ktalmor Sep 30, 2025
7c6b63a
Merge pull request #7137 from ktalmor/KUS-single-blob-queued-ingestion
denrea Sep 30, 2025
331819c
Merge pull request #7139 from MicrosoftDocs/main
learn-build-service-prod[bot] Sep 30, 2025
7949f5f
Merging changes synced from https://github.com/MicrosoftDocs/dataexpl…
Sep 30, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 8 additions & 5 deletions data-explorer/kusto/ingestion-properties.md
Original file line number Diff line number Diff line change
@@ -1,22 +1,25 @@
---
title: Data ingestion properties
description: Learn about the various data ingestion properties.
description: Optimize data ingestion by configuring properties that align with your data formats.
ms.reviewer: tzgitlin
ms.topic: conceptual
ms.date: 08/11/2024
ms.date: 09/25/2025
monikerRange: "azure-data-explorer || microsoft-fabric"
---
# Data ingestion properties

> [!INCLUDE [applies](includes/applies-to-version/applies.md)] [!INCLUDE [fabric](includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](includes/applies-to-version/azure-data-explorer.md)]

Data ingestion is the process by which data is added to a table and is made available for query. You add properties to the ingestion command after the `with` keyword.
Data ingestion adds data to a table and makes it available for query. Add properties to the ingestion command after the `with` keyword.

[!INCLUDE [ingestion-properties](includes/ingestion-properties.md)]

## Related content

* Learn more about [supported data formats](ingestion-supported-formats.md)
::: moniker range="microsoft-fabric"
* [Supported data formats](ingestion-supported-formats.md)
::: moniker-end
::: moniker range="azure-data-explorer"
* Learn more about [data ingestion](/azure/data-explorer/ingest-data-overview)
* [Supported data formats](ingestion-supported-formats.md)
* [Data ingestion](/azure/data-explorer/ingest-data-overview)
::: moniker-end
50 changes: 25 additions & 25 deletions data-explorer/kusto/ingestion-supported-formats.md
Original file line number Diff line number Diff line change
@@ -1,39 +1,39 @@
---
title: Data formats supported for ingestion
description: Learn about the various data and compression formats supported for ingestion.
title: Data Ingestion - Supported Formats and Compression
description: Explore the various data formats like CSV, JSON, Parquet, and more, supported for ingestion. Understand compression options and best practices for data preparation.
ms.reviewer: tzgitlin
ms.topic: conceptual
ms.date: 08/11/2024
ms.date: 09/21/2025
monikerRange: "azure-data-explorer || microsoft-fabric"
---
# Data formats supported for ingestion

> [!INCLUDE [applies](includes/applies-to-version/applies.md)] [!INCLUDE [fabric](includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](includes/applies-to-version/azure-data-explorer.md)]

Data ingestion is the process by which data is added to a table and is made available for query. For all ingestion methods, other than ingest-from-query, the data must be in one of the supported formats. The following table lists and describes the formats that is supported for data ingestion.
Data ingestion adds data to a table and makes it available for query. For all ingestion methods, other than ingest-from-query, the data must be in one of the supported formats. The following table lists and describes the formats that are supported for data ingestion.

> [!NOTE]
> Before you ingest data, make sure that your data is properly formatted and defines the expected fields. We recommend using your preferred validator to confirm the format is valid. For example, you may find the following validators useful to check CSV or JSON files:
>
> * CSV: http://csvlint.io/
> * JSON: https://jsonlint.com/

For more information about why ingestion might fail, see [Ingestion failures](management/ingestion-failures.md)
To learn why ingestion might fail, see [Ingestion failures](management/ingestion-failures.md).
::: moniker range="azure-data-explorer"
and [Ingestion error codes in Azure Data Explorer](/azure/data-explorer/error-codes).
and [Ingestion error codes in Azure Data Explorer](/azure/data-explorer/error-codes).
::: moniker-end

|Format |Extension |Description|
|---------|------------|-----------|
|ApacheAvro|`.avro` |An [AVRO](https://avro.apache.org/docs/current/) format with support for [logical types](https://avro.apache.org/docs/++version++/specification/#Logical+Types). The following compression codecs are supported: `null`, `deflate`, and `snappy`. Reader implementation of the `apacheavro` format is based on the official [Apache Avro library](https://github.com/apache/avro). For information about ingesting Event Hub Capture Avro files, see [Ingesting Event Hub Capture Avro files](/azure/data-explorer/ingest-data-event-hub-overview#schema-mapping-for-event-hub-capture-avro-files). |
|Avro |`.avro` |A legacy implementation for [AVRO](https://avro.apache.org/docs/current/) format based on [.NET library](https://www.nuget.org/packages/Microsoft.Hadoop.Avro). The following compression codecs are supported: `null`, `deflate` (for `snappy` - use `ApacheAvro` data format). |
|ApacheAvro|`.avro` |An [Avro](https://avro.apache.org/docs/current/) format that supports [logical types](https://avro.apache.org/docs/++version++/specification/#Logical+Types). Supported compression codecs: `null`, `deflate`, and `snappy`. The reader implementation of the `apacheavro` format is based on the official [Apache Avro library](https://github.com/apache/avro). For details on ingesting Event Hubs Capture Avro files, see [Ingesting Event Hubs Capture Avro files](/azure/data-explorer/ingest-data-event-hub-overview#schema-mapping-for-event-hub-capture-avro-files). |
|Avro |`.avro` |A legacy implementation of the [Avro](https://avro.apache.org/docs/current/) format based on the [.NET library](https://www.nuget.org/packages/Microsoft.Hadoop.Avro). Supported compression codecs: `null` and `deflate`. To use `snappy`, use the `ApacheAvro` data format. |
|CSV |`.csv` |A text file with comma-separated values (`,`). See [RFC 4180: _Common Format and MIME Type for Comma-Separated Values (CSV) Files_](https://www.ietf.org/rfc/rfc4180.txt).|
|JSON |`.json` |A text file with JSON objects delimited by `\n` or `\r\n`. See [JSON Lines (JSONL)](http://jsonlines.org/).|
|MultiJSON|`.multijson`|A text file with a JSON array of property bags (each representing a record), or any number of property bags delimited by whitespace, `\n` or `\r\n`. Each property bag can be spread on multiple lines.|
|MultiJSON|`.multijson`|A text file with a JSON array of property bags (each representing a record), or any number of property bags delimited by whitespace, `\n`, or `\r\n`. Each property bag can span multiple lines.|
|ORC |`.orc` |An [ORC file](https://en.wikipedia.org/wiki/Apache_ORC).|
|Parquet |`.parquet` |A [Parquet file](https://en.wikipedia.org/wiki/Apache_Parquet). |
|PSV |`.psv` |A text file with pipe-separated values (<code>&#124;</code>). |
|RAW |`.raw` |A text file whose entire contents is a single string value. |
|RAW |`.raw` |A text file whose entire contents are a single string value. |
|SCsv |`.scsv` |A text file with semicolon-separated values (`;`).|
|SOHsv |`.sohsv` |A text file with SOH-separated values. (SOH is ASCII codepoint 1; this format is used by Hive on HDInsight.)|
|TSV |`.tsv` |A text file with tab-separated values (`\t`).|
Expand All @@ -43,42 +43,42 @@ and [Ingestion error codes in Azure Data Explorer](/azure/data-explorer/error-c

> [!NOTE]
>
> * Ingestion from data storage systems that provide ACID functionality on top of regular Parquet format files (e.g. Apache Iceberg, Apache Hudi, Delta Lake) is not supported.
> * Schema-less Avro is not supported.
> * Ingestion from data storage systems that provide ACID functionality on top of regular Parquet format files (for example, Apache Iceberg, Apache Hudi, and Delta Lake) isn't supported.
> * Schemaless Avro isn't supported.

::: moniker range="azure-data-explorer"
For more info on ingesting data using `json` or `multijson` formats, see [ingest json formats](/azure/data-explorer/ingest-json-formats).
For more information about ingesting data by using the `json` or `multijson` formats, see [Ingest JSON formats](/azure/data-explorer/ingest-json-formats).
::: moniker-end

## Supported data compression formats

Blobs and files can be compressed through any of the following compression algorithms:
Compress blobs and files with these algorithms:

|Compression|Extension|
|-----------|---------|
|gzip |.gz |
|zip |.zip |

Indicate compression by appending the extension to the name of the blob or file.
Indicate compression by appending the extension to the blob or file name.

For example:

* `MyData.csv.zip` indicates a blob or a file formatted as CSV, compressed with zip (archive or a single file)
* `MyData.json.gz` indicates a blob or a file formatted as JSON, compressed with gzip.
* `MyData.csv.zip` indicates a blob or file formatted as CSV, compressed with zip (archive or single file).
* `MyData.json.gz` indicates a blob or file formatted as JSON, compressed with gzip.

Blob or file names that don't include the format extensions but just compression (for example, `MyData.zip`) is also supported. In this case, the file format
must be specified as an ingestion property because it cannot be inferred.
Blob or file names that include only the compression extension (for example, `MyData.zip`) are also supported. In this case, specify the file format
as an ingestion property because it can't be inferred.

> [!NOTE]
>
> * Some compression formats keep track of the original file extension as part of the compressed stream. This extension is generally ignored for determining the file format. If the file format can't be determined from the (compressed) blob or file name, it must be specified through the `format` ingestion property.
> * Not to be confused with internal (chunk level) compression codec used by `Parquet`, `AVRO` and `ORC` formats. Internal compression name is usually added to a file name before file format extension, for example: `file1.gz.parquet`, `file1.snappy.avro`, etc.
> * [Deflate64/Enhanced Deflate](https://en.wikipedia.org/wiki/Deflate#Deflate64/Enhanced_Deflate) zip compression method is not supported. Please note that Windows built-in zip compressor may choose to use this compression method on files of size over 2GB.
> * Some compression formats store the original file extension in the compressed stream. Ignore this extension when you determine the file format. If you can't determine the file format from the compressed blob or file name, specify it with the `format` ingestion property.
> * Don't confuse these with internal chunk-level compression codecs used by `Parquet`, `AVRO`, and `ORC` formats. The internal compression name is usually added before the file format extension (for example, `file1.gz.parquet`, `file1.snappy.avro`).
> * The [Deflate64/Enhanced Deflate](https://en.wikipedia.org/wiki/Deflate#Deflate64/Enhanced_Deflate) zip compression method isn't supported. Windows built-in zip compressor can use this method on files larger than 2 GB.

## Related content

* Learn more about [supported data formats](ingestion-supported-formats.md)
* Learn more about [Data ingestion properties](ingestion-properties.md)
* [Supported data formats](ingestion-supported-formats.md)
* [Data ingestion properties](ingestion-properties.md)
::: moniker range="azure-data-explorer"
* Learn more about [data ingestion](/azure/data-explorer/ingest-data-overview)
* [Data ingestion](/azure/data-explorer/ingest-data-overview)
::: moniker-end
Original file line number Diff line number Diff line change
Expand Up @@ -3,23 +3,23 @@ title: .cancel queued ingestion operation command
description: Learn how to use the `.cancel queued operation` command to cancel a long-running operation.
ms.reviewer: vplauzon
ms.topic: reference
ms.date: 03/19/2025
ms.date: 09/30/2025
---
# .cancel queued ingestion operation command (preview)

> [!INCLUDE [applies](../../includes/applies-to-version/applies.md)] [!INCLUDE [azure-data-explorer](../../includes/applies-to-version/azure-data-explorer.md)]
> [!INCLUDE [applies](../../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../../includes/applies-to-version/azure-data-explorer.md)]

The `.cancel queued ingestion operation` command cancels an ingestion operation. This command is useful for aborting an ingestion operation that is taking too long to complete.
The `.cancel queued ingestion operation` command cancels an ingestion operation. Use this command to stop an ingestion operation that takes too long to finish.

The cancel operation command is done on a best effort basis. For example, ongoing ingestion processes or in-flight ingestion, might not get canceled.
The cancel operation command works on a best-effort basis. For example, ongoing ingestion processes or in-flight ingestion might not be canceled.

> [!NOTE]
>
> Queued ingestion commands are run on the data ingestion URI endpoint `https://ingest-<YourClusterName><Region>.kusto.windows.net`.
> Queued ingestion commands run on the data ingestion URI endpoint `https://ingest-<YourClusterName><Region>.kusto.windows.net`.

## Permissions

You must have at least [Table Ingestor](../../access-control/role-based-access-control.md) permissions to run this command.
You need at least [Table Ingestor](../../access-control/role-based-access-control.md) permissions to run this command.

## Syntax

Expand All @@ -31,42 +31,39 @@ You must have at least [Table Ingestor](../../access-control/role-based-access-c

| Name | Type | Required | Description |
|--|--|--|--|
| *IngestionOperationId* | `string` | :heavy_check_mark: | The unique ingestion operation ID returned from the running command.|
| *IngestionOperationId* | `string` | :heavy_check_mark: | The unique ingestion operation ID returned by the running command.|

## Returns

|Output parameter |Type |Description|
|---|---|---|
|IngestionOperationId | `string` |The unique operation identifier.|
|StartedOn | `datetime` |Date/time, in UTC, at which the `.ingest-from-storage-queued` was executed.|
|LastUpdatedOn | `datetime` |Date/time, in UTC, when the status was updated.|
|StartedOn | `datetime` |Date and time, in UTC, when the `.ingest-from-storage-queued` operation is executed.|
|LastUpdatedOn | `datetime` |Date and time, in UTC, when the status is updated.|
|State | `string` |The state of the operation.|
|Discovered | `long` |Count of the blobs that were listed from storage and queued for ingestion.|
|Pending | `long` |Count of the blobs to be ingested.|
|Canceled | `long` |Count of the blobs that were canceled due to a call to the [.cancel queued ingestion operation](cancel-queued-ingestion-operation-command.md) command.|
|Canceled | `long` |Count of the blobs that are canceled due to a call to the [.cancel queued ingestion operation](cancel-queued-ingestion-operation-command.md) command.|
|Ingested | `long` |Count of the blobs that have been ingested.|
|Failed | `long` |Count of the blobs that failed **permanently**.|
|SampleFailedReasons | `string` |A sample of reasons for blob ingestion failures.|
|Failed | `long` |Count of the blobs that fail **permanently**.|
|SampleFailedReasons | `string` |A sample of reasons for blob ingestion failure.|
|Database | `string` |The database where the ingestion process is occurring.|
|Table | `string` | The table where the ingestion process is occurring.|

>[!NOTE]
> If the ingestion operation was initiated with tracking disabled, cancellation commands execute on a best‑effort basis. The returned state may indicate: "Cancellation request received – service will attempt best effort cancellation (tracking isn't enabled on operation)"

## Example

The following example cancels the ingestion of operation `00001111;11112222;00001111-aaaa-2222-bbbb-3333cccc4444`.
This example cancels the ingestion operation `00001111;11112222;00001111-aaaa-2222-bbbb-3333cccc4444`.

```Kusto
.cancel queued ingestion operation '00001111;11112222;00001111-aaaa-2222-bbbb-3333cccc4444'
```

|IngestionOperationId|Started On |Last Updated On |State |Discovered |Pending| Canceled | Ingested |Failed|SampleFailedReasons|Database|Table|
|Ingestion operation ID|Started on|Last updated on|State|Discovered|Pending|Canceled|Ingested|Failed|Sample failed reasons|Database|Table|
|--|--|--|--|--|--|--|--|--|--|--|--|
|00001111;11112222;00001111-aaaa-2222-bbbb-3333cccc4444 |2025-03-20 15:03:11.0000000 ||Canceled | 10 |10 |0 |0 |0 | |TestDatabase|Logs|
|00001111;11112222;00001111-aaaa-2222-bbbb-3333cccc4444|2025-03-20 15:03:11.0000000||Canceled|10|10|0|0|0||TestDatabase|Logs|

## Related content

* [Queued ingestion overview](queued-ingestion-overview.md)
* [Data formats supported for ingestion](../../ingestion-supported-formats.md)
* [.ingest-from-storage-queued command](ingest-from-storage-queued.md)
* [.ingest-from-storage-queued into command](ingest-from-storage-queued.md)
Loading