Skip to content

Commit 9e968f7

Browse files
Merge pull request #7217 from vplauzon/vpl/light-ingest-commands
Vpl/light ingest commands
2 parents bb82385 + 6124db0 commit 9e968f7

File tree

1 file changed

+10
-8
lines changed

1 file changed

+10
-8
lines changed

data-explorer/lightingest.md

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -3,11 +3,11 @@ title: Use LightIngest to ingest data into Azure Data Explorer
33
description: Learn about LightIngest, a command-line utility for ad-hoc data ingestion into Azure Data Explorer.
44
ms.reviewer: tzgitlin
55
ms.topic: how-to
6-
ms.date: 11/03/2025
6+
ms.date: 11/18/2025
77
---
88

99
# Use LightIngest to ingest data into Azure Data Explorer
10-
10+
1111
[LightIngest](https://github.com/Azure/Kusto-Lightingest/blob/main/README.md) is a command-line utility for ad-hoc data ingestion into Azure Data Explorer. The utility can pull source data from a local folder, an Azure blob storage container, or an Amazon S3 bucket.
1212

1313
LightIngest is most useful when you want to ingest a large amount of data, because there's no time constraint on ingestion duration. It's also useful when you want to later query records according to the time they were created, and not the time they were ingested.
@@ -20,13 +20,16 @@ For an example of how to autogenerate a LightIngest command, see [ingest histori
2020

2121
* LightIngest. Get LightIngest in one of two ways:
2222
* [Download LightIngest binaries for your operating system](https://github.com/Azure/Kusto-Lightingest/releases). Make sure to unzip the binaries after download.
23-
23+
2424
* [Install LightIngest as a .NET tool](https://www.nuget.org/packages/Microsoft.Azure.Kusto.LightIngest). This method requires that you have the .NET SDK version 6.0 or higher installed on your machine. Then, run the following command:
2525

2626
```
2727
dotnet tool install -g Microsoft.Azure.Kusto.LightIngest
2828
```
2929
30+
> [!NOTE]
31+
> Consider using the [queued ingestion commands](/kusto/management/data-ingestion/queued-ingestion-use-case?view=azure-data-explorer&preserve-view=true) as they do not require installing software.
32+
3033
## Run LightIngest
3134
3235
To run LightIngest:
@@ -110,18 +113,18 @@ The argument values must include:
110113
> [!IMPORTANT]
111114
> When you specify that the creation time should be overridden, make sure the `Lookback` property in the target table's effective [Extents merge policy](/kusto/management/merge-policy?view=azure-data-explorer&preserve-view=true) aligns with the values in your file or blob paths.
112115
113-
**Examples**
116+
**Examples**
117+
118+
* A blob name that contains the datetime as follows: `historicalvalues19840101.parquet` (the timestamp is four digits for the year, two digits for the month, and two digits for the day of month),
114119
115-
* A blob name that contains the datetime as follows: `historicalvalues19840101.parquet` (the timestamp is four digits for the year, two digits for the month, and two digits for the day of month),
116-
117120
The value for `-creationTimePattern` argument is part of the filename: *"'historicalvalues'yyyyMMdd'.parquet'"*
118121
119122
```kusto
120123
LightIngest "https://ingest-{Cluster name and region}.kusto.windows.net;Fed=True" -db:{Database} -table:Trips -source:"https://{Account}.blob.core.windows.net/{ROOT_CONTAINER};{StorageAccountKey}" -creationTimePattern:"'historicalvalues'yyyyMMdd'.parquet'"
121124
-pattern:"*.parquet" -format:parquet -limit:2 -cr:10.0 -dontWait:true
122125
```
123126
124-
* For a blob URI that refers to hierarchical folder structure, like `https://storageaccount/mycontainer/myfolder/2002/12/01/blobname.extension`,
127+
* For a blob URI that refers to hierarchical folder structure, like `https://storageaccount/mycontainer/myfolder/2002/12/01/blobname.extension`,
125128
126129
The value for `-creationTimePattern` argument is part of the folder structure: *"'folder/'yyyy/MM/dd'/blob'"*
127130
@@ -220,4 +223,3 @@ LightIngest performs three actions that can use managed identity for authenticat
220223
* **Connect to Azure Storage to download blobs**: Use "-ingestmi" to specify a managed identity installed on the Kusto service that has read privileges on the storage container.
221224
222225
* **Connect to Azure Storage to list container blobs**: Use the "-storageMi" argument to specify a managed identity installed on the client VM that has list privileges on the storage container. If you use this method but not the previous one (connect to Azure storage to download blobs), the managed identity must have read privileges as well and a token is passed to the Kusto service to be used for the ingestion. Set all three arguments.
223-

0 commit comments

Comments
 (0)