You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/data-explorer/lightingest.md
+8-8Lines changed: 8 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,7 +11,7 @@ ms.date: 04/01/2020
11
11
12
12
# Install and use LightIngest
13
13
14
-
LightIngest is a command-line utility for ad-hoc data ingestion into Azure Data Explorer.
14
+
LightIngest is a command-line utility for ad-hoc data ingestion into Azure Data Explorer.
15
15
The utility can pull source data from a local folder or from an Azure blob storage container.
16
16
17
17
## Prerequisites
@@ -42,20 +42,20 @@ The utility can pull source data from a local folder or from an Azure blob stora
42
42
>
43
43
>
44
44
45
-
1. Enter `LightIngest` followed by the connection string to the Azure Data Explorer cluster that will manage the ingestion.
45
+
1. Enter LightIngest followed by the connection string to the Azure Data Explorer cluster that will manage the ingestion.
46
46
Enclose the connection string in double quotes and follow the [Kusto connection strings specification](https://docs.microsoft.com/azure/kusto/api/connection-strings/kusto).
47
47
48
48
For example:
49
49
```
50
50
ingest-{Cluster name and region}.kusto.windows.net;AAD Federated Security=True -db:{Database} -table:Trips -source:"https://{Account}.blob.core.windows.net/{ROOT_CONTAINER};{StorageAccountKey}" -pattern:"*.csv.gz" -format:csv -limit:2 -ignoreFirst:true -cr:10.0 -dontWait:true
51
51
```
52
52
53
-
* The recommended method is for `LightIngest` to work with the ingestion endpoint at `https://ingest-{yourClusterNameAndRegion}.kusto.windows.net`. This way, the Azure Data Explorer service can manage the ingestion load, and you can easily recover from transient errors. However, you can also configure `LightIngest` to work directly with the engine endpoint (`https://{yourClusterNameAndRegion}.kusto.windows.net`).
53
+
* The recommended method is for LightIngest to work with the ingestion endpoint at `https://ingest-{yourClusterNameAndRegion}.kusto.windows.net`. This way, the Azure Data Explorer service can manage the ingestion load, and you can easily recover from transient errors. However, you can also configure LightIngest to work directly with the engine endpoint (`https://{yourClusterNameAndRegion}.kusto.windows.net`).
54
54
55
55
> [!Note]
56
56
> If you ingest directly with the engine endpoint, you don't need to include `ingest-` but there won't be a DM feature to protect the engine and improve the ingestion success rate.
57
57
58
-
* For optimal ingestion performance, it's important for LightIngest to know the raw data size and so `LightIngest` will estimate the uncompressed size of local files. However, `LightIngest` might not be able to correctly estimate the raw size of compressed blobs without first downloading them. Therefore, when ingesting compressed blobs, set the `rawSizeBytes` property on the blob metadata to uncompressed data size in bytes.
58
+
* For optimal ingestion performance, it's important for LightIngest to know the raw data size and so LightIngest will estimate the uncompressed size of local files. However, LightIngest might not be able to correctly estimate the raw size of compressed blobs without first downloading them. Therefore, when ingesting compressed blobs, set the `rawSizeBytes` property on the blob metadata to uncompressed data size in bytes.
59
59
60
60
## General command-line arguments
61
61
@@ -79,11 +79,11 @@ The utility can pull source data from a local folder or from an Azure blob stora
79
79
### Using CreationTimePattern argument
80
80
81
81
The `-creationTimePattern` argument extracts the CreationTime property from the file or blob path. The pattern doesn't need to reflect the entire item path, just the section enclosing the timestamp you want to use.
82
-
The value of the argument must include the following:
82
+
83
+
The argument values must include:
83
84
* Constant test immediately preceding the timestamp, enclosed in single quotes
84
85
* The timestamp format, in standard [.NET DateTime notation](https://docs.microsoft.com/dotnet/standard/base-types/custom-date-and-time-format-strings)
85
-
* Constant text immediately following the timestamp
86
-
For example, if blob names end with 'historicalvalues19840101.parquet' (the timestamp is four digits for the year, two digits for the month and two digits for the day of month), the corresponding value for the `-creationTimePattern` argument is:
86
+
* Constant text immediately following the timestamp. For example, if blob names end with 'historicalvalues19840101.parquet' (the timestamp is four digits for the year, two digits for the month, and two digits for the day of month), the corresponding value for the `-creationTimePattern` argument is:
87
87
88
88
```
89
89
ingest-{Cluster name and region}.kusto.windows.net;AAD Federated Security=True -db:{Database} -table:Trips -source:"https://{Account}.blob.core.windows.net/{ROOT_CONTAINER};{StorageAccountKey}" -creationTimePattern:"'historicalvalues'yyyyMMdd'.parquet'"
@@ -104,7 +104,7 @@ ingest-{Cluster name and region}.kusto.windows.net;AAD Federated Security=True -
104
104
|-devTracing |-trace |string |Optional |If set, diagnostic logs are written to a local directory (by default, `RollingLogs` in the current directory, or can be modified by setting the switch value) |
105
105
106
106
## Blob metadata properties
107
-
When used with Azure blobs, `LightIngest` will use certain blob metadata properties to augment the ingestion process.
107
+
When used with Azure blobs, LightIngest will use certain blob metadata properties to augment the ingestion process.
0 commit comments