Changes per Ornat's comments and new content

Author: yossi-karp

articles/data-explorer/lightingest.md

@@ -1,6 +1,6 @@
 ---
-title: LightIngest
-description: Learn about LightIngest, a command-line utility for ad-hoc data ingestion into Kusto
+title: LightIngest is a command-line utility for ingestion into Azure Data Explorer.
+description: Learn about LightIngest, a command-line utility for ad-hoc data ingestion into Azure Data Explorer.
 author: orspod
 ms.author: orspodek
 ms.reviewer: tzgitlin
@@ -11,7 +11,7 @@ ms.date: 03/17/2020
 
 # LightIngest
 
-LightIngest is a command-line utility for ad-hoc data ingestion into Kusto.
+LightIngest is a command-line utility for ad-hoc data ingestion into Azure Data Explorer.
 The utility can pull source data from a local folder or from an Azure blob storage container.
 
 ## Prerequisites
@@ -39,25 +39,24 @@ The utility can pull source data from a local folder or from an Azure blob stora
     >
     >![Command line Help](media/lightingest/lightingest-cmd-line-help.png)
 
-1. (Mandatory) Enter `LightIngest` followed by the connection string to the Kusto cluster that will manage the ingestion.
-    The connection string should be enclosed in double quotes and follow the [Kusto connection strings specification](https://docs.microsoft.com/azure/kusto/api/connection-strings/kusto).
+1. Enter `LightIngest` followed by the connection string to the Azure Data Explorer cluster that will manage the ingestion.
+    Enclose the connection string in double quotes and follow the [Kusto connection strings specification](https://docs.microsoft.com/azure/kusto/api/connection-strings/kusto).
 
     For example:
     ```
     LightIngest "Data Source=https://ingest-tzgitlin.westus.kusto.windows.net;AAD Federated Security=True"  -db:TzviaTest -table:Trips -source:"https://tzgitlinegdemo2.blob.core.windows.net/saadxworkshop1;VXPnUFzvBRLBIqEgcA0hRnSXmq69jVyZMChgUn5BeVwhjLnx4ucHZ8RPGTZ0F2hXHnC/vesoFSMF5f4gepeTJw==" -pattern:"*.csv.gz" -format:csv -limit:2 -ignoreFirst:true -cr:10.0 -dontWait:true
     ```
 
-> [!Note]
-> * It's recommended to configure `LightIngest` to work with the ingestion endpoint at `https://ingest-{yourClusterNameAndRegion}.kusto.windows.net`. This way the Kusto service can manage the ingestion load, and it provides for recovery in case of transient errors. However, you can also configure `LightIngest` to work directly with the engine endpoint (`https://{yourClusterNameAndRegion}.kusto.windows.net`).
-> * Knowing the raw data size is important for optimal ingestion performance. `LightIngest` will estimate the uncompressed size of local files. However, for compressed blobs, `LightIngest` could have difficulties correctly estimating their raw size without first downloading them. When ingesting compressed blobs, it will be helpful for `LightIngest` performance if you set the `rawSizeBytes` property on the blob metadata to uncompressed data size in bytes.
+* The recommended method is for `LightIngest` to work with the ingestion endpoint at `https://ingest-{yourClusterNameAndRegion}.kusto.windows.net`. This way, the Azure Data Explorer service can manage the ingestion load, and you can easily recover from transient errors. However, you can also configure `LightIngest` to work directly with the engine endpoint (`https://{yourClusterNameAndRegion}.kusto.windows.net`).
+* For optimal ingestion performance, it is important for LightIngest to know the raw data size and so `LightIngest` will estimate the uncompressed size of local files. However, `LightIngest` might not be able to correctly estimate the raw size of compressed blobs without first downloading them. Therefore, when ingesting compressed blobs, set the `rawSizeBytes` property on the blob metadata to uncompressed data size in bytes.
 
-## Command line arguments reference
+## General command-line arguments
 
 |Argument name         |Short name   |Type    |Mandatory |Description                                |
 |----------------------|-------------|--------|----------|-------------------------------------------|
-|                      |             |string  |Mandatory |[Kusto Connection String](https://docs.microsoft.com/azure/kusto/api/connection-strings/kusto) specifying the Kusto endpoint that will handle the ingestion. Should be enclosed in double quotes |
-|-database             |-db          |string  |Optional  |Target Kusto database name |
-|-table                |             |string  |Mandatory |Target Kusto table name |
+|                      |             |string  |Mandatory |[Azure Data Explorer Connection String](https://docs.microsoft.com/azure/kusto/api/connection-strings/kusto) specifying the Kusto endpoint that will handle the ingestion. Should be enclosed in double quotes |
+|-database             |-db          |string  |Optional  |Target Azure Data Explorer database name |
+|-table                |             |string  |Mandatory |Target Azure Data Explorer table name |
 |-sourcePath           |-source      |string  |Mandatory |Path to source files or root URI of the blob container. If the data is in blobs, must contain storage account key or SAS. Recommended to enclose in double quotes |
 |-prefix               |             |string  |Optional  |When the source data to ingest resides on blob storage, this URL prefix is shared by all blobs, excluding the container name. For example, if the data is in `MyContainer/Dir1/Dir2`, then the prefix should be `Dir1/Dir2`. Enclosing in double quotes is recommended |
 |-pattern              |             |string  |Optional  |Pattern by which source files/blobs are picked. Supports wildcards. For example, `"*.csv"`. Recommended to enclose in double quotes |
@@ -68,11 +67,11 @@ The utility can pull source data from a local folder or from an Azure blob stora
 |-tag                  |             |string  |Optional  |[Tags](https://docs.microsoft.com/azure/kusto/management/extents-overview#extent-tagging) to associate with the ingested data. Multiple occurrences are permitted |
 |-dontWait             |             |bool    |Optional  |If set to 'true', does not wait for ingestion completion. Useful when ingesting large amounts of files/blobs |
 
-### Additional arguments for advanced scenarios
+### Command-line arguments for advanced scenarios
 
 |Argument name         |Short name   |Type    |Mandatory |Description                                |
 |----------------------|-------------|--------|----------|-------------------------------------------|
-|-compression          |-cr          |double  |Optional  |Compression ratio hint. Useful when ingesting compressed files/blobs to help Kusto assess the raw data size. Calculated as original size divided by compressed size |
+|-compression          |-cr          |double  |Optional  |Compression ratio hint. Useful when ingesting compressed files/blobs to help Azure Data Explorer assess the raw data size. Calculated as original size divided by compressed size |
 |-limit                |-l           |integer |Optional  |If set, limits the ingestion to first N files |
 |-ingestTimeout        |             |integer |Optional  |Timeout in minutes for all ingest operations completion. Defaults to `60`|
 |-forceSync            |             |bool    |Optional  |If set, forces synchronous ingestion. Defaults to `false` |
@@ -82,34 +81,40 @@ The utility can pull source data from a local folder or from an Azure blob stora
 
 ## Usage examples
 
-**Example 1**
+**Ingesting a specific number of blobs in JSON format**
 
-* Ingest two blobs under a specified storage account {Account}, files of `CSV` format matching the pattern `.csv.gz`.
-* Destination is the database {Database}, the table `Trips`, ignoring the first record
+* Ingest two blobs under a specified storage account {Account}, files of `JSON` format matching the pattern `.json`
+* Destination is the database {Database}, the table `SampleData`
 * Data will be ingested at a compression ratio of 10.0
 * LightIngest won't wait for the ingestion to be completed
 
-To use the LightIngest command, below:
-1. Create a table command.
-1. Create a mapping command.
-1. Copy the cluster name and paste it into the LightIngest command {Cluster Name and Region}.
-1. Enter the database name into the LightIngest command {Database}.
-1. Enter the table name into the LightIngest command.
+To use the LightIngest command below:
+1. Create a table command and enter the table name into the LightIngest command, replacing `SampleData`.
+1. Create a mapping command and enter the IngestionMappingRef command, replacing `SampleData_mapping`.
+1. Copy your cluster name and enter it into the LightIngest command, replacing `{Cluster Name and Region}`.
+1. Enter the database name into the LightIngest command, replacing `{Database}`.
+1. Replace `{Account}` with your account name.
 
-```
-LightIngest "Data Source=https://ingest-{Cluster Name and Region}.kusto.windows.net;AAD Federated Security=True"  
-    -db:{Database}
-    -table:Trips
-    -source:"https://{Account}.blob.core.windows.net/saadxworkshop1;VXPnUFzvBRLBIqEgcA0hRnSXmq69jVyZMChgUn5BeVwhjLnx4ucHZ8RPGTZ0F2hXHnC/vesoFSMF5f4gepeTJw==" 
-    -pattern:"*.csv.gz" 
-    -format:csv 
-    -limit:2 
-    -ignoreFirst:true 
-    -cr:10.0 
-    -dontWait:true
-```
+    ```
+    LightIngest "Data Source=https://{Cluster name and region}kusto.windows.net;AAD Federated Security=True"  
+        -db:{Database name} 
+        -table:SampleData 
+        -source:"https://{Account}.blob.core.windows.net/data?sp=rl&st=2020-03-17T14:10:02Z&se=2022-12-31T14:10:00Z&sv=2019-02-02&sr=c&sig=QY6%2B1jAjIBQzkPIatkdDlbr%2FggUyq4gklmt%2FcOUM31Y%3D" 
+        -IngestionMappingRef:SampleData_mapping 
+        -pattern:"*.json" 
+        -format:JSON 
+        -limit:2 
+        -cr:10.0 
+        -dontWait:true
+    ```
+
+     ![Injestion result](media/lightingest/lightingest-cmd-line-result.png)
+
+1. In Azure Data Explorer, open query count.
+    ![Injestion result in Azure Data Explorer](media/lightingest/lightingest-showfailure-count.png)
+
+**Ingesting blobs using a storage account key or a SAS token**
 
-**Example 2**
 * Ingest 10 blobs under specified storage account `ACCOUNT`, in folder `DIR`, under container `CONT`, and matching the pattern `*.csv.gz`
 * Destination is database `DB`, table `TABLE`, and the ingestion mapping `MAPPING` is precreated on the destination
 * The tool will wait until the ingest operations complete
@@ -136,7 +141,8 @@ LightIngest.exe "https://ingest-{clusterAndRegion}.kusto.windows.net;Fed=True;In
   -limit:10
 ```
 
-**Example 3**
+**Ingesting all blobs in a container, not including header rows**
+
 * Ingest all blobs under specified storage account `ACCOUNT`, in folder `DIR1/DIR2`, under container `CONT`, and matching the pattern `*.csv.gz`
 * Destination is database `DB`, table `TABLE`, and the ingestion mapping `MAPPING` is precreated on the destination
 * Source blobs contain header line, so the tool is instructed to drop the first record of each blob
@@ -154,7 +160,8 @@ LightIngest.exe "https://ingest-{clusterAndRegion}.kusto.windows.net;Fed=True"
   -ignoreFirstRow:true
 ```
 
-**Example 4**
+**Ingesting all JSON files from a path**
+
 * Ingest all files under path `PATH`, matching the pattern `*.json`
 * Destination is database `DB`, table `TABLE`, and the ingestion mapping is defined in local file `MAPPING_FILE_PATH`
 * The tool will post the data for ingestion and won't wait for the ingest operations to complete
@@ -169,7 +176,8 @@ LightIngest.exe "https://ingest-{clusterAndRegion}.kusto.windows.net;Fed=True"
   -mappingPath:"MAPPING_FILE_PATH"
 ```
 
-**Example 5**
+**Ingesting files and writing diagnostic trace files**
+
 * Ingest all files under path `PATH`, matching the pattern `*.json`
 * Destination is database `DB`, table `TABLE`, and the ingestion mapping is defined in local file `MAPPING_FILE_PATH`
 * The tool will post the data for ingestion and won't wait for the ingest operations to complete