You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/data-explorer/ingestion-properties.md
+9-16Lines changed: 9 additions & 16 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,33 +11,26 @@ ms.date: 03/19/2020
11
11
12
12
# Azure Data Explorer data ingestion properties
13
13
14
-
Data ingestion is the process by which data is added to a table and is made available for query in Azure Data Explorer. The following table describes the properties supported by Azure Data Explorer. You add properties to the Ingestion command after the `with` keyword.
14
+
Data ingestion is the process by which data is added to a table and is made available for query in Azure Data Explorer. The following table describes the properties supported by Azure Data Explorer. You add properties to the ingestion command after the `with` keyword.
|`ingestionMapping`|A string value that indicates how to map data from the source file to the actual columns in the table. This property requires defining the `format` value with the relevant mapping type. See [data mappings](https://docs.microsoft.com/azure/kusto/management/mappings).|`with (format="json", ingestionMapping = "[{\"column\":\"rownumber\", \"Properties\":{\"Path\":\"$.RowNumber\"}}, {\"column\":\"rowguid\", \"Properties\":{\"Path\":\"$.RowGuid\"}}]")`<br>(deprecated: `avroMapping`, `csvMapping`, `jsonMapping`) |
19
-
|`ingestionMappingReference`|A string value that indicates how to map data from the source file to the actual columns in the table using a named mapping policy object. This property requires defining the `format` value with the relevant mapping type. See [data mappings](https://docs.microsoft.com/azure/kusto/management/mappings).|`with (format="csv", ingestionMappingReference = "Mapping1")`<br>(deprecated: `avroMappingReference`, `csvMappingReference`, `jsonMappingReference`)|
20
-
|`creationTime`|The datetime value (formatted as an ISO8601 string) to use at the creation time of the ingested data extents. If unspecified, the current value (`now()`) will be used. Overriding the default is useful when ingesting older data, so that retention policy will be applied correctly.|`with (creationTime="2017-02-13T11:09:36.7992775Z")`|
18
+
|`ingestionMapping`|A string value that indicates how to map data from the source file to the actual columns in the table. Define the `format` value with the relevant mapping type. See [data mappings](https://docs.microsoft.com/azure/kusto/management/mappings).|`with (format="json", ingestionMapping = "[{\"column\":\"rownumber\", \"Properties\":{\"Path\":\"$.RowNumber\"}}, {\"column\":\"rowguid\", \"Properties\":{\"Path\":\"$.RowGuid\"}}]")`<br>(deprecated: `avroMapping`, `csvMapping`, `jsonMapping`) |
19
+
|`ingestionMappingReference`|A string value that indicates how to map data from the source file to the actual columns in the table using a named mapping policy object. Define the `format` value with the relevant mapping type. See [data mappings](https://docs.microsoft.com/azure/kusto/management/mappings).|`with (format="csv", ingestionMappingReference = "Mapping1")`<br>(deprecated: `avroMappingReference`, `csvMappingReference`, `jsonMappingReference`)|
20
+
|`creationTime`|The datetime value (formatted as an ISO8601 string) to use at the creation time of the ingested data extents. If unspecified, the current value (`now()`) will be used. Overriding the default is useful when ingesting older data, so that the retention policy will be applied correctly.|`with (creationTime="2017-02-13T11:09:36.7992775Z")`|
21
21
|`extend_schema`|A Boolean value that, if specified, instructs the command to extend the schema of the table (defaults to `false`). This option applies only to `.append` and `.set-or-append` commands. The only allowed schema extensions have additional columns added to the table at the end.|If the original table schema is `(a:string, b:int)`, a valid schema extension would be `(a:string, b:int, c:datetime, d:string)`, but `(a:string, c:datetime)` wouldn't be valid|
22
-
|`folder`|For [ingest-from-query](https://docs.microsoft.com/azure/kusto/management/data-ingestion/ingest-from-query) commands, the folder to assign to the table (if the table already exists, this property will override the table's folder)|`with (folder="Tables/Temporary")`|
23
-
|`format`|The data format (see [supported data formats](ingestion-supported-formats.md)|`with (format="csv")`|
22
+
|`folder`|For [ingest-from-query](https://docs.microsoft.com/azure/kusto/management/data-ingestion/ingest-from-query) commands, the folder to assign to the table. If the table already exists, this property will override the table's folder.|`with (folder="Tables/Temporary")`|
23
+
|`format`|The data format (see [supported data formats](ingestion-supported-formats.md)).|`with (format="csv")`|
24
24
|`ingestIfNotExists`|A string value that, if specified, prevents ingestion from succeeding if the table already has data tagged with an `ingest-by:` tag with the same value. This ensures idempotent data ingestion. For more information, see [ingest-by: tags](https://docs.microsoft.com/azure/kusto/management/extents-overview#ingest-by-extent-tags).|The properties `with (ingestIfNotExists='["Part0001"]', tags='["ingest-by:Part0001"]')` indicate that if data with the tag `ingest-by:Part0001` already exists, then don't complete the current ingestion. If it doesn't already exist, this new ingestion should have this tag set (in case a future ingestion attempts to ingest the same data again.)|
25
-
|`ignoreFirstRecord`|A Boolean value that, if set to `true`, indicates that ingestion should ignore the first record of every file. This property is useful for files in `CSV`and similar formats if the first record in the file is a header record specifying the column names. By default, `false` is assumed.|`with (ignoreFirstRecord=false)`|
25
+
|`ignoreFirstRecord`|A Boolean value that, if set to `true`, indicates that ingestion should ignore the first record of every file. This property is useful for files in `CSV`and similar formats, if the first record in the file are the column names. By default, `false` is assumed.|`with (ignoreFirstRecord=false)`|
26
26
|`persistDetails`|A Boolean value that, if specified, indicates that the command should persist the detailed results (even if successful) so that the [.show operation details](https://docs.microsoft.com/azure/kusto/management/operations#show-operation-details) command could retrieve them. Defaults to `false`.|`with (persistDetails=true)`|
27
-
|`policy_ingestiontime`|A Boolean value that, if specified, describes whether to enable the [Ingestion Time Policy](https://docs.microsoft.com/azure/kusto/management/ingestiontimepolicy) on a table that is created by this command. (The default is `true`.)|`with (policy_ingestiontime=false)`|
27
+
|`policy_ingestiontime`|A Boolean value that, if specified, describes whether to enable the [Ingestion Time Policy](https://docs.microsoft.com/azure/kusto/management/ingestiontimepolicy) on a table that is created by this command. The default is `true`.|`with (policy_ingestiontime=false)`|
28
28
|`recreate_schema`|A Boolean value that, if specified, describes whether the command may recreate the schema of the table. This property applies only to the `.set-or-replace` command. This property takes precedence over the `extend_schema` property if both are set.|`with (recreate_schema=true)`|
29
29
|`tags`|A list of [tags](https://docs.microsoft.com/azure/kusto/management/extents-overview#extent-tagging) to associate with the ingested data, formatted as a JSON string |`with (tags="['Tag1', 'Tag2']")`|
30
30
|`validationPolicy`|A JSON string that indicates which validations to run during ingestion. See [Data ingestion](https://docs.microsoft.com/azure/kusto/management/data-ingestion/) for an explanation of the different options.|`with (validationPolicy='{"ValidationOptions":1, "ValidationImplications":1}')` (this is actually the default policy)|
31
31
|`zipPattern`|Use this property when ingesting data from storage that has a ZIP archive. This is a string value indicating the regular expression to use when selecting which files in the ZIP archive to ingest. All other files in the archive will be ignored.|`with (zipPattern="*.csv")`|
32
32
33
-
<!-- TODO: Fill-in the following
34
-
The following table shows which property applies to each method of ingestion.
Copy file name to clipboardExpand all lines: articles/data-explorer/ingestion-supported-formats.md
+5-6Lines changed: 5 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -18,7 +18,7 @@ Data ingestion is the process by which data is added to a table and is made avai
18
18
|avro |`.avro`|An [Avro container file](https://avro.apache.org/docs/current/). The following codes are supported: `null`, `deflate` (`snappy` is currently not supported).|
19
19
|CSV |`.csv`|A text file with comma-separated values (`,`). See [RFC 4180: _Common Format and MIME Type for Comma-Separated Values (CSV) Files_](https://www.ietf.org/rfc/rfc4180.txt).|
20
20
|JSON |`.json`|A text file with JSON objects delimited by `\n` or `\r\n`. See [JSON Lines (JSONL)](http://jsonlines.org/).|
21
-
|multijson|`.multijson`|A text file with a JSON array of property bags (each representing a record), or any number of property bags delimited by whitespace, `\n` or `\r\n`. Each property bag can be spread on multiple lines. (This format is preferred over `JSON`, unless the data is non-property bags.)|
21
+
|multijson|`.multijson`|A text file with a JSON array of property bags (each representing a record), or any number of property bags delimited by whitespace, `\n` or `\r\n`. Each property bag can be spread on multiple lines. This format is preferred over `JSON`, unless the data is non-property bags.|
0 commit comments