Merge pull request #100417 from orspod/2020-1-system-properties

toc changes & system properties

Author: PMEds28

articles/data-explorer/ingest-data-event-hub.md

@@ -3,10 +3,10 @@ title: 'Ingest data from Event Hub into Azure Data Explorer'
 description: 'In this article, you learn how to ingest (load) data into Azure Data Explorer from Event Hub.'
 author: orspod
 ms.author: orspodek
-ms.reviewer: mblythe
+ms.reviewer: tzgitlin
 ms.service: data-explorer
 ms.topic: conceptual
-ms.date: 07/17/2019
+ms.date: 01/08/2020
 
 # Customer intent: As a database administrator, I want to ingest data into Azure Data Explorer from an event hub, so I can analyze streaming data.
 ---
@@ -106,7 +106,7 @@ Now you connect to the event hub from Azure Data Explorer. When this connection
 
     ![Event hub connection](media/ingest-data-event-hub/event-hub-connection.png)
 
-    Data Source:
+    **Data Source:**
 
     **Setting** | **Suggested value** | **Field description**
     |---|---|---|
@@ -117,7 +117,7 @@ Now you connect to the event hub from Azure Data Explorer. When this connection
     | Event system properties | Select relevant properties | The [Event Hub system properties](/azure/service-bus-messaging/service-bus-amqp-protocol-guide#message-annotations). If there are multiple records per event message, the system properties will be added to the first one. When adding system properties, [create](/azure/kusto/management/tables#create-table) or [update](/azure/kusto/management/tables#alter-table-and-alter-merge-table) table schema and [mapping](/azure/kusto/management/mappings) to include the selected properties. |
     | | |
 
-    Target table:
+    **Target table:**
 
     There are two options for routing the ingested data: *static* and *dynamic*. 
     For this article, you use static routing, where you specify the table name, data format, and mapping. Therefore, leave **My data includes routing info** unselected.
@@ -135,6 +135,8 @@ Now you connect to the event hub from Azure Data Explorer. When this connection
     > * Enable GZip compression for static routing by opening a [support request in the Azure portal](https://ms.portal.azure.com/#blade/Microsoft_Azure_Support/HelpAndSupportBlade/overview). Enable GZip compression for dynamic routing as seen in the [sample app](https://github.com/Azure-Samples/event-hubs-dotnet-ingest). 
     > * Avro format and event system properties aren't supported on compression payload.
 
+[!INCLUDE [data-explorer-container-system-properties](../../includes/data-explorer-container-system-properties.md)]
+
 ## Copy the connection string
 
 When you run the [sample app](https://github.com/Azure-Samples/event-hubs-dotnet-ingest) listed in Prerequisites, you need the connection string for the event hub namespace.

articles/data-explorer/ingest-data-iot-hub.md

@@ -1,12 +1,12 @@
 ---
 title: 'Ingest data from IoT Hub into Azure Data Explorer'
 description: 'In this article, you learn how to ingest (load) data into Azure Data Explorer from IoT Hub.'
-author: oflipman
-ms.author: oflipman
-ms.reviewer: orspodek
+author: orspod
+ms.author: orspodek
+ms.reviewer: tzgitlin
 ms.service: data-explorer
 ms.topic: conceptual
-ms.date: 08/27/2019
+ms.date: 01/08/2020
 
 # Customer intent: As a database administrator, I want to ingest data into Azure Data Explorer from an IoT Hub, so I can analyze streaming data.
 ---
@@ -101,6 +101,8 @@ Now you connect to the IoT Hub from Azure Data Explorer. When this connection is
     > * Select **My data includes routing info** to use dynamic routing, where your data includes the necessary routing information as seen in the [sample app](https://github.com/Azure-Samples/event-hubs-dotnet-ingest) comments. If both static and dynamic properties are set, the dynamic properties override the static ones. 
     > * Only events enqueued after you create the data connection are ingested.
 
+[!INCLUDE [data-explorer-container-system-properties](../../includes/data-explorer-container-system-properties.md)]
+
 ## Generate sample data for testing
 
 The simulated device application connects to a device-specific endpoint on your IoT hub and sends simulated temperature and humidity telemetry.

articles/data-explorer/toc.yml

@@ -54,30 +54,30 @@
       items:
       - name: Azure portal
         href: ingest-data-event-hub.md
-      - name: Python
-        href: data-connection-event-hub-python.md
       - name: C#
         href: data-connection-event-hub-csharp.md
+      - name: Python
+        href: data-connection-event-hub-python.md
       - name: Azure Resource Manager template
         href: data-connection-event-hub-resource-manager.md
     - name: Event Grid
       items:
       - name: Azure portal
         href: ingest-data-event-grid.md
-      - name: Python
-        href: data-connection-event-grid-python.md
       - name: C#
         href: data-connection-event-grid-csharp.md
+      - name: Python
+        href: data-connection-event-grid-python.md
       - name: Azure Resource Manager template
         href: data-connection-event-grid-resource-manager.md
     - name: IoT Hub
       items:
       - name: Azure portal
         href: ingest-data-iot-hub.md
-      - name: Python
-        href: data-connection-iot-hub-python.md
       - name: C#
         href: data-connection-iot-hub-csharp.md
+      - name: Python
+        href: data-connection-iot-hub-python.md
       - name: Azure Resource Manager template
         href: data-connection-iot-hub-resource-manager.md
     - name: One-click ingestion

includes/data-explorer-container-system-properties.md

@@ -0,0 +1,37 @@
+---
+author: orspod
+ms.service: data-explorer
+ms.topic: include
+ms.date: 01/08/2020
+ms.author: orspodek
+---
+
+### Event system properties mapping
+
+If you selected **Event system properties** in the **Data Source** section of the table above, go to the [Web UI](https://dataexplorer.azure.com/) to run the relevant KQL command for proper mapping creation.
+
+   **For csv mapping:**
+
+    ```kusto
+    .create table MyTable ingestion csv mapping "CsvMapping1"
+    '['
+    '   { "column" : "messageid", "DataType":"string", "Properties":{"Ordinal":"0"}},'
+    '   { "column" : "userid", "DataType":"string", "Properties":{"Ordinal":"1"}},'
+    '   { "column" : "other", "DataType":"int", "Properties":{"Ordinal":"2"}}'
+    ']'
+    ```
+ 
+   **For json mapping:**
+
+    ```kusto
+    .create table MyTable ingestion json mapping "JsonMapping1"
+    '['
+    '    { "column" : "messageid", "datatype" : "string", "Properties":{"Path":"$.message-id"}},'
+    '    { "column" : "userid", "Properties":{"Path":"$.user-id"}},'
+    '    { "column" : "other", "Properties":{"Path":"$.other"}}'
+    ']'
+    ```
+
+   > [!TIP]
+   > * You must include all selected properties in the mapping. 
+   > * The properties order is important in csv mapping. The system properties must be listed before all other properties and in the same order in which they appear in the **Event system properties** dropdown.