Merge branch 'master' of https://github.com/MicrosoftDocs/azure-docs-pr into 2019-10-authentication

Author: orspod

articles/data-explorer/azure-notebooks.md

@@ -0,0 +1,106 @@
+---
+title: Use Azure Notebooks to analyze data in Azure Data Explorer
+description: This topic shows you how to create a query using an Azure Notebook
+author: orspod
+ms.author: orspodek
+ms.reviewer: adieldar
+ms.service: data-explorer
+ms.topic: conceptual
+ms.date: 04/01/2020
+---
+
+# Use Azure Notebooks to analyze data in Azure Data Explorer
+
+[Azure Notebooks](https://notebooks.azure.com/) is an Azure cloud service that simplifies creating and sharing [Jupyter Notebooks](https://jupyter.org/). Azure Notebooks makes it easy to combine documentation, code, and the results of running the code.
+
+> [!Note]
+> * The following procedure uses the Python client in the Azure Notebooks environment to query Azure Data Explorer. However, you can also use [KQLmagic](https://docs.microsoft.com/azure/data-explorer/kqlmagic) to query Azure Data Explorer.
+> * Some users reported issues authenticating using Edge; until such issues are resolved,
+   use a different browser.
+
+## Create a project
+
+1. Open [Azure Notebooks](https://notebooks.azure.com/) in your browser and sign-in.
+
+1. Select **My Projects** tab in the header. 
+
+    [![](media/azurenotebooks/an-myprojects.png "My projects")](media/azurenotebooks/an-myprojects.png#lightbox)
+
+1. Select **+ New Projects**.
+    
+1. In the **Create New Project** dialog box:
+    1. Enter the **Project Name**.
+    1. Clear the **Public** checkbox.
+        >[!Important]
+        > If you don't clear the Public checkbox, your project will be exposed on the open Internet.
+    1. Select **Create**.
+    
+    ![Create a new project](media/azurenotebooks/an-create-new-project-blank.png)
+
+    The Project ID is created automatically.
+
+## Create a notebook
+
+1. In your new project, create a notebook. The notebook should use a [supported language](https://github.com/Azure/azure-kusto-python#minimum-requirements).
+To create a notebook, select **+ New** and select **Notebook**.
+
+    ![Create new notebook](media/azurenotebooks/an-create-new-notebook-menu.png) 
+
+1. In the **Create New Notebook** dialog box, enter the notebook name.
+
+1. Select **Python 3.6** and select **New**.
+    
+    ![Create new notebook](media/azurenotebooks/an-create-new-notebook.png) 
+    
+1. In your project, select your new notebook.
+
+    ![Select new notebook](media/azurenotebooks/an-select-notebook.png)
+
+    Wait until your Python kernel initializes. When the filled circle icon changes to a hollow circle icon, you can proceed.
+
+    ![Kernel initializes](media/azurenotebooks/an-python-init-icon.png)
+
+1. To import the system module, enter the following commands and select **Run**:
+    ```python
+    import sys
+    sys.version
+    ```
+
+    > [!Note]
+    > To run a cell, you can also press Shift+Enter.
+
+1.  To import SDK classes, enter the following commands and select **Run**:
+    ```python
+    from azure.kusto.data.request import KustoClient, KustoConnectionStringBuilder
+    ```
+
+1.  To build the connection string and start the Kusto client, enter the following commands and select **Run**:  
+    ```python
+    kcsb = KustoConnectionStringBuilder.with_aad_device_authentication("https://help.kusto.windows.net")
+    kc = KustoClient(kcsb)
+    kc.execute("Samples", ".show version")
+    ```
+1. At the prompt, open a new browser tab to [https://aka.ms/devicelogin](https://aka.ms/devicelogin). 
+   
+1. Enter the authorization code, and sign-in your AAD (@microsoft.com) account. The Kusto client `kc` can now authenticate to Azure Data Explorer using your identity.
+
+1. Return to your notebook to see the result of the authentication. 
+
+[![](media/azurenotebooks/an-python-commands.png "Python commands")](media/azurenotebooks/an-python-commands.png#lightbox)
+
+## Execute a Kusto query
+
+1. Enter your Kusto query and select **Run**. For example:
+
+    ```python
+    query= "StormEvents | project State, EventType | take 10"
+    response = kc.execute("Samples", query)
+    for row in response.primary_results[0]:
+        print(", ".join(row))
+    ```    
+
+[![](media/azurenotebooks/an-commands.png "Python commands")](media/azurenotebooks/an-commands.png#lightbox)
+
+## Next steps
+
+* [Kusto-python GitHub repo](https://github.com/Azure/azure-kusto-python)

articles/data-explorer/flow-usage.md

@@ -19,8 +19,6 @@ For more information, see [Microsoft Flow connector (Preview)](flow.md).
 * [Push data to Power BI dataset](#push-data-to-power-bi-dataset)
 * [Conditional queries](#conditional-queries)
 * [Email multiple Azure Data Explorer Flow charts](#email-multiple-azure-data-explorer-flow-charts)
-* [Send a different email to different contacts](#send-a-different-email-to-different-contacts)
-* [Create a custom HTML table](#create-a-custom-html-table)
 
 ## Microsoft Flow connector and SQL
 
@@ -96,23 +94,21 @@ Visualize this information as a pie chart and email it to the team.
 
 ## Email multiple Azure Data Explorer Flow charts
 
-1. Create a new Flow with "Recurrence" trigger, and define the interval of the Flow and the frequency. 
+1. Create a new Flow with the recurrence trigger, and define the interval of the Flow and the frequency. 
 1. Add a new step, with one or more Kusto - Run query and visualize results actions. 
 
     ![Run several queries in a flow](./media/flow-usage/flow-severalqueries.png)
 1. For each Kusto - Run query and visualize result, define the following fields:
-    * Cluster URL (in the *Cluster Name* field)
+    * Cluster URL
     * Database Name
-    * Query and Chart Type (HTML Table/ Pie Chart/ Time Chart/ Bar Chart/ Enter Custom Value).
+    * Query and Chart Type (HTML table, pie chart, time chart, bar chart, or enter a custom value).
 
     ![Visualize results with multiple attachments](./media/flow-usage/flow-visualizeresultsmultipleattachments.png)
 
-    > [!IMPORTANT]
-    > In the *Cluster Name* fields, enter the cluster URL.
-
-1. Add a Send an email action. 
-    * In the *Body* field, insert the required body so that the visualized result of the query is included in the body of the email.
-    * To add an attachment to the email, add Attachment Name and Attachment Content.
+1. Add a Send an email (v2) action: 
+    1. In the body section, select the code view icon.
+    1. In the **Body** field, insert the required BodyHtml so that the visualized result of the query is included in the body of the email.
+    1. To add an attachment to the email, add Attachment Name and Attachment Content.
 
     ![Email multiple attachments](./media/flow-usage/flow-email-multiple-attachments.png)
 
@@ -124,68 +120,6 @@ Results:
 
 [![](./media/flow-usage/flow-resultsmultipleattachments2.png "Results of multiple attachments")](./media/flow-usage/flow-resultsmultipleattachments2.png#lightbox)
 
-## Send a different email to different contacts
-
-You can leverage Microsoft Flow to send different customized emails to different contacts. The email addresses and the email contents are a result of a Kusto query.
-
-Example:
-
-![Dynamic email using a Kusto query](./media/flow-usage/flow-dynamicemailkusto.png)
-
-> [!IMPORTANT]
-> In the *Cluster Name* field, enter the cluster URL.
-
-![Dynamic email in the flow action](./media/flow-usage/flow-dynamicemail.png)
-
-## Create a custom HTML table
-
-You can leverage Microsoft Flow to create and use custom HTML elements, such as a custom HTML table.
-
-The following example demonstrates how to create a custom HTML table. The HTML table will have its rows colored by log level (the same as in Azure Data Explorer).
-
-Follow these instructions to create a similar Flow:
-
-1. Create a new Kusto - Run query and list results action.
-
-    ![List results for an HTML table](./media/flow-usage/flow-listresultforhtmltable.png)
-
-> [!IMPORTANT]
-> In the *Cluster Name* field, enter the cluster URL.
-
-1. Loop over the query results and create the HTML table body: 
-    1. To create a variable to hold the HTML string, select **New step**
-    1. Select **Add an action** and search for Variables. 
-    1. Select **Variables - Initialize variable**. 
-    1. Initialize a string variable as follows:
-
-    ![Initialize a variable](./media/flow-usage/flow-initializevariable.png)
-
-1. Loop over the results:
-    1. Select **New step**.
-    1. Select **Add an action**.
-    1. Search for Variables. 
-    1. Select **Variables - Append to string variable**. 
-    1. Select the variable name that you initialized before, and create the HTML table rows using the query results. 
-    When selecting the query results, Apply to each is automatically added.
-
-    In the example below, the `if` expression is used to define the style of each row:
-
-    ```if(equals(items('Apply_to_each')?['Level'], 'Warning'), 'Yellow', if(equals(items('Apply_to_each')?['Level'], 'Error'), 'red', 'white'))```
-
-    [![](./media/flow-usage/flow-createhtmltableloopcontent.png "Create HTML table loop content")](./media/flow-usage/flow-createhtmltableloopcontent.png#lightbox)
-
-1. Create the full HTML content: 
-    1. Add a new action outside Apply to each. 
-    In the following example the action used is Send an email.
-    1. Define your HTML table using the variable from the previous steps. 
-    1. If you're sending an email, select **Show advanced options** and, under Is HTML, select **Yes**.
-
-    ![Custom HTML table email](./media/flow-usage/flow-customhtmltablemail.png)
-
-Result:
-
-![Custom HTML table email result](./media/flow-usage/flow-customhtmltableresult.png)
-
 ## Next steps
 
 Learn about the [Microsoft Azure Explorer Logic App connector](https://docs.microsoft.com/azure/kusto/tools/logicapps) which is another way to run Kusto queries and commands automatically as part of a scheduled or triggered task.

articles/data-explorer/ingest-data-one-click.md

@@ -1,88 +1,74 @@
 ---
 title: Use one-click ingestion to ingest data into Azure Data Explorer
-description: Learn about how to ingest (load) data into Azure Data Explorer simply using one-click ingestion.
+description: Overview of ingesting (loading) data into Azure Data Explorer simply, using one-click ingestion.
 author: orspod
 ms.author: orspodek
 ms.reviewer: tzgitlin
 ms.service: data-explorer
-ms.topic: conceptual
-ms.date: 10/31/2019
+ms.topic: overview
+ms.date: 03/29/2020
 ---
 
-# Use one-click ingestion to ingest data into Azure Data Explorer
+# What is one-click ingestion? 
 
-This article shows you how to use one-click ingestion for quick ingestion of a new table in JSON or CSV formats. The data can be ingested from storage or a local file into an existing table or a new table. Use the intuitive one-click wizard, and your data ingests within a few minutes. Then, you can edit the table and run queries by using the Azure Data Explorer Web UI.
+One-click ingestion enables you to quickly ingest data and automatically suggest tables and mapping structures, based on a data source in Azure Data Explorer. 
+
+Using the Azure Data Explorer Web UI, you can ingest data from storage (blob file), a local file, or a container (up to 10,000 blobs). You can also define an event grid on a container for continuous ingestion. The data can be ingested into an existing or new table in JSON, CSV, and [other formats](#file-formats). One click ingestion can suggest a structure for a new table and table mapping, based on the data source, and provide an intuitive platform to adjust the table structure of an existing table and table mapping. One click ingestion will ingest the data into the table within only a few minutes.
+
+One-click ingestion is particularly useful when ingesting data for the first time, or when your data's schema is unfamiliar to you.
 
 ## Prerequisites
 
 * If you don't have an Azure subscription, create a [free Azure account](https://azure.microsoft.com/free/) before you begin.
-* Sign in to [the application](https://dataexplorer.azure.com/).
 * Create [an Azure Data Explorer cluster and database](create-cluster-database-portal.md).
-* Sign in to the [Web UI](https://dataexplorer.azure.com/) and [add a connection to your cluster](/azure/data-explorer/web-query-data#add-clusters).
-
-## Ingest new data
-
-1. Right-click the *database* or *table* row in left menu of the Web UI and select **Ingest new data (Preview)**.
-
-    ![Select one-click ingestion in the Web UI](media/ingest-data-one-click/one-click-ingestion-in-webui.png)   
- 
-1. In the **Ingest new data (Preview)** window, select the **Source** tab and complete the **Project Details**:
-
-    * For **Table**, select an existing table name from the drop-down menu or select **Create new** to make a new table.
-    * For **Ingestion type**, select either **from storage** or **from file**.
-      * If you selected **from storage**, select **Link to storage** to add the URL. Use [Blob SAS URL](/azure/vs-azure-tools-storage-explorer-blobs#get-the-sas-for-a-blob-container) for private storage accounts. 
-      * If you selected **from file**, select **Browse** and drag the file into the box.
-    * Select **Edit schema** to view and edit your table column configuration.
- 
-    ![One-click ingestion source details](media/ingest-data-one-click/one-click-ingestion-source.png) 
+* Sign in to the [Azure Data Explorer Web UI](https://dataexplorer.azure.com/) and [add a connection to your cluster](/azure/data-explorer/web-query-data#add-clusters).
 
-    > [!TIP]
-    > If you select **Ingest new data (Preview)** on a *table* row, the selected table name will appear in the **Project Details**.
+## File formats
 
-1. If you selected an existing table, the **Map columns** window opens to map source data columns to target table columns. 
-    * Use **Omit column** to remove a target column from the table.
-    * Use **New column** to add a new column to the table.
+One-click ingestion supports ingesting a new table from source data in any of the following formats:
+* JSON
+* CSV
+* TSV
+* SCSV
+* SOHSV
+* TSVE
+* PSV
 
-    ![Map columns window](media/ingest-data-one-click/one-click-map-columns-window.png)
+## One-click ingestion wizard
 
-1. In the **Schema** tab:
+The one-click ingestion wizard guides you through the one-click ingestion process. 
 
-    * Select **Compression type** from the drop-down menu, and then select either **Uncompressed** or **GZip**.
-    * Select **Data format** from the drop-down menu, and then select **JSON**, **CSV**, **TSV**, **SCSV**, **SOHSV**, **TSVE**, or **PSV**. 
-        * When you select the **JSON** format, you must also select **JSON levels**, from 1 to 10. The levels affect the table column data depiction. 
-        * If you select a format other than JSON, you must select the check box **Include column names** to ignore the heading row of the file.
-    * **Mapping name** is set automatically but can be edited.
-    * If you selected an existing table, you can select **Map columns** to open the **Map columns** window.
+> [!Note]
+> This section describes the wizard in general. The options you select depend on whether you are ingesting into a new or existing table. 
+> For more information, see:
+    > * Ingest into [a new table](one-click-ingestion-new-table.md)
+    > * Ingest into an [existing table](one-click-ingestion-existing-table.md) 
+    
+1. To access the wizard, right-click the *database* or *table* row in left menu of the Azure Data Explorer web UI and select **Ingest new data (preview)**.
 
-    ![One-click ingestion CSV format schema](media/ingest-data-one-click/one-click-csv-format.png)
-
-1. Above the **Editor** pane, select the **v** button to open the editor. In the editor, you can view and copy the automatic queries generated from your inputs. 
-
-1. In the table: 
-    * Right-click new column headers to **Change data type**, **Rename column**, **Delete column**, **Sort ascending**, or **Sort descending**. On existing columns, only data sorting is available. 
-    * Double-click the new column name to edit.
-
-1. Select **Start ingestion** to create a table and mapping and to begin data ingestion.
-
-    ![One-click ingestion JSON format schema](media/ingest-data-one-click/one-click-json-format.png) 
- 
-## Query data
-
-1. In the **Data ingestion completed** window, all three steps will be marked with green check marks if data ingestion finishes successfully.
- 
-    ![One-click data ingestion complete](media/ingest-data-one-click/one-click-data-ingestion-complete.png)
+    ![Select one-click ingestion in the Web UI](media/ingest-data-one-click/one-click-ingestion-in-webui.png)   
 
-1. Select the **v** button to open the query. Copy to the Web UI to edit the query.
+1. The wizard guides you through the following options:
+       * Ingest into an [existing table](one-click-ingestion-existing-table.md)
+       * Ingest into [a new table](one-click-ingestion-new-table.md)
+       * Ingest data from:
+              * Blob storage
+              * A local file
+              * A container
+       * Enter the sample size, from 1 to 10,000 rows (from container only)
+       
+1. When you have successfully selected the data source, a preview of the data is displayed. 
+    If you are ingesting data from a container, you can filter the data so that only files with specific prefixes or file extensions are ingested. For example, you might only want to ingest files with filenames beginning with the word *Europe*, or only files with the extension *.json*. 
 
-1. The menu on the right contains **Quick queries** and **Tools** options. 
+1. Click **Edit schema**. If you are ingesting data into a specific table, you can map the source columns to the target columns and decide whether or not to include column names.
 
-    * **Quick queries** includes links to the Web UI with example queries.
-    * **Tools** includes a link to **Drop commands** on the Web UI, which allow you to troubleshoot issues by running the relevant `.drop` commands.
+1. Start the data ingestion process.
 
-    > [!TIP]
-    > You might lose data when you use `.drop` commands. Use them carefully.
+> [!Note]
+> If your data source is a container, note that Azure Data Explorer's data ingestion aggregation (batching) policy is designed to optimize the ingestion process. By default, the policy is configured to 5 minutes or 500 MB of data, so you may experience latency. See [batching policy](/azure/kusto/concepts/batchingpolicy) for aggregation options. When ingesting data from other sources, the ingestion will take immediate effect.
 
 ## Next steps
 
-* [Query data in Azure Data Explorer Web UI](web-query-data.md)
-* [Write queries for Azure Data Explorer using Kusto Query Language](write-queries.md)
+* Decide if you will use one-click ingestion to ingest data into [an existing table](one-click-ingestion-existing-table.md) or [a new table](one-click-ingestion-new-table.md)
+* [Query data in Azure Data Explorer Web UI](/azure/data-explorer/web-query-data)
+* [Write queries for Azure Data Explorer using Kusto Query Language](/azure/data-explorer/write-queries)