diff --git a/data-explorer/create-cluster-database.md b/data-explorer/create-cluster-database.md index 948f818c04..9ac16e30ac 100644 --- a/data-explorer/create-cluster-database.md +++ b/data-explorer/create-cluster-database.md @@ -4,7 +4,7 @@ description: Learn how to create an Azure Data Explorer cluster and database. ms.reviewer: lugoldbe ms.topic: how-to ms.custom: devx-track-azurepowershell -ms.date: 05/17/2023 +ms.date: 09/28/2025 --- # Create an Azure Data Explorer cluster and database diff --git a/data-explorer/database-script.md b/data-explorer/database-script.md index ef3e5bf308..2338c20e54 100644 --- a/data-explorer/database-script.md +++ b/data-explorer/database-script.md @@ -4,7 +4,7 @@ description: Learn about how to use database script to run a Kusto Query Languag ms.reviewer: docohe ms.topic: how-to ms.custom: devx-track-bicep -ms.date: 02/16/2024 +ms.date: 09/28/2025 --- # Configure a database using a Kusto Query Language script diff --git a/data-explorer/devops.md b/data-explorer/devops.md index 090e183ae1..2722e663d4 100644 --- a/data-explorer/devops.md +++ b/data-explorer/devops.md @@ -1,45 +1,45 @@ --- title: Azure DevOps task for Azure Data Explorer -description: In this article, you learn to create a release pipeline and deploy your schema changes to your database. +description: Create a release pipeline and deploy schema changes to your database. ms.reviewer: shfeldma ms.topic: how-to -ms.date: 07/17/2024 +ms.date: 09/28/2025 #Customer intent: I want to use Azure DevOps to create a release pipeline and deploy --- -# Azure DevOps Task for Azure Data Explorer +# Azure DevOps task for Azure Data Explorer [Azure DevOps Services](https://azure.microsoft.com/services/devops/) provides development collaboration tools such as high-performance pipelines, free private Git repositories, configurable Kanban boards, and extensive automated and continuous testing capabilities. [Azure Pipelines](https://azure.microsoft.com/services/devops/pipelines/) is an Azure DevOps capability that enables you to manage CI/CD to deploy your code with high-performance pipelines that work with any language, platform, and cloud. -[Azure Data Explorer - Pipeline Tools](https://marketplace.visualstudio.com/items?itemName=Azure-Kusto.PublishToADX) is the Azure Pipelines task that enables you to create release pipelines and deploy your database changes to your Azure Data Explorer databases. It's available for free in the [Visual Studio Marketplace](https://marketplace.visualstudio.com/). -This extension includes the following basic tasks: +[Azure Data Explorer - Pipeline Tools](https://marketplace.visualstudio.com/items?itemName=Azure-Kusto.PublishToADX) is the Azure Pipelines task that enables you to create release pipelines and deploy your database changes to your Azure Data Explorer databases. It is available for free in the [Visual Studio Marketplace](https://marketplace.visualstudio.com/). +The extension includes the following basic tasks: -* Azure Data Explorer Command - Run Admin Commands against an Azure Data Explorer cluster -* Azure Data Explorer Query - Run Queries against an Azure Data Explorer cluster and parse the results -* Azure Data Explorer Query Server Gate - Agentless task to Gate releases depending on the query outcome +* Azure Data Explorer command - Run admin commands against an Azure Data Explorer cluster +* Azure Data Explorer query - Run queries against an Azure Data Explorer cluster and parse the results +* Azure Data Explorer query server gate - Agentless task to gate releases depending on the query outcome - :::image type="content" source="media/devops/extension-task-types.png" alt-text="Screenshot of the task types available in the Pipeline Tools extension."::: + :::image type="content" source="media/devops/extension-task-types.png" alt-text="Screenshot of task types available in the Pipeline Tools extension."::: -This document describes a simple example on the use of the **Azure Data Explorer - Pipeline Tools** task to deploy your schema changes to your database. For complete CI/CD pipelines, refer to [Azure DevOps documentation](/azure/devops/user-guide/what-is-azure-devops#vsts). +This document describes a simple example of using the **Azure Data Explorer - Pipeline Tools** task to deploy schema changes to your database. For complete CI/CD pipelines, refer to [Azure DevOps documentation](/azure/devops/user-guide/what-is-azure-devops#vsts). ## Prerequisites * An Azure subscription. Create a [free Azure account](https://azure.microsoft.com/free/). * An Azure Data Explorer cluster and database. [Create a cluster and database](create-cluster-and-database.md). * Azure Data Explorer cluster setup: - * Create Microsoft Entra app by [provisioning a Microsoft Entra application](provision-entra-id-app.md). - * Grant access to your Microsoft Entra App on your Azure Data Explorer database by [managing Azure Data Explorer database permissions](manage-database-permissions.md). + * Create Microsoft Entra app by [provisioning a Microsoft Entra application](provision-entra-id-app.md). + * Grant access to your Microsoft Entra App on your Azure Data Explorer database by [managing Azure Data Explorer database permissions](manage-database-permissions.md). * Azure DevOps setup: - * [Sign up for a free organization](/azure/devops/user-guide/sign-up-invite-teammates). - * [Create an organization](/azure/devops/organizations/accounts/create-organization). - * [Create a project in Azure DevOps](/azure/devops/organizations/projects/create-project). - * [Code with Git](/azure/devops/user-guide/code-with-git). + * [Sign up for a free organization](/azure/devops/user-guide/sign-up-invite-teammates). + * [Create an organization](/azure/devops/organizations/accounts/create-organization). + * [Create a project in Azure DevOps](/azure/devops/organizations/projects/create-project). + * [Code with Git](/azure/devops/user-guide/code-with-git). * Extension Installation: - * If you're the Azure DevOps instance owner, install the extension from the [Marketplace](https://marketplace.visualstudio.com/items?itemName=Azure-Kusto.PublishToADX), otherwise contact your Azure DevOps instance [owner](/azure/devops/organizations/security/look-up-organization-owner) and ask them to install it. + * If you're the Azure DevOps instance owner, install the extension from the [Marketplace](https://marketplace.visualstudio.com/items?itemName=Azure-Kusto.PublishToADX), otherwise contact your Azure DevOps instance [owner](/azure/devops/organizations/security/look-up-organization-owner) and ask them to install it. - :::image type="content" source="media/devops/get-extension.png" alt-text="Screenshot of getting the Pipeline Tools extension in the Visual Studio Marketplace."::: + :::image type="content" source="media/devops/get-extension.png" alt-text="Screenshot of getting the Pipeline Tools extension in the Visual Studio Marketplace."::: - :::image type="content" source="media/devops/extension-install.png" alt-text="Screenshot of installing the Pipeline Tools extension from the Visual Studio Marketplace."::: + :::image type="content" source="media/devops/extension-install.png" alt-text="Screenshot of installing the Pipeline Tools extension from the Visual Studio Marketplace."::: ## Prepare your content for release @@ -47,29 +47,29 @@ You can use the following methods to execute admin commands against a cluster wi :::image type="content" source="media/devops/source-control-options.png" alt-text="Screenshot showing the command source control options."::: -* Use a search pattern to get multiple command files from a local agent folder (Build sources or Release artifacts) +* Use a search pattern to get multiple command files from a local agent folder (build sources or release artifacts). :::image type="content" source="media/devops/local-folder-option.png" alt-text="Screenshot showing the local folder option."::: -* Write commands inline +* Write commands inline. :::image type="content" source="media/devops/inline-option.png" alt-text="Screenshot showing the inline command option."::: -* Specify a file path to get command files directly from git source control (recommended) +* Specify a file path to get command files directly from Git source control (recommended). :::image type="content" source="media/devops/git-option.png" alt-text="Screenshot showing git source control files option."::: - Create the following sample folders (*Functions*, *Policies*, *Tables*) in your Git repository. Copy the files from [the samples repo](https://github.com/Azure/azure-kusto-docs-samples/tree/master/DevOps_release_pipeline) into the respective folders and commit the changes. The sample files are provided to execute the following workflow. + Create the following sample folders (*Functions*, *Policies*, *Tables*) in your Git repository. Copy the files from [the samples repo](https://github.com/Azure/azure-kusto-docs-samples/tree/master/DevOps_release_pipeline) into the respective folders, and commit the changes. The sample files are provided to execute the following workflow. :::image type="content" source="media/devops/create-folders.png" alt-text="Screenshot showing the folders to create in the repo."::: > [!TIP] - > When creating your own workflow, we recommend making your code idempotent. For example, use [`.create-merge table`](/kusto/management/create-merge-table-command?view=azure-data-explorer&preserve-view=true) instead of [`.create table`](/kusto/management/create-table-command?view=azure-data-explorer&preserve-view=true), and use [`.create-or-alter`](/kusto/management/create-alter-function?view=azure-data-explorer&preserve-view=true) function instead of [`.create`](/kusto/management/create-function?view=azure-data-explorer&preserve-view=true) function. + > When creating your own workflow, we recommend making your code idempotent. For example, use [`.create-merge table`](/kusto/management/create-merge-table-command?view=azure-data-explorer&preserve-view=true) instead of [`.create table`](/kusto/management/create-table-command?view=azure-data-explorer&preserve-view=true), and use the [`.create-or-alter`](/kusto/management/create-alter-function?view=azure-data-explorer&preserve-view=true) function instead of the [`.create`](/kusto/management/create-function?view=azure-data-explorer&preserve-view=true) function. ## Create a release pipeline 1. Sign in to your [Azure DevOps organization](https://dev.azure.com/). -1. Select **Pipelines** > **Releases** from left-hand menu and select **New pipeline**. +1. Select **Pipelines** > **Releases** from the left-hand menu, and then select **New pipeline**. :::image type="content" source="media/devops/new-pipeline.png" alt-text="Screenshot showing how to start a new pipeline."::: @@ -77,7 +77,7 @@ You can use the following methods to execute admin commands against a cluster wi :::image type="content" source="media/devops/select-template.png" alt-text="Screenshot showing how to select a template."::: -1. Select **Stage** button. In **Stage** pane, add the **Stage name**. Select **Save** to save your pipeline. +1. Select the **Stage** button. In the **Stage** pane, add the **Stage name**, and then select **Save** to save your pipeline. :::image type="content" source="media/devops/stage-name.png" alt-text="Screenshot showing how to name the pipeline stage."::: @@ -85,7 +85,7 @@ You can use the following methods to execute admin commands against a cluster wi :::image type="content" source="media/devops/add-artifact.png" alt-text="Screenshot showing how to add an artifact."::: -1. In the **Variables** tab, select **+ Add** to create a variable for **Endpoint URL** that is used in the task. Write the **Name** and the **Value** of the endpoint. Select **Save** to save your pipeline. +1. In the **Variables** tab, select **+ Add** to create a variable for **Endpoint URL** used in the task. Enter the **Name** and **Value** of the endpoint, and then select **Save** to save your pipeline. :::image type="content" source="media/devops/create-variable.png" alt-text="Screenshot showing how to create a pipeline variable."::: @@ -95,7 +95,7 @@ You can use the following methods to execute admin commands against a cluster wi ### Create a task deploy the folders -1. In the **Pipeline** tab, select on **1 job, 0 task** to add tasks. +1. In the **Pipeline** tab, select **1 job, 0 task** to add tasks. :::image type="content" source="media/devops/add-task.png" alt-text="Screenshot showing adding a task to the pipeline."::: @@ -125,7 +125,7 @@ You can use the following methods to execute admin commands against a cluster wi :::image type="content" source="media/devops/add-service-endpoint.png" alt-text="Screenshot showing how to add a service connection."::: -1. Select **Save** and then in the **Tasks** tab, verify that there are three tasks: **Deploy Tables**, **Deploy Functions**, and **Deploy Policies**. +1. Select **Save**, and then in the **Tasks** tab, verify that there are three tasks: **Deploy Tables**, **Deploy Functions**, and **Deploy Policies**. :::image type="content" source="media/devops/deploy-all-folders.png" alt-text="Screenshot showing how to deploy all folders."::: @@ -186,7 +186,7 @@ If necessary, create a task to run a query against a cluster and gate the releas ### Run the release -1. Select **+ Release** > **Create release** to create a release. +1. Select **+ Release** > **Create release** to start a release. :::image type="content" source="media/devops/create-release.png" alt-text="Screenshot showing how to create a release."::: @@ -198,7 +198,7 @@ Now the creation of a release pipeline for deployment to preproduction is comple ## Keyless authentication support for Azure Data Explorer DevOps tasks -The extension supports keyless authentication for Azure Data Explorer clusters. Keyless authentication allows you to authenticate to Azure Data Explorer clusters without using a key and is more secure and easier to manage than using a key. +The extension supports keyless authentication for Azure Data Explorer clusters. Keyless authentication lets you authenticate to Azure Data Explorer clusters without using a key. It's more secure and easier to manage. ### Use Federated Identity Credentials (FIC) authentication in an Azure Data Explorer service connection @@ -219,7 +219,7 @@ The extension supports keyless authentication for Azure Data Explorer clusters. * **Subject identifier**: `` where `{DevOps_Org_name}` is the Azure DevOps organization name, `{Project_Name}` is the project name, and `{Service_Connection_Name}` is the service connection name you created earlier. > [!NOTE] - > If there is space in your service connection name, you can using it with space in the field. For example: `sc://MyOrg/MyProject/My Service Connection`. + > If there is space in your service connection name, you can use it with the space in the field. For example: `sc://MyOrg/MyProject/My Service Connection`. * **Name**: Enter a name for the credential. @@ -233,17 +233,17 @@ The extension supports keyless authentication for Azure Data Explorer clusters. :::image type="content" source="media/devops/resource-new.png" alt-text="Screenshot showing how to add an Azure Resource Monitor service connection."::: -1. Under **Authentication method**, select **Workload Identity Federation (automatic)**. Alternatively, you can use the manual **Workload Identity Federation (manual)** option to specify the Workload Identity Federation details, or use the **Managed Identity** option. For more information about setting up a managed identity using Azure Resource Management, see [Azure Resource Manager (ARM) Service Connections](/azure/devops/pipelines/library/connect-to-azure?view=azure-devops&preserve-view=true). +1. Under **Authentication method**, select **Workload Identity Federation (automatic)** to proceed. You can also use the manual **Workload Identity Federation (manual)** option to specify the Workload Identity Federation details or the **Managed Identity** option. Learn more about setting up a managed identity using Azure Resource Management in [Azure Resource Manager (ARM) Service Connections](/azure/devops/pipelines/library/connect-to-azure?view=azure-devops&preserve-view=true). :::image type="content" source="media/devops/resource-types.png" alt-text="Screenshot showing the authentication option for an Azure Resource Monitor service connection"::: 1. Fill out the required details, select **Verify**, and then select **Save**. -## Yaml Pipeline configuration +## Yaml pipeline configuration -The tasks can be configured both via Azure DevOps Web UI and via Yaml code within the [pipeline schema](/azure/devops/pipelines/yaml-schema). +You can configure tasks using the Azure DevOps web UI or YAML code within the [pipeline schema](/azure/devops/pipelines/yaml-schema). -### Admin command sample usage +### Admin command sample ```yaml steps: @@ -257,11 +257,11 @@ steps: authType: 'armserviceconn' connectedServiceARM: '' serialDelay: 1000 - continueOnError: true + `continueOnError: true` condition: ne(variables['ProductVersion'], '') ## Custom condition Sample ``` -### Query sample usage +### Query sample ```yaml steps: @@ -290,7 +290,7 @@ steps: | summarize by Cluster , CmConnectionString , ServiceConnectionString ,DeploymentRing | extend ServiceConnectionString = strcat("#connect ", ServiceConnectionString) | where DeploymentRing == "$(DeploymentRing)" - kustoUrls: 'https://.kusto.windows.net?DatabaseName=' + kustoUrls: 'https://.kusto.windows.net?DatabaseName=' authType: 'kustoserviceconn' connectedServiceName: '' minThreshold: '0' diff --git a/data-explorer/grafana.md b/data-explorer/grafana.md index d98471588b..90a7a5f35c 100644 --- a/data-explorer/grafana.md +++ b/data-explorer/grafana.md @@ -1,22 +1,22 @@ --- title: Visualize data from Azure Data Explorer by using Grafana -description: In this article, you learn to set up Azure Data Explorer as a data source for Grafana, and then visualize data from a sample cluster. +description: Discover how to configure Azure Data Explorer for Grafana and create insightful dashboards using sample data. ms.reviewer: gabil ms.topic: how-to -ms.date: 06/27/2023 +ms.date: 09/28/2025 --- # Visualize data from Azure Data Explorer in Grafana -Grafana is an analytics platform where you can query and visualize data, and then create and share dashboards based on your visualizations. Grafana provides an Azure Data Explorer *plug-in*, which enables you to connect to and visualize data from Azure Data Explorer. The plug-in works with both [Azure Managed Grafana](/azure/managed-grafana/overview) and self-hosted Grafana. +Grafana is an analytics platform where you can query and visualize data, and then create and share dashboards based on your visualizations. Grafana provides an Azure Data Explorer *plugin*, which lets you connect to and visualize data from Azure Data Explorer. The plug-in works with both [Azure Managed Grafana](/azure/managed-grafana/overview) and self-hosted Grafana. -In this article, you learn how to [configure your cluster as a data source for Grafana](#configure-the-data-source) and [visualize data in Grafana](#visualize-data) for Azure Managed Grafana and self-hosted Grafana. To follow along with the examples in this article, [ingest the StormEvents sample data](web-ui-samples-query.md). [!INCLUDE [data-explorer-storm-events](includes/data-explorer-storm-events.md)] +This article shows you how to [configure your cluster as a data source for Grafana](#configure-the-data-source) and [visualize data in Grafana](#visualize-data) for Azure Managed Grafana and self-hosted Grafana. To follow the examples in this article, [ingest the StormEvents sample data](web-ui-samples-query.md). [!INCLUDE [data-explorer-storm-events](includes/data-explorer-storm-events.md)] ## Prerequisites -* For Azure Managed Grafana, an Azure account and [Azure Managed Grafana](/azure/managed-grafana/quickstart-managed-grafana-portal) instance. -* For self-hosted Grafana, [Grafana version 5.3.0 or later](https://docs.grafana.org/installation/) for your operating system and the [Azure Data Explorer plug-in](https://grafana.com/grafana/plugins/grafana-azure-data-explorer-datasource/) for Grafana. You need plug-in version 3.0.5 or later to use the Grafana query builder. -* An Azure Data Explorer cluster and database. You can [create a free cluster](start-for-free-web-ui.md) or [create a full cluster](create-cluster-and-database.md). To decide which is best for you, check the [feature comparison](start-for-free.md#feature-comparison). +* For Azure Managed Grafana, you need an Azure account and an [Azure Managed Grafana](/azure/managed-grafana/quickstart-managed-grafana-portal) instance. +* For self-hosted Grafana, install [Grafana version 5.3.0 or later](https://docs.grafana.org/installation/) for your operating system and the [Azure Data Explorer plug-in](https://grafana.com/grafana/plugins/grafana-azure-data-explorer-datasource/) for Grafana. Use plug-in version 3.0.5 or later to access the Grafana query builder. +* Set up an Azure Data Explorer cluster and database. You can [create a free cluster](start-for-free-web-ui.md) or [create a full cluster](create-cluster-and-database.md). To decide which option works best, review the [feature comparison](start-for-free.md#feature-comparison). ## Configure the data source @@ -26,11 +26,11 @@ To configure Azure Data Explorer as a data source, follow the steps for your Gra #### Add the managed identity to the Viewer role -Managed Grafana creates a system-assigned managed identity for each new workspace, by default. You can use it to access your Azure Data Explorer cluster. +Managed Grafana creates a system-assigned managed identity for each new workspace by default. You can use it to access your Azure Data Explorer cluster. -1. In the Azure portal, go to your Azure Data Explorer cluster. +1. In the Azure portal, open your Azure Data Explorer cluster. -1. In the **Overview** section, select the database that has the *StormEvents* sample data. +1. In the **Overview** section, select the database with the *StormEvents* sample data. :::image type="content" source="includes/media/data-explorer-configure-data-source/select-database.png" alt-text="Screenshot of the Azure Data Explorer overview page and the selection of a sample database."::: @@ -52,7 +52,7 @@ Managed Grafana workspaces come with the Azure Data Explorer plug-in preinstalle 1. Under **Overview**, select the **Endpoint** link to open the Grafana UI. -1. In Grafana, on the left menu, select the gear icon. Then select **Data Sources**. +1. In Grafana, on the left menu, select the gear icon, and then select **Data Sources**. :::image type="content" source="media/grafana/data-sources.png" alt-text="Screenshot of the Grafana settings menu and the option for data sources."::: @@ -64,7 +64,7 @@ Managed Grafana workspaces come with the Azure Data Explorer plug-in preinstalle :::image type="content" source="media/grafana/input-cluster-uri.png" alt-text="Screenshot of the pane for connection details with the box for cluster URL highlighted."::: -1. Select **Save & Test**. +1. Select **Save & test**. ### [Self-hosted Grafana](#tab/self-hosted-grafana) @@ -86,7 +86,7 @@ You can create the service principal in the [Azure portal](#azure-portal) or by ##### Azure CLI -1. Use the following command to create a service principal. Set an appropriate scope and a role type of `reader`. +1. Run the following command to create a service principal. Specify an appropriate scope and set the role type to `reader`. ```azurecli az ad sp create-for-rbac --name "https://{UrlToYourDashboard}:{PortNumber}" --role "reader" \ @@ -215,19 +215,19 @@ To enable results cache rendering, do the following on the **Query Optimizations ##### Enable weak consistency -Clusters are configured with strong consistency. This default configuration guarantees that query results are up to date with all changes in the cluster. +Clusters use strong consistency by default, which guarantees that query results are up to date with all changes in the cluster. When you enable weak consistency, query results can have a lag of 1 to 2 minutes after cluster alterations. However, weak consistency might boost visual rendering time. If immediate consistency isn't critical and performance is marginal, enable weak consistency to improve performance. For more information, see [Query consistency](/kusto/concepts/query-consistency?view=azure-data-explorer&preserve-view=true). -To enable weak consistency, on the **Query Optimizations** pane, select **Data consistency** > **Weak**. +To enable weak consistency, in the **Query Optimizations** pane, select **Data consistency** > **Weak**. --- ## Visualize data -You finished configuring Azure Data Explorer as a data source for Grafana. Now it's time to visualize data. +You've configured Azure Data Explorer as a data source for Grafana. Now it's time to visualize data. -The following basic example uses both the query builder mode and the raw mode of the query editor. We recommend that you view [write queries for Azure Data Explorer](/azure/data-explorer/kusto/query/tutorials/learn-common-operators) for examples of other queries to run against the dataset. +The following basic example uses both the query builder mode and the raw mode of the query editor. View [write queries for Azure Data Explorer](/azure/data-explorer/kusto/query/tutorials/learn-common-operators) for examples of other queries to run against the dataset. 1. In Grafana, on the left menu, select the plus icon. Then select **Dashboard**. @@ -237,17 +237,17 @@ The following basic example uses both the query builder mode and the raw mode of :::image type="content" source="media/grafana/add-graph.png" alt-text="Screenshot of the page for adding a panel, with the graph option highlighted."::: -1. On the graph pane, select **Panel Title** > **Edit**. +1. On the graph pane, select **Panel title** > **Edit**. :::image type="content" source="media/grafana/edit-panel.png" alt-text="Screenshot of the Grafana panel menu, with the edit option highlighted."::: -1. At the bottom of the pane, select **Data Source**, and then select the data source that you configured. +1. At the bottom of the pane, select **Data source**, and then select the data source you configured. :::image type="content" source="media/grafana/select-data-source.png" alt-text="Screenshot of the menu for selecting a data source."::: ### Query builder mode -Use query builder mode to define your query. +Use the query builder mode to define your query. 1. Below the data source, select **Database** and choose your database from the dropdown list. 1. Select **From** and choose your table from the dropdown list. @@ -256,8 +256,8 @@ Use query builder mode to define your query. 1. Now that the table is defined, filter the data: - 1. Select **+** to the right of **Where (filter)** to select one or more columns in your table. - 1. For each filter, define the values by using the applicable operator. This selection is similar to using the [where operator](/kusto/query/where-operator?view=azure-data-explorer&preserve-view=true) in Kusto Query Language. + 1. Select **+** to the right of **Where (filter)** to choose one or more columns in your table. + 1. For each filter, define the values by using the applicable operator. This step is similar to using the [where operator](/kusto/query/where-operator?view=azure-data-explorer&preserve-view=true) in Kusto Query Language. 1. Select the values to present in the table: @@ -296,7 +296,7 @@ Use raw mode to edit your query. :::image type="content" source="media/grafana/last-six-hours.png" alt-text="Screenshot of the default time filter of last six hours."::: -1. Specify a custom range that covers 2007, the year included in the StormEvents sample dataset. Then select **Apply**. +1. Enter a custom range that covers 2007, the year included in the StormEvents sample dataset. Then select **Apply**. :::image type="content" source="media/grafana/custom-date-range.png" alt-text="Screenshot of the custom range control, with a custom date range selected."::: @@ -306,30 +306,30 @@ Use raw mode to edit your query. 1. On the top menu, select the save icon: :::image type="icon" source="media/grafana/save-icon.png":::. -To switch to the query builder mode, select **Switch to builder**. Grafana will convert the query to the available logic in the query builder. The query builder logic is limited, so you might lose manual changes that you made to the query. +To switch to query builder mode, select **Switch to builder**. Grafana will convert the query to the available logic in the query builder. The query builder logic is limited, so you might lose manual changes that you made to the query. :::image type="content" source="media/grafana/raw-mode.png" alt-text="Screenshot of the query window, with the button for switching to the builder highlighted."::: ## Create alerts -1. In **Home Dashboard**, select **Alerting** > **Notification channels** to create a new notification channel. +1. In **Home Dashboard**, select **Alerting** > **Notification channels** to create a notification channel. - :::image type="content" source="media/grafana/create-notification-channel.png" alt-text="Screenshot of the dashboard, with the option for creating a notification channel highlighted."::: + :::image type="content" source="media/grafana/create-notification-channel.png" alt-text="Screenshot of the dashboard with the option for creating a notification channel highlighted."::: -1. Enter a name and type under **New Notification Channel**, and then select **Save**. +1. Enter a name and type under **New Notification Channel**, then select **Save**. - :::image type="content" source="media/grafana/new-notification-channel-adx.png" alt-text="Screenshot of the window for creating a new notification channel."::: + :::image type="content" source="media/grafana/new-notification-channel-adx.png" alt-text="Screenshot of the window for creating a notification channel."::: -1. On the dashboard, select **Edit** from the dropdown list. +1. On the dashboard, select **Edit** from the drop-down list. - :::image type="content" source="media/grafana/edit-panel-4-alert.png" alt-text="Screenshot of the dashboard panel, with the Edit menu command highlighted."::: + :::image type="content" source="media/grafana/edit-panel-4-alert.png" alt-text="Screenshot of the dashboard panel with the Edit menu command highlighted."::: -1. Select the alert bell icon to open the **Alert** pane. Select **Create Alert**, and then complete the properties for the alert. +1. Select the alert bell icon to open the **Alert** pane. Select **Create Alert**, then complete the properties for the alert. - :::image type="content" source="media/grafana/alert-properties.png" alt-text="Screenshot of the pane for selecting alert properties."::: + :::image type="content" source="media/grafana/alert-properties.png" alt-text="Screenshot of the pane for selecting the alert properties."::: -1. Select the **Save dashboard** icon to save your changes. +1. Select the **Save dashboard** icon to save the changes. ## Related content -* [Write queries for Azure Data Explorer](/azure/data-explorer/kusto/query/tutorials/learn-common-operators) +* Learn how to [write queries for Azure Data Explorer](/azure/data-explorer/kusto/query/tutorials/learn-common-operators). diff --git a/data-explorer/integrate-visualize-overview.md b/data-explorer/integrate-visualize-overview.md index dd710ee3c6..2e7915d3f2 100644 --- a/data-explorer/integrate-visualize-overview.md +++ b/data-explorer/integrate-visualize-overview.md @@ -3,7 +3,7 @@ title: Visualization integrations overview description: Learn about the available visualize integrations. ms.reviewer: aksdi ms.topic: conceptual -ms.date: 01/14/2024 +ms.date: 09/28/2025 # CustomerIntent: As a data ingestor, I want to know what visualize connectors and tools are available, so that I can choose the right one for my use case. --- # Visualization integrations overview diff --git a/data-explorer/kusto/management/alter-merge-query-acceleration-policy-command.md b/data-explorer/kusto/management/alter-merge-query-acceleration-policy-command.md new file mode 100644 index 0000000000..5e1d3c37bb --- /dev/null +++ b/data-explorer/kusto/management/alter-merge-query-acceleration-policy-command.md @@ -0,0 +1,95 @@ +--- +title: .alter-merge query acceleration policy Command +description: Learn how to use the .alter-merge query acceleration policy command to accelerate queries over external delta tables. +ms.reviewer: sharmaanshul +ms.topic: reference +ms.date: 09/16/2025 +--- + +# `.alter-merge query acceleration policy` command + +> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] + +Alters the [query acceleration policy](query-acceleration-policy.md) properties of a specific external delta table. + +> [!TIP] +> Use `.alter-merge` to add or update properties without replacing the whole policy. Array properties are merged (new elements are added, existing values preserved). +> Use `.alter` to fully replace the policy, including overwriting arrays. + +For limitations, see [Limitations](query-acceleration-policy.md#limitations). + +## Permissions + +You must have at least [Database Admin](../access-control/role-based-access-control.md) permissions to run this command. + +## Syntax + +`.alter-merge` `external` `table` _ExternalTableName_ `policy` `query_acceleration` '_JSON-serialized policy_' + +## Parameters + +| Name | Type | Required | Description | +| ------------------------ | -------- | ------------------ | ----------------------------------------------------------------- | +| _ExternalTableName_ | `string` | :heavy_check_mark: | The name of the external delta table. | +| _JSON-serialized policy_ | `string` | :heavy_check_mark: | String literal holding a [JSON property bag](#json-property-bag). | + +### JSON property bag + +| Property | Type | Required | Description | +| ---------- | ---------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| IsEnabled | `Boolean` | | Indicates whether the policy is enabled. This property is required if no query acceleration policy is defined on the external table. | +| Hot | `Timespan` | | The hot period defined in the query acceleration policy. Minimum value = 1 d. This property is required if no query acceleration policy is defined on the external table. | +| HotWindows | `DateTime` | | One or more optional time windows. Delta data files created within these time windows are accelerated. | +| MaxAge | `Timespan` | | The external table returns accelerated data if the last index refresh time is greater than @now - MaxAge. Otherwise, external table operates in nonaccelerated mode. Default is 5 minutes. Minimum is 1 minute. | + +> [!NOTE] +> Query acceleration is applied to data within a specific time period, defined as `timespan`, starting from the `modificationTime` as stated for each file in the [delta log](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#add-file-and-remove-file). + +### Example + +In case the external table has query acceleration policy defined: + +```json +{ "Hot": "1.00:00:00" } +``` + +In case the external table doesn't have query acceleration policy defined: + +```json +{ "IsEnabled": true, "Hot": "1.00:00:00" } +``` + + +## Returns + +The command returns a table with one record that includes the modified policy object. + +| Column | Type | Description | +| ------------- | -------- | --------------------------------------------------------------------------------------------- | +| PolicyName | `string` | The name of the policy - `QueryAcceleration` | +| EntityName | `string` | The fully qualified name of the entity: `[DatabaseName].[ExternalTableName]` | +| Policy | `string` | A JSON-serialization of the query acceleration policy that is set on the external delta table | +| ChildEntities | `string` | The child entities this policy affects - `null` | +| EntityType | `string` | The type of the entity the policy applies to - `ExternalTable` | + +## Example + +In case the external table has query acceleration policy defined: + +```Kusto +.alter-merge external table MyExternalTable policy query_acceleration '{"Hot": "1.00:00:00", "MaxAge" : "00:05:00"}' +``` + +In case the external table doesn't have query acceleration policy defined: + +```Kusto +.alter-merge external table MyExternalTable policy query_acceleration '{"IsEnabled": true, "Hot": "1.00:00:00", "MaxAge" : "00:05:00"}' +``` + +## Related content + +- [Query acceleration policy](query-acceleration-policy.md) +- [.alter query acceleration policy command](alter-query-acceleration-policy-command.md) +- [.delete query acceleration policy command](delete-query-acceleration-policy-command.md) +- [.show query acceleration policy command](show-query-acceleration-policy-command.md) +- [.show external table operations query_acceleration statistics](show-external-table-operations-query-acceleration-statistics.md) diff --git a/data-explorer/kusto/management/alter-query-acceleration-policy-command.md b/data-explorer/kusto/management/alter-query-acceleration-policy-command.md index ed0148e97f..a8ed84f264 100644 --- a/data-explorer/kusto/management/alter-query-acceleration-policy-command.md +++ b/data-explorer/kusto/management/alter-query-acceleration-policy-command.md @@ -1,12 +1,12 @@ --- title: .alter query acceleration policy command -description: Learn how to use the ".alter query acceleration policy command" to accelerate queries over external delta tables. +description: Learn how to use the .alter query acceleration policy command to accelerate queries over external delta tables. ms.reviewer: sharmaanshul ms.topic: reference -ms.date: 11/19/2024 +ms.date: 09/16/2025 --- -# .alter query acceleration policy command +# `.alter query acceleration policy` command > [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] @@ -84,6 +84,7 @@ The command returns a table with one record that includes the modified policy ob ## Related content - [Query acceleration policy](query-acceleration-policy.md) +- [.alter-merge query acceleration policy command](alter-merge-query-acceleration-policy-command.md) - [.delete query acceleration policy command](delete-query-acceleration-policy-command.md) - [.show query acceleration policy command](show-query-acceleration-policy-command.md) - [.show external table operations query_acceleration statistics](show-external-table-operations-query-acceleration-statistics.md) diff --git a/data-explorer/kusto/management/delete-query-acceleration-policy-command.md b/data-explorer/kusto/management/delete-query-acceleration-policy-command.md index df4a9723a3..a11889582b 100644 --- a/data-explorer/kusto/management/delete-query-acceleration-policy-command.md +++ b/data-explorer/kusto/management/delete-query-acceleration-policy-command.md @@ -1,15 +1,15 @@ --- -title: ".delete query acceleration policy command" +title: .delete query acceleration policy command description: Learn how to use the .delete query acceleration policy command to accelerate queries over external delta tables. ms.reviewer: sharmaanshul ms.topic: reference -ms.date: 11/19/2024 +ms.date: 09/16/2025 --- -# .delete query acceleration policy command +# `.delete query acceleration policy` command > [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] -Deletes the [query acceleration policy](query-acceleration-policy.md) of a specific external delta table. +Deletes the [`query acceleration policy`](query-acceleration-policy.md) of a specific external delta table. ## Permissions @@ -35,5 +35,6 @@ You must have at least [Database Admin](../access-control/role-based-access-cont * [Query acceleration policy](query-acceleration-policy.md) * [.alter query acceleration policy command](alter-query-acceleration-policy-command.md) +* [.alter-merge query acceleration policy command](alter-merge-query-acceleration-policy-command.md) * [.show query acceleration policy command](show-query-acceleration-policy-command.md) * [.show external table operations query_acceleration statistics](show-external-table-operations-query-acceleration-statistics.md) diff --git a/data-explorer/kusto/management/query-acceleration-policy.md b/data-explorer/kusto/management/query-acceleration-policy.md index f5a052c285..c187c3e849 100644 --- a/data-explorer/kusto/management/query-acceleration-policy.md +++ b/data-explorer/kusto/management/query-acceleration-policy.md @@ -3,13 +3,13 @@ title: Query acceleration policy description: Learn how to use the query acceleration policy to accelerate queries over external delta tables. ms.reviewer: sharmaanshul ms.topic: reference -ms.date: 11/19/2024 +ms.date: 09/16/2025 --- # Query acceleration policy > [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] -An [external table](../query/schema-entities/external-tables.md) is a schema entity that references data stored external to a Kusto database. Queries run over external tables can be less performant than on data that is ingested due to various factors such as network calls to fetch data from storage, the absence of indexes, and more. Query acceleration allows specifying a policy on top of external delta tables. This policy defines a number of days to accelerate data for high-performance queries. +An [external table](../query/schema-entities/external-tables.md) is a schema entity that references data stored external to a Kusto database. Queries run over external tables can be less performant than on data that is ingested due to various factors such as network calls to fetch data from storage, the absence of indexes, and more. Query acceleration allows specifying a policy on top of external delta tables. This policy defines the number of days to accelerate data for high-performance queries. ::: moniker range="azure-data-explorer" Query acceleration is supported in Azure Data Explorer over Azure Data Lake Store Gen2 or Azure blob storage [external tables](external-tables-azure-storage.md). @@ -25,24 +25,25 @@ To enable query acceleration in the Fabric UI, see [Query acceleration over OneL * The number of columns in the external table can't exceed 900. * Delta tables with checkpoint V2 aren't supported. -* Query performance over accelerated external delta tables which have more than 2.5 million data files may not be optimal. +* Query performance over accelerated external delta tables that have more than 2.5 million data files may not be optimal. * The feature assumes delta tables with static advanced features, for example column mapping doesn't change, partitions don't change, and so on. To change advanced features, first disable the policy, and once the change is made, re-enable the policy. * Schema changes on the delta table must also be followed with the respective `.alter` external delta table schema, which might result in acceleration starting from scratch if there was breaking schema change. * Parquet files larger than 1 GB won't be cached. * Manual edits to the delta table are not allowed and can lead to unexpected results. > [!NOTE] -> The query acceleration caching operations are limited by the available query acceleration capacity of your cluster. Run the [.show capacity command](show-capacity-command.md) to view the total, consumed, and remaining query acceleration capacity. +> The query acceleration caching operations are limited by the available query acceleration capacity of your cluster. Run the [`.show capacity command`](show-capacity-command.md) to view the total, consumed, and remaining query acceleration capacity. ## Known issues -* Data in the external delta table that's optimized with the [OPTIMIZE](/azure/databricks/sql/language-manual/delta-optimize) function will need to be reaccelearted. -* If you run frequent MERGE/UPDATE/DELETE operations in delta, the underlying parquet files may be rewritten with changes and Kusto will skip accelerating such files, causing retrieval during query time. +* Data in the external delta table that's optimized with the [OPTIMIZE](/azure/databricks/sql/language-manual/delta-optimize) function needs to be reaccelearted. +* If you run frequent MERGE/UPDATE/DELETE operations in delta, the underlying parquet files may be rewritten with changes and Kusto skips accelerating such files, causing retrieval during query time. * The system assumes that all artifacts under the delta table directory have the same access level to the selected users. Different files having different access permissions under the delta table directory might result with unexpected behavior. ## Commands for query acceleration * [.alter query acceleration policy command](alter-query-acceleration-policy-command.md) +* [.alter-merge query acceleration policy command](alter-merge-query-acceleration-policy-command.md) * [.delete query acceleration policy command](delete-query-acceleration-policy-command.md) * [.show query acceleration policy command](show-query-acceleration-policy-command.md) * [.show external table operations query_acceleration statistics](show-external-table-operations-query-acceleration-statistics.md) diff --git a/data-explorer/kusto/management/show-external-table-operations-query-acceleration-statistics.md b/data-explorer/kusto/management/show-external-table-operations-query-acceleration-statistics.md index a46ae271b3..5d7079f654 100644 --- a/data-explorer/kusto/management/show-external-table-operations-query-acceleration-statistics.md +++ b/data-explorer/kusto/management/show-external-table-operations-query-acceleration-statistics.md @@ -1,11 +1,11 @@ --- title: .show external table operations query_acceleration statistics command -description: Learn how to use the ".show external table operations query_acceleration statistics command" to accelerate queries over external delta tables. +description: Learn how to use the .show external table operations query_acceleration statistics command to accelerate queries over external delta tables. ms.reviewer: sharmaanshul ms.topic: reference -ms.date: 11/19/2024 +ms.date: 09/16/2025 --- -# .show external table operations query_acceleration statistics command +# `.show external table operations query_acceleration statistics` command > [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] @@ -60,5 +60,6 @@ The command returns a table with a record per external table that has a non-null * [Query acceleration policy](query-acceleration-policy.md) * [.alter query acceleration policy command](alter-query-acceleration-policy-command.md) +* [.alter-merge query acceleration policy command](alter-merge-query-acceleration-policy-command.md) * [.delete query acceleration policy command](delete-query-acceleration-policy-command.md) * [.show query acceleration policy command](show-query-acceleration-policy-command.md) diff --git a/data-explorer/kusto/management/show-query-acceleration-policy-command.md b/data-explorer/kusto/management/show-query-acceleration-policy-command.md index b3e71792b9..67f54a10f0 100644 --- a/data-explorer/kusto/management/show-query-acceleration-policy-command.md +++ b/data-explorer/kusto/management/show-query-acceleration-policy-command.md @@ -1,12 +1,12 @@ --- title: .show query acceleration policy command -description: Learn how to use the ".show query acceleration policy command" to accelerate queries over external delta tables. +description: Learn how to use the .show query acceleration policy command to accelerate queries over external delta tables. ms.reviewer: sharmaanshul ms.topic: reference -ms.date: 11/19/2024 +ms.date: 09/16/2025 --- -# .show query acceleration policy command +# `.show query acceleration policy` command > [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] @@ -50,11 +50,12 @@ The command returns a table with a record per external table with the following | PolicyName | EntityName | Policy | ChildEntities | EntityType | | ----------------------- | ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | ------------- | -| QueryAccelerationPolicy | [MyDatabase].[MyExternalTable] | {
"IsEnabled": true,
"Hot": "1.00:00:00",
"HotWindows": [{"MinValue":"2025-07-06 07:53:55.0192810","MaxValue":"2025-07-06 07:53:55.0192814"}], "MaxAge": "5m"} | | ExternalTable | +| QueryAccelerationPolicy | [MyDatabase].[MyExternalTable] | `{
"IsEnabled": true,
"Hot": "1.00:00:00",
"HotWindows": [{"MinValue":"2025-07-06 07:53:55.0192810","MaxValue":"2025-07-06 07:53:55.0192814"}], "MaxAge": "5m"}` | | ExternalTable | ## Related content - [Query acceleration policy](query-acceleration-policy.md) - [.alter query acceleration policy command](alter-query-acceleration-policy-command.md) +- [.alter-merge query acceleration policy command](alter-merge-query-acceleration-policy-command.md) - [.delete query acceleration policy command](delete-query-acceleration-policy-command.md) - [.show external table operations query_acceleration statistics](show-external-table-operations-query-acceleration-statistics.md) diff --git a/data-explorer/kusto/management/toc.yml b/data-explorer/kusto/management/toc.yml index 1c79e167e2..90f2704415 100644 --- a/data-explorer/kusto/management/toc.yml +++ b/data-explorer/kusto/management/toc.yml @@ -561,6 +561,9 @@ items: - name: .alter query acceleration policy command href: alter-query-acceleration-policy-command.md displayName: external table, delta table + - name: .alter-merge query acceleration policy command + href: alter-merge-query-acceleration-policy-command.md + displayName: external table, delta table - name: .delete query acceleration policy command href: delete-query-acceleration-policy-command.md displayName: external table, delta table diff --git a/data-explorer/media/devops/extension-install.png b/data-explorer/media/devops/extension-install.png index 2ad955fb25..90b6b26059 100644 Binary files a/data-explorer/media/devops/extension-install.png and b/data-explorer/media/devops/extension-install.png differ diff --git a/data-explorer/media/devops/get-extension.png b/data-explorer/media/devops/get-extension.png index e8b4d0ec86..4919d27061 100644 Binary files a/data-explorer/media/devops/get-extension.png and b/data-explorer/media/devops/get-extension.png differ diff --git a/data-explorer/net-sdk-ingest-data.md b/data-explorer/net-sdk-ingest-data.md index d38a1fb4f1..a7f34d13f3 100644 --- a/data-explorer/net-sdk-ingest-data.md +++ b/data-explorer/net-sdk-ingest-data.md @@ -1,9 +1,9 @@ --- title: 'Ingest data with Kusto .NET SDK' -description: In this article, you learn how to ingest (load) data into Azure Data Explorer using .NET SDK. +description: Learn how to ingest data into Azure Data Explorer using the Kusto .NET SDK. Follow step-by-step instructions to load, map, and validate your data. ms.reviewer: vladikb ms.topic: how-to -ms.date: 05/08/2023 +ms.date: 09/29/2025 # Customer intent: As a .NET SDK developer, I want to ingest data into Azure Data Explorer so that I can query data to include in my apps. --- @@ -17,13 +17,13 @@ ms.date: 05/08/2023 > * [Go](go-ingest-data.md) > * [Java](java-ingest-data.md) -There are two client libraries for .NET: an [ingest library](https://www.nuget.org/packages/Microsoft.Azure.Kusto.Ingest/) and [a data library](https://www.nuget.org/packages/Microsoft.Azure.Kusto.Data/). For more information on .NET SDK, see [about .NET SDK](/kusto/api/netfx/about-the-sdk?view=azure-data-explorer&preserve-view=true). -These libraries enable you to ingest (load) data into a cluster and query data from your code. In this article, you first create a table and data mapping in a test cluster. You then queue an ingestion to the cluster and validate the results. +There are two client libraries for .NET: an [ingest library](https://www.nuget.org/packages/Microsoft.Azure.Kusto.Ingest/) and a [data library](https://www.nuget.org/packages/Microsoft.Azure.Kusto.Data/). To learn more about the .NET SDK, see [about .NET SDK](/kusto/api/netfx/about-the-sdk?view=azure-data-explorer&preserve-view=true). +These libraries let you ingest (load) data into a cluster and query data from your code. In this article, you create a table and data mapping in a test cluster, queue an ingestion to the cluster, and validate the results. ## Prerequisites * A Microsoft account or a Microsoft Entra user identity. An Azure subscription isn't required. -* A cluster and database. [Create a cluster and database](create-cluster-and-database.md). +* A cluster and database. Learn how to [create a cluster and database](create-cluster-and-database.md). ## Install the ingest library @@ -35,13 +35,13 @@ Install-Package Microsoft.Azure.Kusto.Ingest ### Authentication -To authenticate an application, the SDK uses your Microsoft Entra tenant ID. To find your tenant ID, use the following URL, substituting your domain for *YourDomain*. +To authenticate an application, the SDK uses your Microsoft Entra tenant ID. Use the following URL to find your tenant ID, substituting your domain for *YourDomain*. ```http https://login.microsoftonline.com//.well-known/openid-configuration/ ``` -For example, if your domain is *contoso.com*, the URL is: [https://login.microsoftonline.com/contoso.com/.well-known/openid-configuration/](https://login.microsoftonline.com/contoso.com/.well-known/openid-configuration/). Click this URL to see the results; the first line is as follows. +For example, if your domain is *contoso.com*, the URL is: [https://login.microsoftonline.com/contoso.com/.well-known/openid-configuration/](https://login.microsoftonline.com/contoso.com/.well-known/openid-configuration/). Select this URL to see the results; the first line is as follows. ```console "authorization_endpoint":"https://login.microsoftonline.com/6babcaad-604b-40ac-a9d7-9fd97c0b779f/oauth2/authorize" @@ -49,16 +49,16 @@ For example, if your domain is *contoso.com*, the URL is: [https://login.microso The tenant ID in this case is `aaaabbbb-0000-cccc-1111-dddd2222eeee`. -This example uses an interactive Microsoft Entra user authentication to access the cluster. You can also use Microsoft Entra application authentication with certificate or application secret. Make sure to set the correct values for `tenantId` and `clusterUri` before running this code. +This example uses interactive Microsoft Entra user authentication to access the cluster. You can also use Microsoft Entra application authentication with certificate or application secret. Set the correct values for `tenantId` and `clusterUri` before running this code. The SDK provides a convenient way to set up the authentication method as part of the connection string. For complete documentation on connection strings, see [connection strings](/kusto/api/connection-strings/kusto?view=azure-data-explorer&preserve-view=true). > [!NOTE] -> The current version of the SDK doesn't support interactive user authentication on .NET Core. If required, use Microsoft Entra username/password or application authentication instead. +> The current version of the SDK doesn't support interactive user authentication on .NET Core. If needed, use Microsoft Entra username/password or application authentication instead. ### Construct the connection string -Now you can construct the connection string. You'll create the destination table and mapping in a later step. +Construct the connection string. You create the destination table and mapping in a later step. ```csharp var kustoUri = "https://..kusto.windows.net/"; @@ -68,7 +68,7 @@ var kustoConnectionStringBuilder = new KustoConnectionStringBuilder(kustoUri).Wi ## Set source file information -Set the path for the source file. This example uses a sample file hosted on Azure Blob Storage. The **StormEvents** sample dataset contains weather-related data from the [National Centers for Environmental Information](https://www.ncei.noaa.gov/). +Set the path for the source file. This example uses a sample file hosted on Azure Blob Storage. The **StormEvents** sample dataset includes weather-related data from the [National Centers for Environmental Information](https://www.ncei.noaa.gov/). ```csharp var blobPath = "https://kustosamples.blob.core.windows.net/samplefiles/StormEvents.csv"; @@ -79,7 +79,7 @@ var blobPath = "https://kustosamples.blob.core.windows.net/samplefiles/StormEven Create a table named `StormEvents` that matches the schema of the data in the `StormEvents.csv` file. > [!TIP] -> The following code snippets create an instance of a client for almost every call. This is done to make each snippet individually runnable. In production, the client instances are reentrant, and should be kept as long as needed. A single client instance per URI is sufficient, even when working with multiple databases (database can be specified on a command level). +> The following code snippets create an instance of a client for almost every call. This approach makes each snippet individually runnable. In production, client instances are reentrant and should be kept as long as needed. A single client instance per URI is sufficient, even when working with multiple databases (the database can be specified at the command level). ```csharp var databaseName = ""; @@ -120,8 +120,8 @@ using (var kustoClient = KustoClientFactory.CreateCslAdminProvider(kustoConnecti ## Define ingestion mapping -Map the incoming CSV data to the column names used when creating the table. -Provision a [CSV column mapping object](/kusto/management/create-ingestion-mapping-command?view=azure-data-explorer&preserve-view=true) on that table. +Map incoming CSV data to the column names used when creating the table. +Set up a [CSV column mapping object](/kusto/management/create-ingestion-mapping-command?view=azure-data-explorer&preserve-view=true) on that table. ```csharp var tableMappingName = "StormEvents_CSV_Mapping"; @@ -164,7 +164,7 @@ using (var kustoClient = KustoClientFactory.CreateCslAdminProvider(kustoConnecti ## Define batching policy for your table -Batching incoming data optimizes data shard size, which is controlled by the [ingestion batching policy](/kusto/management/batching-policy?view=azure-data-explorer&preserve-view=true). Modify the policy with the [ingestion batching policy management command](/kusto/management/show-table-ingestion-batching-policy?view=azure-data-explorer&preserve-view=true). Use this policy to reduce latency of slowly arriving data. +Batching incoming data optimizes data shard size. The [ingestion batching policy](/kusto/management/batching-policy?view=azure-data-explorer&preserve-view=true) controls this batching. Modify the policy using the [ingestion batching policy management command](/kusto/management/show-table-ingestion-batching-policy?view=azure-data-explorer&preserve-view=true). This policy reduces latency for slowly arriving data. ```csharp using (var kustoClient = KustoClientFactory.CreateCslAdminProvider(kustoConnectionStringBuilder)) @@ -182,19 +182,19 @@ using (var kustoClient = KustoClientFactory.CreateCslAdminProvider(kustoConnecti } ``` -We recommend defining a `Raw Data Size` value for ingested data and incrementally decreasing the size towards 250 MB, while checking if performance improves. +Define a `Raw Data Size` value for ingested data and gradually decrease the size to 250 MB to check if performance improves. -You can use the `Flush Immediately` property to skip batching, although this isn't recommended for large-scale ingestion as it can cause poor performance. +Use the `Flush Immediately` property to skip batching. However, this isn't recommended for large-scale ingestion because it can cause poor performance. ## Queue a message for ingestion -Queue a message to pull data from blob storage and ingest the data. A connection is established to the ingestion cluster, and another client is created to work with that endpoint. +Queue a message to pull data from blob storage and ingest it. Establish a connection to the ingestion cluster, and create another client to work with that endpoint. > [!TIP] > The following code snippets create an instance of a client for almost every call. This is done to make each snippet individually runnable. In production, the client instances are reentrant, and should be kept as long as needed. A single client instance per URI is sufficient, even when working with multiple databases (database can be specified on a command level). ```csharp -var ingestUri = "https://ingest-..kusto.windows.net"; +var ingestUri = "https://ingest-..kusto.windows.net"; // Replace and with your values var ingestConnectionStringBuilder = new KustoConnectionStringBuilder(ingestUri).WithAadUserPromptAuthentication(tenantId); using var ingestClient = KustoIngestFactory.CreateQueuedIngestClient(ingestConnectionStringBuilder); var properties = new KustoQueuedIngestionProperties(databaseName, tableName) @@ -207,12 +207,12 @@ var properties = new KustoQueuedIngestionProperties(databaseName, tableName) }, IgnoreFirstRecord = true }; -await ingestClient.IngestFromStorageAsync(blobPath, properties); +await ingestClient.IngestFromStorageAsync(blobPath, properties); // Ensure blobPath points to the correct storage location ``` -## Validate data was ingested into the table +## Validate data ingestion into the table -Wait five to ten minutes for the queued ingestion to schedule the ingestion and load the data into your cluster. Then run the following code to get the count of records in the `StormEvents` table. +Wait 5 to 10 minutes for the queued ingestion to schedule and load the data into your cluster. Run the following code to get the record count in the `StormEvents` table. ```csharp using var cslQueryProvider = KustoClientFactory.CreateCslQueryProvider(kustoConnectionStringBuilder); @@ -223,14 +223,14 @@ Console.WriteLine(results.Single()); ## Run troubleshooting queries -Sign in to [https://dataexplorer.azure.com](https://dataexplorer.azure.com) and connect to your cluster. Run the following command in your database to see if there were any ingestion failures in the last four hours. Replace the database name before running. +Sign in to [https://dataexplorer.azure.com](https://dataexplorer.azure.com) and connect to your cluster. Run the following command in your database to check for ingestion failures in the last four hours. Replace the database name before running. ```kusto .show ingestion failures | where FailedOn > ago(4h) and Database == "" ``` -Run the following command to view the status of all ingestion operations in the last four hours. Replace the database name before running. +Run the following command to check the status of all ingestion operations in the last four hours. Replace the database name before running. ```kusto .show operations @@ -240,7 +240,7 @@ Run the following command to view the status of all ingestion operations in the ## Clean up resources -If you plan to follow our other articles, keep the resources you created. If not, run the following command in your database to clean up the `StormEvents` table. +If you plan to follow other articles, keep the resources you created. If not, run the following command in your database to clean up the `StormEvents` table. ```kusto .drop table StormEvents @@ -248,4 +248,4 @@ If you plan to follow our other articles, keep the resources you created. If not ## Related content -* [Write queries](/azure/data-explorer/kusto/query/tutorials/learn-common-operators) +* Learn more about [writing queries](/azure/data-explorer/kusto/query/tutorials/learn-common-operators). diff --git a/data-explorer/open-telemetry-connector.md b/data-explorer/open-telemetry-connector.md index d11581c082..d685c5245a 100644 --- a/data-explorer/open-telemetry-connector.md +++ b/data-explorer/open-telemetry-connector.md @@ -1,7 +1,7 @@ --- title: Ingest data from OpenTelemetry to Azure Data Explorer description: Learn how to use Azure Data Explorer as an OpenTelemetry sink. -ms.date: 10/03/2022 +ms.date: 09/28/2025 ms.topic: how-to ms.reviewer: ramacg --- @@ -10,26 +10,28 @@ ms.reviewer: ramacg [!INCLUDE [real-time-analytics-connectors-note](includes/real-time-analytics-connectors-note.md)] -[OpenTelemetry](https://opentelemetry.io/docs/concepts/what-is-opentelemetry/) (OTel) is an open framework for application observability. The instrumentation is hosted by the Cloud Native Computing Foundation (CNCF), which provides standard interfaces for observability data, including [metrics](https://opentelemetry.io/docs/concepts/observability-primer/#reliability--metrics), [logs](https://opentelemetry.io/docs/concepts/observability-primer/#logs), and [traces](https://opentelemetry.io/docs/concepts/observability-primer/#distributed-traces). The OTel Collector is made up of the following three components: **receivers** deal with how to get data into the Collector, **processors** determine what to do with received data, and **exporters** are responsible for where to send the received data. +[OpenTelemetry](https://opentelemetry.io/docs/concepts/what-is-opentelemetry/) (OTel) is an open framework for application observability. The instrumentation is hosted by the Cloud Native Computing Foundation (CNCF). It provides standard interfaces for observability data, including [metrics](https://opentelemetry.io/docs/concepts/observability-primer/#reliability--metrics), [logs](https://opentelemetry.io/docs/concepts/observability-primer/#logs), and [traces](https://opentelemetry.io/docs/concepts/observability-primer/#distributed-traces). The OTel Collector includes three components: **receivers**, which get data into the Collector; **processors**, which determine what to do with the received data; and **exporters**, which send the received data to a destination. The [Azure Data Explorer exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/azuredataexplorerexporter) supports ingestion of data from many receivers into Azure Data Explorer. > [!NOTE] -> * The configuration settings are summarized in the [readme documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/azuredataexplorerexporter/README.md). -> * For the exporter source code, see [Azure Data Explorer exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/azuredataexplorerexporter). +> +> * Configuration settings are summarized in the [readme documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/azuredataexplorerexporter/README.md). +> * For the exporter source code, see [Azure Data Explorer exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/azuredataexplorerexporter). In this article, you learn how to: > [!div class="checklist"] -> * Set up your environment -> * Configure the Azure Data Explorer exporter -> * Run the sample application -> * Query incoming data +> +> * Set up your environment. +> * Configure the Azure Data Explorer exporter. +> * Run the sample application. +> * Query incoming data. ## Prerequisites -* An Azure subscription. Create a [free Azure account](https://azure.microsoft.com/free/) -* A cluster and a database: [Quickstart: Create an Azure Data Explorer cluster and database](create-cluster-and-database.md) +* An Azure subscription. Create a [free Azure account](https://azure.microsoft.com/free/). +* A cluster and a database. Learn how to [create an Azure Data Explorer cluster and database](create-cluster-and-database.md). ## Set up your environment @@ -39,21 +41,18 @@ In this section, you prepare your environment to use the OTel exporter. ### Create a Microsoft Entra app registration -Microsoft Entra application authentication is used for applications that need to access Azure Data Explorer without a user present. To ingest data using the OTel exporter, you need to create and register a Microsoft Entra service principal, and then authorize this principal to ingest data an Azure Data Explorer database. +Microsoft Entra application authentication lets applications access Azure Data Explorer without a user present. To ingest data using the OTel exporter, create and register a Microsoft Entra service principal, and authorize this principal to ingest data into an Azure Data Explorer database. -1. Using your Azure Data Explorer cluster, follow steps 1-7 in [Create a Microsoft Entra application registration in Azure Data Explorer](provision-entra-id-app.md). -1. Save the following values to be used in later steps: - * Application (client) ID - * Directory (tenant) ID - * Client secret key value +1. On your Azure Data Explorer cluster, follow steps 1–7 in [Create a Microsoft Entra application registration in Azure Data Explorer](provision-entra-id-app.md). +1. Save the following values for later steps: ### Grant the Microsoft Entra app permissions -1. In the query tab of the [web UI](https://dataexplorer.azure.com/), connect to your cluster. For more information on how to connect, see [Add clusters](web-query-data.md#add-clusters). +1. In the query tab of the [web UI](https://dataexplorer.azure.com/), connect to your cluster. To learn how to connect, see [Add clusters](web-query-data.md#add-clusters). 1. Browse to the database in which you want to ingest data. -1. Run the following management command, replacing the placeholders. Replace *DatabaseName* with the name of the target database and *ApplicationID* with the previously saved value. This command grants the app the [database ingestor](/kusto/access-control/role-based-access-control?view=azure-data-explorer&preserve-view=true) role. For more information, see [Manage database security roles](/kusto/management/manage-database-security-roles?view=azure-data-explorer&preserve-view=true). +1. Run the following management command, replacing the placeholders. Replace *DatabaseName* with the target database name and *ApplicationID* with the saved value. ```kusto .add database ingestors ('aadapp=') 'Azure Data Explorer App Registration' @@ -65,9 +64,9 @@ Microsoft Entra application authentication is used for applications that need to ### Create target tables 1. Browse to [Azure Data Explorer web UI](https://dataexplorer.azure.com/). -1. Select **Query** from the left menu. +1. Select **Query** from the left menu. 1. Expand the target cluster in the left pane. -1. Select the target database to give your queries the correct context. +1. Select the target database to give your queries correct context. 1. Run the following commands to create tables and schema mapping for the incoming data: ```kusto @@ -80,11 +79,11 @@ Microsoft Entra application authentication is used for applications that need to ### Set up streaming ingestion -Azure Data Explorer has two main types of ingestion: batching and streaming. For more information, see [batching vs streaming ingestion](ingest-data-overview.md#continuous-data-ingestion). The *streaming* method is called *managed* in the Azure Data Explorer exporter configuration. Streaming ingestion may be a good choice for you if you need the logs and traces are to be available in near real time. However, streaming ingestion uses more resources than batched ingestion. The OTel framework itself batches data, which should be considered when choosing which method to use for ingestion. +Azure Data Explorer has two main types of ingestion: batching and streaming. For more information, see [batching vs streaming ingestion](ingest-data-overview.md#continuous-data-ingestion). The *streaming* method is called *managed* in the Azure Data Explorer exporter configuration. Streaming ingestion might be a good choice if you need logs and traces available in near real time. However, streaming ingestion uses more resources than batched ingestion. The OTel framework batches data, which you should consider when choosing an ingestion method. > [!NOTE] -> [Streaming ingestion](ingest-data-streaming.md) must be enabled on Azure Data Explorer cluster to enable the `managed` option. -> You can check if streaming is enabled using the [.show database streaming ingestion policy](/kusto/management/show-database-streaming-ingestion-policy-command?view=azure-data-explorer&preserve-view=true) command. +> [Streaming ingestion](ingest-data-streaming.md) must be enabled on the Azure Data Explorer cluster to use the `managed` option. +> Check if streaming is enabled by running [.show database streaming ingestion policy](/kusto/management/show-database-streaming-ingestion-policy-command?view=azure-data-explorer&preserve-view=true). Run the following command for each of the three tables to enable streaming ingestion: @@ -100,26 +99,26 @@ In order to ingest your OpenTelemetry data into Azure Data Explorer, you need [d |Field | Description | Suggested setting| |---|---|---| - | Exporters| Type of exporter | Azure Data Explorer | + | Exporters| Type of exporter | Azure Data Explorer | | cluster_uri | URI of the Azure Data Explorer cluster that holds the database and tables | https:// <cluster>.kusto.windows.net | | application_id | Client ID| <application id> | | application_key| Client secret | <application key> | | tenant_id | Tenant | <application tenant>| - | db_name | Database that receives the logs | oteldb, or other database you have already created - | metrics_table_name | The target table in the database db_name that stores exported metric data. | OTELMetrics - | logs_table_name | The target table in the database db_name that stores exported logs data. | OTELLogs - | traces_table_name | The target table in the database db_name that stores exported traces data. | OTELTraces - | ingestion_type | Type of ingestion: managed (streaming) or batched | managed - | metrics_table_json_mapping | Optional parameter. Default table mapping is defined during table creation based on OTeL [metrics attributes](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/azuredataexplorerexporter#metrics). The default mapping can be changed using this parameter. | <json metrics_table_name mapping> - | logs_table_json_mapping | Optional parameter. Default table mapping is defined during table creation based on OTeL [logs attributes](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/azuredataexplorerexporter#logs). The default mapping can be changed using this parameter. | <json logs_table_name mapping> - | traces_table_json_mapping | Optional parameter. Default table mapping is defined during table creation based on OTeL [trace attributes](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/azuredataexplorerexporter#traces). The default mapping can be changed using this parameter. |<json traces_table_name mapping> - | traces | Services: traces components to enable | receivers: [otlp]
processors: [batch]
exporters: [azuredataexplorer] - | metrics | Services: metrics components to enable | receivers: [otlp]
processors: [batch]
exporters: [azuredataexplorer] - | logs | Services: logs components to enable | receivers: [otlp]
processors: [batch]
exporters: [ azuredataexplorer] - -1. Use the "--config" flag to run the Azure Data Explorer exporter. - -The following is an example configuration for the Azure Data Explorer exporter: + | db_name | Database that receives the logs | oteldb, or other database you created. | + | metrics_table_name | The target table in the database db_name that stores exported metric data. | OTELMetrics | + | logs_table_name | The target table in the database db_name that stores exported logs data. | OTELLogs | + | traces_table_name | The target table in the database db_name that stores exported traces data. | OTELTraces | + | ingestion_type | Type of ingestion: managed (streaming) or batched | managed | + | metrics_table_json_mapping | Optional parameter. Default table mapping is defined during table creation based on OTeL [metrics attributes](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/azuredataexplorerexporter#metrics). The default mapping can be changed using this parameter. | <json metrics_table_name mapping> | + | logs_table_json_mapping | Optional parameter. Default table mapping is defined during table creation based on OTeL [logs attributes](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/azuredataexplorerexporter#logs). The default mapping can be changed using this parameter. | <json logs_table_name mapping> | + | traces_table_json_mapping | Optional parameter. Default table mapping is defined during table creation based on OTeL [trace attributes](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/azuredataexplorerexporter#traces). The default mapping can be changed using this parameter. |<json traces_table_name mapping> | + | traces | Services: traces components to enable | receivers: [otlp]
processors: [batch]
exporters: [azuredataexplorer] | + | metrics | Services: metrics components to enable | receivers: [otlp]
processors: [batch]
exporters: [azuredataexplorer] | + | logs | Services: logs components to enable | receivers: [otlp]
processors: [batch]
exporters: [ azuredataexplorer] | + +1. To run the Azure Data Explorer exporter, use the "--config" flag. + +This is an example configuration for the Azure Data Explorer exporter: ```yaml --- @@ -162,7 +161,7 @@ service: ## Collect data with a sample application -Now that the collector is configured, you need to send data to be ingested. In this example. you use the sample [spring pet clinic](https://github.com/spring-projects/spring-petclinic) application with the java OTeL collector agent. +Now that the collector is configured, you need to send data to be ingested. In this example. You use the sample [spring pet clinic](https://github.com/spring-projects/spring-petclinic) application with the java OTeL collector agent. 1. Download the collector agent here: [Open telemetry collector agent](https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases). @@ -198,7 +197,7 @@ Now that the collector is configured, you need to send data to be ingested. In t ## Query incoming data -Once the sample app has run, your data has been ingested into the defined tables in Azure Data Explorer. These tables were created in a database that was defined in the OTel collector configuration, as *oteldb*. The tables you've created were defined in the OTel collector configuration. In this example, you've created three tables: *OTELMetrics*, *OTELLogs*, and *OTELTraces*. In this section, you query each table separately to get a small selection of the available data. +Once the sample app has run, your data has been ingested into the defined tables in Azure Data Explorer. These tables are created in a database that was defined in the OTel collector configuration, as *oteldb*. The tables you created are defined in the OTel collector configuration. In this example, you created three tables: *OTELMetrics*, *OTELLogs*, and *OTELTraces*. In this section, you query each table separately to get a small selection of the available data. 1. Browse to [Azure Data Explorer web UI](https://dataexplorer.azure.com/). 1. Select **Query** from the left menu. @@ -249,13 +248,12 @@ Once the sample app has run, your data has been ingested into the defined tables |573c0e4e002a9f7281f6d63eafe4ef87|dab70d0ba8902c5e| |87d003d6-02c1-4f3d-8972-683243c35642|STATUS_CODE_UNSET|SPAN_KIND_CLIENT |2022-07-01T13:17:59Z |2022-07-01T13:17:59Z |{"telemetry.auto.version":"1.14.0", "os.description":"Windows 11 10.0", "process.executable.path":"C:\\Program Files\\Java\\jdk-18.0.1.1;bin;java.exe", "process.runtime.description":"Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 18.0.1.1+2-6", "service.name":"my-service", "process.runtime.name":"Java(TM) SE Runtime Environment", "telemetry.sdk.language":"java", "telemetry.sdk.name":"opentelemetry", "host.arch":"amd64", "host.name":"DESKTOP-SFS7RUQ", "process.pid":34316, "process.runtime.version":"18.0.1.1+2-6", "os.type":"windows", "process.command_line":"C:\\Program Files\\Java\\jdk-18.0.1.1;bin;java.exe -javaagent:./opentelemetry-javaagent.jar", "telemetry.sdk.version":"1.14.0"}|{"db.user":"sa", "thread.id":1, "db.name":"87d003d6-02c1-4f3d-8972-683243c35642", "thread.name":"main", "db.system":"h2", "scope.name":"io.opentelemetry.jdbc", "scope.version":"1.14.0-alpha", "db.connection_string":"h2:mem:", "db.statement":"DROP TABLE vet_specialties IF EXISTS"}|[] |[] | |84a9a8c4009d91476da02dfa40746c13|3cd4c0e91717969a| |87d003d6-02c1-4f3d-8972-683243c35642|STATUS_CODE_UNSET|SPAN_KIND_CLIENT |2022-07-01T13:17:59Z |2022-07-01T13:17:59Z |{"telemetry.auto.version":"1.14.0", "os.description":"Windows 11 10.0", "process.executable.path":"C:\\Program Files\\Java\\jdk-18.0.1.1;bin;java.exe", "process.runtime.description":"Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 18.0.1.1+2-6", "service.name":"my-service", "process.runtime.name":"Java(TM) SE Runtime Environment", "telemetry.sdk.language":"java", "telemetry.sdk.name":"opentelemetry", "host.arch":"amd64", "host.name":"DESKTOP-SFS7RUQ", "process.pid":34316, "process.runtime.version":"18.0.1.1+2-6", "os.type":"windows", "process.command_line":"C:\\Program Files\\Java\\jdk-18.0.1.1;bin;java.exe -javaagent:./opentelemetry-javaagent.jar", "telemetry.sdk.version":"1.14.0"}|{"db.user":"sa", "thread.id":1, "db.name":"87d003d6-02c1-4f3d-8972-683243c35642", "thread.name":"main", "db.system":"h2", "scope.name":"io.opentelemetry.jdbc", "scope.version":"1.14.0-alpha", "db.connection_string":"h2:mem:", "db.statement":"DROP TABLE vets IF EXISTS"} |[] |[] | - ### Further data processing -Using update policies, the collected data can further be processed as per application need. For more information, see [Update policy overview](/kusto/management/update-policy?view=azure-data-explorer&preserve-view=true). +Use update policies to further process the collected data, as per application need. For more information, see [Update policy overview](/kusto/management/update-policy?view=azure-data-explorer&preserve-view=true). 1. The following example exports histogram metrics to a histo-specific table with buckets and aggregates. Run the following command in the query pane of the Azure Data Explorer web UI: - + ```kusto .create table HistoBucketData (Timestamp: datetime, MetricName: string , MetricType: string , Value: double, LE: double, Host: string , ResourceAttributes: dynamic, MetricAttributes: dynamic ) diff --git a/data-explorer/python-ingest-data.md b/data-explorer/python-ingest-data.md index c9f6b31ed4..d73a62bd75 100644 --- a/data-explorer/python-ingest-data.md +++ b/data-explorer/python-ingest-data.md @@ -3,7 +3,7 @@ title: 'Ingest data using the Azure Data Explorer Python library' description: In this article, you learn how to ingest (load) data into Azure Data Explorer using Python. ms.reviewer: vladikbr ms.topic: how-to -ms.date: 05/08/2023 +ms.date: 09/30/2025 # Customer intent: As a Python developer, I want to ingest data into Azure Data Explorer so that I can query data to include in my apps. --- @@ -11,25 +11,26 @@ ms.date: 05/08/2023 # Ingest data using the Azure Data Explorer Python library > [!div class="op_single_selector"] +> > * [.NET](net-sdk-ingest-data.md) > * [Python](python-ingest-data.md) > * [Node](node-ingest-data.md) > * [Go](go-ingest-data.md) > * [Java](java-ingest-data.md) -In this article, you ingest data using the Azure Data Explorer Python library. Azure Data Explorer is a fast and highly scalable data exploration service for log and telemetry data. Azure Data Explorer provides two client libraries for Python: an [ingest library](https://github.com/Azure/azure-kusto-python/tree/master/azure-kusto-ingest) and [a data library](https://github.com/Azure/azure-kusto-python/tree/master/azure-kusto-data). These libraries enable you to ingest, or load, data into a cluster and query data from your code. +In this article, you use the Azure Data Explorer Python library to ingest data. Azure Data Explorer is a fast, scalable data exploration service for log and telemetry data. Azure Data Explorer provides two client libraries for Python: an [ingest library](https://github.com/Azure/azure-kusto-python/tree/master/azure-kusto-ingest) and [a data library](https://github.com/Azure/azure-kusto-python/tree/master/azure-kusto-data). These libraries enable you to ingest, or load, data into a cluster and query data from your code. -First, create a table and data mapping in a cluster. You then queue ingestion to the cluster and validate the results. +First, create a table and data mapping in a cluster. Then, queue ingestion to the cluster and validate the results. ## Prerequisites * A Microsoft account or a Microsoft Entra user identity. An Azure subscription isn't required. -* An Azure Data Explorer cluster and database. [Create a cluster and database](create-cluster-and-database.md). -* [Python 3.4+](https://www.python.org/downloads/). +* An Azure Data Explorer cluster and database. Learn how to [create a cluster and database](create-cluster-and-database.md). +* Install [Python 3.4+](https://www.python.org/downloads/). ## Install the data and ingest libraries -Install *azure-kusto-data* and *azure-kusto-ingest*. +Install the *azure-kusto-data* and *azure-kusto-ingest* libraries. ```python pip install azure-kusto-data @@ -46,7 +47,7 @@ from azure.kusto.data.exceptions import KustoServiceError from azure.kusto.data.helpers import dataframe_from_result_table ``` -To authenticate an application, Azure Data Explorer uses your Microsoft Entra tenant ID. To find your tenant ID, use the following URL, replacing your domain for *YourDomain*. +To authenticate an application, Azure Data Explorer uses your Microsoft Entra tenant ID. To find your tenant ID, use the following URL, replacing your domain with *YourDomain*. ```http https://login.microsoftonline.com//.well-known/openid-configuration/ @@ -58,7 +59,7 @@ For example, if your domain is *contoso.com*, the URL is: [https://login.microso "authorization_endpoint":"https://login.microsoftonline.com/6babcaad-604b-40ac-a9d7-9fd97c0b779f/oauth2/authorize" ``` -The tenant ID in this case is `aaaabbbb-0000-cccc-1111-dddd2222eeee`. Set the values for AAD_TENANT_ID, KUSTO_URI, KUSTO_INGEST_URI, and KUSTO_DATABASE before running this code. +The tenant ID in this case is `aaaabbbb-0000-cccc-1111-dddd2222eeee`. Set the values for `AAD_TENANT_ID`, `KUSTO_URI`, `KUSTO_INGEST_URI`, and `KUSTO_DATABASE` before running this code. ```python AAD_TENANT_ID = "" @@ -67,7 +68,7 @@ KUSTO_INGEST_URI = "https://ingest-..kusto.windows.net/" KUSTO_DATABASE = "" ``` -Now construct the connection string. The following example uses device authentication to access the cluster. You can also use [managed identity](managed-identities-overview.md) authentication, [Microsoft Entra application certificate](https://github.com/Azure/azure-kusto-python/blob/master/azure-kusto-data/tests/sample.py#L24), [Microsoft Entra application key](https://github.com/Azure/azure-kusto-python/blob/master/azure-kusto-data/tests/sample.py#L20), and [Microsoft Entra user and password](https://github.com/Azure/azure-kusto-python/blob/master/azure-kusto-data/tests/sample.py#L34). +Construct the connection string. The following example uses device authentication to access the cluster. You can also use [managed identity](managed-identities-overview.md) authentication, [Microsoft Entra application certificate](https://github.com/Azure/azure-kusto-python/blob/master/azure-kusto-data/tests/sample.py#L24), [Microsoft Entra application key](https://github.com/Azure/azure-kusto-python/blob/master/azure-kusto-data/tests/sample.py#L20), or [Microsoft Entra user and password](https://github.com/Azure/azure-kusto-python/blob/master/azure-kusto-data/tests/sample.py#L34). You create the destination table and mapping in a later step. @@ -84,7 +85,7 @@ DESTINATION_TABLE_COLUMN_MAPPING = "StormEvents_CSV_Mapping" ## Set source file information -Import additional classes and set constants for the data source file. This example uses a sample file hosted on Azure Blob Storage. The **StormEvents** sample dataset contains weather-related data from the [National Centers for Environmental Information](https://www.ncei.noaa.gov/). +Import additional classes and set constants for the data source file. This example uses a sample file hosted on Azure Blob Storage. The **StormEvents** sample dataset has weather-related data from the [National Centers for Environmental Information](https://www.ncei.noaa.gov/). ```python from azure.kusto.data import DataFormat @@ -92,17 +93,17 @@ from azure.kusto.ingest import QueuedIngestClient, IngestionProperties, FileDesc CONTAINER = "samplefiles" ACCOUNT_NAME = "kustosamples" -SAS_TOKEN = "" # If relevant add SAS token +SAS_TOKEN = "" # If relevant, add SAS token FILE_PATH = "StormEvents.csv" -FILE_SIZE = 64158321 # in bytes +FILE_SIZE = 64158321 # In bytes BLOB_PATH = "https://" + ACCOUNT_NAME + ".blob.core.windows.net/" + \ - CONTAINER + "/" + FILE_PATH + SAS_TOKEN + CONTAINER + "/" + FILE_PATH + SAS_TOKEN # Construct the full blob path ``` ## Create a table on your cluster -Create a table that matches the schema of the data in the StormEvents.csv file. When this code runs, it returns a message like the following message: *To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code F3W4VWZDM to authenticate*. Follow the steps to sign in, then return to run the next code block. Subsequent code blocks that make a connection require you to sign in again. +Create a table that matches the schema of the data in the StormEvents.csv file. When this code runs, it returns a message like this: *To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code F3W4VWZDM to authenticate*. Follow the steps to sign in, then return to run the next code block. Subsequent code blocks that make a connection require you to sign in again. ```python KUSTO_CLIENT = KustoClient(KCSB_DATA) @@ -115,10 +116,10 @@ dataframe_from_result_table(RESPONSE.primary_results[0]) ## Define ingestion mapping -Map incoming CSV data to the column names and data types used when creating the table. This maps source data fields to destination table columns +Map incoming CSV data to the column names and data types used when creating the table. This process maps source data fields to destination table columns. ```python -CREATE_MAPPING_COMMAND = """.create table StormEvents ingestion csv mapping 'StormEvents_CSV_Mapping' '[{"Name":"StartTime","datatype":"datetime","Ordinal":0}, {"Name":"EndTime","datatype":"datetime","Ordinal":1},{"Name":"EpisodeId","datatype":"int","Ordinal":2},{"Name":"EventId","datatype":"int","Ordinal":3},{"Name":"State","datatype":"string","Ordinal":4},{"Name":"EventType","datatype":"string","Ordinal":5},{"Name":"InjuriesDirect","datatype":"int","Ordinal":6},{"Name":"InjuriesIndirect","datatype":"int","Ordinal":7},{"Name":"DeathsDirect","datatype":"int","Ordinal":8},{"Name":"DeathsIndirect","datatype":"int","Ordinal":9},{"Name":"DamageProperty","datatype":"int","Ordinal":10},{"Name":"DamageCrops","datatype":"int","Ordinal":11},{"Name":"Source","datatype":"string","Ordinal":12},{"Name":"BeginLocation","datatype":"string","Ordinal":13},{"Name":"EndLocation","datatype":"string","Ordinal":14},{"Name":"BeginLat","datatype":"real","Ordinal":16},{"Name":"BeginLon","datatype":"real","Ordinal":17},{"Name":"EndLat","datatype":"real","Ordinal":18},{"Name":"EndLon","datatype":"real","Ordinal":19},{"Name":"EpisodeNarrative","datatype":"string","Ordinal":20},{"Name":"EventNarrative","datatype":"string","Ordinal":21},{"Name":"StormSummary","datatype":"dynamic","Ordinal":22}]'""" +CREATE_MAPPING_COMMAND = """.create table StormEvents ingestion csv mapping 'StormEvents_CSV_Mapping' '[{"Name":"StartTime","datatype":"datetime","Ordinal":0}, {"Name":"EndTime","datatype":"datetime","Ordinal":1}, {"Name":"EpisodeId","datatype":"int","Ordinal":2}, {"Name":"EventId","datatype":"int","Ordinal":3}, {"Name":"State","datatype":"string","Ordinal":4}, {"Name":"EventType","datatype":"string","Ordinal":5}, {"Name":"InjuriesDirect","datatype":"int","Ordinal":6}, {"Name":"InjuriesIndirect","datatype":"int","Ordinal":7}, {"Name":"DeathsDirect","datatype":"int","Ordinal":8}, {"Name":"DeathsIndirect","datatype":"int","Ordinal":9}, {"Name":"DamageProperty","datatype":"int","Ordinal":10}, {"Name":"DamageCrops","datatype":"int","Ordinal":11}, {"Name":"Source","datatype":"string","Ordinal":12}, {"Name":"BeginLocation","datatype":"string","Ordinal":13}, {"Name":"EndLocation","datatype":"string","Ordinal":14}, {"Name":"BeginLat","datatype":"real","Ordinal":16}, {"Name":"BeginLon","datatype":"real","Ordinal":17}, {"Name":"EndLat","datatype":"real","Ordinal":18}, {"Name":"EndLon","datatype":"real","Ordinal":19}, {"Name":"EpisodeNarrative","datatype":"string","Ordinal":20}, {"Name":"EventNarrative","datatype":"string","Ordinal":21}, {"Name":"StormSummary","datatype":"dynamic","Ordinal":22}]'""" RESPONSE = KUSTO_CLIENT.execute_mgmt(KUSTO_DATABASE, CREATE_MAPPING_COMMAND) @@ -132,7 +133,7 @@ Queue a message to pull data from blob storage and ingest that data into Azure D ```python INGESTION_CLIENT = QueuedIngestClient(KCSB_INGEST) -# All ingestion properties are documented here: https://learn.microsoft.com/azure/kusto/management/data-ingest#ingestion-properties +# All ingestion properties are documented here: Learn more in [Azure Data Explorer ingestion properties](https://learn.microsoft.com/azure/kusto/management/data-ingest#ingestion-properties). INGESTION_PROPERTIES = IngestionProperties(database=KUSTO_DATABASE, table=DESTINATION_TABLE, data_format=DataFormat.CSV, ingestion_mapping_reference=DESTINATION_TABLE_COLUMN_MAPPING, additional_properties={'ignoreFirstRecord': 'true'}) # FILE_SIZE is the raw size of the data in bytes @@ -140,12 +141,12 @@ BLOB_DESCRIPTOR = BlobDescriptor(BLOB_PATH, FILE_SIZE) INGESTION_CLIENT.ingest_from_blob( BLOB_DESCRIPTOR, ingestion_properties=INGESTION_PROPERTIES) -print('Done queuing up ingestion with Azure Data Explorer') +print('Ingestion queued successfully with Azure Data Explorer') ``` ## Query data that was ingested into the table -Wait for five to 10 minutes for the queued ingestion to schedule the ingest and load the data into Azure Data Explorer. Then run the following code to get the count of records in the StormEvents table. +Wait 5 to 10 minutes for the queued ingestion to schedule and load the data into Azure Data Explorer. Run the following code to count the records in the StormEvents table. ```python QUERY = "StormEvents | count" @@ -157,14 +158,14 @@ dataframe_from_result_table(RESPONSE.primary_results[0]) ## Run troubleshooting queries -Sign in to [https://dataexplorer.azure.com](https://dataexplorer.azure.com) and connect to your cluster. Run the following command in your database to see if there were any ingestion failures in the last four hours. Replace the database name before running. +Sign in to [https://dataexplorer.azure.com](https://dataexplorer.azure.com) and connect to your cluster. Run this command in your database to check for ingestion failures in the last four hours. Replace the database name before running. ```Kusto .show ingestion failures | where FailedOn > ago(4h) and Database == "" ``` -Run the following command to view the status of all ingestion operations in the last four hours. Replace the database name before running. +Run this command to view the status of all ingestion operations in the last four hours. Replace the database name before running. ```Kusto .show operations @@ -174,9 +175,9 @@ Run the following command to view the status of all ingestion operations in the ## Clean up resources -If you plan to follow our other articles, keep the resources you created. If not, run the following command in your database to clean up the StormEvents table. +If you plan to follow other articles, keep the resources you created. If not, run the following command in your database to clean up the StormEvents table. -```Kusto +```kusto .drop table StormEvents ``` diff --git a/data-explorer/security-network-private-endpoint.md b/data-explorer/security-network-private-endpoint.md index c79be7777f..afea61bab5 100644 --- a/data-explorer/security-network-private-endpoint.md +++ b/data-explorer/security-network-private-endpoint.md @@ -1,36 +1,36 @@ --- title: Private endpoints for Azure Data Explorer -description: In this article, you'll learn about private endpoints for Azure Data Explorer. +description: Discover how private endpoints enhance Azure Data Explorer security by enabling private network access and blocking public connections. ms.reviewer: basaba ms.topic: how-to -ms.date: 04/05/2022 +ms.date: 09/28/2025 --- # Private endpoints for Azure Data Explorer -You can use [private endpoints](/azure/private-link/private-endpoint-overview) for your cluster to allow clients on a [virtual network](/azure/virtual-network/virtual-networks-overview) to securely access data over a [private link](/azure/private-link/private-link-overview). Private endpoints use private IP addresses from your virtual network address space to connect you privately to your cluster. Network traffic between clients on the virtual network and the cluster, traverses over the virtual network and a private link on the [Microsoft backbone network](/azure/networking/microsoft-global-network), eliminating exposure from the public internet. +Use [private endpoints](/azure/private-link/private-endpoint-overview) for your cluster to let clients on a [virtual network](/azure/virtual-network/virtual-networks-overview) securely access data over a [private link](/azure/private-link/private-link-overview). Private endpoints use private IP addresses from your virtual network address space to connect to your cluster securely. Network traffic between clients on the virtual network and the cluster traverses the virtual network and a private link on the [Microsoft backbone network](/azure/networking/microsoft-global-network), avoiding exposure to the public internet. -Using private endpoints for your cluster enables you to: +Private endpoints for your cluster let you: -* Secure your cluster by configuring the firewall to block all connections on the public endpoint to the cluster. -* Increase security for the virtual network by enabling you to block exfiltration of data from the virtual network. -* Securely connect to clusters from on-premises networks that connect to the virtual network using a [VPN gateway](/azure/vpn-gateway/vpn-gateway-about-vpngateways) or [ExpressRoutes](/azure/expressroute/expressroute-locations) with private-peering. +* Secure the cluster by setting the firewall to block all connections on the public endpoint. +* Increase virtual network security by blocking data exfiltration. +* Securely connect to clusters from on-premises networks through a [VPN gateway](/azure/vpn-gateway/vpn-gateway-about-vpngateways) or [ExpressRoutes](/azure/expressroute/expressroute-locations) with private peering. ## Overview -A private endpoint is a special network interface for an Azure service in your virtual network that is assigned IP addresses from the IP address range of your virtual network. When you create a private endpoint for your cluster, it provides secure connectivity between clients on your virtual network and your cluster. The connection between the private endpoint and the cluster uses a secure private link. +A private endpoint is a special network interface for an Azure service in your virtual network. It's assigned IP addresses from the IP address range of your virtual network. Creating a private endpoint for your cluster provides secure connectivity between clients on your virtual network and your cluster. The private endpoint and the cluster connect using a secure private link. -:::image type="content" source="media/security-network-private-endpoint/pe-diagram-detail.png" alt-text="Diagram showing the schema of the private endpoint architecture."::: +:::image type="content" source="media/security-network-private-endpoint/pe-diagram-detail.png" alt-text="Screenshot of the private endpoint architecture schema for Azure Data Explorer."::: -Applications in the virtual network can seamlessly connect to the cluster over the private endpoint. The connection strings and authorization mechanisms are the same as you'd use to connect to a public endpoint. +Applications in the virtual network connect to the cluster over the private endpoint. The connection strings and authorization mechanisms are the same as those used to connect to a public endpoint. -When you create a private endpoint for cluster in your virtual network, a consent request is sent for approval to the cluster owner. If the user requesting the creation of the private endpoint is also an owner of the cluster, the request is automatically approved. Cluster owners can manage consent requests and private endpoints for the cluster in the Azure portal, under **Private endpoints**. +Creating a private endpoint for a cluster in your virtual network sends a consent request to the cluster owner for approval. If the user requesting the creation of the private endpoint is also an owner of the cluster, the request is automatically approved. Cluster owners can manage consent requests and private endpoints for the cluster in the Azure portal, under **Private endpoints**. -You can secure your cluster to only accept connections from your virtual network by configuring the cluster firewall to deny access through its public endpoint by default. You don't need a firewall rule to allow traffic from a virtual network that has a private endpoint because the cluster firewall only controls access for the public endpoint. In contrast, private endpoints rely on the consent flow for granting subnets access to the cluster. +Secure your cluster to accept connections only from your virtual network by configuring the cluster firewall to deny access through its public endpoint by default. You don't need a firewall rule to allow traffic from a virtual network that has a private endpoint because the cluster firewall only controls access for the public endpoint. In contrast, private endpoints rely on the consent flow for granting subnets access to the cluster. -## Plan the size of subnet in your virtual network +## Plan the size of the subnet in your virtual network -The size of the subnet used to host a private endpoint for a cluster can't be altered once the subnet is deployed. The private endpoint consumes multiple IP addresses in your virtual network. In extreme scenarios, such as high-end ingestion, the number of IP addresses consumed by the private endpoint might increase. This increase is caused by an increased number of transient storage accounts required as staging accounts for ingesting into your cluster. If the scenario is relevant in your environment, you must plan for it when determining the size for the subnet. +The size of the subnet that hosts a private endpoint for a cluster can't be changed after the subnet is deployed. The private endpoint consumes multiple IP addresses in your virtual network. In extreme scenarios, like high-end ingestion, the number of IP addresses used by the private endpoint can increase. The cause of this increase is due an increased number of transient storage accounts required as staging accounts for ingesting into your cluster. If this scenario applies to your environment, plan for it when determining the subnet size. > [!NOTE] > The relevant ingestion scenarios that would be responsible for scaling out the transient storage accounts are [ingestion from a local file](/kusto/api/netfx/kusto-ingest-client-examples?view=azure-data-explorer&preserve-view=true#ingest-from-local-file) and [async ingestion from a blob](/kusto/api/netfx/kusto-ingest-client-examples?view=azure-data-explorer&preserve-view=true#async-ingestion-from-a-single-azure-blob). @@ -46,9 +46,9 @@ Use the following information to help you determine the total number of IP addre | **Total** | **13** | > [!NOTE] -> The absolute minimum size for the subnet must be **/28** (14 usable IP addresses). If you plan to create an Azure Data Explorer cluster for extreme ingestion workloads you are on the safe side with a **/24** netmask. +> The absolute minimum size for the subnet is **/28** (14 usable IP addresses). If you plan to create an Azure Data Explorer cluster for extreme ingestion workloads, use a **/24** netmask to stay on the safe side. -If you created a subnet that is too small, you can delete it and create a new one with a larger address range. Once you've recreated the subnet, you can create a new private endpoint for the cluster. +If you created a subnet that is too small, you can delete it and create a new one with a larger address range. After recreating the subnet, create a new private endpoint for the cluster. ## Connect to a private endpoint @@ -62,21 +62,21 @@ By default, Azure Data Explorer creates a [private DNS zone](/azure/dns/private- > [!IMPORTANT] > For optimal configuration, we recommend that you align your deployment with the recommendations in the [Private Endpoint and DNS configuration at Scale](/azure/cloud-adoption-framework/ready/azure-best-practices/private-link-and-dns-integration-at-scale) Cloud Adoption Framework article. Use the information in the article to automate Private DNS entry creation using Azure Policies, making it easier to manage your deployment as you scale. -:::image type="content" source="media/security-network-private-endpoint/pe-dns-config-inline.png" alt-text="Screenshot of the DNS configuration page, showing the DNS configuration of the private endpoint." lightbox="media/security-network-private-endpoint/pe-dns-config.png"::: +:::image type="content" source="media/security-network-private-endpoint/pe-dns-config-inline.png" alt-text="Screenshot of the DNS configuration page, showing the DNS configuration of the private endpoint." lightbox="media/security-network-private-endpoint/pe-dns-config-inline.png"::: -Azure Data Explorer creates multiple customer visible FQDNs as part of the private endpoint deployment. In addition to the *query* and *ingestion* FQDN it comes with several FQDNs for blob / table / queue endpoints (needed for ingestion scenarios) +Azure Data Explorer creates multiple customer visible FQDNs as part of the private endpoint deployment. In addition to the *query* and *ingestion* FQDN, it comes with several FQDNs for blob / table / queue endpoints (needed for ingestion scenarios). ## Disable public access To increase security, you also can disable public access to the cluster in the Azure portal. -:::image type="content" source="media/security-network-private-endpoint/pe-disable-public-access.png" alt-text="Screenshot of the networking page, showing the disable public access option."::: +:::image type="content" source="media/security-network-private-endpoint/pe-disable-public-access.png" alt-text="Screenshot of the networking page in Azure Data Explorer showing the disable public access option." lightbox="media/security-network-private-endpoint/pe-disable-public-access.png"::: ## Managed private endpoints -You can use a managed private endpoint to either enable the cluster to securely access your ingestion- or query-related services via their private endpoint. This allows the Azure Data Explorer cluster to access your resources via a private IP address. +You can use a managed private endpoint to either enable the cluster to securely access your ingestion- or query-related services via their private endpoint. The Azure Data Explorer cluster can then access your resources via a private IP address. -:::image type="content" source="media/security-network-private-endpoint/pe-mpe.png" alt-text="Diagram showing the schema of the managed private endpoint architecture."::: +:::image type="content" source="media/security-network-private-endpoint/pe-mpe.png" alt-text="Diagram of the managed private endpoint architecture for Azure Data Explorer." lightbox="media/security-network-private-endpoint/pe-mpe.png"::: > [!NOTE] > We recommend using Managed Identity connect to [Azure Storage](/azure/storage/common/storage-network-security?tabs=azure-portal#grant-access-to-trusted-azure-services) and [Azure Event Hubs](/azure/event-hubs/event-hubs-ip-filtering#trusted-microsoft-services) instead of managed private endpoints. To connect using managed identities, configure the Azure Storage or Event Hubs resources to recognize Azure Data Explorer as a trusted service. Then, use [Managed Identity](/azure/data-explorer/managed-identities-overview) to grant access by creating a network rule exception for trusted Azure services.``` @@ -98,7 +98,7 @@ Private endpoints aren't supported for virtual network injected Azure Data Explo ## Implications on cost -Private endpoints or managed private endpoints are resources that incur additional costs. The cost varies depending on the selected solution architecture. For more information, see [Azure Private Link pricing](https://azure.microsoft.com/pricing/details/private-link/). +Private endpoints or managed private endpoints are resources that incur extra costs. The cost varies depending on the selected solution architecture. For more information, see [Azure Private Link pricing](https://azure.microsoft.com/pricing/details/private-link/). ## Related content