diff --git a/data-explorer/.openpublishing.redirection.json b/data-explorer/.openpublishing.redirection.json index b2e7f8ae07..8fa2683914 100644 --- a/data-explorer/.openpublishing.redirection.json +++ b/data-explorer/.openpublishing.redirection.json @@ -25,6 +25,11 @@ "redirect_url": "/sql/azure-data-studio/notebooks/notebooks-kqlmagic?context=/azure/data-explorer/context/context", "redirect_document_id": false }, + { + "source_path": "delete-data.md", + "redirect_url": "/kusto/concepts/delete-data?view=azure-data-explorer&preserve-view=true", + "redirect_document_id": false + }, { "source_path": "connect-from-databricks.md", "redirect_url": "/azure/data-explorer/spark-connector", @@ -425,6 +430,11 @@ "redirect_url": "/kusto/set-timeout-limits?view=azure-data-explorer&preserve-view=true", "redirect_document_id": false }, + { + "source_path": "debug-inline-python.md", + "redirect_url": "/kusto/debug-inline-python", + "redirect_document_id": false + }, { "source_path": "excel-connector.md", "redirect_url": "/azure/data-explorer/excel", diff --git a/data-explorer/debug-inline-python.md b/data-explorer/debug-inline-python.md deleted file mode 100644 index f69644b92e..0000000000 --- a/data-explorer/debug-inline-python.md +++ /dev/null @@ -1,95 +0,0 @@ ---- -title: Debug Kusto Query Language inline Python using VS Code - Azure Data Explorer -description: Learn how to debug Kusto Query Language (KQL) inline Python using VS Code. -ms.reviewer: adieldar -ms.topic: how-to -ms.date: 11/08/2022 ---- - -# Debug Kusto Query Language inline Python using VS Code - -Azure Data Explorer supports running Python code embedded in Kusto Query Language using the [python() plugin](/kusto/query/python-plugin?view=azure-data-explorer&preserve-view=true). The plugin runtime is hosted in a sandbox, an isolated and secure Python environment. The python() plugin capability extends Kusto Query Language native functionalities with the huge archive of OSS Python packages. This extension enables you to run advanced algorithms, such as machine learning, artificial intelligence, statistical, and time series as part of the query. - -Kusto Query Language tools aren't convenient for developing and debugging Python algorithms. Therefore, develop the algorithm on your favorite Python-integrated development environment such as Jupyter, PyCharm, VS, or VS Code. When the algorithm is complete, copy and paste into KQL. To improve and streamline this workflow, Azure Data Explorer supports integration between Kusto Explorer or Web UI clients and VS Code for authoring and debugging KQL inline Python code. - -> [!NOTE] -> This workflow can only be used to debug relatively small input tables (up to few MB). Therefore, you may need to limit the input for debugging. If you need to process a large table, limit it for debugging using `| take`, `| sample`, or `where rand() < 0.x`. - -## Prerequisites - -* An Azure subscription. Create a [free Azure account](https://azure.microsoft.com/free/). -* An Azure Data Explorer cluster and database. [Create a cluster and database](create-cluster-and-database.md). -* Install Python [Anaconda Distribution](https://www.anaconda.com/download). In **Advanced Options**, select **Add Anaconda to my PATH environment variable**. -* Install [Visual Studio Code](https://code.visualstudio.com/Download). -* Install [Python extension for Visual Studio Code](https://marketplace.visualstudio.com/items?itemName=ms-python.python). - -## Run your query in your client application - -1. In your client application, prefix a query containing inline Python with `set query_python_debug;` -1. Run the query. - * Kusto Explorer: VS Code is automatically launched with the *debug_python.py* script. - * Kusto Web UI: - 1. Download and save *debug_python.py*, *df.txt*, and *kargs.txt*. In window, select **Allow**. **Save** files in selected directory. - - ![Web UI downloads the inline python files.](media/debug-inline-python/webui-inline-python.png) - - 1. Right-click *debug_python.py* and open with VS Code. - The *debug_python.py* script contains the inline Python code, from the KQL query, prefixed by the template code to initialize the input dataframe from *df.txt* and the dictionary of parameters from *kargs.txt*. - -1. In VS Code, launch the VS Code debugger: **Debug** > **Start Debugging (F5)**, select **Python** configuration. The debugger will launch and automatically breakpoint to debug the inline code. - -### How does inline Python debugging in VS Code work? - -1. The query is parsed and executed in the server until the required `| evaluate python()` clause is reached. -1. The Python sandbox is invoked but instead of running the code, it serializes the input table, the dictionary of parameters, and the code, and sends them back to the client. -1. These three objects are saved in three files: *df.txt*, *kargs.txt*, and *debug_python.py* in the selected directory (Web UI) or in the client %TEMP% directory (Kusto Explorer). -1. VS Code is launched, preloaded with the *debug_python.py* file that contains a prefix code to initialize df and kargs from their respective files, followed by the Python script embedded in the KQL query. - -## Query example - -1. Run the following KQL query in your client application: - - ```kusto - range x from 1 to 4 step 1 - | evaluate python(typeof(*, x4:int), - 'exp = kargs["exp"]\n' - 'result = df\n' - 'result["x4"] = df["x"].pow(exp)\n' - , bag_pack('exp', 4)) - ``` - - See the resulting table: - - | x | x4 | - |---------|---------| - | 1 | 1 | - | 2 | 16 | - | 3 | 81 | - | 4 | 256 | - -1. Run the same KQL query in your client application using `set query_python_debug;`: - - ```kusto - set query_python_debug; - range x from 1 to 4 step 1 - | evaluate python(typeof(*, x4:int), - 'exp = kargs["exp"]\n' - 'result = df\n' - 'result["x4"] = df["x"].pow(exp)\n' - , bag_pack('exp', 4)) - ``` - -1. VS Code is launched: - - ![launch VS Code.](media/debug-inline-python/launch-vs-code.png) - -1. VS Code debugs and prints 'result' dataframe in the debug console: - - ![VS Code debug.](media/debug-inline-python/debug-vs-code.png) - -> [!NOTE] -> There may be differences between the Python sandbox image and your local installation. [Check the sandbox image for specific packages by querying the plugin](/kusto/functions-library/get-packages-version-fl?view=azure-data-explorer&preserve-view=true). -> [!NOTE] -> In case there are errors when launching VS code debugging session try setting the default terminal profile to command prompt: -> * Press CTRL + SHIFT + P to open the Command Palette. -> * Search for "Terminal: Select Default Profile" and select "Command Prompt" diff --git a/data-explorer/delete-data.md b/data-explorer/delete-data.md deleted file mode 100644 index 836f94c46a..0000000000 --- a/data-explorer/delete-data.md +++ /dev/null @@ -1,79 +0,0 @@ ---- -title: Delete data from Azure Data Explorer -description: This article describes delete scenarios in Azure Data Explorer, including purge, dropping extents and retention based deletes. -ms.reviewer: avneraa -ms.topic: how-to -ms.date: 02/01/2022 ---- - -# Delete data from Azure Data Explorer - - - -Azure Data Explorer supports several ways to delete data from a table. Use the following information to help you choose which deletion method is best for your use case. - -| Use case | Considerations | Method | -|--|--|--| -| Delete all data from a table | | [Use the `.clear table data` command](#delete-all-data-in-a-table) | -| Routinely delete old data | Use if you need an automated deletion solution | [Use a retention policy](#delete-data-using-a-retention-policy) | -| Bulk delete specific data by extents | Only use if you are an expert user | [Use the `.drop extents` command](#delete-data-by-dropping-extents) | -| Delete records based on their content | - Storage artifacts that contain the deleted records aren't necessarily deleted
- Deleted records can't be recovered (regardless of any retention or recoverability settings)
- Use if you need a quick way to delete records | [Use soft delete](#soft-delete) | -| Delete records based on their content | - Storage artifacts that contain the deleted records are deleted
- Deleted records can't be recovered (regardless of any retention or recoverability settings)
- Requires significant system resources and time to complete | [Use purge](#purge) | - -The following sections describe the different deletion methods. - -## Delete all data in a table - -To delete all data in a table, use the [.clear table data](/kusto/management/clear-table-data-command?view=azure-data-explorer&preserve-view=true) command. This is the most efficient way to remove all data from a table. - -Syntax: - -```kusto -.clear table data -``` - -## Delete data using a retention policy - -Automatically delete data based on a [retention policy](/kusto/management/retention-policy?view=azure-data-explorer&preserve-view=true). You can set the retention policy at the database or table level. There is no guarantee as to when the deletion occurs, but it will not be deleted before the retention period. This is a very efficient and convenient way to remove old data. - -Consider a database or table that is set for 90 days of retention. If only 60 days of data are needed, delete the older data as follows: - -```kusto -.alter-merge database policy retention softdelete = 60d - -.alter-merge table policy retention softdelete = 60d -``` - -## Delete data by dropping extents - -[Extent (data shard)](/kusto/management/extents-overview?view=azure-data-explorer&preserve-view=true) is the internal structure where data is stored. Each extent can hold up to millions of records. Extents can be deleted individually or as a group using [drop extent(s) commands](/kusto/management/drop-extents?view=azure-data-explorer&preserve-view=true). - -### Examples - -You can delete all rows in a table or just a specific extent. - -- Delete all rows in a table: - - ```kusto - .drop extents from TestTable - ``` - -- Delete a specific extent: - - ```kusto - .drop extent e9fac0d2-b6d5-4ce3-bdb4-dea052d13b42 - ``` - -## Delete individual rows - -Both purge and soft delete can be used for deleting individual rows, but while soft delete doesn't necessarily delete the storage artifacts that contain records to delete, purge does delete all such storage artifacts. - -Both methods prevent deleted records from being recovered, regardless of any retention or recoverability settings. The deletion process is final and irreversible. - -### Soft delete - -With [soft delete](/kusto/concepts/data-soft-delete?view=azure-data-explorer&preserve-view=true), data is not necessarily deleted from storage artifacts. This method marks all matching records as deleted, so that they will be filtered out in queries, and doesn't require significant system resources. - -### Purge - -With [purge](/kusto/concepts/data-purge?view=azure-data-explorer&preserve-view=true), extents that have one or more records to be deleted, are replaced with new extents in which those records do not exist. This deletion process isn't immediate, requires significant system resources, and can take a whole day to complete. diff --git a/data-explorer/kusto/.openpublishing.redirection.json b/data-explorer/kusto/.openpublishing.redirection.json index 39dfbe6bb2..f39e62265e 100644 --- a/data-explorer/kusto/.openpublishing.redirection.json +++ b/data-explorer/kusto/.openpublishing.redirection.json @@ -26,8 +26,8 @@ "redirect_document_id": false }, { - "source_path": "concepts/delete-data.md", - "redirect_url": "/azure/data-explorer/delete-data", + "source_path": "data-explorer/delete-data.md", + "redirect_url": "/kusto/concepts/delete-data", "redirect_document_id": false }, { diff --git a/data-explorer/kusto/concepts/data-soft-delete.md b/data-explorer/kusto/concepts/data-soft-delete.md index 9f4adebb9c..28d82844aa 100644 --- a/data-explorer/kusto/concepts/data-soft-delete.md +++ b/data-explorer/kusto/concepts/data-soft-delete.md @@ -4,11 +4,11 @@ description: This article describes data soft delete. ms.reviewer: slneimer ms.topic: reference ms.date: 08/11/2024 -monikerRange: "azure-data-explorer" +monikerRange: "azure-data-explorer || microsoft-fabric" --- # Soft delete overview -> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] +> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] The ability to delete individual records is supported. Record deletion is commonly achieved using one of the following methods: diff --git a/data-explorer/kusto/concepts/delete-data.md b/data-explorer/kusto/concepts/delete-data.md new file mode 100644 index 0000000000..6832ab4cb3 --- /dev/null +++ b/data-explorer/kusto/concepts/delete-data.md @@ -0,0 +1,95 @@ +--- +title: Delete data +description: This article describes delete scenarios, including purge, dropping extents and retention based deletes. +ms.reviewer: avneraa +ms.topic: how-to +ms.date: 10/11/2024 +--- +# Delete data + +> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] + +Delete data from a table is supported in several ways. Use the following information to help you choose which deletion method is best for your use case. + +:::moniker range="azure-data-explorer" +| Use case | Considerations | Method | +|--|--|--| +| Delete all data from a table. | | [Use the `.clear table data` command](#delete-all-data-in-a-table) | +| Routinely delete old data. | Use if you need an automated deletion solution. | [Use a retention policy](#delete-data-using-a-retention-policy) | +| Bulk delete specific data by extents. | Only use if you're an expert user. | [Use the `.drop extents` command](#delete-data-by-dropping-extents) | +| Delete records based on their content. | - Storage artifacts that contain the deleted records aren't necessarily deleted.
- Deleted records can't be recovered (regardless of any retention or recoverability settings).
- Use if you need a quick way to delete records. | [Use soft delete](#soft-delete)| +| Delete records based on their content. | - Storage artifacts that contain the deleted records are deleted.
- Deleted records can't be recovered (regardless of any retention or recoverability settings).
- Requires significant system resources and time to complete. | [Use purge](#purge) | +:::moniker-end + +:::moniker range="microsoft-fabric" +| Use case | Considerations | Method | +|--|--|--| +| Delete all data from a table. | | [Use the `.clear table data` command](#delete-all-data-in-a-table) | +| Routinely delete old data. | Use if you need an automated deletion solution. | [Use a retention policy](#delete-data-using-a-retention-policy) | +| Bulk delete specific data by extents. | Only use if you're an expert user. | [Use the `.drop extents` command](#delete-data-by-dropping-extents) | +| Delete records based on their content. | - Storage artifacts that contain the deleted records aren't necessarily deleted.
- Deleted records can't be recovered (regardless of any retention or recoverability settings).
- Use if you need a quick way to delete records. | [Use soft delete](#delete-individual-rows) | +:::moniker-end + +The following sections describe the different deletion methods. + +## Delete all data in a table + +To delete all data in a table, use the [.clear table data](../management/clear-table-data-command.md) command. This command is the most efficient way to remove all data from a table. + +Syntax: + +```kusto +.clear table data +``` + +## Delete data using a retention policy + +Automatically delete data based on a [retention policy](../management/retention-policy.md). You can set the retention policy at the database or table level. There's no guarantee as to when the deletion occurs, but it will not be deleted before the retention period. This is an efficient and convenient way to remove old data. + +Consider a database or table that is set for 90 days of retention. If only 60 days of data are needed, delete the older data as follows: + +```kusto +.alter-merge database policy retention softdelete = 60d + +.alter-merge table policy retention softdelete = 60d +``` + +## Delete data by dropping extents + +[Extent (data shard)](../management/extents-overview.md) is the internal structure where data is stored. Each extent can hold up to millions of records. Extents can be deleted individually or as a group using [drop extent(s) commands](../management/drop-extents.md). + +### Examples + +You can delete all rows in a table or just a specific extent. + +- Delete all rows in a table: + + ```kusto + .drop extents from TestTable + ``` + +- Delete a specific extent: + + ```kusto + .drop extent e9fac0d2-b6d5-4ce3-bdb4-dea052d13b42 + ``` + +## Delete individual rows + +:::moniker range="azure-data-explorer" +Both purge and soft delete can be used for deleting individual rows. Soft delete doesn't necessarily delete the storage artifacts that contain records to delete, and purge does delete all such storage artifacts. + +Both methods prevent deleted records from being recovered, regardless of any retention or recoverability settings. The deletion process is final and irreversible. + +### Soft delete + +With [soft delete](data-soft-delete.md), data isn't necessarily deleted from storage artifacts. This method marks all matching records as deleted, so that they'll be filtered out in queries, and doesn't require significant system resources. + +### Purge + +With [purge](data-purge.md), extents that have one or more records to be deleted, are replaced with new extents in which those records don't exist. This deletion process isn't immediate, requires significant system resources, and can take a whole day to complete. +::: moniker-end + +:::moniker range="microsoft-fabric" +[Soft delete](data-soft-delete.md) can be used for deleting individual rows. Data isn't necessarily deleted from storage artifacts. Soft delete prevent deleted records from being recovered, regardless of any retention or recoverability settings. The deletion process is final and irreversible. This method marks all matching records as deleted, so that they'll be filtered out in queries, and doesn't require significant system resources. +::: moniker-end diff --git a/data-explorer/kusto/debug-inline-python.md b/data-explorer/kusto/debug-inline-python.md new file mode 100644 index 0000000000..a32fedcf19 --- /dev/null +++ b/data-explorer/kusto/debug-inline-python.md @@ -0,0 +1,125 @@ +--- +title: Debug Kusto Query Language inline Python using Visual Studio Code +description: Learn how to debug Kusto Query Language (KQL) inline Python using Visual Studio Code. +ms.reviewer: adieldar +ms.topic: how-to +ms.date: 11/14/2024 +--- + +# Debug Kusto Query Language inline Python using Visual Studio Code + +> [!INCLUDE [applies](includes/applies-to-version/applies.md)] [!INCLUDE [fabric](includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](includes/applies-to-version/azure-data-explorer.md)] + +You can embed Python code in Kusto Query Language queries using the [python() plugin](/kusto/query/python-plugin?view=azure-data-explorer&preserve-view=true). The plugin runtime is hosted in a sandbox, an isolated and secure Python environment. The python() plugin capability extends Kusto Query Language native functionalities with the huge archive of OSS Python packages. This extension enables you to run advanced algorithms, such as machine learning, artificial intelligence, statistical, and time series as part of the query. + +Kusto Query Language tools aren't convenient for developing and debugging Python algorithms. Therefore, develop the algorithm on your favorite Python-integrated development environment such as Jupyter, PyCharm, Visual Studio, or Visual Studio Code. When the algorithm is complete, copy and paste into KQL. To improve and streamline this workflow, Kusto Explorer or Web UI clients can integrate with Visual Studio Code for authoring and debugging KQL inline Python code. + +> [!NOTE] +> This workflow can only be used to debug relatively small input tables (up to few MB). Therefore, you may need to limit the input for debugging. If you need to process a large table, limit it for debugging using `| take`, `| sample`, or `where rand() < 0.x`. + +## Prerequisites + +::: moniker range="azure-data-explorer" + +* An Azure subscription. Create a [free Azure account](https://azure.microsoft.com/free/). +* An Azure Data Explorer cluster and database. [Create a cluster and database](/azure/data-explorer/create-cluster-and-database). +* Install Python [Anaconda Distribution](https://www.anaconda.com/download). In **Advanced Options**, select **Add Anaconda to my PATH environment variable**. +* Install [Visual Studio Code](https://code.visualstudio.com/Download). +* Install [Python extension for Visual Studio Code](https://marketplace.visualstudio.com/items?itemName=ms-python.python). +* Enable the Python plugin. For more information, see [Manage language extensions in your Azure Data Explorer cluster](/azure/data-explorer/language-extensions#enable-language-extensions-on-your-cluster). +::: moniker-end + +::: moniker range="microsoft-fabric" + +* A [workspace](/fabric/get-started/create-workspaces) with a Microsoft Fabric-enabled [capacity](/fabric/enterprise/licenses). +* A database. [Create a KQL database](/fabric/real-time-intelligence/create-database). +* Install Python [Anaconda Distribution](https://www.anaconda.com/download). In **Advanced Options**, select **Add Anaconda to my PATH environment variable**. +* Install [Visual Studio Code](https://code.visualstudio.com/Download). +* Install [Python extension for Visual Studio Code](https://marketplace.visualstudio.com/items?itemName=ms-python.python). +* [Enable the Python plugin](/fabric/real-time-intelligence/python-plugin). + +::: moniker-end + +## Enable Python debugging in Visual Studio Code + +::: moniker range="azure-data-explorer" + +1. In your client application, prefix a query containing inline Python with `set query_python_debug;` +1. Run the query. + * Kusto Explorer: Visual Studio Code is automatically launched with the *debug_python.py* script. + * Kusto Web UI: + 1. Download and save *debug_python.py*, *df.txt*, and *kargs.txt*. In window, select **Allow**. **Save** files in selected directory. + 1. Right-click *debug_python.py* and open with Visual Studio Code. + The *debug_python.py* script contains the inline Python code, from the KQL query, prefixed by the template code to initialize the input dataframe from *df.txt* and the dictionary of parameters from *kargs.txt*. + +1. In Visual Studio Code, launch the Visual Studio Code debugger: **Run** > **Start Debugging (F5)**, select **Python** configuration. The debugger launches and automatically sets a breakpoint to debug the inline code. +::: moniker-end + +::: moniker range="microsoft-fabric" + +1. In your client application, prefix a query containing inline Python with `set query_python_debug;` +1. Run the query. + * Kusto Explorer: Visual Studio Code is automatically launched with the *debug_python.py* script. + * KQL queryset: + 1. Download and save *debug_python.py*, *df.txt*, and *kargs.txt*. In window, select **Allow**. **Save** files in selected directory. + 1. Right-click *debug_python.py* and open with Visual Studio Code. + The *debug_python.py* script contains the inline Python code, from the KQL query, prefixed by the template code to initialize the input dataframe from *df.txt* and the dictionary of parameters from *kargs.txt*. + +1. In Visual Studio Code, launch the Visual Studio Code debugger: **Run** > **Start Debugging (F5)**, select **Python** configuration. The debugger launches and automatically sets a breakpoint to debug the inline code. +::: moniker-end + +### How does inline Python debugging in Visual Studio Code work? + +1. The query is parsed and executed in the server until the required `| evaluate python()` clause is reached. +1. The Python sandbox is invoked but instead of running the code, it serializes the input table, the dictionary of parameters, and the code, and sends them back to the client. +1. These three objects are saved in three files: *df.txt*, *kargs.txt*, and *debug_python.py* in the selected directory (Web UI) or in the client %TEMP% directory (Kusto Explorer). +1. Visual Studio Code is launched, preloaded with the *debug_python.py* file that contains a prefix code to initialize df and kargs from their respective files, followed by the Python script embedded in the KQL query. + +## Query example + +1. Run the following KQL query in your client application: + + ```kusto + range x from 1 to 4 step 1 + | evaluate python(typeof(*, x4:int), + 'exp = kargs["exp"]\n' + 'result = df\n' + 'result["x4"] = df["x"].pow(exp)\n' + , bag_pack('exp', 4)) + ``` + + See the resulting table: + + | x | x4 | + |---------|---------| + | 1 | 1 | + | 2 | 16 | + | 3 | 81 | + | 4 | 256 | + +1. Run the same KQL query in your client application using `set query_python_debug;`: + + ```kusto + set query_python_debug; + range x from 1 to 4 step 1 + | evaluate python(typeof(*, x4:int), + 'exp = kargs["exp"]\n' + 'result = df\n' + 'result["x4"] = df["x"].pow(exp)\n' + , bag_pack('exp', 4)) + ``` + +1. Visual Studio Code is launched: + + ![launch Visual Studio Code.](media/debug-inline-python/launch-vs-code.png) + +1. Visual Studio Code debugs and prints 'result' dataframe in the debug console: + + ![VS Code debug.](media/debug-inline-python/debug-vs-code.png) + +> [!NOTE] +> There may be differences between the Python sandbox image and your local installation. [Check the sandbox image for specific packages by querying the plugin](/kusto/functions-library/get-packages-version-fl?view=azure-data-explorer&preserve-view=true). +> [!NOTE] +> In case there are errors when launching Visual Studio Code debugging session try setting the default terminal profile to command prompt: +> * Press CTRL + SHIFT + P to open the Command Palette. +> * Search for "Terminal: Select Default Profile" and select "Command Prompt" diff --git a/data-explorer/kusto/management/drop-extents.md b/data-explorer/kusto/management/drop-extents.md index f557447472..1ae52fc0e3 100644 --- a/data-explorer/kusto/management/drop-extents.md +++ b/data-explorer/kusto/management/drop-extents.md @@ -4,11 +4,11 @@ description: Learn how to use the `.drop extents` command to drop extents from a ms.reviewer: orspodek ms.topic: reference ms.date: 08/11/2024 -monikerRange: "azure-data-explorer" +monikerRange: "azure-data-explorer || microsoft-fabric" --- # .drop extents command -> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] +> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] Drops extents from a specified database or table. diff --git a/data-explorer/kusto/management/extent-tags.md b/data-explorer/kusto/management/extent-tags.md index 0c93d7b8ae..29675bfae6 100644 --- a/data-explorer/kusto/management/extent-tags.md +++ b/data-explorer/kusto/management/extent-tags.md @@ -4,11 +4,11 @@ description: Learn how to create and use extent tags. ms.reviewer: orspodek ms.topic: reference ms.date: 08/11/2024 -monikerRange: "azure-data-explorer" +monikerRange: "azure-data-explorer || microsoft-fabric" --- # Extent tags -> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] +> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] An *extent tag* is a string that describes properties common to all data in an [extent](extents-overview.md). For example, during data ingestion, you can append an extent tag to signify the source of the ingested data. Then, you can use this tag for analysis. diff --git a/data-explorer/kusto/management/extents-overview.md b/data-explorer/kusto/management/extents-overview.md index ed97f7a6fb..fc8c1f2a88 100644 --- a/data-explorer/kusto/management/extents-overview.md +++ b/data-explorer/kusto/management/extents-overview.md @@ -3,12 +3,12 @@ title: Extents (data shards) description: This article describes Extents (data shards). ms.reviewer: orspodek ms.topic: reference -ms.date: 08/11/2024 -monikerRange: "azure-data-explorer" +ms.date: 11/11/2024 +monikerRange: "microsoft-fabric || azure-data-explorer" --- # Extents (data shards) -> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] +> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] Tables are partitioned into *extents*, or *data shards*. Each extent is a horizontal segment of the table that contains data and metadata such as its creation time and optional tags. The union of all these extents contains the entire dataset of the table. Extents are evenly distributed across nodes in the cluster, and they're cached in both local SSD and memory for optimized performance. diff --git a/data-explorer/kusto/management/sharding-policy.md b/data-explorer/kusto/management/sharding-policy.md index b5418d548d..00c11c5dbc 100644 --- a/data-explorer/kusto/management/sharding-policy.md +++ b/data-explorer/kusto/management/sharding-policy.md @@ -3,12 +3,12 @@ title: Data sharding policy description: Learn how to use the data sharding policy to define if and how extents in the database are created. ms.reviewer: orspodek ms.topic: reference -ms.date: 08/11/2024 -monikerRange: "azure-data-explorer" +ms.date: 11/11/2024 +monikerRange: "microsoft-fabric || azure-data-explorer" --- # Data sharding policy -> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] +> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] The sharding policy defines if and how [extents (data shards)](../management/extents-overview.md) in your cluster are created. You can only query data in an extent once it's created. diff --git a/data-explorer/kusto/management/show-extents.md b/data-explorer/kusto/management/show-extents.md index 4feb1ddf03..39c5f569c7 100644 --- a/data-explorer/kusto/management/show-extents.md +++ b/data-explorer/kusto/management/show-extents.md @@ -4,12 +4,12 @@ description: Learn how to use the `.show extents` command to show the extents fo ms.reviewer: orspodek ms.topic: reference ms.date: 08/11/2024 -monikerRange: "azure-data-explorer" +monikerRange: "azure-data-explorer || microsoft-fabric" --- # .show extents -> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] +> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] > [!NOTE] > Data shards are called **extents**, and all commands use "extent" or "extents" as a synonym. @@ -101,29 +101,28 @@ If `hot` is specified - shows only extents that expected to be in the hot cache. Shows information about extents (data shards) that are present in the cluster. - If `hot` is specified - shows only extents that are expected to be in the hot cache. ## Returns |Output parameter |Type |Description | |---|---|---| -|ExtentId | `guid` |ID of the extent -|DatabaseName | `string` |Database that the extent belongs to -|TableName | `string` |Table that the extents belong to -|MaxCreatedOn | `datetime` |Date-time when the extent was created. For a merged extent, the maximum of creation times among source extents -|OriginalSize |Double |Original size in bytes of the extent data -|ExtentSize |Double |Size of the extent in memory (compressed + index) -|CompressedSize |Double |Compressed size of the extent data in memory -|IndexSize |Double |Index size of the extent data -|Blocks | `long` |Number of data blocks in the extent -|Segments | `long` |Number of data segments in the extent -|ExtentContainerId | `string` | ID of the extent container the extent is in -|RowCount | `long` |Number of rows in the extent -|MinCreatedOn | `datetime` |Date-time when the extent was created. For a merged extent, the minimum of creation times among the source extents -|Tags| `string` |Tags, if any, defined for the extent -|Kind| `string` |The kind of the storage engine that created the extent -|DeletedRowCount| `long` |Number of deleted rows in the extent +|ExtentId | `guid` |ID of the extent | +|DatabaseName | `string` |Database that the extent belongs to. | +|TableName | `string` |Table that the extents belong to. | +|MaxCreatedOn | `datetime` |Date-time when the extent was created. For a merged extent, the maximum of creation times among source extents. | +|OriginalSize |Double |Original size in bytes of the extent data. | +|ExtentSize |Double |Size of the extent in memory (compressed + index). | +|CompressedSize |Double |Compressed size of the extent data in memory. | +|IndexSize |Double |Index size of the extent data. | +|Blocks | `long` |Number of data blocks in the extent. | +|Segments | `long` |Number of data segments in the extent. | +|ExtentContainerId | `string` | ID of the extent container the extent is in. | +|RowCount | `long` |Number of rows in the extent. | +|MinCreatedOn | `datetime` |Date-time when the extent was created. For a merged extent, the minimum of creation times among the source extents. | +|Tags| `string` |Tags, if any, defined for the extent. | +|Kind| `string` |The kind of the storage engine that created the extent. | +|DeletedRowCount| `long` |Number of deleted rows in the extent. | ## Examples diff --git a/data-explorer/kusto/management/show-table-sharding-policy-command.md b/data-explorer/kusto/management/show-table-sharding-policy-command.md index bb92e9bac1..1552be8ee7 100644 --- a/data-explorer/kusto/management/show-table-sharding-policy-command.md +++ b/data-explorer/kusto/management/show-table-sharding-policy-command.md @@ -3,12 +3,12 @@ title: .show table policy sharding command description: Learn how to use the `.show table policy sharding` command to show the table's sharding policy. ms.reviewer: yonil ms.topic: reference -ms.date: 08/11/2024 -monikerRange: "azure-data-explorer" +ms.date: 11/11/2024 +monikerRange: "azure-data-explorer || microsoft-fabric" --- # .show table policy sharding command -> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] +> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] Show the table sharding policy. Use the [sharding policy](../management/sharding-policy.md) to manage data sharding for databases and tables. diff --git a/data-explorer/kusto/management/toc.yml b/data-explorer/kusto/management/toc.yml index 4e1d849a1f..32bba73e97 100644 --- a/data-explorer/kusto/management/toc.yml +++ b/data-explorer/kusto/management/toc.yml @@ -925,6 +925,8 @@ items: items: - name: Clear cached external artifacts href: clear-external-artifacts-cache-command.md + - name: Delete data + href: /kusto/concepts/delete-data?view=azure-data-explorer&preserve-view=true - name: Data purge href: ../concepts/data-purge.md - name: Data soft delete diff --git a/data-explorer/media/debug-inline-python/debug-vs-code.png b/data-explorer/kusto/media/debug-inline-python/debug-vs-code.png similarity index 100% rename from data-explorer/media/debug-inline-python/debug-vs-code.png rename to data-explorer/kusto/media/debug-inline-python/debug-vs-code.png diff --git a/data-explorer/media/debug-inline-python/launch-vs-code.png b/data-explorer/kusto/media/debug-inline-python/launch-vs-code.png similarity index 100% rename from data-explorer/media/debug-inline-python/launch-vs-code.png rename to data-explorer/kusto/media/debug-inline-python/launch-vs-code.png diff --git a/data-explorer/media/debug-inline-python/webui-inline-python.png b/data-explorer/kusto/media/debug-inline-python/webui-inline-python.png similarity index 100% rename from data-explorer/media/debug-inline-python/webui-inline-python.png rename to data-explorer/kusto/media/debug-inline-python/webui-inline-python.png diff --git a/data-explorer/kusto/query/parse-json-function.md b/data-explorer/kusto/query/parse-json-function.md index e4f4a13c19..8eea15f892 100644 --- a/data-explorer/kusto/query/parse-json-function.md +++ b/data-explorer/kusto/query/parse-json-function.md @@ -3,7 +3,8 @@ title: parse_json() function description: Learn how to use the parse_json() function to return an object of type `dynamic`. ms.reviewer: alexans ms.topic: reference -ms.date: 08/11/2024 +ms.date: 11/19/2024 +monikerRange: "microsoft-fabric || azure-data-explorer" --- # parse_json() @@ -37,39 +38,51 @@ An object of type `dynamic` that is determined by the value of *json*: ## Example -In the following example, when `context_custom_metrics` is a `string` -that looks like this: +In the following example, when `context_custom_metrics` is a `string` that looks like this: ```json {"duration":{"value":118.0,"count":5.0,"min":100.0,"max":150.0,"stdDev":0.0,"sampledValue":118.0,"sum":118.0}} ``` -then the following query retrieves the value of the `duration` slot -in the object, and from that it retrieves two slots, `duration.value` and - `duration.min` (`118.0` and `110.0`, respectively). +then the following query retrieves the value of the `duration` slot in the object, and from that it retrieves two slots, `duration.value` and `duration.min` (`118.0` and `110.0`, respectively). + +:::moniker range="azure-data-explorer" +> [!div class="nextstepaction"] +> Run the query +::: moniker-end ```kusto -T -| extend d=parse_json(context_custom_metrics) -| extend duration_value=d.duration.value, duration_min=d["duration"]["min"] +datatable(context_custom_metrics:string) +[ + '{"duration":{"value":118.0,"count":5.0,"min":100.0,"max":150.0,"stdDev":0.0,"sampledValue":118.0,"sum":118.0}}' +] +| extend d = parse_json(context_custom_metrics) +| extend duration_value = d.duration.value, duration_min = d.duration.min ``` **Notes** -It's common to have a JSON string describing a property bag in which -one of the "slots" is another JSON string. +It's common to have a JSON string describing a property bag in which one of the "slots" is another JSON string. For example: +:::moniker range="azure-data-explorer" +> [!div class="nextstepaction"] +> Run the query +::: moniker-end + ```kusto let d='{"a":123, "b":"{\\"c\\":456}"}'; print d ``` -In such cases, it isn't only necessary to invoke `parse_json` twice, but also -to make sure that in the second call, `tostring` is used. Otherwise, the -second call to `parse_json` will just pass on the input to the output as-is, -because its declared type is `dynamic`. +In such cases, it isn't only necessary to invoke `parse_json` twice, but also to make sure that in the second call, `tostring` is used. Otherwise, the second call to `parse_json` will just pass on the input to the output as-is, because its declared type is `dynamic`. + +:::moniker range="azure-data-explorer" +> [!div class="nextstepaction"] +> Run the query +::: moniker-end + ```kusto let d='{"a":123, "b":"{\\"c\\":456}"}'; @@ -78,4 +91,4 @@ print d_b_c=parse_json(tostring(parse_json(d).b)).c ## Related content -* [parse operator](parse-operator.md) \ No newline at end of file +* [parse operator](parse-operator.md) diff --git a/data-explorer/kusto/query/toc.yml b/data-explorer/kusto/query/toc.yml index 6c593fbccb..300d692e4a 100644 --- a/data-explorer/kusto/query/toc.yml +++ b/data-explorer/kusto/query/toc.yml @@ -7,6 +7,8 @@ items: href: t-sql.md - name: Set timeout limits href: ../set-timeout-limits.md + - name: Debug KQL inline Python + href: ../debug-inline-python.md - name: Best practices for KQL queries items: - name: Best practices for KQL queries diff --git a/data-explorer/toc.yml b/data-explorer/toc.yml index 9f3d77ebb2..4d39c3fb04 100644 --- a/data-explorer/toc.yml +++ b/data-explorer/toc.yml @@ -340,8 +340,6 @@ items: href: power-apps-connector.md - name: Query data using Python href: python-query-data.md - - name: Debug KQL inline Python - href: debug-inline-python.md - name: Query data using T-SQL displayName: explain, sql, tsql href: t-sql.md @@ -498,10 +496,10 @@ items: href: table-retention-policy-wizard.md - name: Data items: + - name: Delete data + href: /kusto/concepts/delete-data?view=azure-data-explorer&preserve-view=true - name: Purge data - Azure portal href: data-purge-portal.md - - name: Delete data - href: delete-data.md - name: Deal with duplicate data href: dealing-with-duplicates.md - name: Monitor