Repo sync for protected branch

data-explorer/kusto/management/alter-query-acceleration-policy-command.md

@@ -0,0 +1,70 @@
+---
+title: .alter query acceleration policy command (preview)
+description: Learn how to use the ".alter query acceleration policy command" to accelerate queries over external delta tables.
+ms.reviewer: sharmaanshul
+ms.topic: reference
+ms.date: 11/19/2024
+---
+# .alter query acceleration policy command (preview)
+
+> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)]
+
+Alters the [query acceleration policy](query-acceleration-policy.md) of a specific external delta table.
+
+For limitations, see [Limitations](query-acceleration-policy.md#limitations).
+
+## Permissions
+
+You must have at least [Database Admin](../access-control/role-based-access-control.md) permissions to run this command.
+
+## Syntax
+
+`.alter` `external` `table` *ExternalTableName* `policy` `query_acceleration` '*JSON-serialized policy*'
+
+## Parameters
+
+| Name                     | Type     | Required           | Description                                 |
+| ------------------------ | -------- | ------------------ | ------------------------------------------- |
+| *ExternalTableName*      | `string` | :heavy_check_mark: | The name of the external delta table.       |
+| *JSON-serialized policy* | `string` | :heavy_check_mark: | String literal holding a [JSON property bag](#json-property-bag). |
+
+### JSON property bag
+
+| Property  | Type       | Required           | Description                                                                  |
+| --------- | ---------- | ------------------ | ---------------------------------------------------------------------------- |
+| IsEnabled | `Boolean`  | :heavy_check_mark: | Indicates whether the policy is enabled.                                     |
+| Hot       | `Timespan` | :heavy_check_mark: | The hot period defined in the query acceleration policy. Minimum value = 1 d. |
+
+> [!NOTE]
+> Query acceleration is applied to data within a specific time period, defined as `timespan`, starting from the `modificationTime` as stated for each file in the [delta log](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#add-file-and-remove-file). 
+
+### Example
+
+```json
+{"IsEnabled": true, "Hot": "1.00:00:00"}
+```
+
+## Returns
+
+The command returns a table with one record that includes the modified policy object.
+
+| Column        | Type     | Description                                                                    |
+| ------------- | -------- | ------------------------------------------------------------------------------ |
+| PolicyName    | `string` | The name of the policy - `QueryAcceleration`                                   |
+| EntityName    | `string` | The fully qualified name of the entity: `[DatabaseName].[ExternalTableName]`   |
+| Policy        | `string` | A JSON-serialization of the query acceleration policy that is set on the external delta table |
+| ChildEntities | `string` | The child entities this policy affects - `null`                                |
+| EntityType    | `string` | The type of the entity the policy applies to - `ExternalTable`                 |
+
+## Example
+
+```Kusto
+.alter external table MyExternalTable policy query_acceleration '{"IsEnabled": true, "Hot": "1.00:00:00"}'
+```
+
+## Related content
+
+* [Query acceleration policy](query-acceleration-policy.md)
+* [.delete query acceleration policy command](delete-query-acceleration-policy-command.md)
+* [.show query acceleration policy command](show-query-acceleration-policy-command.md)
+* [.show external table operations query_acceleration statistics](show-external-table-operations-query-acceleration-statistics.md)

data-explorer/kusto/management/delete-query-acceleration-policy-command.md

@@ -0,0 +1,39 @@
+---
+title: ".delete query acceleration policy command (preview)"
+description: Learn how to use the .delete query acceleration policy command to accelerate queries over external delta tables.
+ms.reviewer: sharmaanshul
+ms.topic: reference
+ms.date: 11/19/2024
+---
+# .delete query acceleration policy command (preview)
+
+> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)]
+
+Deletes the [query acceleration policy](query-acceleration-policy.md) of a specific external delta table.
+
+## Permissions
+
+You must have at least [Database Admin](../access-control/role-based-access-control.md) permissions to run this command.
+
+## Syntax
+
+`.delete` `external` `table` *ExternalTableName* `policy` `query_acceleration`
+
+## Parameters
+
+| Name                | Type     | Required           | Description                     |
+| ------------------- | -------- | ------------------ | ------------------------------- |
+| *ExternalTableName* | `string` | :heavy_check_mark: | The name of the external delta table. |
+
+## Example
+
+```Kusto
+.delete external table MyExternalTable policy query_acceleration
+```
+
+## Related content
+
+* [Query acceleration policy](query-acceleration-policy.md)
+* [.alter query acceleration policy command](alter-query-acceleration-policy-command.md)
+* [.show query acceleration policy command](show-query-acceleration-policy-command.md)
+* [.show external table operations query_acceleration statistics](show-external-table-operations-query-acceleration-statistics.md)

data-explorer/kusto/management/external-tables-delta-lake.md

@@ -14,6 +14,8 @@ The commands in this article can be used to create or alter a delta [external ta
 > [!NOTE]
 > If the table exists, the `.create` command will fail with an error. Use `.create-or-alter` or `.alter` to modify existing tables.
 
+To accelerate queries over external delta tables, see [Query acceleration policy](query-acceleration-policy.md).
+
 ## Permissions
 
 To `.create` requires at least [Database User](../access-control/role-based-access-control.md) permissions, and to `.alter` requires at least [Table Admin](../access-control/role-based-access-control.md) permissions.

data-explorer/kusto/management/query-acceleration-policy.md

@@ -0,0 +1,47 @@
+---
+title: Query acceleration policy (preview)
+description: Learn how to use the query acceleration policy to accelerate queries over external delta tables.
+ms.reviewer: sharmaanshul
+ms.topic: reference
+ms.date: 11/19/2024
+---
+# Query acceleration policy (preview)
+
+> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)]
+
+An [external table](../query/schema-entities/external-tables.md) is a schema entity that references data stored external to a Kusto database. Queries run over external tables can be less performant than on data that is ingested due to various factors such as network calls to fetch data from storage, the absence of indexes, and more. Query acceleration allows specifying a policy on top of external delta tables. This policy defines a number of days to accelerate data for high-performance queries.
+
+::: moniker range="azure-data-explorer"
+Query acceleration is supported in Azure Data Explorer over Azure Data Lake Store Gen2 or Azure blob storage [external tables](external-tables-azure-storage.md).
+::: moniker-end
+
+::: moniker range="microsoft-fabric"
+Query acceleration is supported in Eventhouse over OneLake, Azure Data Lake Store Gen2, or Azure blob storage [external tables](/fabric/real-time-intelligence/onelake-shortcuts).
+
+To enable query acceleration in the Fabric UI, see [Query acceleration over OneLake shortcuts](https://go.microsoft.com/fwlink/?linkid=2296674).
+::: moniker-end
+
+## Limitations
+
+* The number of columns in the external table can't exceed 900.
+* Query performance over accelerated external delta tables which have partitions may not be optimal during preview.
+* The feature assumes delta tables with static advanced features, for example column mapping doesn't change, partitions don't change, and so on. To change advanced features, first disable the policy, and once the change is made, re-enable the policy.
+* Schema changes on the delta table must also be followed with the respective `. alter` external delta table schema, which might result in acceleration starting from scratch if there was breaking schema change.
+* Index-based pruning isn't supported for partitions.
+* Parquet files with a compressed size higher than 6 GB won't be cached.
+::: moniker range="azure-data-explorer"
+* Query acceleration isn't supported for external tables with impersonation authentication.
+::: moniker-end
+
+## Known issues
+
+* Data in the external delta table that is optimized with the [OPTIMIZE](/azure/databricks/sql/language-manual/delta-optimize) function will need to be reaccelearted.
+* If you run frequent MERGE/UPDATE/DELETE operations in delta, the underlying parquet files may be rewritten with changes and Kusto will skip accelerating such files, causing retrieval during query time.
+* The system assumes that all artifacts under the delta table directory have the same access level to the selected users. Different files having different access permissions under the delta table directory might result with unexpected behavior.
+
+## Commands for query acceleration
+
+* [.alter query acceleration policy command](alter-query-acceleration-policy-command.md)
+* [.delete query acceleration policy command](delete-query-acceleration-policy-command.md)
+* [.show query acceleration policy command](show-query-acceleration-policy-command.md)
+* [.show external table operations query_acceleration statistics](show-external-table-operations-query-acceleration-statistics.md)

data-explorer/kusto/management/show-external-table-operations-query-acceleration-statistics.md

@@ -0,0 +1,63 @@
+---
+title: .show external table operations query_acceleration statistics command (preview)
+description: Learn how to use the ".show external table operations query_acceleration statistics command" to accelerate queries over external delta tables.
+ms.reviewer: sharmaanshul
+ms.topic: reference
+ms.date: 11/19/2024
+---
+# .show external table operations query_acceleration statistics command (preview)
+
+> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)]
+
+Shows statistics for [query acceleration](query-acceleration-policy.md) for a specific external delta table, or all external delta tables in the database that have a query acceleration policy set.
+
+## Permissions
+
+You must have at least [Database monitor or Database viewer](../access-control/role-based-access-control.md) permissions to run this command.
+
+## Syntax
+
+`.show` `external` `table` *ExternalTableName* `operations` `query_acceleration` `statistics`
+
+`.show` `external` `tables` `operations` `query_acceleration` `statistics`
+
+## Parameters
+
+| Name                | Type     | Required           | Description                     |
+| ------------------- | -------- | ------------------ | ------------------------------- |
+| *ExternalTableName* | `string` |  | The name of the external table. |
+
+## Returns
+
+The command returns a table with a record per external table that has a non-null policy, with the following columns:
+
+| Column                          | Type       | Description                                                                                          |
+| ------------------------------- | ---------- | ---------------------------------------------------------------------------------------------------- |
+| ExternalTableName               | `string`   | The name of the external table                                                                       |
+| IsEnabled                       | `bool`     | Indicates whether the external has a query acceleration policy enabled                               |
+| Hot                             | `timespan` | The hot period defined in the query acceleration policy                                              |
+| HotSize                         | `long`     | The total size on disk (in bytes) of artifacts accelerated due to the query acceleration policy           |
+| LastUpdatedDateTime             | `datetime` | Indicates the last datetime when the internal representation of the table was successfully refreshed |
+| AccelerationPendingArtifactSize | `long`     | The total size of artifacts that are pending acceleration                                            |
+|AccelerationCompletePercentage | `double`  | The percentage of artifacts that are accelerated, out of the total artifacts that are able to be accelerated           |
+|NotHealthyReason               | `string`  | Describes the reason for query acceleration not being healthy. Empty if healthy                      |
+
+## Example
+
+```Kusto
+.show external table MyExternalTable operations query_acceleration statistics
+```
+
+**Output**
+
+| ExternalTableName | IsEnabled | Hot | HotSize | LastUpdatedDateTime | AccelerationPendingArtifactSize | AccelerationCompletePercentage | NotHealthyReason |
+| --- | --- | --- | --- | --- | --- | --- |
+| MyExternalTable | True | 1.00:00:00 | 56877928187 | 2024-08-13 19:54:47.5868860 | 0 | 100 | |
+| MyExternalTable2 | True | 1.00:00:00 | 60467660293 | 2024-08-13 19:54:47.5868860 | 0 | 100 | |
+
+## Related content
+
+* [Query acceleration policy](query-acceleration-policy.md)
+* [.alter query acceleration policy command](alter-query-acceleration-policy-command.md)
+* [.delete query acceleration policy command](delete-query-acceleration-policy-command.md)
+* [.show query acceleration policy command](show-query-acceleration-policy-command.md)

data-explorer/kusto/management/show-query-acceleration-policy-command.md

@@ -0,0 +1,59 @@
+---
+title: .show query acceleration policy command (preview)
+description: Learn how to use the ".show query acceleration policy command" to accelerate queries over external delta tables.
+ms.reviewer: sharmaanshul
+ms.topic: reference
+ms.date: 11/19/2024
+---
+# .show query acceleration policy command (preview)
+
+> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)]
+
+Shows the [query acceleration policy](query-acceleration-policy.md) of a specific external delta table, or for all external delta tables in the database that have a query acceleration policy set.
+
+## Permissions
+
+You must have at least [Database monitor or Database viewer](../access-control/role-based-access-control.md) permissions to run this command.
+
+## Syntax
+
+`.show` `external` `table` *ExternalTableName* `policy` `query_acceleration`
+
+`.show` `external` `table` `*` `policy` `query_acceleration`
+
+## Parameters
+
+| Name                | Type     | Required           | Description                     |
+| ------------------- | -------- | ------------------ | ------------------------------- |
+| *ExternalTableName* | `string` |  | The name of the external delta table. |
+
+## Returns
+
+The command returns a table with a record per external table with the following columns:
+
+| Column        | Type     | Description                                                                    |
+| ------------- | -------- | ------------------------------------------------------------------------------ |
+| PolicyName    | `string` | The name of the policy - `QueryAcceleration`                                   |
+| EntityName    | `string` | The fully qualified name of the entity: `[DatabaseName].[ExternalTableName]`   |
+| Policy        | `string` | A JSON-serialization of the query acceleration policy that is set on the table |
+| ChildEntities | `string` | The child entities this policy affects - `null`                                |
+| EntityType    | `string` | The type of the entity the policy applies to - `ExternalTable`                 |
+
+## Example
+
+```Kusto
+.show external table MyExternalTable policy query_acceleration
+```
+
+**Output**
+
+| PolicyName              | EntityName                     | Policy                                                                             | ChildEntities | EntityType    |
+| ----------------------- | ------------------------------ | ---------------------------------------------------------------------------------- | ------------- | ------------- |
+| QueryAccelerationPolicy | [MyDatabase].[MyExternalTable] | {<br>  "IsEnabled": true,<br>  "Hot": "1.00:00:00"}                || ExternalTable |
+
+## Related content
+
+* [Query acceleration policy](query-acceleration-policy.md)
+* [.alter query acceleration policy command](alter-query-acceleration-policy-command.md)
+* [.delete query acceleration policy command](delete-query-acceleration-policy-command.md)
+* [.show external table operations query_acceleration statistics](show-external-table-operations-query-acceleration-statistics.md)

data-explorer/kusto/management/toc.yml

@@ -546,6 +546,23 @@ items:
             displayName: .show table partitioning policy
           - name: .show database extents partitioning statistics command
             href: show-database-extents-partitioning-statistics.md
+      - name: Query acceleration
+        items:
+          - name: Query acceleration policy
+            href: query-acceleration-policy.md
+            displayName: external table, delta table
+          - name: .alter query acceleration policy command 
+            href: alter-query-acceleration-policy-command.md
+            displayName: external table, delta table
+          - name: .delete query acceleration policy command
+            href: delete-query-acceleration-policy-command.md
+            displayName: external table, delta table
+          - name: .show external table operations query acceleration statistcs command
+            href: show-external-table-operations-query-acceleration-statistics.md
+            displayName: external table, delta table 
+          - name: .show query acceleration policy command
+            href: show-query-acceleration-policy-command.md
+            displayName: external table, delta table
       - name: Query weak consistency
         items:
           - name: Query weak consistency policy

data-explorer/kusto/query/external-table-function.md

@@ -16,6 +16,8 @@ References an [external table](schema-entities/external-tables.md) by name.
 > The `external_table` function has similar restrictions as the [table](table-function.md) function.
 > Standard [query limits](../concepts/query-limits.md) apply to external table queries as well.
 
+To accelerate queries over external delta tables, see [Query acceleration policy](../management/query-acceleration-policy.md).
+
 ## Syntax
 
 `external_table(` *TableName* [`,` *MappingName* ] `)`

data-explorer/kusto/query/schema-entities/external-tables.md

@@ -42,6 +42,8 @@ Use the following commands to manage external tables:
 For more information about how to query external tables, and ingested and uningested data, see [Query data in Azure Data Lake using Azure Data Explorer](/azure/data-explorer/data-lake-query-data).
 ::: moniker-end
 
+To accelerate queries over external delta tables, see [Query acceleration policy](../../management/query-acceleration-policy.md).
+
 > [!NOTE]
 >
 > * The maximum limit of external tables per database is 1,000.