Skip to content

Commit 73bb920

Browse files
learn-build-service-prod[bot]learn-build-service-prod[bot]
authored
Merge pull request #2444 from MicrosoftDocs/main638676211927135269sync_temp
For protected branch, push strategy should use PR and merge to target branch method to work around git push error
2 parents b0cd11e + 518cae3 commit 73bb920

9 files changed

+301
-0
lines changed

‎data-explorer/kusto/management/alter-query-acceleration-policy-command.md‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
---
2+
title: .alter query acceleration policy command (preview)
3+
description: Learn how to use the ".alter query acceleration policy command" to accelerate queries over external delta tables.
4+
ms.reviewer: sharmaanshul
5+
ms.topic: reference
6+
ms.date: 11/19/2024
7+
---
8+
# .alter query acceleration policy command (preview)
9+
10+
> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)]
11+
12+
Alters the [query acceleration policy](query-acceleration-policy.md) of a specific external delta table.
13+
14+
For limitations, see [Limitations](query-acceleration-policy.md#limitations).
15+
16+
## Permissions
17+
18+
You must have at least [Database Admin](../access-control/role-based-access-control.md) permissions to run this command.
19+
20+
## Syntax
21+
22+
`.alter` `external` `table` *ExternalTableName* `policy` `query_acceleration` '*JSON-serialized policy*'
23+
24+
## Parameters
25+
26+
| Name | Type | Required | Description |
27+
| ------------------------ | -------- | ------------------ | ------------------------------------------- |
28+
| *ExternalTableName* | `string` | :heavy_check_mark: | The name of the external delta table. |
29+
| *JSON-serialized policy* | `string` | :heavy_check_mark: | String literal holding a [JSON property bag](#json-property-bag). |
30+
31+
### JSON property bag
32+
33+
| Property | Type | Required | Description |
34+
| --------- | ---------- | ------------------ | ---------------------------------------------------------------------------- |
35+
| IsEnabled | `Boolean` | :heavy_check_mark: | Indicates whether the policy is enabled. |
36+
| Hot | `Timespan` | :heavy_check_mark: | The hot period defined in the query acceleration policy. Minimum value = 1 d. |
37+
38+
> [!NOTE]
39+
> Query acceleration is applied to data within a specific time period, defined as `timespan`, starting from the `modificationTime` as stated for each file in the [delta log](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#add-file-and-remove-file).
40+
41+
### Example
42+
43+
```json
44+
{"IsEnabled": true, "Hot": "1.00:00:00"}
45+
```
46+
47+
## Returns
48+
49+
The command returns a table with one record that includes the modified policy object.
50+
51+
| Column | Type | Description |
52+
| ------------- | -------- | ------------------------------------------------------------------------------ |
53+
| PolicyName | `string` | The name of the policy - `QueryAcceleration` |
54+
| EntityName | `string` | The fully qualified name of the entity: `[DatabaseName].[ExternalTableName]` |
55+
| Policy | `string` | A JSON-serialization of the query acceleration policy that is set on the external delta table |
56+
| ChildEntities | `string` | The child entities this policy affects - `null` |
57+
| EntityType | `string` | The type of the entity the policy applies to - `ExternalTable` |
58+
59+
## Example
60+
61+
```Kusto
62+
.alter external table MyExternalTable policy query_acceleration '{"IsEnabled": true, "Hot": "1.00:00:00"}'
63+
```
64+
65+
## Related content
66+
67+
* [Query acceleration policy](query-acceleration-policy.md)
68+
* [.delete query acceleration policy command](delete-query-acceleration-policy-command.md)
69+
* [.show query acceleration policy command](show-query-acceleration-policy-command.md)
70+
* [.show external table operations query_acceleration statistics](show-external-table-operations-query-acceleration-statistics.md)

‎data-explorer/kusto/management/delete-query-acceleration-policy-command.md‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
---
2+
title: ".delete query acceleration policy command (preview)"
3+
description: Learn how to use the .delete query acceleration policy command to accelerate queries over external delta tables.
4+
ms.reviewer: sharmaanshul
5+
ms.topic: reference
6+
ms.date: 11/19/2024
7+
---
8+
# .delete query acceleration policy command (preview)
9+
10+
> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)]
11+
12+
Deletes the [query acceleration policy](query-acceleration-policy.md) of a specific external delta table.
13+
14+
## Permissions
15+
16+
You must have at least [Database Admin](../access-control/role-based-access-control.md) permissions to run this command.
17+
18+
## Syntax
19+
20+
`.delete` `external` `table` *ExternalTableName* `policy` `query_acceleration`
21+
22+
## Parameters
23+
24+
| Name | Type | Required | Description |
25+
| ------------------- | -------- | ------------------ | ------------------------------- |
26+
| *ExternalTableName* | `string` | :heavy_check_mark: | The name of the external delta table. |
27+
28+
## Example
29+
30+
```Kusto
31+
.delete external table MyExternalTable policy query_acceleration
32+
```
33+
34+
## Related content
35+
36+
* [Query acceleration policy](query-acceleration-policy.md)
37+
* [.alter query acceleration policy command](alter-query-acceleration-policy-command.md)
38+
* [.show query acceleration policy command](show-query-acceleration-policy-command.md)
39+
* [.show external table operations query_acceleration statistics](show-external-table-operations-query-acceleration-statistics.md)

‎data-explorer/kusto/management/external-tables-delta-lake.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,8 @@ The commands in this article can be used to create or alter a delta [external ta
1414
> [!NOTE]
1515
> If the table exists, the `.create` command will fail with an error. Use `.create-or-alter` or `.alter` to modify existing tables.
1616
17+
To accelerate queries over external delta tables, see [Query acceleration policy](query-acceleration-policy.md).
18+
1719
## Permissions
1820

1921
To `.create` requires at least [Database User](../access-control/role-based-access-control.md) permissions, and to `.alter` requires at least [Table Admin](../access-control/role-based-access-control.md) permissions.

‎data-explorer/kusto/management/query-acceleration-policy.md‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
---
2+
title: Query acceleration policy (preview)
3+
description: Learn how to use the query acceleration policy to accelerate queries over external delta tables.
4+
ms.reviewer: sharmaanshul
5+
ms.topic: reference
6+
ms.date: 11/19/2024
7+
---
8+
# Query acceleration policy (preview)
9+
10+
> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)]
11+
12+
An [external table](../query/schema-entities/external-tables.md) is a schema entity that references data stored external to a Kusto database. Queries run over external tables can be less performant than on data that is ingested due to various factors such as network calls to fetch data from storage, the absence of indexes, and more. Query acceleration allows specifying a policy on top of external delta tables. This policy defines a number of days to accelerate data for high-performance queries.
13+
14+
::: moniker range="azure-data-explorer"
15+
Query acceleration is supported in Azure Data Explorer over Azure Data Lake Store Gen2 or Azure blob storage [external tables](external-tables-azure-storage.md).
16+
::: moniker-end
17+
18+
::: moniker range="microsoft-fabric"
19+
Query acceleration is supported in Eventhouse over OneLake, Azure Data Lake Store Gen2, or Azure blob storage [external tables](/fabric/real-time-intelligence/onelake-shortcuts).
20+
21+
To enable query acceleration in the Fabric UI, see [Query acceleration over OneLake shortcuts](https://go.microsoft.com/fwlink/?linkid=2296674).
22+
::: moniker-end
23+
24+
## Limitations
25+
26+
* The number of columns in the external table can't exceed 900.
27+
* Query performance over accelerated external delta tables which have partitions may not be optimal during preview.
28+
* The feature assumes delta tables with static advanced features, for example column mapping doesn't change, partitions don't change, and so on. To change advanced features, first disable the policy, and once the change is made, re-enable the policy.
29+
* Schema changes on the delta table must also be followed with the respective `. alter` external delta table schema, which might result in acceleration starting from scratch if there was breaking schema change.
30+
* Index-based pruning isn't supported for partitions.
31+
* Parquet files with a compressed size higher than 6 GB won't be cached.
32+
::: moniker range="azure-data-explorer"
33+
* Query acceleration isn't supported for external tables with impersonation authentication.
34+
::: moniker-end
35+
36+
## Known issues
37+
38+
* Data in the external delta table that is optimized with the [OPTIMIZE](/azure/databricks/sql/language-manual/delta-optimize) function will need to be reaccelearted.
39+
* If you run frequent MERGE/UPDATE/DELETE operations in delta, the underlying parquet files may be rewritten with changes and Kusto will skip accelerating such files, causing retrieval during query time.
40+
* The system assumes that all artifacts under the delta table directory have the same access level to the selected users. Different files having different access permissions under the delta table directory might result with unexpected behavior.
41+
42+
## Commands for query acceleration
43+
44+
* [.alter query acceleration policy command](alter-query-acceleration-policy-command.md)
45+
* [.delete query acceleration policy command](delete-query-acceleration-policy-command.md)
46+
* [.show query acceleration policy command](show-query-acceleration-policy-command.md)
47+
* [.show external table operations query_acceleration statistics](show-external-table-operations-query-acceleration-statistics.md)

‎data-explorer/kusto/management/show-external-table-operations-query-acceleration-statistics.md‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
---
2+
title: .show external table operations query_acceleration statistics command (preview)
3+
description: Learn how to use the ".show external table operations query_acceleration statistics command" to accelerate queries over external delta tables.
4+
ms.reviewer: sharmaanshul
5+
ms.topic: reference
6+
ms.date: 11/19/2024
7+
---
8+
# .show external table operations query_acceleration statistics command (preview)
9+
10+
> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)]
11+
12+
Shows statistics for [query acceleration](query-acceleration-policy.md) for a specific external delta table, or all external delta tables in the database that have a query acceleration policy set.
13+
14+
## Permissions
15+
16+
You must have at least [Database monitor or Database viewer](../access-control/role-based-access-control.md) permissions to run this command.
17+
18+
## Syntax
19+
20+
`.show` `external` `table` *ExternalTableName* `operations` `query_acceleration` `statistics`
21+
22+
`.show` `external` `tables` `operations` `query_acceleration` `statistics`
23+
24+
## Parameters
25+
26+
| Name | Type | Required | Description |
27+
| ------------------- | -------- | ------------------ | ------------------------------- |
28+
| *ExternalTableName* | `string` | | The name of the external table. |
29+
30+
## Returns
31+
32+
The command returns a table with a record per external table that has a non-null policy, with the following columns:
33+
34+
| Column | Type | Description |
35+
| ------------------------------- | ---------- | ---------------------------------------------------------------------------------------------------- |
36+
| ExternalTableName | `string` | The name of the external table |
37+
| IsEnabled | `bool` | Indicates whether the external has a query acceleration policy enabled |
38+
| Hot | `timespan` | The hot period defined in the query acceleration policy |
39+
| HotSize | `long` | The total size on disk (in bytes) of artifacts accelerated due to the query acceleration policy |
40+
| LastUpdatedDateTime | `datetime` | Indicates the last datetime when the internal representation of the table was successfully refreshed |
41+
| AccelerationPendingArtifactSize | `long` | The total size of artifacts that are pending acceleration |
42+
|AccelerationCompletePercentage | `double` | The percentage of artifacts that are accelerated, out of the total artifacts that are able to be accelerated |
43+
|NotHealthyReason | `string` | Describes the reason for query acceleration not being healthy. Empty if healthy |
44+
45+
## Example
46+
47+
```Kusto
48+
.show external table MyExternalTable operations query_acceleration statistics
49+
```
50+
51+
**Output**
52+
53+
| ExternalTableName | IsEnabled | Hot | HotSize | LastUpdatedDateTime | AccelerationPendingArtifactSize | AccelerationCompletePercentage | NotHealthyReason |
54+
| --- | --- | --- | --- | --- | --- | --- |
55+
| MyExternalTable | True | 1.00:00:00 | 56877928187 | 2024-08-13 19:54:47.5868860 | 0 | 100 | |
56+
| MyExternalTable2 | True | 1.00:00:00 | 60467660293 | 2024-08-13 19:54:47.5868860 | 0 | 100 | |
57+
58+
## Related content
59+
60+
* [Query acceleration policy](query-acceleration-policy.md)
61+
* [.alter query acceleration policy command](alter-query-acceleration-policy-command.md)
62+
* [.delete query acceleration policy command](delete-query-acceleration-policy-command.md)
63+
* [.show query acceleration policy command](show-query-acceleration-policy-command.md)

‎data-explorer/kusto/management/show-query-acceleration-policy-command.md‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
---
2+
title: .show query acceleration policy command (preview)
3+
description: Learn how to use the ".show query acceleration policy command" to accelerate queries over external delta tables.
4+
ms.reviewer: sharmaanshul
5+
ms.topic: reference
6+
ms.date: 11/19/2024
7+
---
8+
# .show query acceleration policy command (preview)
9+
10+
> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)]
11+
12+
Shows the [query acceleration policy](query-acceleration-policy.md) of a specific external delta table, or for all external delta tables in the database that have a query acceleration policy set.
13+
14+
## Permissions
15+
16+
You must have at least [Database monitor or Database viewer](../access-control/role-based-access-control.md) permissions to run this command.
17+
18+
## Syntax
19+
20+
`.show` `external` `table` *ExternalTableName* `policy` `query_acceleration`
21+
22+
`.show` `external` `table` `*` `policy` `query_acceleration`
23+
24+
## Parameters
25+
26+
| Name | Type | Required | Description |
27+
| ------------------- | -------- | ------------------ | ------------------------------- |
28+
| *ExternalTableName* | `string` | | The name of the external delta table. |
29+
30+
## Returns
31+
32+
The command returns a table with a record per external table with the following columns:
33+
34+
| Column | Type | Description |
35+
| ------------- | -------- | ------------------------------------------------------------------------------ |
36+
| PolicyName | `string` | The name of the policy - `QueryAcceleration` |
37+
| EntityName | `string` | The fully qualified name of the entity: `[DatabaseName].[ExternalTableName]` |
38+
| Policy | `string` | A JSON-serialization of the query acceleration policy that is set on the table |
39+
| ChildEntities | `string` | The child entities this policy affects - `null` |
40+
| EntityType | `string` | The type of the entity the policy applies to - `ExternalTable` |
41+
42+
## Example
43+
44+
```Kusto
45+
.show external table MyExternalTable policy query_acceleration
46+
```
47+
48+
**Output**
49+
50+
| PolicyName | EntityName | Policy | ChildEntities | EntityType |
51+
| ----------------------- | ------------------------------ | ---------------------------------------------------------------------------------- | ------------- | ------------- |
52+
| QueryAccelerationPolicy | [MyDatabase].[MyExternalTable] | {<br> "IsEnabled": true,<br> "Hot": "1.00:00:00"} || ExternalTable |
53+
54+
## Related content
55+
56+
* [Query acceleration policy](query-acceleration-policy.md)
57+
* [.alter query acceleration policy command](alter-query-acceleration-policy-command.md)
58+
* [.delete query acceleration policy command](delete-query-acceleration-policy-command.md)
59+
* [.show external table operations query_acceleration statistics](show-external-table-operations-query-acceleration-statistics.md)

‎data-explorer/kusto/management/toc.yml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -546,6 +546,23 @@ items:
546546
displayName: .show table partitioning policy
547547
- name: .show database extents partitioning statistics command
548548
href: show-database-extents-partitioning-statistics.md
549+
- name: Query acceleration
550+
items:
551+
- name: Query acceleration policy
552+
href: query-acceleration-policy.md
553+
displayName: external table, delta table
554+
- name: .alter query acceleration policy command
555+
href: alter-query-acceleration-policy-command.md
556+
displayName: external table, delta table
557+
- name: .delete query acceleration policy command
558+
href: delete-query-acceleration-policy-command.md
559+
displayName: external table, delta table
560+
- name: .show external table operations query acceleration statistcs command
561+
href: show-external-table-operations-query-acceleration-statistics.md
562+
displayName: external table, delta table
563+
- name: .show query acceleration policy command
564+
href: show-query-acceleration-policy-command.md
565+
displayName: external table, delta table
549566
- name: Query weak consistency
550567
items:
551568
- name: Query weak consistency policy

‎data-explorer/kusto/query/external-table-function.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,8 @@ References an [external table](schema-entities/external-tables.md) by name.
1616
> The `external_table` function has similar restrictions as the [table](table-function.md) function.
1717
> Standard [query limits](../concepts/query-limits.md) apply to external table queries as well.
1818
19+
To accelerate queries over external delta tables, see [Query acceleration policy](../management/query-acceleration-policy.md).
20+
1921
## Syntax
2022

2123
`external_table(` *TableName* [`,` *MappingName* ] `)`

‎data-explorer/kusto/query/schema-entities/external-tables.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,8 @@ Use the following commands to manage external tables:
4242
For more information about how to query external tables, and ingested and uningested data, see [Query data in Azure Data Lake using Azure Data Explorer](/azure/data-explorer/data-lake-query-data).
4343
::: moniker-end
4444

45+
To accelerate queries over external delta tables, see [Query acceleration policy](../../management/query-acceleration-policy.md).
46+
4547
> [!NOTE]
4648
>
4749
> * The maximum limit of external tables per database is 1,000.

0 commit comments

Comments
 (0)