Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ title: Continuous data export
description: This article describes Continuous data export.
ms.reviewer: yifats
ms.topic: reference
ms.date: 12/08/2024
ms.date: 07/30/2025
---
# Continuous data export overview

Expand Down Expand Up @@ -101,12 +101,14 @@ Followed by:
<| T | where cursor_before_or_at("636751928823156645")
```

::: moniker range="azure-data-explorer"
## Continuous export from a table with Row Level Security

To create a continuous export job with a query that references a table with [Row Level Security policy](../../management/row-level-security-policy.md), you must:

* Provide a managed identity as part of the continuous export configuration. For more information, see [Use a managed identity to run a continuous export job](continuous-export-with-managed-identity.md).
* Use [impersonation](../../api/connection-strings/storage-connection-strings.md#impersonation) authentication for the external table to which the data is exported.
::: moniker-end

## Continuous export to delta table - Preview

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ title: .create or alter continuous-export
description: This article describes how to create or alter continuous data export.
ms.reviewer: yifats
ms.topic: reference
ms.date: 12/08/2024
ms.date: 07/30/2025
---
# .create or alter continuous-export

Expand Down Expand Up @@ -31,11 +31,15 @@ You must have at least [Database Admin](../../access-control/role-based-access-c
| *T1*, *T2* | `string` | | A comma-separated list of fact tables in the query. If not specified, all tables referenced in the query are assumed to be fact tables. If specified, tables *not* in this list are treated as dimension tables and aren't scoped, so all records participate in all exports. See [continuous data export overview](continuous-data-export.md) for details. |
| *propertyName*, *propertyValue* | `string` | | A comma-separated list of optional [properties](#supported-properties).|

::: moniker range="azure-data-explorer"
> [!NOTE]
> If the target external table uses [impersonation](../../api/connection-strings/storage-connection-strings.md#impersonation) authentication, you must specify a managed identity to run the continuous export. For more information, see [Use a managed identity to run a continuous export job](continuous-export-with-managed-identity.md).
::: moniker-end

## Supported properties

::: moniker range="azure-data-explorer"

| Property | Type | Description |
|--|--|--|
| `intervalBetweenRuns` | `Timespan` | The time span between continuous export executions. Must be greater than 1 minute. |
Expand All @@ -46,6 +50,20 @@ You must have at least [Database Admin](../../access-control/role-based-access-c
| `managedIdentity` | `string` | The managed identity for which the continuous export job runs. The managed identity can be an object ID, or the `system` reserved word. For more information, see [Use a managed identity to run a continuous export job](continuous-export-with-managed-identity.md#use-a-managed-identity-to-run-a-continuous-export-job). |
| `isDisabled` | `bool` | Disable or enable the continuous export. Default is false. |

::: moniker-end
::: moniker range="microsoft-fabric"

| Property | Type | Description |
|--|--|--|
| `intervalBetweenRuns` | `Timespan` | The time span between continuous export executions. Must be greater than 1 minute. |
| `forcedLatency` | `Timespan` | An optional period of time to limit the query to records ingested before a specified period relative to the current time. This property is useful if, for example, the query performs some aggregations or joins, and you want to make sure all relevant records have been ingested before running the export. |
| `sizeLimit` | `long` | The size limit in bytes of a single storage artifact written before compression. Valid range: 100 MB (default) to 1 GB. |
| `distributed` | `bool` | Disable or enable distributed export. Setting to false is equivalent to `single` distribution hint. Default is true. |
| `parquetRowGroupSize` | `int` | Relevant only when data format is Parquet. Controls the row group size in the exported files. Default row group size is 100,000 records. |
| `isDisabled` | `bool` | Disable or enable the continuous export. Default is false. |

::: moniker-end

## Example

The following example creates or alters a continuous export `MyExport` that exports data from the `T` table to `ExternalBlob`. The data exports occur every hour, and have a defined forced latency and size limit per storage artifact.
Expand Down
49 changes: 36 additions & 13 deletions data-explorer/kusto/management/external-tables-azure-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ title: Create and alter Azure Storage external tables
description: This article describes how to create and alter external tables based on Azure Blob Storage or Azure Data Lake
ms.reviewer: orspodek
ms.topic: reference
ms.date: 08/11/2024
ms.date: 07/30/2025
---

# Create and alter Azure Storage external tables
Expand All @@ -13,13 +13,15 @@ ms.date: 08/11/2024
The commands in this article can be used to create or alter an Azure Storage [external table](../query/schema-entities/external-tables.md) in the database from which the command is executed. An Azure Storage external table references data located in Azure Blob Storage, Azure Data Lake Store Gen1, or Azure Data Lake Store Gen2.

> [!NOTE]
> If the table exists, the `.create` command will fail with an error. Use `.create-or-alter` or `.alter` to modify existing tables.
> If the table exists, the `.create` command fails with an error. Use `.create-or-alter` or `.alter` to modify existing tables.

## Permissions

To `.create` requires at least [Database User](../access-control/role-based-access-control.md) permissions, and to `.alter` requires at least [Table Admin](../access-control/role-based-access-control.md) permissions.

:::moniker range="azure-data-explorer"
To `.create-or-alter` an external table using managed identity authentication requires [AllDatabasesAdmin](../access-control/role-based-access-control.md) permissions.
:::moniker-end

## Syntax

Expand All @@ -38,22 +40,24 @@ To `.create-or-alter` an external table using managed identity authentication re
|*Schema*| `string` | :heavy_check_mark:|The external data schema is a comma-separated list of one or more column names and [data types](../query/scalar-data-types/index.md), where each item follows the format: *ColumnName* `:` *ColumnType*. If the schema is unknown, use [infer\_storage\_schema](../query/infer-storage-schema-plugin.md) to infer the schema based on external file contents.|
|*Partitions*| `string` || A comma-separated list of columns by which the external table is partitioned. Partition column can exist in the data file itself, or as part of the file path. See [partitions formatting](#partitions-formatting) to learn how this value should look.|
|*PathFormat*| `string` ||An external data folder URI path format to use with partitions. See [path format](#path-format).|
|*DataFormat*| `string` | :heavy_check_mark:|The data format, which can be any of the [ingestion formats](../ingestion-supported-formats.md). We recommend using the `Parquet` format for external tables to improve query and export performance, unless you use `JSON` paths mapping. When using an external table for [export scenario](data-export/export-data-to-an-external-table.md), you're limited to the following formats: `CSV`, `TSV`, `JSON` and `Parquet`.|
|*StorageConnectionString*| `string` | :heavy_check_mark:|One or more comma-separated paths to Azure Blob Storage blob containers, Azure Data Lake Gen 2 file systems or Azure Data Lake Gen 1 containers, including credentials. The external table storage type is determined by the provided connection strings. See [storage connection strings](../api/connection-strings/storage-connection-strings.md).|
|*DataFormat*| `string` | :heavy_check_mark:|The data format, which can be any of the [ingestion formats](../ingestion-supported-formats.md). We recommend using the `Parquet` format for external tables to improve query and export performance, unless you use `JSON` paths mapping. When using an external table for [export scenario](data-export/export-data-to-an-external-table.md), you're limited to the following formats: `CSV`, `TSV`, `JSON`, and `Parquet`.|
|*StorageConnectionString*| `string` | :heavy_check_mark:|One or more comma-separated paths to Azure Blob Storage blob containers, Azure Data Lake Gen 2 file systems or Azure Data Lake Gen 1 containers, including credentials. The provided connection string determines the external table storage type. See [storage connection strings](../api/connection-strings/storage-connection-strings.md).|
|*Property*| `string` ||A key-value property pair in the format *PropertyName* `=` *PropertyValue*. See [optional properties](#optional-properties).|

> [!NOTE]
> CSV files with non-identical schema might result in data appearing shifted or missing. We recommend separating CSV files with distinct schemas to separate storage containers and defining an external table for each storage container with the proper schema.
> CSV files with nonidentical schema might result in data appearing shifted or missing. We recommend separating CSV files with distinct schemas to separate storage containers and defining an external table for each storage container with the proper schema.

> [!TIP]
> Provide more than a single storage account to avoid storage throttling while [exporting](data-export/export-data-to-an-external-table.md) large amounts of data to the external table. Export will distribute the writes between all accounts provided.
> Provide more than a single storage account to avoid storage throttling while [exporting](data-export/export-data-to-an-external-table.md) large amounts of data to the external table. Export distributes the writes between all accounts provided.

## Authentication and authorization

The authentication method to access an external table is based on the connection string provided during its creation, and the permissions required to access the table vary depending on the authentication method.

The following table lists the supported authentication methods for Azure Storage external tables and the permissions needed to read or write to the table.

::: moniker range="azure-data-explorer"

| Authentication method | Azure Blob Storage / Data Lake Storage Gen2 | Data Lake Storage Gen1 |
|--|--|--|
|[Impersonation](../api/connection-strings/storage-connection-strings.md#impersonation)|**Read permissions:** Storage Blob Data Reader<br/>**Write permissions:** Storage Blob Data Contributor|**Read permissions:** Reader<br/>**Write permissions:** Contributor|
Expand All @@ -62,6 +66,18 @@ The following table lists the supported authentication methods for Azure Storage
|[Microsoft Entra access token](../api/connection-strings/storage-connection-strings.md#microsoft-entra-access-token)|No additional permissions required.|No additional permissions required.|
|[Storage account access key](../api/connection-strings/storage-connection-strings.md#storage-account-access-key)|No additional permissions required.|This authentication method isn't supported in Gen1.|

::: moniker-end
::: moniker range="microsoft-fabric"

| Authentication method | Azure Blob Storage / Data Lake Storage Gen2 | Data Lake Storage Gen1 |
|--|--|--|
|[Impersonation](../api/connection-strings/storage-connection-strings.md#impersonation)|**Read permissions:** Storage Blob Data Reader<br/>**Write permissions:** Storage Blob Data Contributor|**Read permissions:** Reader<br/>**Write permissions:** Contributor|
|[Shared Access (SAS) token](../api/connection-strings/storage-connection-strings.md#shared-access-sas-token)|**Read permissions:** List + Read<br/>**Write permissions:** Write|This authentication method isn't supported in Gen1.|
|[Microsoft Entra access token](../api/connection-strings/storage-connection-strings.md#microsoft-entra-access-token)|No additional permissions required.|No additional permissions required.|
|[Storage account access key](../api/connection-strings/storage-connection-strings.md#storage-account-access-key)|No additional permissions required.|This authentication method isn't supported in Gen1.|

::: moniker-end

[!INCLUDE [partitions-formatting](../includes/partitions-formatting.md)]

### Path format
Expand Down Expand Up @@ -125,12 +141,12 @@ external_table("ExternalTable")
| `compressed` | `bool` | Only relevant for the [export scenario](data-export/export-data-to-an-external-table.md).<br>If set to true, the data is exported in the format specified by the `compressionType` property. For the read path, compression is automatically detected. |
| `compressionType` | `string` | Only relevant for the [export scenario](data-export/export-data-to-an-external-table.md).<br>The compression type of exported files. For non-Parquet files, only `gzip` is allowed. For Parquet files, possible values include `gzip`, `snappy`, `lz4_raw`, `brotli`, and `zstd`. Default is `gzip`. For the read path, compression type is automatically detected. |
| `includeHeaders` | `string` | For delimited text formats (CSV, TSV, ...), specifies whether files contain a header. Possible values are: `All` (all files contain a header), `FirstFile` (first file in a folder contains a header), `None` (no files contain a header). |
| `namePrefix` | `string` | If set, specifies the prefix of the files. On write operations, all files will be written with this prefix. On read operations, only files with this prefix are read. |
| `fileExtension` | `string` | If set, specifies the extension of the files. On write, files names will end with this suffix. On read, only files with this file extension will be read. |
| `namePrefix` | `string` | If set, specifies the prefix of the files. On write operations, all files are written with this prefix. On read operations, only files with this prefix are read. |
| `fileExtension` | `string` | If set, specifies the extension of the files. On write, files names end with this suffix. On read, only files with this file extension are read. |
| `encoding` | `string` | Specifies how the text is encoded: `UTF8NoBOM` (default) or `UTF8BOM`. |
| `sampleUris` | `bool` | If set, the command result provides several examples of simulated external data files URI as they're expected by the external table definition. This option helps validate whether the *Partitions* and *PathFormat* parameters are defined properly. |
| `filesPreview` | `bool` | If set, one of the command result tables contains a preview of [.show external table artifacts](show-external-table-artifacts.md) command. Like `sampleUri`, the option helps validate the *Partitions* and *PathFormat* parameters of external table definition. |
| `validateNotEmpty` | `bool` | If set, the connection strings are validated for having content in them. The command will fail if the specified URI location doesn't exist, or if there are insufficient permissions to access it. |
| `validateNotEmpty` | `bool` | If set, the connection strings are validated for having content in them. The command fails if the specified URI location doesn't exist, or if there are insufficient permissions to access it. |
| `dryRun` | `bool` | If set, the external table definition isn't persisted. This option is useful for validating the external table definition, especially in conjunction with the `filesPreview` or `sampleUris` parameter. |

> [!NOTE]
Expand All @@ -141,7 +157,7 @@ external_table("ExternalTable")

### File filtering logic

When querying an external table, performance is improved by filtering out irrelevant external storage files. The process of iterating files and deciding whether a file should be processed is as follows:
When you query an external table, performance is improved by filtering out irrelevant external storage files. The process of iterating files and deciding whether a file should be processed is as follows:

1. Build a URI pattern that represents a place where files are found. Initially, the URI pattern equals a connection string provided as part of the external table definition. If there are any partitions defined, they're rendered using *PathFormat*, then appended to the URI pattern.

Expand All @@ -158,9 +174,9 @@ Once all the conditions are met, the file is fetched and processed.

## Examples

### Non-partitioned external table
### Nonpartitioned external table

In the following non-partitioned external table, the files are expected to be placed directly under the container(s) defined:
In the following nonpartitioned external table, the files are expected to be placed directly under the container(s) defined:

```kusto
.create external table ExternalTable (x:long, s:string)
Expand Down Expand Up @@ -250,8 +266,15 @@ external_table("ExternalTable")
## Related content

::: moniker range="azure-data-explorer"

* [Query external tables](/azure/data-explorer/data-lake-query-data).
::: moniker-end
* [Export data to an external table](data-export/export-data-to-an-external-table.md).
* [Continuous data export to an external table](data-export/continuous-data-export.md).

::: moniker-end
::: moniker range="microsoft-fabric"

* [Export data to an external table](data-export/export-data-to-an-external-table.md).
* [Continuous data export to an external table](data-export/continuous-data-export.md).

::: moniker-end
Loading