Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
76f7782
Learn Editor: Update external-tables-azure-storage.md
yaela8 Jan 26, 2025
f95d040
Learn Editor: Update show-external-tables.md
yaela8 Jan 26, 2025
40bb745
Learn Editor: Update export-data-to-storage.md
yaela8 Jan 26, 2025
7aa9fc2
Learn Editor: Update export-data-to-storage.md
yaela8 Jan 26, 2025
723e43e
replace branch PR 6542
ktalmor Jan 26, 2025
164d361
Merge branch 'main' of https://github.com/ktalmor/dataexplorer-docs-p…
ktalmor Jan 28, 2025
6f0489e
Replace help cluster include
ktalmor Jan 28, 2025
ed161a0
Merge branch 'main' of https://github.com/MicrosoftDocs/dataexplorer-…
ktalmor Jan 29, 2025
41bb62e
Update data-explorer/kusto/management/data-export/export-data-to-stor…
yaela8 Feb 2, 2025
67fdf88
Update data-explorer/kusto/management/external-tables-azure-storage.md
yaela8 Feb 2, 2025
7e4d2bc
Update data-explorer/kusto/management/external-tables-azure-storage.md
yaela8 Feb 2, 2025
5346e37
Update data-explorer/kusto/management/show-external-tables.md
yaela8 Feb 2, 2025
f0d7d73
Fix broken link in show-external-tables.md
yaela8 Feb 3, 2025
ab04489
add compression properties to external delta tables doc
yaela8 Feb 3, 2025
62bddf3
Add introductory example sentences
ktalmor Feb 3, 2025
e8e36e2
Added sentence
ktalmor Feb 3, 2025
604280e
Update data-explorer/kusto/management/data-export/export-data-to-stor…
shsagir Feb 4, 2025
b51cce4
Update data-explorer/kusto/management/external-tables-delta-lake.md
shsagir Feb 4, 2025
4999ba6
Apply suggestions from code review
shsagir Feb 4, 2025
0428e15
Fix note formatting in set-statement.md
ktalmor Feb 4, 2025
f8e651b
Clarify documentation for external table properties
shsagir Feb 4, 2025
bd5d56c
Clarify descriptions in export-data-to-storage.md
shsagir Feb 4, 2025
1dc9667
Clarify `persistDetails` description in documentation
shsagir Feb 4, 2025
988144e
Update project-rename-operator.md example section
ktalmor Feb 5, 2025
bd5563c
Update example description in set-statement.md
ktalmor Feb 6, 2025
8a6ae1f
conflict resolved
ktalmor Feb 6, 2025
f28a77c
Merge branch 'wi-364896-batch7' of https://github.com/ktalmor/dataexp…
ktalmor Feb 6, 2025
f621087
Merge branch 'main' into wi-364896-batch7
ktalmor Feb 6, 2025
a6a8a6c
Apply suggestions from code review
shsagir Feb 9, 2025
6c6021a
align wording and table markdown
yaela8 Feb 9, 2025
27a64d8
Merge branch 'main' of https://github.com/MicrosoftDocs/dataexplorer-…
ktalmor Feb 10, 2025
dc8a773
resolved conflict
ktalmor Feb 10, 2025
9c31c3b
Merge branch 'wi-364896-batch7' of https://github.com/ktalmor/dataexp…
ktalmor Feb 10, 2025
001dce7
Merge pull request #6572 from yaela8/docs-editor/external-tables-azur…
v-ccolin Feb 10, 2025
cdf5b87
Merge remote-tracking branch 'upstream/main' into wi-364896-batch7
ktalmor Feb 10, 2025
0ab34b0
Merge pull request #6580 from ktalmor/wi-364896-batch7
v-ccolin Feb 10, 2025
0626dff
Merge pull request #6625 from MicrosoftDocs/main
v-ccolin Feb 10, 2025
4bef368
Merging changes synced from https://github.com/MicrosoftDocs/dataexpl…
Feb 10, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -23,32 +23,32 @@ You must have at least [Database Viewer](../../access-control/role-based-access-

## Parameters

| Name | Type | Required | Description |
|--|--|--|--|
| `async` | `string` | | If specified, the command runs in asynchronous mode. See [asynchronous mode](#asynchronous-mode). |
| `compressed` | `string` | | If specified, the output storage artifacts are compressed as `.gz` files. See the `compressionType` [supported property](#supported-properties) for compressing Parquet files as snappy. |
| *OutputDataFormat* | `string` | :heavy_check_mark: | Indicates the data format of the storage artifacts written by the command. Supported values are: `csv`, `tsv`, `json`, and `parquet`. |
| *StorageConnectionString* | `string` | | One or more [storage connection strings](../../api/connection-strings/storage-connection-strings.md) that indicate which storage to write the data to. More than one storage connection string might be specified for scalable writes. Each such connection string must indicate the credentials to use when writing to storage. For example, when writing to Azure Blob Storage, the credentials can be the storage account key, or a shared access key (SAS) with the permissions to read, write, and list blobs. |
| *PropertyName*, *PropertyValue* | `string` | | A comma-separated list of key-value property pairs. See [supported properties](#supported-properties).|
| Name | Type | Required | Description |
|-- |-- |-- |-- |
| *async* | `string` | | If specified, the command runs in asynchronous mode. See [asynchronous mode](#asynchronous-mode). |
| *compressed* | `bool` | | If specified, the output storage artifacts are compressed in the format specified by the `compressionType` [supported property](#supported-properties). |
| *OutputDataFormat* | `string` | :heavy_check_mark: | The data format of the storage artifacts written by the command. Supported values are: `csv`, `tsv`, `json`, and `parquet`. |
| *StorageConnectionString* | `string` | | One or more [storage connection strings](../../api/connection-strings/storage-connection-strings.md) that specify which storage to write the data to. More than one storage connection string might be specified for scalable writes. Each such connection string must specify the credentials to use when writing to storage. For example, when writing to Azure Blob Storage, the credentials can be the storage account key, or a shared access key (SAS) with the permissions to read, write, and list blobs. |
| *PropertyName*, *PropertyValue* | `string` | | A comma-separated list of key-value property pairs. See [supported properties](#supported-properties).|

> [!NOTE]
> We highly recommended exporting data to storage that is colocated in the same region as the database itself. This includes data that is exported so it can be transferred to another cloud service in other regions. Writes should be done locally, while reads can happen remotely.

## Supported properties

| Property | Type | Description |
|--|--|--|
| `includeHeaders` | `string` | For `csv`/`tsv` output, controls the generation of column headers. Can be one of `none` (default; no header lines emitted), `all` (emit a header line into every storage artifact), or `firstFile` (emit a header line into the first storage artifact only). |
| `fileExtension` | `string` | Indicates the "extension" part of the storage artifact (for example, `.csv` or `.tsv`). If compression is used, `.gz` is appended as well. |
| `namePrefix` | `string` | Indicates a prefix to add to each generated storage artifact name. A random prefix is used if left unspecified. |
| `encoding` | `string` | Indicates how to encode the text: `UTF8NoBOM` (default) or `UTF8BOM`. |
| `compressionType` | `string` | Indicates the type of compression to use. Possible values are `gzip` or `snappy`. Default is `gzip`. `snappy` can (optionally) be used for `parquet` format. |
| `distribution` | `string` | Distribution hint (`single`, `per_node`, `per_shard`). If value equals `single`, a single thread writes to storage. Otherwise, export writes from all nodes executing the query in parallel. See [evaluate plugin operator](../../query/evaluate-operator.md). Defaults to `per_shard`. |
| `persistDetails` | `bool` | Indicates that the command should persist its results (see `async` flag). Defaults to `true` in async runs, but can be turned off if the caller doesn't require the results). Defaults to `false` in synchronous executions, but can be turned on in those as well. |
| `sizeLimit` | `long` | The size limit in bytes of a single storage artifact written before compression. Valid range: 100 MB (default) to 4 GB. |
| `parquetRowGroupSize` | `int` | Relevant only when data format is Parquet. Controls the row group size in the exported files. Default row group size is 100,000 records. |
| `distributed` | `bool` | Disable or enable distributed export. Setting to false is equivalent to `single` distribution hint. Default is true. |
| `parquetDatetimePrecision` | `string` | Specifies the precision to use when exporting `datetime` values to Parquet. Possible values are millisecond and microsecond. Default is millisecond. |
| Property | Type | Description |
|-- |-- |-- |
| `includeHeaders` | `string` | For `csv`/`tsv` output, controls the generation of column headers. Can be one of `none` (default; no header lines emitted), `all` (emit a header line into every storage artifact), or `firstFile` (emit a header line into the first storage artifact only). |
| `fileExtension` | `string` | The "extension" part of the storage artifact (for example, `.csv` or `.tsv`). If compression is used, `.gz` is appended as well. |
| `namePrefix` | `string` | The prefix to add to each generated storage artifact name. A random prefix is used if left unspecified. |
| `encoding` | `string` | The encoding for text. Possible values include: `UTF8NoBOM` (default) or `UTF8BOM`. |
| `compressionType` | `string` | The type of compression to use. For non-Parquet files, only `gzip` is allowed. For Parquet files, possible values include `gzip`, `snappy`, `lz4_raw`, `brotli`, and `zstd`. Default is `gzip`. |
| `distribution` | `string` | Distribution hint (`single`, `per_node`, `per_shard`). If value equals `single`, a single thread writes to storage. Otherwise, export writes from all nodes executing the query in parallel. See [evaluate plugin operator](../../query/evaluate-operator.md). Defaults to `per_shard`. |
| `persistDetails` | `bool` | If `true`, the command persists its results (see `async` flag). Defaults to `true` in async runs, but can be turned off if the caller doesn't require the results. Defaults to `false` in synchronous executions, but can be turned on in those as well. |
| `sizeLimit` | `long` | The size limit in bytes of a single storage artifact written before compression. Valid range: 100 MB (default) to 4 GB. |
| `parquetRowGroupSize` | `int` | Relevant only when data format is Parquet. Controls the row group size in the exported files. Default row group size is 100,000 records. |
| `distributed` | `bool` | Disable or enable distributed export. Setting to false is equivalent to `single` distribution hint. Default is true. |
| `parquetDatetimePrecision` | `string` | The precision to use when exporting `datetime` values to Parquet. Possible values are millisecond and microsecond. Default is millisecond. |

## Authentication and authorization

Expand Down
25 changes: 13 additions & 12 deletions data-explorer/kusto/management/external-tables-azure-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,19 +118,20 @@ external_table("ExternalTable")

## Optional properties

| Property | Type | Description |
|------------------|----------|-------------------------------------------------------------------------------------|
| `folder` | `string` | Table's folder |
| `docString` | `string` | String documenting the table |
| `compressed` | `bool` | If set, indicates whether the files are compressed as `.gz` files (used in [export scenario](data-export/export-data-to-an-external-table.md) only) |
| `includeHeaders` | `string` | For delimited text formats (CSV, TSV, ...), indicates whether files contain a header. Possible values are: `All` (all files contain a header), `FirstFile` (first file in a folder contains a header), `None` (no files contain a header). |
| `namePrefix` | `string` | If set, indicates the prefix of the files. On write operations, all files will be written with this prefix. On read operations, only files with this prefix are read. |
| `fileExtension` | `string` | If set, indicates file extensions of the files. On write, files names will end with this suffix. On read, only files with this file extension will be read. |
| `encoding` | `string` | Indicates how the text is encoded: `UTF8NoBOM` (default) or `UTF8BOM`. |
| `sampleUris` | `bool` | If set, the command result provides several examples of simulated external data files URI as they're expected by the external table definition. This option helps validate whether the *Partitions* and *PathFormat* parameters are defined properly. |
| `filesPreview` | `bool` | If set, one of the command result tables contains a preview of [.show external table artifacts](show-external-table-artifacts.md) command. Like `sampleUri`, the option helps validate the *Partitions* and *PathFormat* parameters of external table definition. |
| Property | Type | Description |
|------------------ |----------|-------------------------------------------------------------------------------------|
| `folder` | `string` | Table's folder |
| `docString` | `string` | String documenting the table |
| `compressed` | `bool` | Only relevant for the [export scenario](data-export/export-data-to-an-external-table.md).<br>If set to true, the data is exported in the format specified by the `compressionType` property. For the read path, compression is automatically detected. |
| `compressionType` | `string` | Only relevant for the [export scenario](data-export/export-data-to-an-external-table.md).<br>The compression type of exported files. For non-Parquet files, only `gzip` is allowed. For Parquet files, possible values include `gzip`, `snappy`, `lz4_raw`, `brotli`, and `zstd`. Default is `gzip`. For the read path, compression type is automatically detected. |
| `includeHeaders` | `string` | For delimited text formats (CSV, TSV, ...), specifies whether files contain a header. Possible values are: `All` (all files contain a header), `FirstFile` (first file in a folder contains a header), `None` (no files contain a header). |
| `namePrefix` | `string` | If set, specifies the prefix of the files. On write operations, all files will be written with this prefix. On read operations, only files with this prefix are read. |
| `fileExtension` | `string` | If set, specifies the extension of the files. On write, files names will end with this suffix. On read, only files with this file extension will be read. |
| `encoding` | `string` | Specifies how the text is encoded: `UTF8NoBOM` (default) or `UTF8BOM`. |
| `sampleUris` | `bool` | If set, the command result provides several examples of simulated external data files URI as they're expected by the external table definition. This option helps validate whether the *Partitions* and *PathFormat* parameters are defined properly. |
| `filesPreview` | `bool` | If set, one of the command result tables contains a preview of [.show external table artifacts](show-external-table-artifacts.md) command. Like `sampleUri`, the option helps validate the *Partitions* and *PathFormat* parameters of external table definition. |
| `validateNotEmpty` | `bool` | If set, the connection strings are validated for having content in them. The command will fail if the specified URI location doesn't exist, or if there are insufficient permissions to access it. |
| `dryRun` | `bool` | If set, the external table definition isn't persisted. This option is useful for validating the external table definition, especially in conjunction with the `filesPreview` or `sampleUris` parameter. |
| `dryRun` | `bool` | If set, the external table definition isn't persisted. This option is useful for validating the external table definition, especially in conjunction with the `filesPreview` or `sampleUris` parameter. |

> [!NOTE]
> The external table isn't accessed during creation, only during query and export. Use the `validateNotEmpty` optional property during creation to make sure the table definition is valid and the storage is accessible.
Expand Down
18 changes: 10 additions & 8 deletions data-explorer/kusto/management/external-tables-delta-lake.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,14 +54,16 @@ The supported authentication methods are the same as those supported by [Azure S

## Optional properties

| Property | Type | Description |
|------------------|----------|------------------------------------------------------------------------------------|
| `folder` | `string` | Table's folder |
| `docString` | `string` | String documenting the table |
| `namePrefix` | `string` | If set, indicates the prefix of the files. On write operations, all files will be written with this prefix. On read operations, only files with this prefix are read. |
| `fileExtension` | `string` | If set, indicates file extensions of the files. On write, files names will end with this suffix. On read, only files with this file extension will be read. |
| `encoding` | `string` | Indicates how the text is encoded: `UTF8NoBOM` (default) or `UTF8BOM`. |
| `dryRun` | `bool` | If set, the external table definition isn't persisted. This option is useful for validating the external table definition, especially in conjunction with the `filesPreview` or `sampleUris` parameter. |
| Property | Type | Description |
|------------------ |---------- |------------------------------------------------------------------------------------|
| `folder` | `string` | Table's folder |
| `docString` | `string` | String documenting the table |
| `compressed` | `bool` | Only relevant for the [export scenario](data-export/export-data-to-an-external-table.md).<br>If set to true, the data is exported in the format specified by the `compressionType` property. For the read path, compression is automatically detected. |
| `compressionType` | `string` | Only relevant for the [export scenario](data-export/export-data-to-an-external-table.md).<br>The compression type of exported files. For non-Parquet files, only `gzip` is allowed. For Parquet files, possible values include `gzip`, `snappy`, `lz4_raw`, `brotli`, and `zstd`. Default is `gzip`. For the read path, compression type is automatically detected. |
| `namePrefix` | `string` | If set, specifies the prefix of the files. On write operations, all files will be written with this prefix. On read operations, only files with this prefix are read. |
| `fileExtension` | `string` | If set, specifies extension of the files. On write, files names will end with this suffix. On read, only files with this file extension will be read. |
| `encoding` | `string` | Specifies how the text is encoded: `UTF8NoBOM` (default) or `UTF8BOM`. |
| `dryRun` | `bool` | If set, the external table definition isn't persisted. This option is useful for validating the external table definition, especially in conjunction with the `filesPreview` or `sampleUris` parameter. |

> [!NOTE]
> The external delta table is accessed during creation, to infer the partitioning information and, optionally, the schema. Make sure that the table definition is valid and that the storage is accessible.
Expand Down
2 changes: 1 addition & 1 deletion data-explorer/kusto/management/show-external-tables.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ You must have at least Database User, Database Viewer, Database Monitor to run t
| TableType | `string` | Type of external table |
| Folder | `string` | Table's folder |
| DocString | `string` | String documenting the table |
| Properties | `string` | Table's JSON serialized properties (specific to the type of table) |
| Properties | `string` | Table's JSON serialized properties (specific to the type of table; For more information, see [Create and alter Azure Storage external tables](external-tables-azure-storage.md) or [Create and alter delta external tables on Azure Storage](external-tables-delta-lake.md). |

## Example

Expand Down
2 changes: 2 additions & 0 deletions data-explorer/kusto/query/let-statement.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,8 @@ To optimize multiple uses of the `let` statement within a single query, see [Opt

## Examples

The examples in this section show how to use the syntax to help you get started.

[!INCLUDE [help-cluster](../includes/help-cluster-note.md)]

The query examples show the syntax and example usage of the operator, statement, or function.
Expand Down
14 changes: 10 additions & 4 deletions data-explorer/kusto/query/project-operator.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,10 @@ A table with columns that were named as arguments. Contains same number of rows

## Examples

The examples in this section show how to use the syntax to help you get started.

[!INCLUDE [help-cluster](../includes/help-cluster-note.md)]

### Only show specific columns

Only show the `EventId`, `State`, `EventType` of the `StormEvents` table.
Expand All @@ -61,7 +65,9 @@ StormEvents
| project EventId, State, EventType
```

The following results table shows only the top 10 results.
**Output**

The table shows the first 10 results.

|EventId|State|EventType|
|--|--|--|
Expand Down Expand Up @@ -92,7 +98,9 @@ StormEvents
| where TotalInjuries > 5
```

The following table shows only the first 10 results.
**Output**

The table shows the first 10 results.

|StartLocation| TotalInjuries|
|--|--|
Expand All @@ -108,9 +116,7 @@ The following table shows only the first 10 results.
|COLLIERVILLE| 6|
|...|...|

::: moniker range="microsoft-fabric || azure-data-explorer || azure-monitor || microsoft-sentinel"
## Related content

* [`extend`](extend-operator.md)
* [series_stats](series-stats-function.md)
::: moniker-end
12 changes: 7 additions & 5 deletions data-explorer/kusto/query/project-rename-operator.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ title: project-rename operator
description: Learn how to use the project-rename operator to rename columns in the output table.
ms.reviewer: alexans
ms.topic: reference
ms.date: 08/11/2024
ms.date: 01/20/2025
---
# project-rename operator

Expand All @@ -29,7 +29,9 @@ Renames columns in the output table.

A table that has the columns in the same order as in an existing table, with columns renamed.

## Examples
## Example

If you have a table with columns a, b, and c, and you want to rename a to new_a and b to new_b while keeping the same order, the query would look like this:

:::moniker range="azure-data-explorer"
> [!div class="nextstepaction"]
Expand All @@ -43,6 +45,6 @@ print a='a', b='b', c='c'

**Output**

|new_a|new_b|c|
|---|---|---|
|a|b|c|
| new_a | new_b | c |
|--|--|--|
| a | b | c |
Loading