From d4727af340636f606081ff1dfffdffd1565d520c Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Sun, 26 Jan 2025 18:38:08 +0200 Subject: [PATCH 01/36] replace branch PR 6555 (1) --- data-explorer/kusto/query/alias-statement.md | 2 +- data-explorer/kusto/query/batches.md | 6 +++-- data-explorer/kusto/query/let-statement.md | 4 +++- .../kusto/query/parse-kv-operator.md | 14 ++++++----- data-explorer/kusto/query/parse-operator.md | 9 ++++++-- .../kusto/query/parse-where-operator.md | 2 +- .../kusto/query/partition-operator.md | 23 +++++++++++-------- .../kusto/query/pattern-statement.md | 4 ++-- data-explorer/kusto/query/print-operator.md | 6 +++-- .../kusto/query/query-parameters-statement.md | 8 ++++++- 10 files changed, 50 insertions(+), 28 deletions(-) diff --git a/data-explorer/kusto/query/alias-statement.md b/data-explorer/kusto/query/alias-statement.md index 0ce2adbd05..3fbd9a0625 100644 --- a/data-explorer/kusto/query/alias-statement.md +++ b/data-explorer/kusto/query/alias-statement.md @@ -51,7 +51,7 @@ The alias must be defined according to the following syntax, where *Eventhouse* ## Examples -[!INCLUDE [help-cluster](../includes/help-cluster-samples-stormevents.md)] +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] First, count the number of records in that table. diff --git a/data-explorer/kusto/query/batches.md b/data-explorer/kusto/query/batches.md index eec2c34b76..36062d2d91 100644 --- a/data-explorer/kusto/query/batches.md +++ b/data-explorer/kusto/query/batches.md @@ -3,7 +3,7 @@ title: Batches description: This article describes Batches. ms.reviewer: alexans ms.topic: reference -ms.date: 01/13/2025 +ms.date: 01/22/2025 --- # Batches @@ -18,7 +18,9 @@ A query can include multiple tabular expression statements, as long as they're d ## Examples -[!INCLUDE [help-cluster](../includes/help-cluster-samples-stormevents.md)] +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] + +Consider the following examples: ### Name tabular results diff --git a/data-explorer/kusto/query/let-statement.md b/data-explorer/kusto/query/let-statement.md index df7d3b2e60..44c4982e3d 100644 --- a/data-explorer/kusto/query/let-statement.md +++ b/data-explorer/kusto/query/let-statement.md @@ -65,7 +65,9 @@ To optimize multiple uses of the `let` statement within a single query, see [Opt ## Examples -[!INCLUDE [help-cluster](../includes/help-cluster-samples-stormevents.md)] +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] + +Consider the following examples: ### Define scalar values diff --git a/data-explorer/kusto/query/parse-kv-operator.md b/data-explorer/kusto/query/parse-kv-operator.md index 28a6e544a0..bab970afed 100644 --- a/data-explorer/kusto/query/parse-kv-operator.md +++ b/data-explorer/kusto/query/parse-kv-operator.md @@ -3,7 +3,7 @@ title: parse-kv operator description: Learn how to use the parse-kv operator to represent structured information extracted from a string expression in a key/value form. ms.reviewer: alexans ms.topic: reference -ms.date: 08/11/2024 +ms.date: 01/22/2025 --- # parse-kv operator @@ -52,16 +52,18 @@ The original input tabular expression *T*, extended with columns per specified k > [!NOTE] > -> * If a key doesn't appear in a record, the corresponding column value will either be `null` or an empty string, depending on the column type. +> * If a key doesn't appear in a record, the corresponding column value is either `null` or an empty string, depending on the column type. > * Only keys that are listed in the operator are extracted. > * The first appearance of a key is extracted, and subsequent values are ignored. -> * When extracting keys and values, leading and trailing white spaces are ignored. +> * When you extract keys and values, leading and trailing white spaces are ignored. ## Examples +Consider the following examples: + ### Extraction with well-defined delimiters -In the following example, keys and values are separated by well defined delimiters. These delimeters are comma and colon characters. +In this query, keys and values are separated by well defined delimiters. These delimeters are comma and colon characters. :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] @@ -101,7 +103,7 @@ print str='src=10.1.1.123 dst=10.1.1.124 bytes=125 failure="connection aborted" |--|--|--|--|--| |2021-01-01 10:00:54.0000000| 10.1.1.123| 10.1.1.124| 125| connection aborted| -The following example uses different opening and closing quotes: +This query uses different opening and closing quotes: :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] @@ -221,7 +223,7 @@ print str="2021-01-01T10:00:34 [INFO] ThreadId:458745723, Machine:Node001, Text: ### Extraction using regex -When no delimiters define text structure well enough, regular expression-based extraction can be useful. +When no delimiters define text structure enough, regular expression-based extraction can be useful. :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] diff --git a/data-explorer/kusto/query/parse-operator.md b/data-explorer/kusto/query/parse-operator.md index efade40e5f..949e9469a9 100644 --- a/data-explorer/kusto/query/parse-operator.md +++ b/data-explorer/kusto/query/parse-operator.md @@ -3,7 +3,7 @@ title: parse operator description: Learn how to use the parse operator to parse the value of a string expression into one or more calculated columns. ms.reviewer: alexans ms.topic: reference -ms.date: 08/11/2024 +ms.date: 01/22/2025 monikerRange: "microsoft-fabric || azure-data-explorer || azure-monitor || microsoft-sentinel " --- # parse operator @@ -37,7 +37,7 @@ Evaluates a string expression and parses its value into one or more calculated c > * If the parsed *expression* isn't of type `string`, it will be converted to type `string`. > * Use [`project`](project-operator.md) if you also want to drop or rename some columns. -### Supported kind values +### Supported `kind` values |Text|Description| |--|--| @@ -67,9 +67,12 @@ The input table extended according to the list of columns that are provided to t ## Examples +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] + The `parse` operator provides a streamlined way to `extend` a table by using multiple `extract` applications on the same `string` expression. This result is useful, when the table has a `string` column that contains several values that you want to break into individual columns. For example, a column that's produced by a developer trace ("`printf`"/"`Console.WriteLine`") statement. ### Parse and extend results + In the following example, the column `EventText` of table `Traces` contains strings of the form `Event: NotifySliceRelease (resourceName={0}, totalSlices={1}, sliceNumber={2}, lockTime={3}, releaseTime={4}, previousLockTime={5})`. The operation extends the table with six columns: `resourceName`, `totalSlices`, `sliceNumber`, `lockTime`, `releaseTime`, and `previousLockTime`. @@ -136,6 +139,8 @@ Leads |ebere@relecloud.com|ebere|relecloud.com| |abi@fourthcoffee.com|abi|www.fourthcoffee.com| +## Regex mode examples + ### Regex mode In the following example, regular expressions are used to parse and extract data from the `EventText` column. The extracted data is projected into new fields. diff --git a/data-explorer/kusto/query/parse-where-operator.md b/data-explorer/kusto/query/parse-where-operator.md index dce21ca740..860d290e64 100644 --- a/data-explorer/kusto/query/parse-where-operator.md +++ b/data-explorer/kusto/query/parse-where-operator.md @@ -3,7 +3,7 @@ title: parse-where operator description: Learn how to use the parse-where operator to parse the value of a string expression into one or more calculated columns. ms.reviewer: alexans ms.topic: reference -ms.date: 08/11/2024 +ms.date: 01/20/2025 --- # parse-where operator diff --git a/data-explorer/kusto/query/partition-operator.md b/data-explorer/kusto/query/partition-operator.md index dc4b38805d..32f9ef3f09 100644 --- a/data-explorer/kusto/query/partition-operator.md +++ b/data-explorer/kusto/query/partition-operator.md @@ -3,7 +3,7 @@ title: partition operator description: Learn how to use the partition operator to partition the records of the input table into multiple subtables. ms.reviewer: alexans ms.topic: reference -ms.date: 08/11/2024 +ms.date: 01/22/2025 --- # partition operator @@ -11,7 +11,7 @@ ms.date: 08/11/2024 The partition operator partitions the records of its input table into multiple subtables according to values in a key column. The operator runs a subquery on each subtable, and produces a single output table that is the union of the results of all subqueries. -This operator is useful when you need to perform a subquery only on a subset of rows that belongs to the same partition key, and not query the whole dataset. These subqueries could include aggregate functions, window functions, top *N* and others. +The partition operator is useful when you need to perform a subquery only on a subset of rows that belong to the same partition key, and not a query of the whole dataset. These subqueries could include aggregate functions, window functions, top *N* and others. The partition operator supports several strategies of subquery operation: @@ -36,16 +36,16 @@ The partition operator supports several strategies of subquery operation: | *Column*| `string` | :heavy_check_mark: | The name of a column in *T* whose values determine how to partition the input tabular source.| | *TransformationSubQuery*| `string` | :heavy_check_mark: | A tabular transformation expression. The source is implicitly the subtables produced by partitioning the records of *T*. Each subtable is homogenous on the value of *Column*.

The expression must provide only one tabular result and shouldn't have other types of statements, such as `let` statements.| | *SubQueryWithSource*| `string` | :heavy_check_mark: | A tabular expression that includes its own tabular source, such as a table reference. This syntax is only supported with the [legacy strategy](#legacy-strategy). The subquery can only reference the key column, *Column*, from *T*. To reference the column, use the syntax `toscalar(`*Column*`)`.

The expression must provide only one tabular result and shouldn't have other types of statements, such as `let` statements.| -| *Hints*| `string` | | Zero or more space-separated parameters in the form of: *HintName* `=` *Value* that control the behavior of the operator. See the [supported hints](#supported-hints) per strategy type. +| *Hints*| `string` | | Zero or more space-separated parameters in the form of: *HintName* `=` *Value* that control the behavior of the operator. See the [supported hints](#supported-hints) per strategy type.| ### Supported hints |Hint name|Type|Strategy|Description| |--|--|--|--| |`hint.shufflekey`| `string` | [shuffle](#shuffle-strategy) | The partition key used to run the partition operator with the `shuffle` strategy. | -|`hint.materialized`| `bool` | [legacy](#legacy-strategy) | If set to `true`, will materialize the source of the `partition` operator. The default value is `false`. | +|`hint.materialized`| `bool` | [legacy](#legacy-strategy) | If set to `true`, outputs the source of the `partition` operator. The default value is `false`. | |`hint.concurrency`| `int` | [legacy](#legacy-strategy) | Determines how many partitions to run in parallel. The default value is `16`.| -|`hint.spread`| `int` | [legacy](#legacy-strategy) | Determines how to distribute the partitions among cluster nodes. The default value is `1`.

For example, if there are *N* partitions and the spread hint is set to *P*, then the *N* partitions will be processed by *P* different cluster nodes equally in parallel/sequentially depending on the concurrency hint.| +|`hint.spread`| `int` | [legacy](#legacy-strategy) | Determines how to distribute the partitions among cluster nodes. The default value is `1`.

For example, if there are *N* partitions and the spread hint is set to *P*, then the *N* partitions are processed by *P* different cluster nodes equally, in parallel/sequentially depending on the concurrency hint.| ## Returns @@ -120,10 +120,13 @@ If the subquery is a tabular transformation without a tabular source, the source To use this strategy, specify `hint.strategy=legacy` or omit any other strategy indication. > [!NOTE] -> An error will occur if the partition column, *Column*, contains more than 64 distinct values. +> An error occurs if the partition column, *Column*, contains more than 64 distinct values. ## Examples +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] + +Consider the following examples: ### Find top values @@ -144,7 +147,7 @@ StormEvents ) ``` -**Output** +**Output** |EventType|State|Events|Injuries| |---|---|---|---| @@ -180,7 +183,7 @@ StormEvents ) ``` -**Output** +**Output** |EventType|TotalInjueries| |---|---| @@ -212,7 +215,7 @@ StormEvents | count ``` -**Output** +**Output** |Count| |---| @@ -238,7 +241,7 @@ range x from 1 to 2 step 1 | count ``` -**Output** +**Output** |Count| |---| diff --git a/data-explorer/kusto/query/pattern-statement.md b/data-explorer/kusto/query/pattern-statement.md index d21b6850f6..43e45db85d 100644 --- a/data-explorer/kusto/query/pattern-statement.md +++ b/data-explorer/kusto/query/pattern-statement.md @@ -56,11 +56,11 @@ For more information, see [Working with middle-tier applications](#work-with-mid | *PathArgType* | `string` | | The type of the *PathArgType* argument. Possible values: `string` | | *ArgValue* | `string` | :heavy_check_mark: | The *ArgName* and optional *PathName* tuple values to be mapped to an *expression*. | | *PathValue* | `string` | | The value to map for *PathName*. | -| *expression* | `string` | :heavy_check_mark: | A tabular or lambda expression that references a function returning tabular data. For example: `Logs | where Timestamp > ago(1h)` | +| *expression* | `string` | :heavy_check_mark: | A tabular or lambda expression that references a function returning tabular data. For example: `Logs | where Timestamp > ago(1h)` ## Examples -[!INCLUDE [help-cluster](../includes/help-cluster-samples-stormevents.md)] +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] In these examples, a pattern is defined. diff --git a/data-explorer/kusto/query/print-operator.md b/data-explorer/kusto/query/print-operator.md index 1107e88c82..fe731644de 100644 --- a/data-explorer/kusto/query/print-operator.md +++ b/data-explorer/kusto/query/print-operator.md @@ -1,9 +1,9 @@ --- -title: print operator +title: print operator description: Learn how to use the print operator to output a single row with one or more scalar expression results as columns. ms.reviewer: alexans ms.topic: reference -ms.date: 11/20/2024 +ms.date: 01/20/2025 --- # print operator @@ -30,6 +30,8 @@ A table with one or more columns and a single row. Each column returns the corre ## Examples +Consider the following examples: + ### Print sum and variable value The following example outputs a row with two columns. One column contains the sum of a series of numbers and the other column contains the value of the variable, `x`. diff --git a/data-explorer/kusto/query/query-parameters-statement.md b/data-explorer/kusto/query/query-parameters-statement.md index 3eb7e68154..f7e74d648d 100644 --- a/data-explorer/kusto/query/query-parameters-statement.md +++ b/data-explorer/kusto/query/query-parameters-statement.md @@ -44,7 +44,13 @@ To reference query parameters, the query text, or functions it uses, must first ## Example -[!INCLUDE [help-cluster](../includes/help-cluster-samples-stormevents.md)] +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] + +Consider the following examples: + +### Declare query parameters + + this query retrieves storm events from the *StormEvents* table where the total number of direct and indirect injuries exceeds a specified threshold (default is 90). It then projects the *EpisodeId*, *EventType*, and the total number of injuries for each of these events. :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] From eca7a9a7e96a59c542f7aee38c3da3ec1d8346d3 Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Sun, 26 Jan 2025 19:05:07 +0200 Subject: [PATCH 02/36] replace branch PR 6555 (2) --- data-explorer/kusto/query/range-operator.md | 31 ++++--- data-explorer/kusto/query/reduce-operator.md | 48 ++++++----- .../kusto/query/restrict-statement.md | 2 +- .../kusto/query/sample-distinct-operator.md | 33 ++++++- data-explorer/kusto/query/sample-operator.md | 54 ++++++++++-- data-explorer/kusto/query/scan-operator.md | 16 +++- data-explorer/kusto/query/search-operator.md | 85 +++++++++++++++++-- .../kusto/query/serialize-operator.md | 30 ++++++- data-explorer/kusto/query/set-statement.md | 2 +- data-explorer/kusto/query/shuffle-query.md | 2 + 10 files changed, 251 insertions(+), 52 deletions(-) diff --git a/data-explorer/kusto/query/range-operator.md b/data-explorer/kusto/query/range-operator.md index 0f5418dcf4..1cf1458feb 100644 --- a/data-explorer/kusto/query/range-operator.md +++ b/data-explorer/kusto/query/range-operator.md @@ -3,7 +3,7 @@ title: range operator description: Learn how to use the range operator to generate a single-column table of values. ms.reviewer: alexans ms.topic: reference -ms.date: 01/07/2025 +ms.date: 01/22/2025 --- # range operator @@ -39,6 +39,8 @@ whose values are *start*, *start* `+` *step*, ... up to and until *stop*. ## Examples +Consider the following examples: + ### Range over the past seven days The following example creates a table with entries for the current time stamp extended over the past seven days, once a day. @@ -114,13 +116,13 @@ let MyTimeline = range MyMonthHour from MyMonthStart to now() step StepBy **Output** -| MyMonthHour | MyMonthHourinUnixTime | DateOnly | TimeOnly | -|--------------|------------------------|---------------|------------------------------| -| 2023-02-01 | 00:00:00.0000000 | 1675209600 | 2023-02-01 00:00:00.0000000 | -| 2023-02-01 | 04:32:02.4000000 | 1675225922.4 | 2023-02-01 00:00:00.0000000 | -| 2023-02-01 | 09:04:04.8000000 | 1675242244.8 | 2023-02-01 00:00:00.0000000 | -| 2023-02-01 | 13:36:07.2000000 | 1675258567.2 | 2023-02-01 00:00:00.0000000 | -| ... | ... | ... | ... | +| MyMonthHour | MyMonthHourinUnixTime | DateOnly | TimeOnly | +|--|--|--|--| +| 2023-02-01 | 00:00:00.0000000 | 1675209600 | 2023-02-01 00:00:00.0000000 | +| 2023-02-01 | 04:32:02.4000000 | 1675225922.4 | 2023-02-01 00:00:00.0000000 | +| 2023-02-01 | 09:04:04.8000000 | 1675242244.8 | 2023-02-01 00:00:00.0000000 | +| 2023-02-01 | 13:36:07.2000000 | 1675258567.2 | 2023-02-01 00:00:00.0000000 | +| ... | ... | ... | ... | ### Incremented steps @@ -134,16 +136,19 @@ whose type is `long` and results in values from one to eight incremented by thre ```kusto range Steps from 1 to 8 step 3 +``` + +**Output** | Steps | -|-------| -| 1 | -| 4 | -| 7 | +|--| +| 1 | +| 4 | +| 7 | ### Traces over a time range -The following example shows how the `range` operator can be used to create a dimension table that is used to introduce zeros where the source data has no values. It takes timestamps from the last four hours and counts traces for each one minute interval. When there are no traces for a specific interval, the count is zero. +The following example shows how the `range` operator can be used to create a dimension table that is used to introduce zeros where the source data has no values. It takes timestamps from the last four hours and counts traces for each one-minute interval. When there are no traces for a specific interval, the count is zero. ```kusto range TIMESTAMP from ago(4h) to now() step 1m diff --git a/data-explorer/kusto/query/reduce-operator.md b/data-explorer/kusto/query/reduce-operator.md index b507ef2e09..3e7e129dd1 100644 --- a/data-explorer/kusto/query/reduce-operator.md +++ b/data-explorer/kusto/query/reduce-operator.md @@ -3,7 +3,7 @@ title: reduce operator description: Learn how to use the reduce operator to group a set of strings together based on value similarity. ms.reviewer: alexans ms.topic: reference -ms.date: 08/11/2024 +ms.date: 01/22/2025 --- # reduce operator @@ -44,8 +44,12 @@ For example, the result of `reduce by city` might include: ## Examples +Consider the following examples: + ### Small threshold value +This query generates a range of numbers, creates a new column with concatenated strings and random integers, and then groups the rows by the new column with specific reduction parameters. + :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] > Run the query @@ -65,6 +69,8 @@ range x from 1 to 1000 step 1 ### Large threshold value +This query generates a range of numbers, creates a new column with concatenated strings and random integers, and then groups the rows by the new column with specific reduction parameters. + :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] > Run the query @@ -78,21 +84,23 @@ range x from 1 to 1000 step 1 **Output** -|Pattern |Count|Representative | -|----------------|-----|-----------------| -|MachineLearning*|177|MachineLearningX9| -|MachineLearning*|102|MachineLearningX0| -|MachineLearning*|106|MachineLearningX1| -|MachineLearning*|96|MachineLearningX6| -|MachineLearning*|110|MachineLearningX4| -|MachineLearning*|100|MachineLearningX3| -|MachineLearning*|99|MachineLearningX8| -|MachineLearning*|104|MachineLearningX7| -|MachineLearning*|106|MachineLearningX2| +The result includes only those groups where the MyText value appears in at least 90% of the rows. + +| Pattern | Count | Representative | +|--|--|--| +| MachineLearning* | 177 | MachineLearningX9 | +| MachineLearning* | 102 | MachineLearningX0 | +| MachineLearning* | 106 | MachineLearningX1 | +| MachineLearning* | 96 | MachineLearningX6 | +| MachineLearning* | 110 | MachineLearningX4 | +| MachineLearning* | 100 | MachineLearningX3 | +| MachineLearning* | 99 | MachineLearningX8 | +| MachineLearning* | 104 | MachineLearningX7 | +| MachineLearning* | 106 | MachineLearningX2 | -### Behavior of Characters parameter +### Behavior of `Characters` parameter -If the *Characters* parameter is unspecified, then every non-ascii numeric character becomes a term separator. +If the `Characters` parameter is unspecified, then every non-ascii numeric character becomes a term separator. :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] @@ -105,11 +113,11 @@ range x from 1 to 10 step 1 | project str = strcat("foo", "Z", tostring(x)) | re **Output** -|Pattern|Count|Representative| +| Pattern | Count | Representative | |--|--|--| -|others|10|| +| others | 10 | | -However, if you specify that "Z" is a separator, then it's as if each value in `str` is 2 terms: `foo` and `tostring(x)`: +However, if you specify that "Z" is a separator, then it's as if each value in `str` is two terms: `foo` and `tostring(x)`: :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] @@ -129,12 +137,12 @@ range x from 1 to 10 step 1 | project str = strcat("foo", "Z", tostring(x)) | re ### Apply `reduce` to sanitized input The following example shows how one might apply the `reduce` operator to a "sanitized" -input, in which GUIDs in the column being reduced are replaced prior to reducing +input, in which GUIDs in the column being reduced are replaced before reducing: ```kusto // Start with a few records from the Trace table. Trace | take 10000 -// We will reduce the Text column which includes random GUIDs. +// Then reduce the Text column which includes random GUIDs. // As random GUIDs interfere with the reduce operation, replace them all // by the string "GUID". | extend Text=replace_regex(Text, @"[[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12}", @"GUID") @@ -145,7 +153,7 @@ Trace | take 10000 ## Related content -[autocluster](autocluster-plugin.md) +* [autocluster](autocluster-plugin.md) > [!NOTE] > The implementation of `reduce` operator is largely based on the paper [A Data Clustering Algorithm for Mining Patterns From Event Logs](https://ristov.github.io/publications/slct-ipom03-web.pdf), by Risto Vaarandi. diff --git a/data-explorer/kusto/query/restrict-statement.md b/data-explorer/kusto/query/restrict-statement.md index e7db0058ed..ec4730610e 100644 --- a/data-explorer/kusto/query/restrict-statement.md +++ b/data-explorer/kusto/query/restrict-statement.md @@ -38,7 +38,7 @@ The middle-tier application can prefix the user's query with a **logical model** ## Examples -[!INCLUDE [help-cluster](../includes/help-cluster-samples-stormevents.md)] +Consider the following examples: ### Let statement diff --git a/data-explorer/kusto/query/sample-distinct-operator.md b/data-explorer/kusto/query/sample-distinct-operator.md index d70b42e966..e89dae24f4 100644 --- a/data-explorer/kusto/query/sample-distinct-operator.md +++ b/data-explorer/kusto/query/sample-distinct-operator.md @@ -3,7 +3,7 @@ title: sample-distinct operator description: Learn how to use the sample-distinct operator to return a column that contains up to the specified number of distinct values of the requested columns. ms.reviewer: alexans ms.topic: reference -ms.date: 08/11/2024 +ms.date: 01/21/2025 --- # sample-distinct operator @@ -34,6 +34,8 @@ The operator tries to return an answer as quickly as possible rather than trying ## Examples +[!INCLUDE [help-cluster-note](../includes/help-cluster-note.md)] + Get 10 distinct values from a population :::moniker range="azure-data-explorer" @@ -45,6 +47,20 @@ Get 10 distinct values from a population StormEvents | sample-distinct 10 of EpisodeId ``` +**Output** + +| EpisodeId | +|--| +| 11074 | +| 11078 | +| 11749 | +| 12554 | +| 12561 | +| 13183 | +| 11780 | +| 11781 | +| 12826 | + Sample a population and do further computation without exceeding the query limits in the summarize :::moniker range="azure-data-explorer" @@ -58,3 +74,18 @@ StormEvents | where EpisodeId in (sampleEpisodes) | summarize totalInjuries=sum(InjuriesDirect) by EpisodeId ``` + +**Output** + +| EpisodeId | totalInjuries | +|--|--| +| 11091 | 0 | +| 11074 | 0 | +| 11078 | 0 | +| 11749 | 0 | +| 12554 | 3 | +| 12561 | 0 | +| 13183 | 0 | +| 11780 | 0 | +| 11781 | 0 | +| 12826 | 0 | diff --git a/data-explorer/kusto/query/sample-operator.md b/data-explorer/kusto/query/sample-operator.md index acf67124ec..e4d6cefe84 100644 --- a/data-explorer/kusto/query/sample-operator.md +++ b/data-explorer/kusto/query/sample-operator.md @@ -3,7 +3,7 @@ title: sample operator description: Learn how to use the sample operator to return up to the specified number of rows from the input table. ms.reviewer: alexans ms.topic: reference -ms.date: 08/11/2024 +ms.date: 01/22/2025 --- # sample operator @@ -14,7 +14,7 @@ Returns up to the specified number of random rows from the input table. > [!NOTE] > > * `sample` is geared for speed rather than even distribution of values. Specifically, it means that it will not produce 'fair' results if used after operators that union 2 datasets of different sizes (such as a `union` or `join` operators). It's recommended to use `sample` right after the table reference and filters. -> * `sample` is a non-deterministic operator, and will return different result set each time it is evaluated during the query. For example, the following query yields two different rows (even if one would expect to return the same row twice). +> * `sample` is a non-deterministic operator, and returns a different result set each time it's evaluated during the query. For example, the following query yields two different rows (even if one would expect to return the same row twice). ## Syntax @@ -31,6 +31,14 @@ Returns up to the specified number of random rows from the input table. ## Examples +[!INCLUDE [help-cluster-note](../includes/help-cluster-note.md)] + +Consider the following examples: + +### Generate a sample + +This query creates a range of numbers, samples one value, and then duplicates that sample. + :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] > Run the query @@ -46,8 +54,8 @@ union (_sample), (_sample) | x | | --- | -| 83 | -| 3 | +| 74 | +| 63 | To ensure that in example above `_sample` is calculated once, one can use [materialize()](materialize-function.md) function: @@ -66,8 +74,10 @@ union (_sample), (_sample) | x | | --- | -| 34 | -| 34 | +| 24 | +| 24 | + +### Generate a sample of a certain percentage of data To sample a certain percentage of your data (rather than a specified number of rows), you can use @@ -80,7 +90,23 @@ To sample a certain percentage of your data (rather than a specified number of r StormEvents | where rand() < 0.1 ``` -To sample keys rather than rows (for example - sample 10 Ids and get all rows for these Ids) you can use [`sample-distinct`](sample-distinct-operator.md) in combination with the `in` operator. +**Output** + +The table contains the first few rows of the output. Run the query to view the full result. + +| StartTime | EndTime | EpisodeId | EventId | State | EventType | +|--|--|--|--|--|--| +| 2007-01-01T00:00:00Z | 2007-01-20T10:24:00Z | 2403 | 11914 | INDIANA | Flood | +| 2007-01-01T00:00:00Z | 2007-01-24T18:47:00Z | 2408 | 11930 | INDIANA | Flood | +| 2007-01-01T00:00:00Z | 2007-01-01T12:00:00Z | 1979 | 12631 | DELAWARE | Heavy Rain | +| 2007-01-01T00:00:00Z | 2007-01-01T00:00:00Z | 2592 | 13208 | NORTH CAROLINA | Thunderstorm Wind | +| 2007-01-01T00:00:00Z | 2007-01-31T23:59:00Z | 1492 | 7069 | MINNESOTA | Drought | +| 2007-01-01T00:00:00Z | 2007-01-31T23:59:00Z | 2240 | 10858 | TEXAS | Drought | +|...|...|...|...|...|...| + +### Generate a sample of keys + +To sample keys rather than rows (for example - sample 10 Ids and get all rows for these Ids), you can use [`sample-distinct`](sample-distinct-operator.md) in combination with the `in` operator. :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] @@ -92,3 +118,17 @@ let sampleEpisodes = StormEvents | sample-distinct 10 of EpisodeId; StormEvents | where EpisodeId in (sampleEpisodes) ``` + +**Output** + +The table contains the first few rows of the output. Run the query to view the full result. + +| StartTime | EndTime | EpisodeId | EventId | State | EventType | +|--|--|--|--|--|--| +| 2007-09-18T20:00:00Z | 2007-09-19T18:00:00Z | 11074 | 60904 | FLORIDA | Heavy Rain | +| 2007-09-20T21:57:00Z | 2007-09-20T22:05:00Z | 11078 | 60913 | FLORIDA | Tornado | +| 2007-09-29T08:11:00Z | 2007-09-29T08:11:00Z | 11091 | 61032 | ATLANTIC SOUTH | Waterspout | +| 2007-12-07T14:00:00Z | 2007-12-08T04:00:00Z | 13183 | 73241 | AMERICAN SAMOA | Flash Flood | +| 2007-12-11T21:45:00Z | 2007-12-12T16:45:00Z | 12826 | 70787 | KANSAS | Flood | +| 2007-12-13T09:02:00Z | 2007-12-13T10:30:00Z | 11780 | 64725 | KENTUCKY | Flood | +|...|...|...|...|...|...| diff --git a/data-explorer/kusto/query/scan-operator.md b/data-explorer/kusto/query/scan-operator.md index 77fcff2949..10876e9589 100644 --- a/data-explorer/kusto/query/scan-operator.md +++ b/data-explorer/kusto/query/scan-operator.md @@ -3,7 +3,7 @@ title: scan operator description: Learn how to use the scan operator to scan data, match, and build sequences based on the predicates. ms.reviewer: alexans ms.topic: reference -ms.date: 08/11/2024 +ms.date: 01/22/2025 --- # scan operator @@ -81,6 +81,8 @@ For a detailed example of this logic, see the [scan logic walkthrough](#scan-log ## Examples +[!INCLUDE [help-cluster-note](../includes/help-cluster-note.md)] + ### Cumulative sum Calculate the cumulative sum for an input column. The result of this example is equivalent to using [row_cumsum()](row-cumsum-function.md). @@ -334,6 +336,18 @@ Events ) ``` +**Output** + +| Ts | Event | m_id | +|--|--|--| +| 00:01:00 | Start | 0 | +| 00:02:00 | B | 0 | +| 00:03:00 | D | 0 | +| 00:04:00 | Stop | 0 | +| 00:08:00 | Start | 1 | +| 00:11:00 | E | 1 | +| 00:12:00 | Stop | 1 | + ### The state Think of the state of the `scan` operator as a table with a row for each step, in which each step has its own state. This state contains the latest values of the columns and declared variables from all of the previous steps and the current step. To learn more, see [State](#state). diff --git a/data-explorer/kusto/query/search-operator.md b/data-explorer/kusto/query/search-operator.md index 838bacfcdb..160ff2a670 100644 --- a/data-explorer/kusto/query/search-operator.md +++ b/data-explorer/kusto/query/search-operator.md @@ -3,7 +3,7 @@ title: search operator description: Learn how to use the search operator to search for a text pattern in multiple tables and columns. ms.reviewer: alexans ms.topic: reference -ms.date: 08/11/2024 +ms.date: 01/21/2025 --- # search operator @@ -24,14 +24,14 @@ Searches a text pattern in multiple tables and columns. | Name | Type | Required | Description | |--|--|--|--| -| *T* | `string` | | The tabular data source to be searched over, such as a table name, a [union operator](union-operator.md), or the results of a tabular query. Cannot appear together with *TableSources*.| +| *T* | `string` | | The tabular data source to be searched over, such as a table name, a [union operator](union-operator.md), or the results of a tabular query. Can't appear together with *TableSources*.| | *CaseSensitivity* | `string` | | A flag that controls the behavior of all `string` scalar operators, such as `has`, with respect to case sensitivity. Valid values are `default`, `case_insensitive`, `case_sensitive`. The options `default` and `case_insensitive` are synonymous, since the default behavior is case insensitive.| -| *TableSources* | `string` | | A comma-separated list of "wildcarded" table names to take part in the search. The list has the same syntax as the list of the [union operator](union-operator.md). Cannot appear together with *TabularSource*.| +| *TableSources* | `string` | | A comma-separated list of "wildcarded" table names to take part in the search. The list has the same syntax as the list of the [union operator](union-operator.md). Can't appear together with *TabularSource*.| | *SearchPredicate* | `string` | :heavy_check_mark: | A boolean expression to be evaluated for every record in the input. If it returns `true`, the record is outputted. See [Search predicate syntax](#search-predicate-syntax).| ### Search predicate syntax -The *SearchPredicate* allows you to search for specific terms in all columns of a table. The operator that will be applied to a search term depends on the presence and placement of a wildcard asterisk (`*`) in the term, as shown in the following table. +The *SearchPredicate* allows you to search for specific terms in all columns of a table. The operator that is applied to a search term depends on the presence and placement of a wildcard asterisk (`*`) in the term, as shown in the following table. |Literal |Operator | |----------|-----------| @@ -74,15 +74,19 @@ Use boolean expressions to combine conditions and create more complex searches. ## Remarks -Unlike the [find operator](find-operator.md), the `search` operator does not support the following: +Unlike the [find operator](find-operator.md), the `search` operator doesn't support the following syntax: -1. `withsource=`: The output will always include a column called `$table` of type `string` whose value +1. `withsource=`: The output always includes a column called `$table` of type `string` whose value is the table name from which each record was retrieved (or some system-generated name if the source isn't a table but a composite expression). 2. `project=`, `project-smart`: The output schema is equivalent to `project-smart` output schema. ## Examples +[!INCLUDE [help-cluster-note](../includes/help-cluster-note.md)] + +Consider the following exaples: + ### Global term search Search for a term over all unrestricted tables and views of the database in scope. @@ -96,7 +100,18 @@ Search for a term over all unrestricted tables and views of the database in scop search "Green" ``` -The output contains records from the `Customers`, `Products`, and `SalesTable` tables. The `Customers` records shows all customers with the last name "Green", and the `Products` and `SalesTable` records shows products with some mention of "Green". +The output contains records from the `Customers`, `Products`, and `SalesTable` tables. The `Customers` records show all customers with the last name "Green", and the `Products` and `SalesTable` records shows products with some mention of "Green". + +**Output** + +| $table | CityName | ContinentName | CustomerKey | Education | FirstName | Gender | LastName | +|--|--|--|--|--|--|--|--| +| Customers | Ballard | North America | 16549 | Partial College | Mason | M | Green | +| Customers | Bellingham | North America | 2070 | High School | Adam | M | Green | +| Customers | Bellingham | North America | 10658 | Bachelors | Sara | F | Green | +| Customers | Beverly Hills | North America | 806 | Graduate Degree | Richard | M | Green | +| Customers | Beverly Hills | North America | 7674 | Graduate Degree | James | M | Green | +| Customers | Burbank | North America | 5241 | Graduate Degree | Madeline | F | Green | ### Conditional global term search @@ -111,6 +126,18 @@ Search for records that match both terms over all unrestricted tables and views search "Green" and ("Deluxe" or "Proseware") ``` +**Output** + +| $table | ProductName | Manufacturer | ColorName | ClassName | ProductCategoryName | +|--|--|--|--|--|--| +| Products | Contoso 8GB Clock & Radio MP3 Player X850 Green | Contoso, Ltd | Green | Deluxe | Audio | +| Products | Proseware Scan Jet Digital Flat Bed Scanner M300 Green | Proseware, Inc. | Green | Regular | Computers | +| Products | Proseware All-In-One Photo Printer M200 Green | Proseware, Inc. | Green | Regular | Computers | +| Products | Proseware Ink Jet Wireless All-In-One Printer M400 Green | Proseware, Inc. | Green | Regular | Computers | +| Products | Proseware Ink Jet Instant PDF Sheet-Fed Scanner M300 Green | Proseware, Inc. | Green | Regular | Computers | +| Products | Proseware Desk Jet All-in-One Printer, Scanner, Copier M350 Green | Proseware, Inc. | Green | Regular | Computers | +| Products | Proseware Duplex Scanner M200 Green | Proseware, Inc. | Green | Regular | Computers | + ### Search a specific table Search only in the `Customers` table. @@ -124,6 +151,17 @@ Search only in the `Customers` table. search in (Products) "Green" ``` +**Output** + +| $table | ProductName | Manufacturer | ColorName | +|--|--|--|--| +| Products | Contoso 4G MP3 Player E400 Green | Contoso, Ltd | Green | +| Products | Contoso 8GB Super-Slim MP3/Video Player M800 Green | Contoso, Ltd | Green | +| Products | Contoso 16GB Mp5 Player M1600 Green | Contoso, Ltd | Green | +| Products | Contoso 8GB Clock & Radio MP3 Player X850 Green | Contoso, Ltd | Green | +| Products | NT Wireless Bluetooth Stereo Headphones M402 Green | Northwind Traders | Green | +| Products | NT Wireless Transmitter and Bluetooth Headphones M150 Green | Northwind Traders | Green | + ### Case-sensitive search Search for records that match both case-sensitive terms over all unrestricted tables and views of the database in scope. @@ -137,6 +175,18 @@ Search for records that match both case-sensitive terms over all unrestricted ta search kind=case_sensitive "blue" ``` +**Output** + +| $table | ProductName | Manufacturer | ColorName | ClassName | +|--|--|--|--|--| +| Products | Contoso 16GB New Generation MP5 Player M1650 blue | Contoso, Ltd | blue | Regular | +| Products | Contoso Bright Light battery E20 blue | Contoso, Ltd | blue | Economy | +| Products | Litware 120mm Blue LED Case Fan E901 blue | Litware, Inc. | blue | Economy | +| NewSales | Litware 120mm Blue LED Case Fan E901 blue | Litware, Inc. | blue | Economy | +| NewSales | Litware 120mm Blue LED Case Fan E901 blue | Litware, Inc. | blue | Economy | +| NewSales | Litware 120mm Blue LED Case Fan E901 blue | Litware, Inc. | blue | Economy | +| NewSales | Litware 120mm Blue LED Case Fan E901 blue | Litware, Inc. | blue | Economy | + ### Search specific columns Search for a term in the "FirstName" and "LastName" columns over all unrestricted tables and views of the database in scope. @@ -150,6 +200,17 @@ Search for a term in the "FirstName" and "LastName" columns over all unrestricte search FirstName:"Aaron" or LastName:"Hughes" ``` +**Output** + +| $table | CustomerKey | Education | FirstName | Gender | LastName | +|--|--|--|--|--|--| +| Customers | 18285 | High School | Riley | F | Hughes | +| Customers | 802 | Graduate Degree | Aaron | M | Sharma | +| Customers | 986 | Bachelors | Melanie | F | Hughes | +| Customers | 12669 | High School | Jessica | F | Hughes | +| Customers | 13436 | Graduate Degree | Mariah | F | Hughes | +| Customers | 10152 | Graduate Degree | Aaron | M | Campbell | + ### Limit search by timestamp Search for a term over all unrestricted tables and views of the database in scope if the term appears in a record with a date greater than the given date. @@ -163,6 +224,16 @@ Search for a term over all unrestricted tables and views of the database in scop search "Hughes" and DateKey > datetime('2009-01-01') ``` +**Output** + +| $table | DateKey | SalesAmount_real | +|--|--|--| +| SalesTable | 2021-12-13T00:00:00Z | 446.4715 | +| SalesTable | 2021-12-13T00:00:00Z | 120.555 | +| SalesTable | 2021-12-13T00:00:00Z | 48.4405 | +| SalesTable | 2021-12-13T00:00:00Z | 39.6435 | +| SalesTable | 2021-12-13T00:00:00Z | 56.9905 | + ## Performance Tips |#|Tip|Prefer|Over| diff --git a/data-explorer/kusto/query/serialize-operator.md b/data-explorer/kusto/query/serialize-operator.md index 1969735169..9471233d86 100644 --- a/data-explorer/kusto/query/serialize-operator.md +++ b/data-explorer/kusto/query/serialize-operator.md @@ -3,7 +3,7 @@ title: serialize operator description: Learn how to use the serialize operator to mark the input row set as serialized and ready for window functions. ms.reviewer: alexans ms.topic: reference -ms.date: 08/11/2024 +ms.date: 01/21/2025 --- # serialize operator @@ -28,6 +28,8 @@ The operator has a declarative meaning. It marks the input row set as serialized ## Examples +[!INCLUDE [help-cluster-note](../includes/help-cluster-note.md)] + ### Serialize subset of rows by condition :::moniker range="azure-data-explorer" @@ -41,6 +43,19 @@ TraceLogs | serialize ``` +**Output** + +This table only shows the top 5 query results. + +| Timestamp | Node | Component | ClientRequestId | Message | +|--|--|--|--|--| +| 2014-03-08T12:24:55.5464757Z | Engine000000000757 | INGESTOR_GATEWAY | 5a848f70-9996-eb17-15ed-21b8eb94bf0e | $$IngestionCommand table=fogEvents format=json | +| 2014-03-08T12:24:56.0929514Z | Engine000000000757 | DOWNLOADER | 5a848f70-9996-eb17-15ed-21b8eb94bf0e | Downloading file path: ""https://benchmarklogs3.blob.core.windows.net/benchmark/2014/IMAGINEFIRST0_1399_0.json.gz"" | +| 2014-03-08T12:25:40.3574831Z | Engine000000000341 | INGESTOR_EXECUTER | 5a848f70-9996-eb17-15ed-21b8eb94bf0e | IngestionCompletionEvent: finished ingestion file path: ""https://benchmarklogs3.blob.core.windows.net/benchmark/2014/IMAGINEFIRST0_1399_0.json.gz"" | +| 2014-03-08T12:25:40.9039588Z | Engine000000000341 | DOWNLOADER | 5a848f70-9996-eb17-15ed-21b8eb94bf0e | Downloading file path: ""https://benchmarklogs3.blob.core.windows.net/benchmark/2014/IMAGINEFIRST0_1399_1.json.gz"" | +| 2014-03-08T12:26:25.1684905Z | Engine000000000057 | INGESTOR_EXECUTER | 5a848f70-9996-eb17-15ed-21b8eb94bf0e | IngestionCompletionEvent: finished ingestion file path: ""https://benchmarklogs3.blob.core.windows.net/benchmark/2014/IMAGINEFIRST0_1399_1.json.gz"" | +|...|...|...|...|...| + ### Add row number to the serialized table To add a row number to the serialized table, use the [row_number()](row-number-function.md) function. @@ -56,6 +71,19 @@ TraceLogs | serialize rn = row_number() ``` +**Output** + +This table only shows the top 5 query results. + +| Timestamp | rn | Node | Component | ClientRequestId | Message | +|--|--|--|--|--|--| +| 2014-03-08T13:00:01.6638235Z | 1 | Engine000000000899 | INGESTOR_EXECUTER | 5a848f70-9996-eb17-15ed-21b8eb94bf0e | IngestionCompletionEvent: finished ingestion file path: ""https://benchmarklogs3.blob.core.windows.net/benchmark/2014/IMAGINEFIRST0_1399_46.json.gz"" | +| 2014-03-08T13:00:02.2102992Z | 2 | Engine000000000899 | DOWNLOADER | 5a848f70-9996-eb17-15ed-21b8eb94bf0e | Downloading file path: ""https://benchmarklogs3.blob.core.windows.net/benchmark/2014/IMAGINEFIRST0_1399_47.json.gz"" | +| 2014-03-08T13:00:46.4748309Z | 3 | Engine000000000584 | INGESTOR_EXECUTER | 5a848f70-9996-eb17-15ed-21b8eb94bf0e | IngestionCompletionEvent: finished ingestion file path: ""https://benchmarklogs3.blob.core.windows.net/benchmark/2014/IMAGINEFIRST0_1399_47.json.gz"" | +| 2014-03-08T13:00:47.0213066Z | 4 | Engine000000000584 | DOWNLOADER | 5a848f70-9996-eb17-15ed-21b8eb94bf0e | Downloading file path: ""https://benchmarklogs3.blob.core.windows.net/benchmark/2014/IMAGINEFIRST0_1399_48.json.gz"" | +| 2014-03-08T13:01:31.2858383Z | 5 | Engine000000000380 | INGESTOR_EXECUTER | 5a848f70-9996-eb17-15ed-21b8eb94bf0e | IngestionCompletionEvent: finished ingestion file path: ""https://benchmarklogs3.blob.core.windows.net/benchmark/2014/IMAGINEFIRST0_1399_48.json.gz"" | +|...|...|...|...|...| + ## Serialization behavior of operators The output row set of the following operators is marked as serialized. diff --git a/data-explorer/kusto/query/set-statement.md b/data-explorer/kusto/query/set-statement.md index 4aceeb710a..49d2095c28 100644 --- a/data-explorer/kusto/query/set-statement.md +++ b/data-explorer/kusto/query/set-statement.md @@ -35,7 +35,7 @@ Request properties aren't formally a part of the Kusto Query Language and may be ## Example -[!INCLUDE [help-cluster](../includes/help-cluster-samples-stormevents.md)] +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] ```kusto set querytrace; diff --git a/data-explorer/kusto/query/shuffle-query.md b/data-explorer/kusto/query/shuffle-query.md index d12ee17050..4235c8a649 100644 --- a/data-explorer/kusto/query/shuffle-query.md +++ b/data-explorer/kusto/query/shuffle-query.md @@ -69,6 +69,8 @@ In some cases, the `hint.strategy = shuffle` is ignored, and the query won't run ## Examples +[!INCLUDE [help-cluster-note](../includes/help-cluster-note.md)] + ## Use summarize with shuffle The `shuffle` strategy query with `summarize` operator shares the load on all cluster nodes, where each node processes one partition of the data. From 41d3a8c99e588ea62fe9723d5bf83a5a4e4bb1fa Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Tue, 28 Jan 2025 17:03:27 +0200 Subject: [PATCH 03/36] Replace help cluster include --- .../kusto/includes/help-cluster-samples-stormevents.md | 6 ------ data-explorer/kusto/query/let-statement.md | 2 +- data-explorer/kusto/query/pattern-statement.md | 2 +- data-explorer/kusto/query/query-parameters-statement.md | 2 +- data-explorer/kusto/query/tabular-expression-statements.md | 2 +- 5 files changed, 4 insertions(+), 10 deletions(-) delete mode 100644 data-explorer/kusto/includes/help-cluster-samples-stormevents.md diff --git a/data-explorer/kusto/includes/help-cluster-samples-stormevents.md b/data-explorer/kusto/includes/help-cluster-samples-stormevents.md deleted file mode 100644 index 3b2b161c57..0000000000 --- a/data-explorer/kusto/includes/help-cluster-samples-stormevents.md +++ /dev/null @@ -1,6 +0,0 @@ ---- -ms.topic: include -ms.date: 01/13/2025 ---- - -In the [help cluster](https://dataexplorer.azure.com/clusters/help/), there's a `Samples` database with a `StormEvents` table. \ No newline at end of file diff --git a/data-explorer/kusto/query/let-statement.md b/data-explorer/kusto/query/let-statement.md index df7d3b2e60..f76b5de2b0 100644 --- a/data-explorer/kusto/query/let-statement.md +++ b/data-explorer/kusto/query/let-statement.md @@ -65,7 +65,7 @@ To optimize multiple uses of the `let` statement within a single query, see [Opt ## Examples -[!INCLUDE [help-cluster](../includes/help-cluster-samples-stormevents.md)] +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] ### Define scalar values diff --git a/data-explorer/kusto/query/pattern-statement.md b/data-explorer/kusto/query/pattern-statement.md index d21b6850f6..c93378ff9e 100644 --- a/data-explorer/kusto/query/pattern-statement.md +++ b/data-explorer/kusto/query/pattern-statement.md @@ -60,7 +60,7 @@ For more information, see [Working with middle-tier applications](#work-with-mid ## Examples -[!INCLUDE [help-cluster](../includes/help-cluster-samples-stormevents.md)] +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] In these examples, a pattern is defined. diff --git a/data-explorer/kusto/query/query-parameters-statement.md b/data-explorer/kusto/query/query-parameters-statement.md index 3eb7e68154..694f38fbc3 100644 --- a/data-explorer/kusto/query/query-parameters-statement.md +++ b/data-explorer/kusto/query/query-parameters-statement.md @@ -44,7 +44,7 @@ To reference query parameters, the query text, or functions it uses, must first ## Example -[!INCLUDE [help-cluster](../includes/help-cluster-samples-stormevents.md)] +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] diff --git a/data-explorer/kusto/query/tabular-expression-statements.md b/data-explorer/kusto/query/tabular-expression-statements.md index 3b5d738cb5..4afdee8191 100644 --- a/data-explorer/kusto/query/tabular-expression-statements.md +++ b/data-explorer/kusto/query/tabular-expression-statements.md @@ -41,7 +41,7 @@ A tabular data source produces sets of records, to be further processed by tabul ## Examples -[!INCLUDE [help-cluster](../includes/help-cluster-samples-stormevents.md)] +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] ### Filter rows by condition From 7cee0001483a18bf734ba7620540959a4d86ecf1 Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Tue, 28 Jan 2025 17:26:24 +0200 Subject: [PATCH 04/36] reset file --- .github/pull_request_template.md | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 9b5681847d..c9787e417b 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -1,12 +1,6 @@ -# Thank you for contributing to Azure Data Explorer documentation +# Thank you for contributing to Kusto documentation -## Fill out these items before submitting your pull request: - -If you are working internally at Microsoft: - -- **Provide a link to an Azure DevOps Boards work item that tracks this feature/update.** - -- **Who is your Docs team contact?** \@mention them individually tag them and let them review the PR before signing off. +Please add a brief comment outlining the purpose of this PR. Add links to any relevant references such as DevOps work items. ## Make sure you've done the following: From dd088dfdf28e91c1d765d91a70f817b17c462442 Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Tue, 28 Jan 2025 17:29:09 +0200 Subject: [PATCH 05/36] reset file --- data-explorer/kusto/access-control/index.md | 48 ++++++++++++++++++--- 1 file changed, 42 insertions(+), 6 deletions(-) diff --git a/data-explorer/kusto/access-control/index.md b/data-explorer/kusto/access-control/index.md index 5817c6017f..0e2026fb0d 100644 --- a/data-explorer/kusto/access-control/index.md +++ b/data-explorer/kusto/access-control/index.md @@ -1,10 +1,10 @@ --- title: Access Control Overview description: This article describes Access control. -ms.reviewer: orspodek +ms.reviewer: yogilad ms.topic: reference ms.custom: has-adal-ref -ms.date: 08/11/2024 +ms.date: 01/26/2025 --- # Access control overview @@ -15,7 +15,7 @@ Access control is based on authentication and authorization. Each query and comm ::: moniker-end :::moniker range="microsoft-fabric" -Access control is based on authentication and authorization. Each query and command on a Fabric resource, such as a KQL database, must pass both authentication and authorization checks. +Access control is based on authentication and authorization. Each query and command on a Fabric resource, such as a database, must pass both authentication and authorization checks. ::: moniker-end * [Authentication](#authentication): Validates the identity of the security principal making a request @@ -33,7 +33,7 @@ The main authentication scenarios are as follows: * [Single page application (SPA) authentication](/azure/active-directory/develop/msal-authentication-flows#authorization-code): Allows client-side SPA web applications to sign in users and get tokens to access your database. This flow must be implemented with MSAL. > [!NOTE] -> For user and application authentication, we recommend using the [Kusto client libraries](../api/client-libraries.md). If you require On-behalf-of (OBO) or Single-Page Application (SPA) authentication, you'll need to use MSAL directly as these flows aren't supported by the client libraries. For more information, see [Authenticate with Microsoft Authentication Library (MSAL)](../api/rest/authenticate-with-msal.md). +> For user and application authentication, we recommend using the [Kusto client libraries](../api/client-libraries.md). If you require On-behalf-of (OBO) or Single-Page Application (SPA) authentication, you must use MSAL directly as the client libraries don't support these flows. For more information, see [Authenticate with Microsoft Authentication Library (MSAL)](../api/rest/authenticate-with-msal.md). ### User authentication @@ -70,16 +70,52 @@ The association of security principals to security roles can be defined individu ## Group authorization -Authorization can be granted to Microsoft Entra ID groups by assigning one or more roles to the group. +Authorization can be granted to Microsoft Entra ID groups by assigning one or more roles to the group. -When the authorization of a user or application principal is checked, the system first checks for an explicit role assignment permitting the specific action. If no such role assignment exists, the system then analyzes the principal's membership across all groups that could potentially authorize the action. If the principal is confirmed to be a member of any of these groups, the requested action is authorized. Otherwise, if the principal is not a member of any such groups, the action doesn't pass the authorization check and the action isn't allowed. +When checking authorization for a user or application principal, the system first looks for an explicit role assignment that permits the specific action. If the role assignment doesn't exists, then the system checks the principal's membership in all groups that could authorize the action. + +If the principal is a member of a group with appropriate permissions, the requested action is authorized. Otherwise, the action doesn't pass the authorization check and is disallowed. > [!NOTE] > > [!INCLUDE [Cached Group Membership](../includes/cached-group-membership.md)] +### Force group membership refresh + +Principals can force a refresh of group membership information. This capability is useful in scenarios where just-in-time (JIT) privileged access services, such as Microsoft Entra Privileged Identity Management (PIM), are used to obtain higher privileges on a resource. + +#### Refresh for a specific group + +Principals can force a refresh of group membership *for a specific group*. However, the following restrictions apply: + +* A refresh can be requested up to 10 times per hour per principal. +* The requesting principal must be a member of the group at the time of the request. + +The request results in an error if either of these conditions aren't met. + +To reevaluate the current principal's membership of a group, run the following command: + +```kusto +.clear cluster cache groupmembership with (group='') +``` + +Use the group's fully qualified name (FQN). For more information, see [Referencing Microsoft Entra principals and groups](../management/reference-security-principals.md#referencing-microsoft-entra-principals-and-groups). + +#### Refresh for other principals + +A privileged principal can request a refresh **for other principals**. The requesting principal must have [AllDatabaseMonitor](role-based-access-control.md) access for the target service. Privileged principals can also run the previous command without restrictions. + +To refresh another principal’s group membership, run the following command: + +> In the following command, replace `` with your own principal fully qualified name (FQN) and `` with your own group FQN. For more information, see [Referencing Microsoft Entra principals and groups](../management/reference-security-principals.md#referencing-microsoft-entra-principals-and-groups). + +```kusto +.clear cluster cache groupmembership with (principal='', group='') +``` + ## Related content * Understand [Kusto role-based access control](role-based-access-control.md). * For user or application authentication, use the [Kusto client libraries](../api/client-libraries.md). * For OBO or SPA authentication, see [How to authenticate with Microsoft Authentication Library (MSAL)](../api/rest/authenticate-with-msal.md). +* For referencing principals and groups, see [Referencing Microsoft Entra principals and groups](../management/reference-security-principals.md). From 9a3f3ce2abe35e95894dc83bd2e24d3701bd775b Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Tue, 28 Jan 2025 17:30:57 +0200 Subject: [PATCH 06/36] reset files --- data-explorer/kusto/management/row-level-security-policy.md | 1 + .../security-network-managed-private-endpoint-create.md | 3 +++ data-explorer/security-network-private-endpoint.md | 3 +++ 3 files changed, 7 insertions(+) diff --git a/data-explorer/kusto/management/row-level-security-policy.md b/data-explorer/kusto/management/row-level-security-policy.md index 29dccbaf0d..30c0c4c478 100644 --- a/data-explorer/kusto/management/row-level-security-policy.md +++ b/data-explorer/kusto/management/row-level-security-policy.md @@ -41,6 +41,7 @@ For more information, see [management commands for managing the Row Level Securi * When it's referenced by an [update policy](update-policy.md) query, while the update policy is not configured with a managed identity. * When it's referenced by a [continuous export](../management/data-export/continuous-data-export.md) that uses an authentication method other than impersonation. * When a [restricted view access policy](restricted-view-access-policy.md) is configured for the table. +* The RLS query can't reference other tables that have Row Level Security policy enabled. * The RLS query can't reference tables located in other databases. ## Examples diff --git a/data-explorer/security-network-managed-private-endpoint-create.md b/data-explorer/security-network-managed-private-endpoint-create.md index e06bee9cf9..5a5aa3ae76 100644 --- a/data-explorer/security-network-managed-private-endpoint-create.md +++ b/data-explorer/security-network-managed-private-endpoint-create.md @@ -10,6 +10,9 @@ ms.date: 11/18/2024 Managed private endpoints are required to connect to Azure resources that are highly protected. They're one-way private connections that allow Azure Data Explorer to connect to other protected services. In this article, you'll learn how to create a managed private endpoint and connect it to your data source. +> [!NOTE] +> We recommend using Managed Identity connect to [Azure Storage](/azure/storage/common/storage-network-security?tabs=azure-portal#grant-access-to-trusted-azure-services) and [Azure Event Hubs](/azure/event-hubs/event-hubs-ip-filtering#trusted-microsoft-services) instead of managed private endpoints. To connect using managed identities, configure the Azure Storage or Event Hubs resources to recognize Azure Data Explorer as a trusted service. Then, use [Managed Identity](/azure/data-explorer/managed-identities-overview) to grant access by creating a network rule exception for trusted Azure services. + ## Prerequisites * An Azure subscription. Create a [free Azure account](https://azure.microsoft.com/free/). diff --git a/data-explorer/security-network-private-endpoint.md b/data-explorer/security-network-private-endpoint.md index 2e705307c2..c79be7777f 100644 --- a/data-explorer/security-network-private-endpoint.md +++ b/data-explorer/security-network-private-endpoint.md @@ -78,6 +78,9 @@ You can use a managed private endpoint to either enable the cluster to securely :::image type="content" source="media/security-network-private-endpoint/pe-mpe.png" alt-text="Diagram showing the schema of the managed private endpoint architecture."::: +> [!NOTE] +> We recommend using Managed Identity connect to [Azure Storage](/azure/storage/common/storage-network-security?tabs=azure-portal#grant-access-to-trusted-azure-services) and [Azure Event Hubs](/azure/event-hubs/event-hubs-ip-filtering#trusted-microsoft-services) instead of managed private endpoints. To connect using managed identities, configure the Azure Storage or Event Hubs resources to recognize Azure Data Explorer as a trusted service. Then, use [Managed Identity](/azure/data-explorer/managed-identities-overview) to grant access by creating a network rule exception for trusted Azure services.``` + ### Supported services Azure Data Explorer supports creating managed private endpoints to the following services: From bb034898a0d6f48783767510c7592b2997152eca Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Tue, 28 Jan 2025 17:55:08 +0200 Subject: [PATCH 07/36] smaill fixes --- data-explorer/kusto/query/let-statement.md | 2 +- data-explorer/kusto/query/partition-operator.md | 2 +- data-explorer/kusto/query/pattern-statement.md | 2 +- data-explorer/kusto/query/print-operator.md | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/data-explorer/kusto/query/let-statement.md b/data-explorer/kusto/query/let-statement.md index 44c4982e3d..69d0573d21 100644 --- a/data-explorer/kusto/query/let-statement.md +++ b/data-explorer/kusto/query/let-statement.md @@ -67,7 +67,7 @@ To optimize multiple uses of the `let` statement within a single query, see [Opt [!INCLUDE [help-cluster](../includes/help-cluster-note.md)] -Consider the following examples: +Review the examples and run them in your Data Explorer query page. ### Define scalar values diff --git a/data-explorer/kusto/query/partition-operator.md b/data-explorer/kusto/query/partition-operator.md index 32f9ef3f09..8dcad5686c 100644 --- a/data-explorer/kusto/query/partition-operator.md +++ b/data-explorer/kusto/query/partition-operator.md @@ -126,7 +126,7 @@ To use this strategy, specify `hint.strategy=legacy` or omit any other strategy [!INCLUDE [help-cluster](../includes/help-cluster-note.md)] -Consider the following examples: +Review the examples and run them in your Data Explorer query page. ### Find top values diff --git a/data-explorer/kusto/query/pattern-statement.md b/data-explorer/kusto/query/pattern-statement.md index 43e45db85d..c8a143368f 100644 --- a/data-explorer/kusto/query/pattern-statement.md +++ b/data-explorer/kusto/query/pattern-statement.md @@ -56,7 +56,7 @@ For more information, see [Working with middle-tier applications](#work-with-mid | *PathArgType* | `string` | | The type of the *PathArgType* argument. Possible values: `string` | | *ArgValue* | `string` | :heavy_check_mark: | The *ArgName* and optional *PathName* tuple values to be mapped to an *expression*. | | *PathValue* | `string` | | The value to map for *PathName*. | -| *expression* | `string` | :heavy_check_mark: | A tabular or lambda expression that references a function returning tabular data. For example: `Logs | where Timestamp > ago(1h)` +| *expression* | `string` | :heavy_check_mark: | A tabular or lambda expression that references a function returning tabular data. For example: `Logs | where Timestamp > ago(1h)`| ## Examples diff --git a/data-explorer/kusto/query/print-operator.md b/data-explorer/kusto/query/print-operator.md index fe731644de..36802d8e61 100644 --- a/data-explorer/kusto/query/print-operator.md +++ b/data-explorer/kusto/query/print-operator.md @@ -30,7 +30,7 @@ A table with one or more columns and a single row. Each column returns the corre ## Examples -Consider the following examples: +Review the examples and run them in your Data Explorer query page. ### Print sum and variable value From d555244c75511a0e6b2ae94b1cf70de7756fa9e5 Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Tue, 28 Jan 2025 18:29:23 +0200 Subject: [PATCH 08/36] small change --- data-explorer/kusto/query/range-operator.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/query/range-operator.md b/data-explorer/kusto/query/range-operator.md index 1cf1458feb..54d58ae7b9 100644 --- a/data-explorer/kusto/query/range-operator.md +++ b/data-explorer/kusto/query/range-operator.md @@ -39,7 +39,7 @@ whose values are *start*, *start* `+` *step*, ... up to and until *stop*. ## Examples -Consider the following examples: +Review the examples and run them in your Data Explorer query page. ### Range over the past seven days From 43cad46b1432807cee141ff742f9d4ad8872cd29 Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Wed, 29 Jan 2025 14:32:24 +0200 Subject: [PATCH 09/36] added example description --- data-explorer/kusto/query/print-operator.md | 2 -- data-explorer/kusto/query/set-statement.md | 2 ++ 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/data-explorer/kusto/query/print-operator.md b/data-explorer/kusto/query/print-operator.md index 36802d8e61..eb679f2e79 100644 --- a/data-explorer/kusto/query/print-operator.md +++ b/data-explorer/kusto/query/print-operator.md @@ -30,8 +30,6 @@ A table with one or more columns and a single row. Each column returns the corre ## Examples -Review the examples and run them in your Data Explorer query page. - ### Print sum and variable value The following example outputs a row with two columns. One column contains the sum of a series of numbers and the other column contains the value of the variable, `x`. diff --git a/data-explorer/kusto/query/set-statement.md b/data-explorer/kusto/query/set-statement.md index 49d2095c28..0d3c42d949 100644 --- a/data-explorer/kusto/query/set-statement.md +++ b/data-explorer/kusto/query/set-statement.md @@ -35,6 +35,8 @@ Request properties aren't formally a part of the Kusto Query Language and may be ## Example +This query enables query tracing and then fetches the first 100 records from the Events table. + [!INCLUDE [help-cluster](../includes/help-cluster-note.md)] ```kusto From b109644a7bc6efd185b42f0bf8c3881ec79175bd Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Wed, 29 Jan 2025 15:25:33 +0200 Subject: [PATCH 10/36] removed sentence --- data-explorer/kusto/query/partition-operator.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/data-explorer/kusto/query/partition-operator.md b/data-explorer/kusto/query/partition-operator.md index 8dcad5686c..8f020d6148 100644 --- a/data-explorer/kusto/query/partition-operator.md +++ b/data-explorer/kusto/query/partition-operator.md @@ -126,8 +126,6 @@ To use this strategy, specify `hint.strategy=legacy` or omit any other strategy [!INCLUDE [help-cluster](../includes/help-cluster-note.md)] -Review the examples and run them in your Data Explorer query page. - ### Find top values In some cases, it's more performant and easier to write a query using the `partition` operator than using the [`top-nested`](top-nested-operator.md) operator. The following query runs a subquery calculating `summarize` and `top` for each `State` starting with `W`: "WYOMING", "WASHINGTON", "WEST VIRGINIA", and "WISCONSIN". From 89b13a6dbf023df71200227e5dfb4c781eb2babc Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Wed, 29 Jan 2025 15:31:08 +0200 Subject: [PATCH 11/36] removed sentence --- data-explorer/kusto/query/range-operator.md | 2 -- data-explorer/kusto/query/reduce-operator.md | 2 -- 2 files changed, 4 deletions(-) diff --git a/data-explorer/kusto/query/range-operator.md b/data-explorer/kusto/query/range-operator.md index 54d58ae7b9..01d78473e7 100644 --- a/data-explorer/kusto/query/range-operator.md +++ b/data-explorer/kusto/query/range-operator.md @@ -39,8 +39,6 @@ whose values are *start*, *start* `+` *step*, ... up to and until *stop*. ## Examples -Review the examples and run them in your Data Explorer query page. - ### Range over the past seven days The following example creates a table with entries for the current time stamp extended over the past seven days, once a day. diff --git a/data-explorer/kusto/query/reduce-operator.md b/data-explorer/kusto/query/reduce-operator.md index 3e7e129dd1..b589aa3040 100644 --- a/data-explorer/kusto/query/reduce-operator.md +++ b/data-explorer/kusto/query/reduce-operator.md @@ -44,8 +44,6 @@ For example, the result of `reduce by city` might include: ## Examples -Consider the following examples: - ### Small threshold value This query generates a range of numbers, creates a new column with concatenated strings and random integers, and then groups the rows by the new column with specific reduction parameters. From af6fe33aec163875bd84e6e00141188a8128b0f5 Mon Sep 17 00:00:00 2001 From: "Daniel Dror (Dubovski)" Date: Fri, 31 Jan 2025 14:22:59 -0800 Subject: [PATCH 12/36] Update callout-policy.md callout policy is wrong. see https://learn.microsoft.com/en-us/kusto/query/ai-embed-text-plugin?view=azure-data-explorer#configure-managed-identity-and-callout-policies for reference --- data-explorer/kusto/management/callout-policy.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/management/callout-policy.md b/data-explorer/kusto/management/callout-policy.md index e4c417ea38..fd71080034 100644 --- a/data-explorer/kusto/management/callout-policy.md +++ b/data-explorer/kusto/management/callout-policy.md @@ -38,7 +38,7 @@ Callout policies are managed at cluster-level and are classified into the follow | sandbox_artifacts | Controls sandboxed plugins ([python](../query/python-plugin.md) and [R](../query/r-plugin.md)). | | external_data | Controls access to external data through [external tables](../query/schema-entities/external-tables.md) or [externaldata](../query/externaldata-operator.md) operator. | | webapi | Controls access to http endpoints. | -| ai_embed_text | Controls the [ai_embed_text plugin)](../query/ai-embed-text-plugin.md). | +| azure_openai | Controls calls to Azure OpenAI plugins (e.g. the embedding plugin [ai_embed_text plugin)](../query/ai-embed-text-plugin.md). | ## Predefined callout policies From 747c03df5587f0347198dd8bc66bd9a92d691b28 Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Mon, 3 Feb 2025 17:53:31 +0200 Subject: [PATCH 13/36] Add introductory examples to multiple Kusto query operator docs. --- data-explorer/kusto/query/parse-kv-operator.md | 3 ++- data-explorer/kusto/query/parse-operator.md | 2 ++ data-explorer/kusto/query/parse-where-operator.md | 4 ++++ data-explorer/kusto/query/partition-operator.md | 3 +++ data-explorer/kusto/query/pattern-statement.md | 3 ++- data-explorer/kusto/query/print-operator.md | 4 ++++ data-explorer/kusto/query/query-parameters-statement.md | 4 ++-- 7 files changed, 19 insertions(+), 4 deletions(-) diff --git a/data-explorer/kusto/query/parse-kv-operator.md b/data-explorer/kusto/query/parse-kv-operator.md index bab970afed..e17a59a07f 100644 --- a/data-explorer/kusto/query/parse-kv-operator.md +++ b/data-explorer/kusto/query/parse-kv-operator.md @@ -59,7 +59,8 @@ The original input tabular expression *T*, extended with columns per specified k ## Examples -Consider the following examples: +The examples in this section show how to use the syntax to help you get started. + ### Extraction with well-defined delimiters diff --git a/data-explorer/kusto/query/parse-operator.md b/data-explorer/kusto/query/parse-operator.md index 949e9469a9..446017ca70 100644 --- a/data-explorer/kusto/query/parse-operator.md +++ b/data-explorer/kusto/query/parse-operator.md @@ -67,6 +67,8 @@ The input table extended according to the list of columns that are provided to t ## Examples +The examples in this section show how to use the syntax to help you get started. + [!INCLUDE [help-cluster](../includes/help-cluster-note.md)] The `parse` operator provides a streamlined way to `extend` a table by using multiple `extract` applications on the same `string` expression. This result is useful, when the table has a `string` column that contains several values that you want to break into individual columns. For example, a column that's produced by a developer trace ("`printf`"/"`Console.WriteLine`") statement. diff --git a/data-explorer/kusto/query/parse-where-operator.md b/data-explorer/kusto/query/parse-where-operator.md index 860d290e64..1ca66e3ed3 100644 --- a/data-explorer/kusto/query/parse-where-operator.md +++ b/data-explorer/kusto/query/parse-where-operator.md @@ -70,6 +70,10 @@ The input table, which is extended according to the list of columns that are pro ## Examples +The examples in this section show how to use the syntax to help you get started. + +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] + The `parse-where` operator provides a streamlined way to `extend` a table by using multiple `extract` applications on the same `string` expression. This is most useful when the table has a `string` column that contains several values that you want to break into individual columns. For example, you can break up a column that was produced by a developer trace ("`printf`"/"`Console.WriteLine`") statement. ### Using `parse` diff --git a/data-explorer/kusto/query/partition-operator.md b/data-explorer/kusto/query/partition-operator.md index 8f020d6148..12f8f03f88 100644 --- a/data-explorer/kusto/query/partition-operator.md +++ b/data-explorer/kusto/query/partition-operator.md @@ -124,8 +124,11 @@ To use this strategy, specify `hint.strategy=legacy` or omit any other strategy ## Examples +The examples in this section show how to use the syntax to help you get started. + [!INCLUDE [help-cluster](../includes/help-cluster-note.md)] + ### Find top values In some cases, it's more performant and easier to write a query using the `partition` operator than using the [`top-nested`](top-nested-operator.md) operator. The following query runs a subquery calculating `summarize` and `top` for each `State` starting with `W`: "WYOMING", "WASHINGTON", "WEST VIRGINIA", and "WISCONSIN". diff --git a/data-explorer/kusto/query/pattern-statement.md b/data-explorer/kusto/query/pattern-statement.md index c8a143368f..79a50b1b44 100644 --- a/data-explorer/kusto/query/pattern-statement.md +++ b/data-explorer/kusto/query/pattern-statement.md @@ -60,9 +60,10 @@ For more information, see [Working with middle-tier applications](#work-with-mid ## Examples +The examples in this section show how to use the syntax to help you get started. + [!INCLUDE [help-cluster](../includes/help-cluster-note.md)] -In these examples, a pattern is defined. ### Define a simple pattern diff --git a/data-explorer/kusto/query/print-operator.md b/data-explorer/kusto/query/print-operator.md index eb679f2e79..6c1b280ecc 100644 --- a/data-explorer/kusto/query/print-operator.md +++ b/data-explorer/kusto/query/print-operator.md @@ -30,6 +30,10 @@ A table with one or more columns and a single row. Each column returns the corre ## Examples +The examples in this section show how to use the syntax to help you get started. + +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] + ### Print sum and variable value The following example outputs a row with two columns. One column contains the sum of a series of numbers and the other column contains the value of the variable, `x`. diff --git a/data-explorer/kusto/query/query-parameters-statement.md b/data-explorer/kusto/query/query-parameters-statement.md index f7e74d648d..b8566ea1cd 100644 --- a/data-explorer/kusto/query/query-parameters-statement.md +++ b/data-explorer/kusto/query/query-parameters-statement.md @@ -44,9 +44,9 @@ To reference query parameters, the query text, or functions it uses, must first ## Example -[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] +The examples in this section show how to use the syntax to help you get started. -Consider the following examples: +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] ### Declare query parameters From ad72002f4886410a54d4ae7fceebecc6c023a666 Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Mon, 3 Feb 2025 18:07:32 +0200 Subject: [PATCH 14/36] Add introductory example sentence --- data-explorer/kusto/query/range-operator.md | 4 ++++ data-explorer/kusto/query/reduce-operator.md | 4 ++++ data-explorer/kusto/query/sample-distinct-operator.md | 8 +++++--- data-explorer/kusto/query/sample-operator.md | 6 +++--- data-explorer/kusto/query/scan-operator.md | 4 +++- data-explorer/kusto/query/search-operator.md | 6 +++--- data-explorer/kusto/query/serialize-operator.md | 4 +++- data-explorer/kusto/query/shuffle-query.md | 4 +++- 8 files changed, 28 insertions(+), 12 deletions(-) diff --git a/data-explorer/kusto/query/range-operator.md b/data-explorer/kusto/query/range-operator.md index 01d78473e7..ba95ffde4d 100644 --- a/data-explorer/kusto/query/range-operator.md +++ b/data-explorer/kusto/query/range-operator.md @@ -39,6 +39,10 @@ whose values are *start*, *start* `+` *step*, ... up to and until *stop*. ## Examples +The example in this section shows how to use the syntax to help you get started. + +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] + ### Range over the past seven days The following example creates a table with entries for the current time stamp extended over the past seven days, once a day. diff --git a/data-explorer/kusto/query/reduce-operator.md b/data-explorer/kusto/query/reduce-operator.md index b589aa3040..e9ef5c2657 100644 --- a/data-explorer/kusto/query/reduce-operator.md +++ b/data-explorer/kusto/query/reduce-operator.md @@ -44,6 +44,10 @@ For example, the result of `reduce by city` might include: ## Examples +The example in this section shows how to use the syntax to help you get started. + +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] + ### Small threshold value This query generates a range of numbers, creates a new column with concatenated strings and random integers, and then groups the rows by the new column with specific reduction parameters. diff --git a/data-explorer/kusto/query/sample-distinct-operator.md b/data-explorer/kusto/query/sample-distinct-operator.md index e89dae24f4..c72fbf3ee6 100644 --- a/data-explorer/kusto/query/sample-distinct-operator.md +++ b/data-explorer/kusto/query/sample-distinct-operator.md @@ -34,9 +34,11 @@ The operator tries to return an answer as quickly as possible rather than trying ## Examples -[!INCLUDE [help-cluster-note](../includes/help-cluster-note.md)] +The example in this section shows how to use the syntax to help you get started. + +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] -Get 10 distinct values from a population +### Get 10 distinct values from a population :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] @@ -61,7 +63,7 @@ StormEvents | sample-distinct 10 of EpisodeId | 11781 | | 12826 | -Sample a population and do further computation without exceeding the query limits in the summarize +### Further compute the sample values :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] diff --git a/data-explorer/kusto/query/sample-operator.md b/data-explorer/kusto/query/sample-operator.md index e4d6cefe84..5a1c271a58 100644 --- a/data-explorer/kusto/query/sample-operator.md +++ b/data-explorer/kusto/query/sample-operator.md @@ -31,9 +31,9 @@ Returns up to the specified number of random rows from the input table. ## Examples -[!INCLUDE [help-cluster-note](../includes/help-cluster-note.md)] - -Consider the following examples: +The example in this section shows how to use the syntax to help you get started. + +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] ### Generate a sample diff --git a/data-explorer/kusto/query/scan-operator.md b/data-explorer/kusto/query/scan-operator.md index 10876e9589..20f570c182 100644 --- a/data-explorer/kusto/query/scan-operator.md +++ b/data-explorer/kusto/query/scan-operator.md @@ -81,7 +81,9 @@ For a detailed example of this logic, see the [scan logic walkthrough](#scan-log ## Examples -[!INCLUDE [help-cluster-note](../includes/help-cluster-note.md)] +The example in this section shows how to use the syntax to help you get started. + +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] ### Cumulative sum diff --git a/data-explorer/kusto/query/search-operator.md b/data-explorer/kusto/query/search-operator.md index 160ff2a670..28a8230704 100644 --- a/data-explorer/kusto/query/search-operator.md +++ b/data-explorer/kusto/query/search-operator.md @@ -83,9 +83,9 @@ Unlike the [find operator](find-operator.md), the `search` operator doesn't supp ## Examples -[!INCLUDE [help-cluster-note](../includes/help-cluster-note.md)] - -Consider the following exaples: +The example in this section shows how to use the syntax to help you get started. + +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] ### Global term search diff --git a/data-explorer/kusto/query/serialize-operator.md b/data-explorer/kusto/query/serialize-operator.md index 9471233d86..fc03a90b44 100644 --- a/data-explorer/kusto/query/serialize-operator.md +++ b/data-explorer/kusto/query/serialize-operator.md @@ -28,7 +28,9 @@ The operator has a declarative meaning. It marks the input row set as serialized ## Examples -[!INCLUDE [help-cluster-note](../includes/help-cluster-note.md)] +The example in this section shows how to use the syntax to help you get started. + +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] ### Serialize subset of rows by condition diff --git a/data-explorer/kusto/query/shuffle-query.md b/data-explorer/kusto/query/shuffle-query.md index 4235c8a649..db21d5cdbd 100644 --- a/data-explorer/kusto/query/shuffle-query.md +++ b/data-explorer/kusto/query/shuffle-query.md @@ -69,7 +69,9 @@ In some cases, the `hint.strategy = shuffle` is ignored, and the query won't run ## Examples -[!INCLUDE [help-cluster-note](../includes/help-cluster-note.md)] +The example in this section shows how to use the syntax to help you get started. + +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] ## Use summarize with shuffle From 9f12402c2b94fd9f223e00f0a56b6ab697680b2c Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Mon, 3 Feb 2025 18:13:20 +0200 Subject: [PATCH 15/36] Update shuffle-query.md --- data-explorer/kusto/query/shuffle-query.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/data-explorer/kusto/query/shuffle-query.md b/data-explorer/kusto/query/shuffle-query.md index db21d5cdbd..d11cd7dfcf 100644 --- a/data-explorer/kusto/query/shuffle-query.md +++ b/data-explorer/kusto/query/shuffle-query.md @@ -73,7 +73,7 @@ The example in this section shows how to use the syntax to help you get started. [!INCLUDE [help-cluster](../includes/help-cluster-note.md)] -## Use summarize with shuffle +### Use summarize with shuffle The `shuffle` strategy query with `summarize` operator shares the load on all cluster nodes, where each node processes one partition of the data. @@ -94,7 +94,7 @@ StormEvents |---| |67| -## Use join with shuffle +### Use join with shuffle :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] @@ -121,7 +121,7 @@ StormEvents |---| |103| -## Use make-series with shuffle +### Use make-series with shuffle :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] @@ -310,7 +310,7 @@ lineitem | consume ``` -## Use join with shuffle to improve performance +### Use join with shuffle to improve performance The following example shows how using `shuffle` strategy with the `join` operator improves performance. From 1c4a4c97b6e26916f64b5c58badce5b7efae4b40 Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Tue, 4 Feb 2025 10:29:35 +0200 Subject: [PATCH 16/36] formatting Co-authored-by: Shlomo Sagir <51323195+shsagir@users.noreply.github.com> --- data-explorer/kusto/query/reduce-operator.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/query/reduce-operator.md b/data-explorer/kusto/query/reduce-operator.md index e9ef5c2657..55cd417f5e 100644 --- a/data-explorer/kusto/query/reduce-operator.md +++ b/data-explorer/kusto/query/reduce-operator.md @@ -102,7 +102,7 @@ The result includes only those groups where the MyText value appears in at least ### Behavior of `Characters` parameter -If the `Characters` parameter is unspecified, then every non-ascii numeric character becomes a term separator. +If the *Characters* parameter is unspecified, then every non-ascii numeric character becomes a term separator. :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] From f182a80b87c3ce5d407682db87807906da083ec2 Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Tue, 4 Feb 2025 10:39:25 +0200 Subject: [PATCH 17/36] separated code from text --- data-explorer/kusto/query/reduce-operator.md | 25 +++++++++++++------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/data-explorer/kusto/query/reduce-operator.md b/data-explorer/kusto/query/reduce-operator.md index 55cd417f5e..ed32e9d8c4 100644 --- a/data-explorer/kusto/query/reduce-operator.md +++ b/data-explorer/kusto/query/reduce-operator.md @@ -3,7 +3,7 @@ title: reduce operator description: Learn how to use the reduce operator to group a set of strings together based on value similarity. ms.reviewer: alexans ms.topic: reference -ms.date: 01/22/2025 +ms.date: 02/04/2025 --- # reduce operator @@ -141,17 +141,26 @@ range x from 1 to 10 step 1 | project str = strcat("foo", "Z", tostring(x)) | re The following example shows how one might apply the `reduce` operator to a "sanitized" input, in which GUIDs in the column being reduced are replaced before reducing: +Start with a few records from the Trace table. + ```kusto -// Start with a few records from the Trace table. Trace | take 10000 -// Then reduce the Text column which includes random GUIDs. -// As random GUIDs interfere with the reduce operation, replace them all -// by the string "GUID". +```kusto + +Then reduce the Text column which includes random GUIDs. +As random GUIDs interfere with the reduce operation, replace them all +by the string "GUID". + +```kusto | extend Text=replace_regex(Text, @"[[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12}", @"GUID") -// Now perform the reduce. In case there are other "quasi-random" identifiers with embedded '-' -// or '_' characters in them, treat these as non-term-breakers. +```kusto + +Now perform the reduce. In case there are other "quasi-random" identifiers with embedded '-' +or '_' characters in them, treat these as non-term-breakers. + +```kusto | reduce by Text with characters="-_" -``` +```kusto ## Related content From 7b2e5cafbcbd8ee327b781d048053748c6e0c76b Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Tue, 4 Feb 2025 10:44:29 +0200 Subject: [PATCH 18/36] Fix code block --- data-explorer/kusto/query/reduce-operator.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/data-explorer/kusto/query/reduce-operator.md b/data-explorer/kusto/query/reduce-operator.md index ed32e9d8c4..e5e3101d11 100644 --- a/data-explorer/kusto/query/reduce-operator.md +++ b/data-explorer/kusto/query/reduce-operator.md @@ -152,16 +152,18 @@ As random GUIDs interfere with the reduce operation, replace them all by the string "GUID". ```kusto -| extend Text=replace_regex(Text, @"[[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12}", @"GUID") +extend Text=replace_regex(Text, @"[[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12}", @"GUID") ```kusto Now perform the reduce. In case there are other "quasi-random" identifiers with embedded '-' or '_' characters in them, treat these as non-term-breakers. ```kusto -| reduce by Text with characters="-_" +reduce by Text with characters="-_" ```kusto +``` + ## Related content * [autocluster](autocluster-plugin.md) From 06f5bdfea51e02bd4fec65f69b353b99f7df1d65 Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Tue, 4 Feb 2025 11:00:39 +0200 Subject: [PATCH 19/36] fixed code box --- data-explorer/kusto/query/reduce-operator.md | 17 ++++------------- 1 file changed, 4 insertions(+), 13 deletions(-) diff --git a/data-explorer/kusto/query/reduce-operator.md b/data-explorer/kusto/query/reduce-operator.md index e5e3101d11..2beab6e6de 100644 --- a/data-explorer/kusto/query/reduce-operator.md +++ b/data-explorer/kusto/query/reduce-operator.md @@ -142,26 +142,17 @@ The following example shows how one might apply the `reduce` operator to a "sani input, in which GUIDs in the column being reduced are replaced before reducing: Start with a few records from the Trace table. - -```kusto -Trace | take 10000 -```kusto - Then reduce the Text column which includes random GUIDs. As random GUIDs interfere with the reduce operation, replace them all by the string "GUID". - -```kusto -extend Text=replace_regex(Text, @"[[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12}", @"GUID") -```kusto - Now perform the reduce. In case there are other "quasi-random" identifiers with embedded '-' or '_' characters in them, treat these as non-term-breakers. ```kusto -reduce by Text with characters="-_" -```kusto - +Trace +| take 10000 +| extend Text = replace(@"[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}", "GUID", Text) +| reduce by Text with characters="-_" ``` ## Related content From 7a594395e573156d92114329f90209df01db425d Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Tue, 4 Feb 2025 13:15:25 +0200 Subject: [PATCH 20/36] Update search examples --- data-explorer/kusto/query/search-operator.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/data-explorer/kusto/query/search-operator.md b/data-explorer/kusto/query/search-operator.md index 28a8230704..97b74eb9b7 100644 --- a/data-explorer/kusto/query/search-operator.md +++ b/data-explorer/kusto/query/search-operator.md @@ -89,7 +89,9 @@ The example in this section shows how to use the syntax to help you get started. ### Global term search -Search for a term over all unrestricted tables and views of the database in scope. +Search for the term Green in the database. + +The output finds records with the term *Green* as a last name or a color in the `Customers`, `Products`, and `SalesTable` tables. :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] @@ -97,11 +99,9 @@ Search for a term over all unrestricted tables and views of the database in scop ::: moniker-end ```kusto -search "Green" + search "Green" ``` -The output contains records from the `Customers`, `Products`, and `SalesTable` tables. The `Customers` records show all customers with the last name "Green", and the `Products` and `SalesTable` records shows products with some mention of "Green". - **Output** | $table | CityName | ContinentName | CustomerKey | Education | FirstName | Gender | LastName | @@ -115,7 +115,7 @@ The output contains records from the `Customers`, `Products`, and `SalesTable` t ### Conditional global term search -Search for records that match both terms over all unrestricted tables and views of the database in scope. +Search for records that contain the term *Green* and one of either terms *Deluze* or *Proseware* in the database. :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] @@ -140,7 +140,7 @@ search "Green" and ("Deluxe" or "Proseware") ### Search a specific table -Search only in the `Customers` table. +Search fore the term *Green* only in the `Customers` table. :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] @@ -164,7 +164,7 @@ search in (Products) "Green" ### Case-sensitive search -Search for records that match both case-sensitive terms over all unrestricted tables and views of the database in scope. +Search for records that match both case-sensitive terms in the database. :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] @@ -189,7 +189,7 @@ search kind=case_sensitive "blue" ### Search specific columns -Search for a term in the "FirstName" and "LastName" columns over all unrestricted tables and views of the database in scope. +Search for the terms *Aaron* and *Hughes*, in the "FirstName" and "LastName" columns respectively, in the database. :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] @@ -213,7 +213,7 @@ search FirstName:"Aaron" or LastName:"Hughes" ### Limit search by timestamp -Search for a term over all unrestricted tables and views of the database in scope if the term appears in a record with a date greater than the given date. +Search for the term *Hughes* in the database, if the term appears in a record with a date greater than the given date in 'datetime'. :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] From 01549630cbfe47f7f92b59ee4a01d56b0c22c5d2 Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Tue, 4 Feb 2025 13:19:50 +0200 Subject: [PATCH 21/36] typo --- data-explorer/kusto/query/search-operator.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/query/search-operator.md b/data-explorer/kusto/query/search-operator.md index 97b74eb9b7..bb05771f95 100644 --- a/data-explorer/kusto/query/search-operator.md +++ b/data-explorer/kusto/query/search-operator.md @@ -140,7 +140,7 @@ search "Green" and ("Deluxe" or "Proseware") ### Search a specific table -Search fore the term *Green* only in the `Customers` table. +Search for the term *Green* only in the `Customers` table. :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] From 8d913a7f37a9a4ecfa00439969dd919b28491140 Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Tue, 4 Feb 2025 13:20:59 +0200 Subject: [PATCH 22/36] Fix case-sensitive search term description --- data-explorer/kusto/query/search-operator.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/query/search-operator.md b/data-explorer/kusto/query/search-operator.md index bb05771f95..a9ca178368 100644 --- a/data-explorer/kusto/query/search-operator.md +++ b/data-explorer/kusto/query/search-operator.md @@ -164,7 +164,7 @@ search in (Products) "Green" ### Case-sensitive search -Search for records that match both case-sensitive terms in the database. +Search for records that match the case-sensitive term in the database. :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] From 752d30a1f2a27353769a2a8ad9593eaab503a292 Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Tue, 4 Feb 2025 13:43:59 +0200 Subject: [PATCH 23/36] remove empty line Co-authored-by: Shlomo Sagir <51323195+shsagir@users.noreply.github.com> --- data-explorer/kusto/query/parse-kv-operator.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/data-explorer/kusto/query/parse-kv-operator.md b/data-explorer/kusto/query/parse-kv-operator.md index e17a59a07f..66bdc148ea 100644 --- a/data-explorer/kusto/query/parse-kv-operator.md +++ b/data-explorer/kusto/query/parse-kv-operator.md @@ -60,8 +60,6 @@ The original input tabular expression *T*, extended with columns per specified k ## Examples The examples in this section show how to use the syntax to help you get started. - - ### Extraction with well-defined delimiters In this query, keys and values are separated by well defined delimiters. These delimeters are comma and colon characters. From de14616005982575191020603bc9b255afc76366 Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Tue, 4 Feb 2025 13:44:27 +0200 Subject: [PATCH 24/36] Terminology fix Co-authored-by: Shlomo Sagir <51323195+shsagir@users.noreply.github.com> --- data-explorer/kusto/query/partition-operator.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/query/partition-operator.md b/data-explorer/kusto/query/partition-operator.md index 12f8f03f88..9e88f8006a 100644 --- a/data-explorer/kusto/query/partition-operator.md +++ b/data-explorer/kusto/query/partition-operator.md @@ -43,7 +43,7 @@ The partition operator supports several strategies of subquery operation: |Hint name|Type|Strategy|Description| |--|--|--|--| |`hint.shufflekey`| `string` | [shuffle](#shuffle-strategy) | The partition key used to run the partition operator with the `shuffle` strategy. | -|`hint.materialized`| `bool` | [legacy](#legacy-strategy) | If set to `true`, outputs the source of the `partition` operator. The default value is `false`. | +|`hint.materialized`| `bool` | [legacy](#legacy-strategy) | If set to `true`, materializes the source of the `partition` operator. The default value is `false`. | |`hint.concurrency`| `int` | [legacy](#legacy-strategy) | Determines how many partitions to run in parallel. The default value is `16`.| |`hint.spread`| `int` | [legacy](#legacy-strategy) | Determines how to distribute the partitions among cluster nodes. The default value is `1`.

For example, if there are *N* partitions and the spread hint is set to *P*, then the *N* partitions are processed by *P* different cluster nodes equally, in parallel/sequentially depending on the concurrency hint.| From 8fc77239bac8b97b1701d81ae420738a3eb83769 Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Tue, 4 Feb 2025 13:44:52 +0200 Subject: [PATCH 25/36] Type Co-authored-by: Shlomo Sagir <51323195+shsagir@users.noreply.github.com> --- data-explorer/kusto/query/query-parameters-statement.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/query/query-parameters-statement.md b/data-explorer/kusto/query/query-parameters-statement.md index b8566ea1cd..cf721a064a 100644 --- a/data-explorer/kusto/query/query-parameters-statement.md +++ b/data-explorer/kusto/query/query-parameters-statement.md @@ -50,7 +50,7 @@ The examples in this section show how to use the syntax to help you get started. ### Declare query parameters - this query retrieves storm events from the *StormEvents* table where the total number of direct and indirect injuries exceeds a specified threshold (default is 90). It then projects the *EpisodeId*, *EventType*, and the total number of injuries for each of these events. + This query retrieves storm events from the *StormEvents* table where the total number of direct and indirect injuries exceeds a specified threshold (default is 90). It then projects the *EpisodeId*, *EventType*, and the total number of injuries for each of these events. :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] From 6e18d1db349044dd9b135b3a20c0c601d1af4c7d Mon Sep 17 00:00:00 2001 From: Shlomo Sagir <51323195+shsagir@users.noreply.github.com> Date: Tue, 4 Feb 2025 14:39:22 +0200 Subject: [PATCH 26/36] Update data-explorer/kusto/query/parse-operator.md --- data-explorer/kusto/query/parse-operator.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/data-explorer/kusto/query/parse-operator.md b/data-explorer/kusto/query/parse-operator.md index 446017ca70..2a89a34523 100644 --- a/data-explorer/kusto/query/parse-operator.md +++ b/data-explorer/kusto/query/parse-operator.md @@ -141,8 +141,6 @@ Leads |ebere@relecloud.com|ebere|relecloud.com| |abi@fourthcoffee.com|abi|www.fourthcoffee.com| -## Regex mode examples - ### Regex mode In the following example, regular expressions are used to parse and extract data from the `EventText` column. The extracted data is projected into new fields. From 0c115d2e54221cd118957cadc6754148e7881859 Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Wed, 5 Feb 2025 11:46:19 +0200 Subject: [PATCH 27/36] Fix typos in 'delimiter' in parse-kv-operator.md --- data-explorer/kusto/query/parse-kv-operator.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/data-explorer/kusto/query/parse-kv-operator.md b/data-explorer/kusto/query/parse-kv-operator.md index 66bdc148ea..cdba91a80d 100644 --- a/data-explorer/kusto/query/parse-kv-operator.md +++ b/data-explorer/kusto/query/parse-kv-operator.md @@ -14,13 +14,13 @@ Extracts structured information from a string expression and represents the info The following extraction modes are supported: -* [**Specified delimeter**](#specified-delimeter): Extraction based on specified delimiters that dictate how keys/values and pairs are separated from each other. -* [**Non-specified delimeter**](#nonspecified-delimiter): Extraction with no need to specify delimiters. Any nonalphanumeric character is considered a delimiter. +* [**Specified delimiter**](#specified-delimiter): Extraction based on specified delimiters that dictate how keys/values and pairs are separated from each other. +* [**Non-specified delimiter**](#nonspecified-delimiter): Extraction with no need to specify delimiters. Any nonalphanumeric character is considered a delimiter. * [**Regex**](#regex): Extraction based on [regular expressions](regex.md). ## Syntax -### Specified delimeter +### Specified delimiter *T* `|` `parse-kv` *Expression* `as` `(` *KeysList* `)` `with` `(` `pair_delimiter` `=` *PairDelimiter* `,` `kv_delimiter` `=` *KvDelimiter* [`,` `quote` `=` *QuoteChars* ... [`,` `escape` `=` *EscapeChar* ...]] [`,` `greedy` `=` `true`] `)` From 84ab58d3d2b34b384faa6dd0b6272e82a2652070 Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Wed, 5 Feb 2025 11:51:57 +0200 Subject: [PATCH 28/36] Fix typo in query parameters statement documentation --- data-explorer/kusto/query/query-parameters-statement.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/query/query-parameters-statement.md b/data-explorer/kusto/query/query-parameters-statement.md index cf721a064a..7f58a57b53 100644 --- a/data-explorer/kusto/query/query-parameters-statement.md +++ b/data-explorer/kusto/query/query-parameters-statement.md @@ -34,7 +34,7 @@ To reference query parameters, the query text, or functions it uses, must first |Name|Type|Required|Description| |--|--|--|--| |*Name1*| `string` | :heavy_check_mark:|The name of a query parameter used in the query.| -|*Type1*| `string` | :heavy_check_mark:|The corresponding type, such as `string` or `datetime`. The values provided by the user are encoded as strings. The appropriate parse method is applied to the query parameter to get a strongly-typed value.| +|*Type1*| `string` | :heavy_check_mark:|The corresponding type, such as `string` or `datetime`. The values provided by the user are encoded as strings. The appropriate parse method is applied to the query parameter to get a strongly typed value.| |*DefaultValue1*| `string` ||A default value for the parameter. This value must be a literal of the appropriate scalar type.| > [!NOTE] From c3979733ecaaf912a19488015ae13265d7fc8ebb Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Wed, 5 Feb 2025 12:05:41 +0200 Subject: [PATCH 29/36] Specify *ContosoSales* database as scope in search examples --- data-explorer/kusto/query/search-operator.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/data-explorer/kusto/query/search-operator.md b/data-explorer/kusto/query/search-operator.md index a9ca178368..b5ee8d7067 100644 --- a/data-explorer/kusto/query/search-operator.md +++ b/data-explorer/kusto/query/search-operator.md @@ -89,7 +89,7 @@ The example in this section shows how to use the syntax to help you get started. ### Global term search -Search for the term Green in the database. +Search for the term Green in all the tables of the *ContosoSales* database. The output finds records with the term *Green* as a last name or a color in the `Customers`, `Products`, and `SalesTable` tables. @@ -115,7 +115,7 @@ The output finds records with the term *Green* as a last name or a color in the ### Conditional global term search -Search for records that contain the term *Green* and one of either terms *Deluze* or *Proseware* in the database. +Search for records that contain the term *Green* and one of either terms *Deluze* or *Proseware* in the *ContosoSales* database. :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] @@ -164,7 +164,7 @@ search in (Products) "Green" ### Case-sensitive search -Search for records that match the case-sensitive term in the database. +Search for records that match the case-sensitive term in the *ContosoSales* database. :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] @@ -189,7 +189,7 @@ search kind=case_sensitive "blue" ### Search specific columns -Search for the terms *Aaron* and *Hughes*, in the "FirstName" and "LastName" columns respectively, in the database. +Search for the terms *Aaron* and *Hughes*, in the "FirstName" and "LastName" columns respectively, in the *ContosoSales* database. :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] @@ -213,7 +213,7 @@ search FirstName:"Aaron" or LastName:"Hughes" ### Limit search by timestamp -Search for the term *Hughes* in the database, if the term appears in a record with a date greater than the given date in 'datetime'. +Search for the term *Hughes* in the *ContosoSales* database, if the term appears in a record with a date greater than the given date in 'datetime'. :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] From 28aed47b430816fab176b9d7ba109dd1221d721d Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Wed, 5 Feb 2025 14:17:46 +0200 Subject: [PATCH 30/36] Acrolinx checks --- data-explorer/kusto/query/range-operator.md | 2 +- data-explorer/kusto/query/reduce-operator.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/data-explorer/kusto/query/range-operator.md b/data-explorer/kusto/query/range-operator.md index ba95ffde4d..4a8af54716 100644 --- a/data-explorer/kusto/query/range-operator.md +++ b/data-explorer/kusto/query/range-operator.md @@ -26,7 +26,7 @@ Generates a single-column table of values. |--|--|--|--| |*columnName*| `string` | :heavy_check_mark:| The name of the single column in the output table.| |*start*|int, long, real, datetime, or timespan| :heavy_check_mark:| The smallest value in the output.| -|*stop*|int, long, real, datetime, or timespan| :heavy_check_mark:| The highest value being generated in the output or a bound on the highest value if *step* steps over this value.| +|*stop*|int, long, real, datetime, or timespan| :heavy_check_mark:| The highest value being generated in the output or a bound on the highest value if *step* is over this value.| |*step*|int, long, real, datetime, or timespan| :heavy_check_mark:| The difference between two consecutive values.| > [!NOTE] diff --git a/data-explorer/kusto/query/reduce-operator.md b/data-explorer/kusto/query/reduce-operator.md index 2beab6e6de..fb9d1a2392 100644 --- a/data-explorer/kusto/query/reduce-operator.md +++ b/data-explorer/kusto/query/reduce-operator.md @@ -146,7 +146,7 @@ Then reduce the Text column which includes random GUIDs. As random GUIDs interfere with the reduce operation, replace them all by the string "GUID". Now perform the reduce. In case there are other "quasi-random" identifiers with embedded '-' -or '_' characters in them, treat these as non-term-breakers. +or '_' characters in them, treat characters as non-term-breakers. ```kusto Trace From 9feaea2a6e1d8f553c83f78e19710bb4b5593a31 Mon Sep 17 00:00:00 2001 From: Shlomo Sagir <51323195+shsagir@users.noreply.github.com> Date: Thu, 6 Feb 2025 11:35:04 +0200 Subject: [PATCH 31/36] Update data-explorer/kusto/management/callout-policy.md --- data-explorer/kusto/management/callout-policy.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/management/callout-policy.md b/data-explorer/kusto/management/callout-policy.md index fd71080034..3d51c1dc7d 100644 --- a/data-explorer/kusto/management/callout-policy.md +++ b/data-explorer/kusto/management/callout-policy.md @@ -38,7 +38,7 @@ Callout policies are managed at cluster-level and are classified into the follow | sandbox_artifacts | Controls sandboxed plugins ([python](../query/python-plugin.md) and [R](../query/r-plugin.md)). | | external_data | Controls access to external data through [external tables](../query/schema-entities/external-tables.md) or [externaldata](../query/externaldata-operator.md) operator. | | webapi | Controls access to http endpoints. | -| azure_openai | Controls calls to Azure OpenAI plugins (e.g. the embedding plugin [ai_embed_text plugin)](../query/ai-embed-text-plugin.md). | +| azure_openai | Controls calls to Azure OpenAI plugins such as the embedding plugin [ai_embed_text plugin](../query/ai-embed-text-plugin.md). | ## Predefined callout policies From 9b85900e5f597c54100d74c226c263b7bd4c5556 Mon Sep 17 00:00:00 2001 From: Shlomo Sagir <51323195+shsagir@users.noreply.github.com> Date: Thu, 6 Feb 2025 12:37:49 +0200 Subject: [PATCH 32/36] Update data-explorer/kusto/query/parse-kv-operator.md --- data-explorer/kusto/query/parse-kv-operator.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/query/parse-kv-operator.md b/data-explorer/kusto/query/parse-kv-operator.md index cdba91a80d..cb13e0dc7b 100644 --- a/data-explorer/kusto/query/parse-kv-operator.md +++ b/data-explorer/kusto/query/parse-kv-operator.md @@ -3,7 +3,7 @@ title: parse-kv operator description: Learn how to use the parse-kv operator to represent structured information extracted from a string expression in a key/value form. ms.reviewer: alexans ms.topic: reference -ms.date: 01/22/2025 +ms.date: 02/06/2025 --- # parse-kv operator From 87b143dd769ba520f44cb8ba8edec09e822bd428 Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Thu, 6 Feb 2025 12:47:30 +0200 Subject: [PATCH 33/36] Add note on default search behavior --- data-explorer/kusto/query/search-operator.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/data-explorer/kusto/query/search-operator.md b/data-explorer/kusto/query/search-operator.md index b5ee8d7067..5135057375 100644 --- a/data-explorer/kusto/query/search-operator.md +++ b/data-explorer/kusto/query/search-operator.md @@ -29,6 +29,9 @@ Searches a text pattern in multiple tables and columns. | *TableSources* | `string` | | A comma-separated list of "wildcarded" table names to take part in the search. The list has the same syntax as the list of the [union operator](union-operator.md). Can't appear together with *TabularSource*.| | *SearchPredicate* | `string` | :heavy_check_mark: | A boolean expression to be evaluated for every record in the input. If it returns `true`, the record is outputted. See [Search predicate syntax](#search-predicate-syntax).| +> [!NOTE] +> If both tabular data source (*T*) and *TableSources* are omitted, the search is carried over all unrestricted tables and views of the database in scope. + ### Search predicate syntax The *SearchPredicate* allows you to search for specific terms in all columns of a table. The operator that is applied to a search term depends on the presence and placement of a wildcard asterisk (`*`) in the term, as shown in the following table. @@ -51,8 +54,7 @@ You can also restrict the search to a specific column, look for an exact match i Use boolean expressions to combine conditions and create more complex searches. For example, `"error" and x==123` would result in a search for records that have the term `error` in any columns and the value `123` in the `x` column. -> [!NOTE] -> If both *TabularSource* and *TableSources* are omitted, the search is carried over all unrestricted tables and views of the database in scope. + ### Search predicate syntax examples From b6c599d640bea53b2acf8e19d9d6b7e442fa0cad Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Thu, 6 Feb 2025 12:48:42 +0200 Subject: [PATCH 34/36] Update parameter descriptions for search operator --- data-explorer/kusto/query/search-operator.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/data-explorer/kusto/query/search-operator.md b/data-explorer/kusto/query/search-operator.md index 5135057375..a2f401f7ae 100644 --- a/data-explorer/kusto/query/search-operator.md +++ b/data-explorer/kusto/query/search-operator.md @@ -24,9 +24,9 @@ Searches a text pattern in multiple tables and columns. | Name | Type | Required | Description | |--|--|--|--| -| *T* | `string` | | The tabular data source to be searched over, such as a table name, a [union operator](union-operator.md), or the results of a tabular query. Can't appear together with *TableSources*.| +| *T* | `string` | | The tabular data source to be searched over, such as a table name, a [union operator](union-operator.md), or the results of a tabular query. Can't be specified together with *TableSources*.| | *CaseSensitivity* | `string` | | A flag that controls the behavior of all `string` scalar operators, such as `has`, with respect to case sensitivity. Valid values are `default`, `case_insensitive`, `case_sensitive`. The options `default` and `case_insensitive` are synonymous, since the default behavior is case insensitive.| -| *TableSources* | `string` | | A comma-separated list of "wildcarded" table names to take part in the search. The list has the same syntax as the list of the [union operator](union-operator.md). Can't appear together with *TabularSource*.| +| *TableSources* | `string` | | A comma-separated list of "wildcarded" table names to take part in the search. The list has the same syntax as the list of the [union operator](union-operator.md). Can't be specified together with tabular data source (*T*).| | *SearchPredicate* | `string` | :heavy_check_mark: | A boolean expression to be evaluated for every record in the input. If it returns `true`, the record is outputted. See [Search predicate syntax](#search-predicate-syntax).| > [!NOTE] From 2f71060610cb25a7cd63bb157b67193dcd5bae94 Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Thu, 6 Feb 2025 12:51:20 +0200 Subject: [PATCH 35/36] Add example query to serialize-operator.md --- data-explorer/kusto/query/serialize-operator.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/data-explorer/kusto/query/serialize-operator.md b/data-explorer/kusto/query/serialize-operator.md index fc03a90b44..48d33cb620 100644 --- a/data-explorer/kusto/query/serialize-operator.md +++ b/data-explorer/kusto/query/serialize-operator.md @@ -34,6 +34,8 @@ The example in this section shows how to use the syntax to help you get started. ### Serialize subset of rows by condition + This query retrieves all log entries from the *TraceLogs* table that have a specific *ClientRequestId* and preserves the order of these entries during processing. + :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] > Run the query From 6a7c28edc700b066491b46808b11e0e8fca4796c Mon Sep 17 00:00:00 2001 From: Colin Cooper <72402153+v-ccolin@users.noreply.github.com> Date: Thu, 6 Feb 2025 11:41:09 +0000 Subject: [PATCH 36/36] Apply suggestions from code review --- data-explorer/kusto/query/reduce-operator.md | 2 +- data-explorer/kusto/query/search-operator.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/data-explorer/kusto/query/reduce-operator.md b/data-explorer/kusto/query/reduce-operator.md index fb9d1a2392..bc72989891 100644 --- a/data-explorer/kusto/query/reduce-operator.md +++ b/data-explorer/kusto/query/reduce-operator.md @@ -145,7 +145,7 @@ Start with a few records from the Trace table. Then reduce the Text column which includes random GUIDs. As random GUIDs interfere with the reduce operation, replace them all by the string "GUID". -Now perform the reduce. In case there are other "quasi-random" identifiers with embedded '-' +Now perform the reduce operation. In case there are other "quasi-random" identifiers with embedded '-' or '_' characters in them, treat characters as non-term-breakers. ```kusto diff --git a/data-explorer/kusto/query/search-operator.md b/data-explorer/kusto/query/search-operator.md index a2f401f7ae..577edeb832 100644 --- a/data-explorer/kusto/query/search-operator.md +++ b/data-explorer/kusto/query/search-operator.md @@ -117,7 +117,7 @@ The output finds records with the term *Green* as a last name or a color in the ### Conditional global term search -Search for records that contain the term *Green* and one of either terms *Deluze* or *Proseware* in the *ContosoSales* database. +Search for records that contain the term *Green* and one of either terms *Deluxe* or *Proseware* in the *ContosoSales* database. :::moniker range="azure-data-explorer" > [!div class="nextstepaction"]