Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
7d9ba97
pattern-reviewed
ktalmor Jan 12, 2025
1609da1
Merge remote-tracking branch 'upstream/main' into wi-360961-query-review
ktalmor Jan 12, 2025
79fab57
patterns-fixes
ktalmor Jan 12, 2025
1ccb413
Added link to sample
ktalmor Jan 12, 2025
664094a
alias-review
ktalmor Jan 12, 2025
dd7172f
added missing output
ktalmor Jan 13, 2025
bf25e2f
include-help-cluster
ktalmor Jan 13, 2025
3de710e
Added-include-help-cluster
ktalmor Jan 13, 2025
64023ba
Query-statements-review-cont
ktalmor Jan 13, 2025
45c441d
wi-360961-query-review
ktalmor Jan 13, 2025
a629621
Bold
ktalmor Jan 14, 2025
fee6751
fixed table tabs
ktalmor Jan 14, 2025
059b361
renamed include file
ktalmor Jan 15, 2025
a3c6306
Update data-explorer/kusto/query/batches.md
shsagir Jan 15, 2025
82a2b82
fix conceptual tab
ktalmor Jan 15, 2025
5b8880f
Merge branch 'wi-360961-query-review' of https://github.com/ktalmor/d…
ktalmor Jan 15, 2025
5ce0d89
closed section
ktalmor Jan 15, 2025
3828ba7
cleaned tables
ktalmor Jan 15, 2025
1793319
minor fixes
ktalmor Jan 15, 2025
ff6bacb
Apply suggestions from code review
shsagir Jan 19, 2025
c16cabf
Update monitor-queued-ingestion.md
shsagir Jan 20, 2025
2339da1
Merge pull request #6557 from MicrosoftDocs/shsagir-patch-14
shsagir Jan 20, 2025
35136d0
Merge pull request #6528 from ktalmor/wi-360961-query-review
v-ccolin Jan 20, 2025
18fae4b
Merge pull request #6559 from MicrosoftDocs/main
Taojunshen Jan 20, 2025
ad260eb
Merging changes synced from https://github.com/MicrosoftDocs/dataexpl…
Jan 20, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
ms.topic: include
ms.date: 01/13/2025
---

In the [help cluster](https://dataexplorer.azure.com/clusters/help/), there's a `Samples` database with a `StormEvents` table.
7 changes: 4 additions & 3 deletions data-explorer/kusto/query/alias-statement.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,18 +3,19 @@ title: Alias statement
description: Learn how to use an alias statement to define an alias for a database that is used for a query.
ms.reviewer: alexans
ms.topic: reference
ms.date: 08/11/2024
ms.date: 01/12/2025
---
# Alias statement

> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] [!INCLUDE [monitor](../includes/applies-to-version/monitor.md)] [!INCLUDE [sentinel](../includes/applies-to-version/sentinel.md)]

Alias statements allow you to define an alias for databases, which can be used later in the same query.
Alias statements allow you to define an alias for a database, which can be used in the same query.

:::moniker range="azure-data-explorer"
This is useful when you're working with several clusters but want to appear as if you're working on fewer clusters.
The alias must be defined according to the following syntax, where *clustername* and *databasename* are existing and valid entities.


## Syntax

`alias` database *DatabaseAliasName* `=` cluster("https://*clustername*.kusto.windows.net").database("*DatabaseName*")
Expand Down Expand Up @@ -50,7 +51,7 @@ The alias must be defined according to the following syntax, where *Eventhouse*

## Examples

In the [help cluster](https://dataexplorer.azure.com/clusters/help/), there's a `Samples` database with a `StormEvents` table.
[!INCLUDE [help-cluster](../includes/help-cluster-samples-stormevents.md)]

First, count the number of records in that table.

Expand Down
45 changes: 43 additions & 2 deletions data-explorer/kusto/query/batches.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ title: Batches
description: This article describes Batches.
ms.reviewer: alexans
ms.topic: reference
ms.date: 08/11/2024
ms.date: 01/13/2025
---
# Batches

Expand All @@ -18,6 +18,8 @@ A query can include multiple tabular expression statements, as long as they're d

## Examples

[!INCLUDE [help-cluster](../includes/help-cluster-samples-stormevents.md)]

### Name tabular results

The following query produces two tabular results. User agent tools can then display those results with the appropriate name associated with each (`Count of events in Florida` and `Count of events in Guam`, respectively).
Expand All @@ -32,9 +34,25 @@ StormEvents | where State == "FLORIDA" | count | as ['Count of events in Florida
StormEvents | where State == "GUAM" | count | as ['Count of events in Guam']
```

**Output**

### [Count of events in Florida](#tab/florida)

| Count |
| -- |
| 1042 |

### [Count of events in Guam](#tab/guam)

| Count |
| -- |
| 4 |

---

### Share a calculation

Batching is useful for scenarios where a common calculation is shared by multiple subqueries, such as for dashboards. If the common calculation is complex, use the [materialize() function](materialize-function.md) and construct the query so that it will be executed only once:
Batching is useful for scenarios where a common calculation is shared by multiple subqueries, such as for dashboards. If the common calculation is complex, use the [materialize() function](materialize-function.md) and construct the query so that it will be executed only once.

:::moniker range="azure-data-explorer"
> [!div class="nextstepaction"]
Expand All @@ -46,3 +64,26 @@ let m = materialize(StormEvents | summarize n=count() by State);
m | where n > 2000;
m | where n < 10
```

**Output**

### [Table 1](#tab/table-1)

| State | n |
|--|--|
| ILLINOIS | 2022 |
| IOWA | 2337 |
| KANSAS | 3166 |
| MISSOURI | 2016 |
| TEXAS | 4701 |

### [Table 2](#tab/table-2)

| State | n |
|--|--|
| GUAM | 2022 |
| GULF OF ALASKA | 2337 |
| HAWAII WATERS | 3166 |
| LAKE ONTARIO | 2016 |

---
12 changes: 12 additions & 0 deletions data-explorer/kusto/query/let-statement.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,8 @@ To optimize multiple uses of the `let` statement within a single query, see [Opt

## Examples

[!INCLUDE [help-cluster](../includes/help-cluster-samples-stormevents.md)]

### Define scalar values

The following example uses a scalar expression statement.
Expand All @@ -91,6 +93,16 @@ let ['some number'] = 20;
range y from 0 to ['some number'] step 5
```

**Output**

|y|
|---|
|0|
|5|
|10|
|15|
|20|

### Create a user defined function with scalar calculation

This example uses the let statement with arguments for scalar calculation. The query defines function `MultiplyByN` for multiplying two numbers.
Expand Down
51 changes: 37 additions & 14 deletions data-explorer/kusto/query/pattern-statement.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,17 +3,21 @@ title: Pattern statement
description: Learn how to use pattern statements to map string tuples to tabular expressions.
ms.reviewer: alexans
ms.topic: reference
ms.date: 08/11/2024
ms.date: 01/15/2025
monikerRange: "microsoft-fabric || azure-data-explorer"
---

# Pattern statement

> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] [!INCLUDE [monitor](../includes/applies-to-version/monitor.md)] [!INCLUDE [sentinel](../includes/applies-to-version/sentinel.md)]

A **pattern** is a construct that maps string tuples to tabular expressions. Each pattern must *declare* a pattern name and optionally *define* a pattern mapping. Patterns that define a mapping return a tabular expression when invoked. Any two statements must be separated by a semicolon.
A **pattern** is a construct that maps string tuples to tabular expressions.

*Empty patterns* are patterns that are declared but don't define a mapping. When invoked, they return error *SEM0036* along with the details of the missing pattern definitions in the HTTP header. Middle-tier applications that provide a Kusto Query Language (KQL) experience can use the returned details as part of their process to enrich KQL query results.
Each pattern must *declare* a pattern name and optionally *define* a pattern mapping. Patterns that define a mapping return a tabular expression when invoked. Separate any two statements by a semicolon.

Empty patterns are patterns that are declared but don't define a mapping. When invoked, they return error *SEM0036* along with the details of the missing pattern definitions in the HTTP header.

Middle-tier applications that provide a Kusto Query Language (KQL) experience can use the returned details as part of their process to enrich KQL query results.
For more information, see [Working with middle-tier applications](#work-with-middle-tier-applications).

## Syntax
Expand Down Expand Up @@ -56,11 +60,13 @@ For more information, see [Working with middle-tier applications](#work-with-mid

## Examples

In each of the following examples, a pattern is declared, defined, and then invoked.
[!INCLUDE [help-cluster](../includes/help-cluster-samples-stormevents.md)]

### Define simple patterns
In these examples, a pattern is defined.

The following example defines a pattern that maps states to an expression that returns its capital/major city.
### Define a simple pattern

This example defines a pattern that maps states to an expression that returns its capital/major city.

:::moniker range="azure-data-explorer"
> [!div class="nextstepaction"]
Expand All @@ -77,21 +83,23 @@ declare pattern country = (name:string)[state:string]
country("Canada").Alberta
```

**Output**
**Output**

|Capital|
|-------|
|Edmonton|

### Define a scoped pattern

This example defines a pattern to scope data and metrics of application data. The pattern is invoked to return a union of the data.

:::moniker range="azure-data-explorer"
> [!div class="nextstepaction"]
> <a href="https://dataexplorer.azure.com/clusters/help/databases/Samples?query=H4sIAAAAAAAAA62RsQrCMBCGd6Hv8FOHtlDFtrgoHQQXB5+gdAhpLJWahCRCofruJpjBCi7SwJHckfv/fJeG0Z4oBkmMYYrjICVKxETKvqPEdIKfmp02quNtUmkqJPNZDQSLMVjArjgiWZSsq+hIDIlqVyoxQhHeMgy4KHFDBiOwhTZM2vMDUokro8Ybhm5bZmEKJ/EWGPDcf+mfmfWm2lr8re8lbM22N3HyaZLPAZFPILINVlOSYg6S4heJizu33+aueyT3mtTnbo6+8QXib67/+wEAAA==" target="_blank">Run the query</a>
::: moniker-end

The following example defines a pattern that defines some scoped application data.

```kusto
declare pattern App = (applicationId:string)[scope:string]
declare pattern App = (applicationId:string)[scope:string]
{
('a1').['Data'] = { range x from 1 to 5 step 1 | project App = "App #1", Data = x };
('a1').['Metrics'] = { range x from 1 to 5 step 1 | project App = "App #1", Metrics = rand() };
Expand Down Expand Up @@ -147,23 +155,34 @@ union app("ApplicationX").["*"]
| count
```

**Returns semantic error**
**Output semantic error**

> One or more pattern references were not declared. Detected pattern references: ["app('ApplicationX').['*']"]

## Work with middle-tier applications

A middle-tier application provides its users with the ability to use KQL and wants to enhance the experience by enriching the query results with augmented data from its internal service.

To this end, the application provides users with a pattern statement that returns tabular data that their users can use in their queries. The pattern's arguments are the keys the application will use to retrieve the enrichment data. When the user runs the query, the application does not parse the query itself but instead plans to leverage the error returned by an empty pattern to retrieve the keys it requires. So it prepends the query with the empty pattern declaration, sends it to the cluster for processing, and then parses the returned HTTP header to retrieve the values of missing pattern arguments. The application uses these values to look up the enrichment data and builds a new declaration that defines the appropriate enrichment data mapping. Finally, the application prepends the new definition to the user's query, resends it for processing, and returns the result it receives to the user.
To this end, the application provides users with a pattern statement that returns tabular data that their users can use in their queries. The pattern's arguments are the keys the application will use to retrieve the enrichment data.

When the user runs the query, the application doesn't parse the query itself but instead uses the error returned by an empty pattern to retrieve the keys it requires. So it prepends the query with the empty pattern declaration, sends it to the cluster for processing, and then parses the returned HTTP header to retrieve the values of missing pattern arguments. The application uses these values to look up the enrichment data and builds a new declaration that defines the appropriate enrichment data mapping.

Finally, the application prepends the new definition to the query, resends it for processing, and returns the result it receives to the user.

### Example

In the following example, a middle-tier application provides the ability to enrich queries with longitude/latitude locations. The application uses an internal service to map IP addresses to longitude/latitude locations, and provides a pattern called `map_ip_to_longlat` for this purpose. Let's suppose the application gets the following query from the user:
In the examples, a pattern is declared, defined, and then invoked.

#### Declare an empty pattern

In this example, a middle-tier application enriches queries with longitude/latitude locations. The application uses an internal service to map IP addresses to longitude/latitude locations, and provides a pattern called `map_ip_to_longlat`. When the query is run, it returns an error with missing pattern definitions:

```kusto
map_ip_to_longlat("10.10.10.10")
```

#### Declare and define a pattern

The application does not parse this query and hence does not know which IP address (*10.10.10.10*) was passed to the pattern. So it prepends the user query with an empty `map_ip_to_longlat` pattern declaration and sends it for processing:

```kusto
Expand All @@ -174,7 +193,11 @@ map_ip_to_longlat("10.10.10.10")
The application receives the following error in response.
> One or more pattern references were not declared. Detected pattern references: ["map_ip_to_longlat('10.10.10.10')"]

The application inspects the error, determines that the error indicates a missing pattern reference, and retrieves the missing IP address (*10.10.10.10*). It uses the IP address to look up the enrichment data in its internal service and builds a new pattern defining the mapping of the IP address to the corresponding longitude and latitude data. The new pattern is prepended to the user's query and run again. This time the query succeeds because the enrichment data is now declared in the query, and the result is sent to the user.
#### Invoke a pattern

The application inspects the error, determines that the error indicates a missing pattern reference, and retrieves the missing IP address (*10.10.10.10*). It uses the IP address to look up the enrichment data in its internal service and builds a new pattern defining the mapping of the IP address to the corresponding longitude and latitude data. The new pattern is prepended to the user's query and run again.

This time the query succeeds because the enrichment data is now declared in the query, and the result is sent to the user.

:::moniker range="azure-data-explorer"
> [!div class="nextstepaction"]
Expand Down
4 changes: 3 additions & 1 deletion data-explorer/kusto/query/query-parameters-statement.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ title: Query parameters declaration statement
description: Learn how to use the query parameters declaration statement to parameterize queries and protect against injection attacks.
ms.reviewer: alexans
ms.topic: reference
ms.date: 08/11/2024
ms.date: 01/13/2025
monikerRange: "microsoft-fabric || azure-data-explorer"
---
# Query parameters declaration statement
Expand Down Expand Up @@ -44,6 +44,8 @@ To reference query parameters, the query text, or functions it uses, must first

## Example

[!INCLUDE [help-cluster](../includes/help-cluster-samples-stormevents.md)]

:::moniker range="azure-data-explorer"
> [!div class="nextstepaction"]
> <a href="https://dataexplorer.azure.com/clusters/help/databases/Samples?query=H4sIAAAAAAAAA4WNuw7CMBAE+0j5hytBpKAFBBUpXEOPrHgFjvzifAEi8fEkQUBJO9qZNWicZtC1A/enpFl7CDjPvH6o0HYMs3YxnGlLq+V8UxYHiezrG4JkKosn3S8Y9GlqkfeW0QgtvkAF80Y7+hVHL3FsR14nm6OBMhVN1WOfUJFE0e7TGL7/9l+H314eyAAAAA==" target="_blank">Run the query</a>
Expand Down
12 changes: 7 additions & 5 deletions data-explorer/kusto/query/restrict-statement.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ title: Restrict statement
description: Learn how to use the restrict statement to limit tabular views that are visible to subsequent query statements.
ms.reviewer: alexans
ms.topic: reference
ms.date: 08/11/2024
ms.date: 01/13/2025
monikerRange: "microsoft-fabric || azure-data-explorer"
---
# Restrict statement
Expand Down Expand Up @@ -38,9 +38,11 @@ The middle-tier application can prefix the user's query with a **logical model**

## Examples

[!INCLUDE [help-cluster](../includes/help-cluster-samples-stormevents.md)]

### Let statement

The following example uses a [let statement](let-statement.md) appearing before `restrict` statement.
The example uses a [let statement](let-statement.md) appearing before `restrict` statement.

```kusto
// Limit access to 'Test' let statement only
Expand All @@ -50,7 +52,7 @@ restrict access to (Test);

### Tables or functions

The following example uses references to [tables](../management/tables.md) or [functions](../management/functions.md) that are defined in the database metadata.
The example uses references to [tables](../management/tables.md) or [functions](../management/functions.md) that are defined in the database metadata.

```kusto
// Assuming the database that the query uses has table Table1 and Func1 defined in the metadata,
Expand All @@ -61,7 +63,7 @@ restrict access to (database().Table1, database().Func1, database('DB2').Table2)

### Patterns

The following example uses wildcard patterns that can match multiples of [let statements](let-statement.md) or tables/functions.
The example uses wildcard patterns that can match multiples of [let statements](let-statement.md) or tables/functions.

```kusto
let Test1 = () { print x=1 };
Expand All @@ -82,7 +84,7 @@ restrict access to (database('DB2').*);

### Prevent user from querying other user data

The following example shows how a middle-tier application can prepend a user's query
The example shows how a middle-tier application can prepend a user's query
with a logical model that prevents the user from querying any other user's data.

```kusto
Expand Down
4 changes: 3 additions & 1 deletion data-explorer/kusto/query/set-statement.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ title: Set statement
description: Learn how to use the set statement to set a request property for the duration of the query.
ms.reviewer: alexans
ms.topic: reference
ms.date: 08/11/2024
ms.date: 01/13/2025
monikerRange: "microsoft-fabric || azure-data-explorer"
---
# Set statement
Expand Down Expand Up @@ -35,6 +35,8 @@ Request properties aren't formally a part of the Kusto Query Language and may be

## Example

[!INCLUDE [help-cluster](../includes/help-cluster-samples-stormevents.md)]

```kusto
set querytrace;
Events | take 100
Expand Down
8 changes: 5 additions & 3 deletions data-explorer/kusto/query/tabular-expression-statements.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ title: Tabular expression statements
description: Learn how to use tabular expression statements to produce tabular datasets.
ms.reviewer: alexans
ms.topic: reference
ms.date: 08/11/2024
ms.date: 01/13/2025
---
# Tabular expression statements

Expand Down Expand Up @@ -41,9 +41,11 @@ A tabular data source produces sets of records, to be further processed by tabul

## Examples

[!INCLUDE [help-cluster](../includes/help-cluster-samples-stormevents.md)]

### Filter rows by condition

The following query counts the number of records in the `StormEvents` table that have a value of "FLORIDA" in the `State` column.
This query counts the number of records in the `StormEvents` table that have a value of "FLORIDA" in the `State` column.

:::moniker range="azure-data-explorer"
> [!div class="nextstepaction"]
Expand All @@ -64,7 +66,7 @@ StormEvents

### Combine data from two tables

In the following example, the [join](join-operator.md) operator is used to combine records from two tabular data sources: the `StormEvents` table and the `PopulationData` table.
In this example, the [join](join-operator.md) operator is used to combine records from two tabular data sources: the `StormEvents` table and the `PopulationData` table.

:::moniker range="azure-data-explorer"
> [!div class="nextstepaction"]
Expand Down
2 changes: 1 addition & 1 deletion data-explorer/monitor-queued-ingestion.md
Original file line number Diff line number Diff line change
Expand Up @@ -269,5 +269,5 @@ Click away from the settings to get the full chart that compares the number of e
## Related content

* [Monitor Azure Data Explorer performance, health, and usage with metrics](using-metrics.md)
* [Monitor Azure Data Explorer ingestion, commands, queries, and tables using diagnostic logs](using-diagnostic-logs.md)
* [Monitor Azure Data Explorer ingestion, commands, queries, and tables using diagnostic logs](monitor-data-explorer.md#monitor-azure-data-explorer-ingestion-commands-queries-and-tables-using-diagnostic-logs)
* [Use Azure Monitor Insights](/azure/azure-monitor/insights/data-explorer)
Loading