diff --git a/data-explorer/kusto/api/netfx/controlling-tracing.md b/data-explorer/kusto/api/netfx/controlling-tracing.md index d1dbd7977d..df69b5089a 100644 --- a/data-explorer/kusto/api/netfx/controlling-tracing.md +++ b/data-explorer/kusto/api/netfx/controlling-tracing.md @@ -1,74 +1,64 @@ --- -title: Control and suppress SDK client side tracing +title: Control and suppress SDK client side-tracing description: This article describes controlling and suppressing SDK client-side tracing. ms.reviewer: orspodek ms.topic: reference ms.custom: has-adal-ref -ms.date: 08/11/2024 +ms.date: 02/03/2025 --- # Controlling and suppressing Kusto SDK client-side tracing > [!INCLUDE [applies](../../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../../includes/applies-to-version/azure-data-explorer.md)] -The Kusto client libraries use a common platform for tracing. The platform uses a large number of trace sources (`System.Diagnostics.TraceSource`), and each is connected to the default set of trace listeners (`System.Diagnostics.Trace.Listeners`) during its construction. +The Kusto client libraries are instrumented to write traces to local files. By default, the tracing mechanism is disabled but can be enabled programmatically. -If an application has trace listeners associated with the default `System.Diagnostics.Trace` instance -(for example, through its `app.config` file), then the Kusto client libraries will emit traces to those listeners. +## Enable tracing -The tracing can be suppressed or controlled programmatically or through a config file. - -## Suppress tracing programmatically - -To suppress tracing from the Kusto client libraries programmatically, invoke this piece of code when loading the relevant library: +To enable tracing, make sure you have the `Microsoft.Azure.Kusto.Cloud.Platform` [NuGet package](https://www.nuget.org/packages/Microsoft.Azure.Kusto.Cloud.Platform/) installed. Then run the following code: ```csharp -TraceSourceManager.SetTraceVerbosityForAll(TraceVerbosity.Fatal); +using Kusto.Cloud.Platform.Utils; // Requires the NuGet package, Microsoft.Azure.Kusto.Cloud.Platform. + +var manifest = new RollingCsvTraceListener2Manifest +{ + TracesLocalRootPath=@"c:\temp" // The folder where trace files will be written. +}; +RollingCsvTraceListener2.CreateAndInitialize(manifest); +TraceSourceManager.StartupDone(); ``` -## Use a config file to suppress tracing +## Control trace level -To suppress tracing from the client libraries through a config file, modify the file `Kusto.Cloud.Platform.dll.tweaks` (which is included with the `Kusto.Data` library). +Each trace source in the library can have its own default verbosity level. A trace source only writes to file traces whose verbosity is equal to or above its own verbosity level. You can control the verbosity of the trace sources. The following example sets the verbosity level for all trace sources to `Verbose`, ensuring that all traces are written to files: -```xml - - +```csharp +using Kusto.Cloud.Platform.Utils; // Requires the NuGet package, Microsoft.Azure.Kusto.Cloud.Platform. + +TraceSourceManager.SetOverrideTraceVerbosityLevel(TraceVerbosity.Verbose); ``` -> [!NOTE] -> For the tweak to take effect, there must not be a minus sign in the value of `key` +Use the `TraceVerbosity.Fatal` argument to trace only the most severe events. + +## Flush all pending traces -An alternative, is: +Flushing pending trace is recommended when the application hosting the trace system is closed to ensure unwritten traces are saved. It can be done safely even if the tracing system isn't initialized. The following code forces all pending traces to flush to files and recycle all files: ```csharp -Anchor.Tweaks.SetProgrammaticAppSwitch( - "Kusto.Cloud.Platform.Utils.Tracing.OverrideTraceVerbosityLevel", - "0" -); +TraceSourceManager.SuperFlush(SuperFlushMode.Emergency); ``` -## Enable client libraries tracing - -To enable tracing out of the client libraries, enable .NET tracing in your application's *app.config file*. For example, assume that the application `MyApp.exe` uses the Kusto.Data client library. Changing file *MyApp.exe.config* to include the following, will enable `Kusto.Data` tracing the next time that the application starts. - -```xml - - - - - - - - - - - -``` +## Enable MSAL (Microsoft Authentication Library) tracing -The code will configure a trace listener that writes to CSV files in a subdirectory called *RollingLogs*. The subdirectory is located in the process' directory. + Enabling tracing for client libraries automatically enables tracing for [MSAL (Microsoft Authentication Library)](/azure/active-directory/develop/msal-overview). -> [!NOTE] -> Any .NET-compatible trace listener class may be used as well. +## Read trace files -## Enable MSAL (Microsoft Authentication Library) tracing +Once the tracing system is initialized, trace files are written to the specified folder or its subfolders. They're formatted as CSV files with the `.csv` extension. Files that are currently being written use the extension `.csv.in-progress` and are automatically renamed once they're completed. + +Each trace file record includes the following fields: -Once tracing for client libraries is enabled, tracing for [MSAL (Microsoft Authentication Library)](/azure/active-directory/develop/msal-overview) is enabled automatically. +* **Trace record identifier:** Uniquely identifies each trace record. +* **Timestamp:** The timestamp of the trace record. +* **Trace source name:** The name of the trace source. +* **Trace level:** The verbosity level of the trace. +* **Textual content:** The trace record content. \ No newline at end of file diff --git a/data-explorer/kusto/api/powershell/powershell.md b/data-explorer/kusto/api/powershell/powershell.md index 67f7ff5994..87198789a2 100644 --- a/data-explorer/kusto/api/powershell/powershell.md +++ b/data-explorer/kusto/api/powershell/powershell.md @@ -3,7 +3,7 @@ title: Kusto .NET Client Libraries from PowerShell description: This article describes how to use Kusto .NET Client Libraries from PowerShell. ms.reviewer: salevy ms.topic: reference -ms.date: 08/11/2024 +ms.date: 02/02/2025 --- # Use Kusto .NET client libraries from PowerShell @@ -21,7 +21,7 @@ To use the Kusto .NET client libraries in PowerShell: 1. Download [`Microsoft.Azure.Kusto.Tools`](https://www.nuget.org/packages/Microsoft.Azure.Kusto.Tools/). 1. Right-click on the downloaded package. From the menu, select your archiving tool and extract the package contents. If the archiving tool isn't visible from the menu, select **Show more options**. The extraction results in multiple folders, one of which is named *tools*. -1. Inside the *tools* folder, there are different subfolders catering to different PowerShell versions. For PowerShell version 5.1, use the *net472* folder. For PowerShell version 7 or later, use any of the version folders. Copy the path of the relevant folder. +1. Inside the *tools* folder, there are different subfolders catering to different PowerShell versions. For PowerShell version 5.1, use the *net472* folder. For PowerShell version 7 or later, use any of the version folders except the *net472* folder. Copy the path of the relevant folder. 1. From PowerShell, load the libraries, replacing `` with the copied folder path: ```powershell @@ -36,13 +36,16 @@ To use the Kusto .NET client libraries in PowerShell: Once loaded, you can use the libraries to [connect to a cluster and database](#connect-to-a-cluster-and-database). +> [!NOTE] +> The tools in the *net472* folder aren't supported on Linux. + ## Connect to a cluster and database Authenticate to a cluster and database with one of the following methods: * **User authentication:** Prompt the user to verify their identity in a web browser. * **Application authentication:** [Create a Microsoft Entra app](../../access-control/provision-entra-id-app.md) and use the credentials for authentication. -* **Azure CLI authentication:** Sign-in to the Azure CLI on your machine, and Kusto will retrieve the token from Azure CLI. +* **Azure CLI authentication:** Sign-in to the Azure CLI on your machine, and Kusto retrieves the token from Azure CLI. Select the relevant tab. @@ -59,7 +62,7 @@ $kcsb = New-Object Kusto.Data.KustoConnectionStringBuilder($clusterUrl, $databas ### [Application](#tab/app) -[Create an MS Entra app](../../access-control/provision-entra-id-app.md) and grant it access to your database. Then, provide the app credentials in place of the `$applicationId`, `$applicationKey`, and `$authority`. +[Create a Microsoft Entra application](../../access-control/provision-entra-id-app.md) and grant it access to your database. Then, provide the app credentials in place of the `$applicationId`, `$applicationKey`, and `$authority`. ```powershell $clusterUrl = "" @@ -90,7 +93,7 @@ $kcsb = $kcsb.WithAadAzCliAuthentication() Create a query provider and run [Kusto Query Language](../../query/index.md) queries. -The following example defines a simple [take](../../query/take-operator.md) query to sample the data. To run the query, replace `` with the name of a table in your database. Before running the query, the [ClientRequestProperties class](../netfx/client-request-properties.md) is used to set a client request ID and a server timeout. Then, the query is run and the result set is formatted and sorted. +The following example defines a simple [take](../../query/take-operator.md) query to sample the data. To run the query, replace `` with the name of a table in your database. The [ClientRequestProperties class](../netfx/client-request-properties.md) is used to set a client request ID and a server time-out before running the query. Then, the query is run and the result set is formatted and sorted. ```powershell $queryProvider = [Kusto.Data.Net.Client.KustoClientFactory]::CreateCslQueryProvider($kcsb) diff --git a/data-explorer/kusto/query/join-rightouter.md b/data-explorer/kusto/query/join-rightouter.md index fbb6368c86..8aae1bf21f 100644 --- a/data-explorer/kusto/query/join-rightouter.md +++ b/data-explorer/kusto/query/join-rightouter.md @@ -3,7 +3,7 @@ title: rightouter join description: Learn how to use the rightouter join flavor to merge the rows of two tables. ms.reviewer: alexans ms.topic: reference -ms.date: 08/11/2024 +ms.date: 01/21/2025 --- # rightouter join @@ -29,6 +29,8 @@ The `rightouter` join flavor returns all the records from the right side and onl ## Example +This query returns all rows from table Y and any matching rows from table X, filling in NULL values where there is no match from X. + :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] > Run the query diff --git a/data-explorer/kusto/query/join-rightsemi.md b/data-explorer/kusto/query/join-rightsemi.md index 8804be7825..75faabf193 100644 --- a/data-explorer/kusto/query/join-rightsemi.md +++ b/data-explorer/kusto/query/join-rightsemi.md @@ -3,7 +3,7 @@ title: rightsemi join description: Learn how to use the rightsemi join flavor to merge the rows of two tables. ms.reviewer: alexans ms.topic: reference -ms.date: 08/11/2024 +ms.date: 01/21/2025 --- # rightsemi join @@ -29,6 +29,8 @@ The `rightsemi` join flavor returns all records from the right side that match a ## Example +This query filters and returns only those rows from table Y that have a matching key in table X. + :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] > Run the query @@ -59,3 +61,7 @@ X | join kind=rightsemi Y on Key | b | 10 | | c | 20 | | c | 30 | + +## Related content + +* Learn about other [join flavors](join-operator.md#returns) \ No newline at end of file diff --git a/data-explorer/kusto/query/join-time-window.md b/data-explorer/kusto/query/join-time-window.md index 149bfb4737..a92fb8b0c4 100644 --- a/data-explorer/kusto/query/join-time-window.md +++ b/data-explorer/kusto/query/join-time-window.md @@ -3,7 +3,7 @@ title: Joining within time window description: Learn how to perform a time window join operation to match between two large datasets. ms.reviewer: alexans ms.topic: reference -ms.date: 08/11/2024 +ms.date: 01/28/2025 --- # Time window join @@ -11,20 +11,33 @@ ms.date: 08/11/2024 It's often useful to join between two large datasets on some high-cardinality key, such as an operation ID or a session ID, and further limit the right-hand-side ($right) records that need to match up with each left-hand-side ($left) record by adding a restriction on the "time-distance" between `datetime` columns on the left and on the right. -The above operation differs from the usual Kusto join operation, since for the `equi-join` part of matching the high-cardinality key between the left and right datasets, the system can also apply a distance function and use it to considerably speed up the join. +The above operation differs from the usual join operation, since for the `equi-join` part of matching the high-cardinality key between the left and right datasets, the system can also apply a distance function and use it to considerably speed up the join. > [!NOTE] -> A distance function doesn't behave like equality (that is, when both dist(x,y) and dist(y,z) are true it doesn't follow that dist(x,z) is also true.) Internally, we sometimes refer to this as "diagonal join". +> A distance function doesn't behave like equality (that is, when both dist(x,y) and dist(y,z) are true it doesn't follow that dist(x,z) is also true.) This is sometimes referred to as a "diagonal join". -For example, if you want to identify event sequences within a relatively small time window, assume that you have a table `T` with the following schema: +## Example to identify event sequences without time window + +To identify event sequences within a relatively small time window, this example uses a table `T` with the following schema: * `SessionId`: A column of type `string` with correlation IDs. * `EventType`: A column of type `string` that identifies the event type of the record. * `Timestamp`: A column of type `datetime` indicates when the event described by the record happened. +| SessionId | EventType | Timestamp | +|--|--|--| +| 0 | A | 2017-10-01T00:00:00Z | +| 0 | B | 2017-10-01T00:01:00Z | +| 1 | B | 2017-10-01T00:02:00Z | +| 1 | A | 2017-10-01T00:03:00Z | +| 3 | A | 2017-10-01T00:04:00Z | +| 3 | B | 2017-10-01T00:10:00Z | + +The following query creates the dataset and then identifies all the session IDs in which event type `A` was followed by an event type `B` within a `1min` time window. + :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] -> Run the query +> Run the query ::: moniker-end ```kusto @@ -38,38 +51,6 @@ let T = datatable(SessionId:string, EventType:string, Timestamp:datetime) '3', 'B', datetime(2017-10-01 00:10:00), ]; T -``` - -**Output** - -|SessionId|EventType|Timestamp| -|---|---|---| -|0|A|2017-10-01 00:00:00.0000000| -|0|B|2017-10-01 00:01:00.0000000| -|1|B|2017-10-01 00:02:00.0000000| -|1|A|2017-10-01 00:03:00.0000000| -|3|A|2017-10-01 00:04:00.0000000| -|3|B|2017-10-01 00:10:00.0000000| - -**Problem statement** - -Our query should answer the following question: - - Find all the session IDs in which event type `A` was followed by an - event type `B` within a `1min` time window. - -> [!NOTE] -> In the sample data above, the only such session ID is `0`. - -Semantically, the following query answers this question, albeit inefficiently. - -:::moniker range="azure-data-explorer" -> [!div class="nextstepaction"] -> Run the query -::: moniker-end - -```kusto -T | where EventType == 'A' | project SessionId, Start=Timestamp | join kind=inner @@ -84,43 +65,15 @@ T **Output** -|SessionId|Start|End| -|---|---|---| -|0|2017-10-01 00:00:00.0000000|2017-10-01 00:01:00.0000000| +| SessionId | Start | End | +|--|--|--| +| 0 | 2017-10-01 00:00:00.0000000 | 2017-10-01 00:01:00.0000000 | -To optimize this query, we can rewrite it as described below -so that the time window is expressed as a join key. +## Example optimized with time window -**Rewrite the query to account for the time window** +To optimize this query, we can rewrite it to account for the time window. THe time window is expressed as a join key. Rewrite the query so that the `datetime` values are "discretized" into buckets whose size is half the size of the time window. Use *`equi-join`* to compare the bucket IDs. -Rewrite the query so that the `datetime` values are "discretized" into buckets whose size is half the size of the time window. Use Kusto's *`equi-join`* to compare those bucket IDs. - -```kusto -let lookupWindow = 1min; -let lookupBin = lookupWindow / 2.0; // lookup bin = equal to 1/2 of the lookup window -T -| where EventType == 'A' -| project SessionId, Start=Timestamp, - // TimeKey on the left side of the join is mapped to a discrete time axis for the join purpose - TimeKey = bin(Timestamp, lookupBin) -| join kind=inner - ( - T - | where EventType == 'B' - | project SessionId, End=Timestamp, - // TimeKey on the right side of the join - emulates event 'B' appearing several times - // as if it was 'replicated' - TimeKey = range(bin(Timestamp-lookupWindow, lookupBin), - bin(Timestamp, lookupBin), - lookupBin) - // 'mv-expand' translates the TimeKey array range into a column - | mv-expand TimeKey to typeof(datetime) - ) on SessionId, TimeKey -| where (End - Start) between (0min .. lookupWindow) -| project SessionId, Start, End -``` - -**Runnable query reference (with table inlined)** +The query finds pairs of events within the same session (*SessionId*) where an 'A' event is followed by a 'B' event within 1 minute. It projects the session ID, the start time of the 'A' event, and the end time of the 'B' event. :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] @@ -158,13 +111,13 @@ T **Output** -|SessionId|Start|End| -|---|---|---| -|0|2017-10-01 00:00:00.0000000|2017-10-01 00:01:00.0000000| +| SessionId | Start | End | +|--|--|--| +| 0 | 2017-10-01 00:00:00.0000000 | 2017-10-01 00:01:00.0000000 | -**5M data query** +## 5 million data query -The next query emulates a dataset of 5M records and ~1M IDs and runs the query with the technique described above. +The next query emulates an extensive dataset of 5M records and approximately 1M Session IDs and runs the query with the time window technique. :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] @@ -199,6 +152,10 @@ T **Output** -|Count| -|---| -|3344| +| Count | +|--| +| 3344 | + +## Related content + +* [join operator](join-operator.md) diff --git a/data-explorer/kusto/query/let-statement.md b/data-explorer/kusto/query/let-statement.md index f76b5de2b0..96cc37753a 100644 --- a/data-explorer/kusto/query/let-statement.md +++ b/data-explorer/kusto/query/let-statement.md @@ -3,7 +3,7 @@ title: Let statement description: Learn how to use the Let statement to set a variable name to define an expression or a function. ms.reviewer: alexans ms.topic: reference -ms.date: 08/11/2024 +ms.date: 01/28/2025 ms.localizationpriority: high --- # Let statement @@ -67,6 +67,8 @@ To optimize multiple uses of the `let` statement within a single query, see [Opt [!INCLUDE [help-cluster](../includes/help-cluster-note.md)] +The query examples show the syntax and example usage of the operator, statement, or function. + ### Define scalar values The following example uses a scalar expression statement. @@ -95,13 +97,13 @@ range y from 0 to ['some number'] step 5 **Output** -|y| -|---| -|0| -|5| -|10| -|15| -|20| +| y | +|--| +| 0 | +| 5 | +| 10 | +| 15 | +| 20 | ### Create a user defined function with scalar calculation @@ -120,13 +122,13 @@ range x from 1 to 5 step 1 **Output** -|x|result| -|---|---| -|1|5| -|2|10| -|3|15| -|4|20| -|5|25| +| x | result | +|--|--| +| 1 | 5 | +| 2 | 10 | +| 3 | 15 | +| 4 | 20 | +| 5 | 25 | ### Create a user defined function that trims input @@ -145,14 +147,14 @@ range x from 10 to 15 step 1 **Output** -|x|result| -|---|---| -|10|0| -|11|| -|12|2| -|13|3| -|14|4| -|15|5| +| x | result | +|--|--| +| 10 | 0 | +| 11 | | +| 12 | 2 | +| 13 | 3 | +| 14 | 4 | +| 15 | 5 | ### Use multiple let statements @@ -171,9 +173,9 @@ foo2(2) | count **Output** -|result| -|---| -|50| +| result | +|--| +| 50 | ### Create a view or virtual table @@ -192,10 +194,10 @@ search MyColumn == 5 **Output** -|$table|MyColumn| -|---|---| -|Range10|5| -|Range20|5| +| $table | MyColumn | +|--|--| +| Range10 | 5 | +| Range20 | 5 | ### Use a materialize function @@ -226,11 +228,11 @@ on $left.Day1 == $right.Day **Output** -|Day1|Day2|Percentage| -|---|---|---| -|2016-05-01 00:00:00.0000000|2016-05-02 00:00:00.0000000|34.0645725975255| -|2016-05-01 00:00:00.0000000|2016-05-03 00:00:00.0000000|16.618368960101| -|2016-05-02 00:00:00.0000000|2016-05-03 00:00:00.0000000|14.6291376489636| +| Day1 | Day2 | Percentage | +|--|--|--| +| 2016-05-01 00:00:00.0000000 | 2016-05-02 00:00:00.0000000 | 34.0645725975255 | +| 2016-05-01 00:00:00.0000000 | 2016-05-03 00:00:00.0000000 | 16.618368960101 | +| 2016-05-02 00:00:00.0000000 | 2016-05-03 00:00:00.0000000 | 14.6291376489636 | ### Using nested let statements @@ -261,13 +263,13 @@ StormEvents **Output** | State | s_s | -|---|---| +|--|--| | ATLANTIC SOUTH | ATLANTIC SOUTHATLANTIC SOUTH | | FLORIDA | FLORIDAFLORIDA | | FLORIDA | FLORIDAFLORIDA | | GEORGIA | GEORGIAGEORGIA | | MISSISSIPPI | MISSISSIPPIMISSISSIPPI | -|...|...| +| ... | ... | ### Tabular argument with wildcard diff --git a/data-explorer/kusto/query/lookup-operator.md b/data-explorer/kusto/query/lookup-operator.md index 6b5fca567c..a4d87d35f0 100644 --- a/data-explorer/kusto/query/lookup-operator.md +++ b/data-explorer/kusto/query/lookup-operator.md @@ -3,7 +3,7 @@ title: lookup operator description: Learn how to use the lookup operator to extend columns of a fact table. ms.reviewer: alexans ms.topic: reference -ms.date: 12/04/2024 +ms.date: 01/20/2025 --- # lookup operator @@ -75,7 +75,7 @@ A table with: * If `kind` is unspecified or `kind=leftouter`, then in addition to the inner matches, there's a row for every row on the left (and/or right), even if it has no match. In that case, the unmatched output cells contain nulls. * If `kind=inner`, then there's a row in the output for every combination of matching rows from left and right. -## Examples +## Example The following example shows how to perform a left outer join between the `FactTable` and `DimTable`, based on matching values in the `Personal` and `Family` columns. diff --git a/data-explorer/kusto/query/mv-apply-operator.md b/data-explorer/kusto/query/mv-apply-operator.md index cc3d638e4a..22d6fd3f86 100644 --- a/data-explorer/kusto/query/mv-apply-operator.md +++ b/data-explorer/kusto/query/mv-apply-operator.md @@ -3,7 +3,7 @@ title: mv-apply operator description: Learn how to use the mv-apply operator to apply a subquery to each record and union the results of each subquery. ms.reviewer: alexans ms.topic: reference -ms.date: 08/11/2024 +ms.date: 01/28/2025 --- # mv-apply operator @@ -94,8 +94,13 @@ and *SubQuery* has the same syntax of any query statement. ## Examples +Review the examples and run them in your Data Explorer query page. + ### Getting the largest element from the array +The query outputs the smallest even number (2) and the smallest odd number (1). + + :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] > Run the query @@ -114,13 +119,15 @@ _data **Output** -|`xMod2`|l |element| -|-----|------------|-------| -|1 |[1, 3, 5, 7]|7 | -|0 |[2, 4, 6, 8]|8 | +| `xMod2` | l | element | +|--|--|--| +| 1 | [1, 3, 5, 7] | 7 | +| 0 | [2, 4, 6, 8] | 8 | ### Calculating the sum of the largest two elements in an array +The query outputs the sum of the top 2 even numbers (6 + 8 = 14) and the sum of the top 2 odd numbers (5 + 7 = 12). + :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] > Run the query @@ -140,13 +147,15 @@ _data **Output** -|`xMod2`|l |SumOfTop2| -|-----|---------|---------| -|1 |[1,3,5,7]|12 | -|0 |[2,4,6,8]|14 | +| `xMod2` | l | SumOfTop2 | +|--|--|--| +| 1 | [1,3,5,7] | 12 | +| 0 | [2,4,6,8] | 14 | ### Select elements in arrays +The query identifies the top 2 elements from each dynamic array based on the Arr2 values and summarizes them into new lists. + :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] > Run the query @@ -166,14 +175,16 @@ datatable (Val:int, Arr1:dynamic, Arr2:dynamic) **Output** -|Val1|Arr1|Arr2|`NewArr1`|`NewArr2`| -|-----|-----------|--------|-----|-----| -|1 |["A1","A2","A3"]|[10,30,7]|["A2',"A1"]|[30,10] | -|7 |["B1","B2","B5"]|[15,11,50]|["B5","B1"]|[50,15] | -|3 |["C1","C2","C3","C4"]|[6,40,20,8]|["C2","C3"]|[40,20] | +| Val1 | Arr1 | Arr2 | `NewArr1` | `NewArr2` | +|--|--|--|--|--| +| 1 | ["A1","A2","A3"] | [10,30,7] | ["A2',"A1"] | [30,10] | +| 7 | ["B1","B2","B5"] | [15,11,50] | ["B5","B1"] | [50,15] | +| 3 | ["C1","C2","C3","C4"] | [6,40,20,8] | ["C2","C3"] | [40,20] | ### Using `with_itemindex` for working with a subset of the array +The query results in a table with rows where the index is 3 or greater, including the index and element values from the original lists of even and odd numbers. + :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] > Run the query @@ -194,15 +205,17 @@ _data **Output** -|index|element| -|---|---| -|3|7| -|4|9| -|3|8| -|4|10| +| index | element | +|--|--| +| 3 | 7 | +| 4 | 9 | +| 3 | 8 | +| 4 | 10 | ### Using mutiple columns to join element of 2 arrays +The query combines elements from two dynamic arrays into a new concatenated format and then summarizes them into lists. + :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] > Run the query @@ -222,15 +235,14 @@ datatable (Val: int, Arr1: dynamic, Arr2: dynamic) **Output** -|Val|Arr1|Arr2|`Out`| -|---|---|---|---| -|1|["A1","A2","A3"]|["B1","B2","B3"]|["A1_B1","A2_B2","A3_B3"]| -|5|["C1","C2"]|["D1","D2"]|["C1_D1","C2_D2"]| +| Val | Arr1 | Arr2 | `Out` | +|--|--|--|--| +| 1 | ["A1","A2","A3"] | ["B1","B2","B3"] | ["A1_B1","A2_B2","A3_B3"] | +| 5 | ["C1","C2"] | ["D1","D2"] | ["C1_D1","C2_D2"] | ### Applying mv-apply to a property bag -In the following example, `mv-apply` is used in combination with an -inner `mv-expand` to remove values that don't start with "555" from a property bag: +This query dynamically removes properties from the packed values object based on the criteria that their values do not start with "555". The final result contains the original columns with unwanted properties removed. :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] @@ -256,10 +268,10 @@ datatable(SourceNumber: string, TargetNumber: string, CharsCount: long) **Output** -|SourceNumber|TargetNumber|CharsCount|values -|---|---|---|---| -|555-555-1234|555-555-1212|46|{
"SourceNumber": "555-555-1234",
"TargetNumber": "555-555-1212"
}| -|555-555-1212| | |{
"SourceNumber": "555-555-1212"
}| +| SourceNumber | TargetNumber | CharsCount | values | +|--|--|--|--| +| 555-555-1234 | 555-555-1212 | 46 | {
"SourceNumber": "555-555-1234",
"TargetNumber": "555-555-1212"
} | +| 555-555-1212 |   |   | {
"SourceNumber": "555-555-1212"
} | ## Related content diff --git a/data-explorer/kusto/query/mv-expand-operator.md b/data-explorer/kusto/query/mv-expand-operator.md index cf964c8bb0..6bdcd968c6 100644 --- a/data-explorer/kusto/query/mv-expand-operator.md +++ b/data-explorer/kusto/query/mv-expand-operator.md @@ -3,14 +3,13 @@ title: mv-expand operator description: Learn how to use the mv-expand operator to expand multi-value dynamic arrays or property bags into multiple records. ms.reviewer: alexans ms.topic: reference -ms.date: 08/11/2024 +ms.date: 01/20/2025 monikerRange: "microsoft-fabric || azure-data-explorer || azure-monitor || microsoft-sentinel " --- # mv-expand operator > [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] [!INCLUDE [monitor](../includes/applies-to-version/monitor.md)] [!INCLUDE [sentinel](../includes/applies-to-version/sentinel.md)] - Expands multi-value dynamic arrays or property bags into multiple records. `mv-expand` can be described as the opposite of the aggregation operators @@ -66,6 +65,10 @@ Two modes of property bag expansions are supported: ## Examples +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] + +The examples in this section show how to use the syntax to help you get started. + ### Single column - array expansion :::moniker range="azure-data-explorer" @@ -84,12 +87,12 @@ datatable (a: int, b: dynamic) **Output** -|a|b| -|---|---| -|1|10| -|1|20| -|2|a| -|2|b| +| a | b | +|--|--| +| 1 | 10 | +| 1 | 20 | +| 2 | a | +| 2 | b | ### Single column - bag expansion @@ -111,12 +114,12 @@ datatable (a: int, b: dynamic) **Output** -|a|b| -|---|---| -|1|{"prop1": "a1"}| -|1|{"prop2": "b1"}| -|2|{"prop1": "a2"}| -|2|{"prop2": "b2"}| +| a | b | +|--|--| +| 1 | {"prop1": "a1"} | +| 1 | {"prop2": "b1"} | +| 2 | {"prop1": "a2"} | +| 2 | {"prop2": "b2"} | ### Single column - bag expansion to key-value pairs @@ -139,12 +142,12 @@ datatable (a: int, b: dynamic) **Output** -|a|b|key|val| -|---|---|---|---| -|1|["prop1","a1"]|prop1|a1| -|1|["prop2","b1"]|prop2|b1| -|2|["prop1","a2"]|prop1|a2| -|2|["prop2","b2"]|prop2|b2| +| a | b | key | val | +|--|--|--|--| +| 1 | ["prop1","a1"] | prop1 | a1 | +| 1 | ["prop2","b1"] | prop2 | b1 | +| 2 | ["prop1","a2"] | prop1 | a2 | +| 2 | ["prop2","b2"] | prop2 | b2 | ### Zipped two columns @@ -164,11 +167,11 @@ datatable (a: int, b: dynamic, c: dynamic)[ **Output** -|a|b|c| -|---|---|---| -|1|{"prop1":"a"}|5| -|1|{"prop2":"b"}|4| -|1||3| +| a | b | c | +|--|--|--| +| 1 | {"prop1":"a"} | 5 | +| 1 | {"prop2":"b"} | 4 | +| 1 | | 3 | ### Cartesian product of two columns @@ -190,12 +193,12 @@ datatable (a: int, b: dynamic, c: dynamic) **Output** -|a|b|c| -|---|---|---| -|1|{ "prop1": "a"}|5| -|1|{ "prop1": "a"}|6| -|1|{ "prop2": "b"}|5| -|1|{ "prop2": "b"}|6| +| a | b | c | +|--|--|--| +| 1 | { "prop1": "a"} | 5 | +| 1 | { "prop1": "a"} | 6 | +| 1 | { "prop2": "b"} | 5 | +| 1 | { "prop2": "b"} | 6 | ### Convert output @@ -217,7 +220,7 @@ datatable (a: string, b: dynamic, c: dynamic)[ **Output** | ColumnName | ColumnOrdinal | DateType | ColumnType | -|---|---|---|---| +|--|--|--|--| | a | 0 | System.String | `string` | | b | 1 | System.Object | `dynamic` | | c | 2 | System.Int32 | `int` | @@ -241,12 +244,12 @@ range x from 1 to 4 step 1 **Output** -|x|Index| -|---|---| -|1|0| -|2|1| -|3|2| -|4|3| +| x | Index | +|--|--| +| 1 | 0 | +| 2 | 1 | +| 3 | 2 | +| 4 | 3 | ::: moniker range="microsoft-fabric || azure-data-explorer || azure-monitor || microsoft-sentinel" ## Related content @@ -254,4 +257,4 @@ range x from 1 to 4 step 1 * [mv-apply](mv-apply-operator.md) operator. * For the opposite of the mv-expand operator, see [summarize make_list()](make-list-aggregation-function.md). * For expanding dynamic JSON objects into columns using property bag keys, see [bag_unpack()](bag-unpack-plugin.md) plugin. -::: moniker-end \ No newline at end of file +::: moniker-end diff --git a/data-explorer/kusto/query/pattern-statement.md b/data-explorer/kusto/query/pattern-statement.md index c93378ff9e..b50d176806 100644 --- a/data-explorer/kusto/query/pattern-statement.md +++ b/data-explorer/kusto/query/pattern-statement.md @@ -85,9 +85,9 @@ country("Canada").Alberta **Output** -|Capital| -|-------| -|Edmonton| +| Capital | +|--| +| Edmonton | ### Define a scoped pattern @@ -111,18 +111,18 @@ union App('a2').Data, App('a1').Metrics **Output** -|App|Data|Metrics| -|---|----|-------| -|App #2|9| | -|App #2|8| | -|App #2|7| | -|App #2|6| | -|App #2|5| | -|App #1| |0.53674122855537532| -|App #1| |0.78304713305654439| -|App #1| |0.20168860732346555| -|App #1| |0.13249123867679469| -|App #1| |0.19388305330563443| +| App | Data | Metrics | +|--|--|--| +| App #2 | 9 | | +| App #2 | 8 | | +| App #2 | 7 | | +| App #2 | 6 | | +| App #2 | 5 | | +| App #1 | | 0.53674122855537532 | +| App #1 | | 0.78304713305654439 | +| App #1 | | 0.20168860732346555 | +| App #1 | | 0.13249123867679469 | +| App #1 | | 0.19388305330563443 | ### Normalization @@ -214,6 +214,6 @@ map_ip_to_longlat("10.10.10.10") **Output** -|Lat| `Long` | -|---|---| -|37.405992|-122.078515| +| Lat | `Long` | +|--|--| +| 37.405992 | -122.078515 | diff --git a/data-explorer/kusto/query/project-away-operator.md b/data-explorer/kusto/query/project-away-operator.md index f18581ba2c..6f9be84a7a 100644 --- a/data-explorer/kusto/query/project-away-operator.md +++ b/data-explorer/kusto/query/project-away-operator.md @@ -3,7 +3,7 @@ title: project-away operator description: Learn how to use the project-away operator to select columns from the input table to exclude from the output table. ms.reviewer: alexans ms.topic: reference -ms.date: 08/11/2024 +ms.date: 01/20/2025 monikerRange: "microsoft-fabric || azure-data-explorer || azure-monitor || microsoft-sentinel " --- # project-away operator @@ -38,6 +38,8 @@ A table with columns that weren't named as arguments. Contains same number of ro ## Examples +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] + The input table `PopulationData` has 2 columns: `State` and `Population`. Project-away the `Population` column and you're left with a list of state names. :::moniker range="azure-data-explorer" @@ -50,25 +52,27 @@ PopulationData | project-away Population ``` +**Output** + The following table shows only the first 10 results. -|State| -|---| -|ALABAMA| -|ALASKA| -|ARIZONA| -|ARKANSAS| -|CALIFORNIA| -|COLORADO| -|CONNECTICUT| -|DELAWARE| -|DISTRICT OF COLUMBIA| -|FLORIDA| -|...| +| State | +|--| +| ALABAMA | +| ALASKA | +| ARIZONA | +| ARKANSAS | +| CALIFORNIA | +| COLORADO | +| CONNECTICUT | +| DELAWARE | +| DISTRICT OF COLUMBIA | +| FLORIDA | +| ... | ### Project-away using a column name pattern -The following query removes columns starting with the word "session". +This query removes columns starting with the word "session". :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] @@ -80,27 +84,26 @@ ConferenceSessions | project-away session* ``` -The following table shows only the first 10 results. +**Output** + +The table shows only the first 10 results. + +| conference | owner | participants | URL | level | starttime | duration | time_and_duration | kusto_affinity | +|--|--|--|--|--|--|--|--|--| +| PASS Summit 2019 | Avner Aharoni | | | | 2019-11-07T19:15:00Z | | Thu, Nov 7, 11:15 AM-12:15 PM PST | Focused | +| PASS Summit | Rohan Kumar | Ariel Pisetzky | | | 2018-11-07T08:15:00Z | 90 | Wed, Nov 7, 8:15-9:45 am | Mention | +| Intelligent Cloud 2019 | Rohan Kumar | Henning Rauch | | | 2019-04-09T09:00:00Z | 90 | Tue, Apr 9, 9:00-10:30 AM | Mention | +| Ignite 2019 | Jie Feng | | `https://myignite.techcommunity.microsoft.com/sessions/83940` | 100 | 2019-11-06T14:35:00Z | 20 | Wed, Nov 6, 9:35 AM - 9:55 AM | Mention | +| Ignite 2019 | Bernhard Rode | Le Hai Dang, Ricardo Niepel | `https://myignite.techcommunity.microsoft.com/sessions/81596` | 200 | 2019-11-06T16:45:00Z | 45 | Wed, Nov 6, 11:45 AM-12:30 PM | Mention | +| Ignite 2019 | Tzvia Gitlin | Troyna | `https://myignite.techcommunity.microsoft.com/sessions/83933` | 400 | 2019-11-06T17:30:00Z | 75 | Wed, Nov 6, 12:30 PM-1:30 PM | Focused | +| Ignite 2019 | Jie Feng | `https://myignite.techcommunity.microsoft.com/sessions/81057` | 300 | 2019-11-06T20:30:00Z | 45 | Wed, Nov 6, 3:30 PM-4:15 PM | Mention | +| Ignite 2019 | Manoj Raheja | | `https://myignite.techcommunity.microsoft.com/sessions/83939` | 300 | 2019-11-07T18:15:00Z | 20 | Thu, Nov 7, 1:15 PM-1:35 PM | Focused | +| Ignite 2019 | Uri Barash | | `https://myignite.techcommunity.microsoft.com/sessions/81060` | 300 | 2019-11-08T17:30:00Z | 45 | Fri, Nov8, 10:30 AM-11:15 AM | Focused | +| Ignite 2018 | Manoj Raheja | | | 200 | | 20 | | Focused | +| ... | ... | ... | ... | ... | ... | ... | ... | ... | -|conference|owner|participants|URL|level|starttime|duration|time_and_duration|kusto_affinity| -|---|---|---|---|---|---|---|---|---| -|PASS Summit 2019| Avner Aharoni| || |2019-11-07T19:15:00Z| |Thu, Nov 7, 11:15 AM-12:15 PM PST |Focused| -|PASS Summit| Rohan Kumar| Ariel Pisetzky|| |2018-11-07T08:15:00Z| 90 |Wed, Nov 7, 8:15-9:45 am |Mention| -|Intelligent Cloud 2019| Rohan Kumar| Henning Rauch| | |2019-04-09T09:00:00Z| 90| Tue, Apr 9, 9:00-10:30 AM |Mention| -|Ignite 2019| Jie Feng| | `https://myignite.techcommunity.microsoft.com/sessions/83940` | 100| 2019-11-06T14:35:00Z| 20 |Wed, Nov 6, 9:35 AM - 9:55 AM| Mention| -|Ignite 2019| Bernhard Rode| Le Hai Dang, Ricardo Niepel |`https://myignite.techcommunity.microsoft.com/sessions/81596` | 200 |2019-11-06T16:45:00Z| 45| Wed, Nov 6, 11:45 AM-12:30 PM |Mention| -|Ignite 2019| Tzvia Gitlin| Troyna| `https://myignite.techcommunity.microsoft.com/sessions/83933` | 400 |2019-11-06T17:30:00Z| 75| Wed, Nov 6, 12:30 PM-1:30 PM |Focused| -|Ignite 2019| Jie Feng | `https://myignite.techcommunity.microsoft.com/sessions/81057` | 300| 2019-11-06T20:30:00Z| 45 |Wed, Nov 6, 3:30 PM-4:15 PM |Mention| -|Ignite 2019| Manoj Raheja| | `https://myignite.techcommunity.microsoft.com/sessions/83939` | 300| 2019-11-07T18:15:00Z| 20 |Thu, Nov 7, 1:15 PM-1:35 PM| Focused| -|Ignite 2019| Uri Barash| | `https://myignite.techcommunity.microsoft.com/sessions/81060` | 300| 2019-11-08T17:30:00Z| 45 |Fri, Nov8, 10:30 AM-11:15 AM| Focused| -|Ignite 2018| Manoj Raheja| || 200| |20| |Focused| -|...|...|...|...|...|...|...|...|...| - - -::: moniker range="microsoft-fabric || azure-data-explorer || azure-monitor || microsoft-sentinel" ## Related content * To choose what columns from the input to keep in the output, use [project-keep](project-keep-operator.md). * To rename columns, use [`project-rename`](project-rename-operator.md). * To reorder columns, use [`project-reorder`](project-reorder-operator.md). -::: moniker-end \ No newline at end of file diff --git a/data-explorer/kusto/query/project-keep-operator.md b/data-explorer/kusto/query/project-keep-operator.md index 6f67103a26..a4d7ba60a2 100644 --- a/data-explorer/kusto/query/project-keep-operator.md +++ b/data-explorer/kusto/query/project-keep-operator.md @@ -3,7 +3,7 @@ title: project-keep operator description: Learn how to use the project-keep operator to select columns from the input to keep in the output. ms.reviewer: alexans ms.topic: reference -ms.date: 08/11/2024 +ms.date: 01/28/2025 --- # project-keep operator @@ -36,7 +36,9 @@ A table with columns that were named as arguments. Contains same number of rows ## Example -The following query returns columns from the `ConferenceSessions` table that contain the word "session". +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] + +This query returns columns from the `ConferenceSessions` table that contain the word "session". :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] @@ -48,21 +50,23 @@ ConferenceSessions | project-keep session* ``` -The following table shows only the first 10 results. +**Output** + +The output table shows only the first 10 results. -|sessionid|session_title|session_type|session_location| +| sessionid | session_title | session_type | session_location | |--|--|--|--| -|COM64| Focus Group: Azure Data Explorer |Focus Group|Online| -|COM65| Focus Group: Azure Data Explorer |Focus Group|Online| -|COM08| Ask the Team: Azure Data Explorer|Ask the Team|Online| -|COM137| Focus Group: Built-In Dashboard and Smart Auto Scaling Capabilities in Azure Data Explorer|Focus Group| Online| -|CON-PRT157| Roundtable: Monitoring and managing your Azure Data Explorer deployments|Roundtable|Online| -|CON-PRT103| Roundtable: Advanced Kusto query language topics|Roundtable| Online| -|CON-PRT157| Roundtable: Monitoring and managing your Azure Data Explorer deployments|Roundtable|Online| -|CON-PRT103| Roundtable: Advanced Kusto query language topics|Roundtable|Online| -|CON-PRT130| Roundtable: Data exploration and visualization with Azure Data Explorer |Roundtable |Online| -|CON-PRT130| Roundtable: Data exploration and visualization with Azure Data Explorer |Roundtable |Online| -|...|...|...|...| +| COM64 | Focus Group: Azure Data Explorer | Focus Group | Online | +| COM65 | Focus Group: Azure Data Explorer | Focus Group | Online | +| COM08 | Ask the Team: Azure Data Explorer | Ask the Team | Online | +| COM137 | Focus Group: Built-In Dashboard and Smart Auto Scaling Capabilities in Azure Data Explorer | Focus Group | Online | +| CON-PRT157 | Roundtable: Monitoring and managing your Azure Data Explorer deployments | Roundtable | Online | +| CON-PRT103 | Roundtable: Advanced Kusto query language topics | Roundtable | Online | +| CON-PRT157 | Roundtable: Monitoring and managing your Azure Data Explorer deployments | Roundtable | Online | +| CON-PRT103 | Roundtable: Advanced Kusto query language topics | Roundtable | Online | +| CON-PRT130 | Roundtable: Data exploration and visualization with Azure Data Explorer | Roundtable | Online | +| CON-PRT130 | Roundtable: Data exploration and visualization with Azure Data Explorer | Roundtable | Online | +| ... | ... | ... | ... | ## Related content