You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The Kusto client libraries use a common platform for tracing. The platform uses a large number of trace sources (`System.Diagnostics.TraceSource`), and each is connected to the default set of trace listeners (`System.Diagnostics.Trace.Listeners`) during its construction.
13
+
The Kusto client libraries are instrumented to write traces to local files. By default, the tracing mechanism is disabled but can be enabled programmatically.
14
14
15
-
If an application has trace listeners associated with the default `System.Diagnostics.Trace` instance
16
-
(for example, through its `app.config` file), then the Kusto client libraries will emit traces to those listeners.
15
+
## Enable tracing
17
16
18
-
The tracing can be suppressed or controlled programmatically or through a config file.
19
-
20
-
## Suppress tracing programmatically
21
-
22
-
To suppress tracing from the Kusto client libraries programmatically, invoke this piece of code when loading the relevant library:
17
+
To enable tracing, make sure you have the `Microsoft.Azure.Kusto.Cloud.Platform`[NuGet package](https://www.nuget.org/packages/Microsoft.Azure.Kusto.Cloud.Platform/) installed. Then run the following code:
To suppress tracing from the client libraries through a config file, modify the file `Kusto.Cloud.Platform.dll.tweaks` (which is included with the `Kusto.Data` library).
32
+
Each trace source in the library can have its own default verbosity level. A trace source only writes to file traces whose verbosity is equal to or above its own verbosity level. You can control the verbosity of the trace sources. The following example sets the verbosity level for all trace sources to `Verbose`, ensuring that all traces are written to files:
31
33
32
-
```xml
33
-
<!--Overrides the default trace verbosity level-->
> For the tweak to take effect, there must not be a minus sign in the value of `key`
40
+
Use the `TraceVerbosity.Fatal` argument to trace only the most severe events.
41
+
42
+
## Flush all pending traces
39
43
40
-
An alternative, is:
44
+
Flushing pending trace is recommended when the application hosting the trace system is closed to ensure unwritten traces are saved. It can be done safely even if the tracing system isn't initialized. The following code forces all pending traces to flush to files and recycle all files:
To enable tracing out of the client libraries, enable .NET tracing in your application's *app.config file*. For example, assume that the application `MyApp.exe` uses the Kusto.Data client library. Changing file *MyApp.exe.config* to include the following, will enable `Kusto.Data` tracing the next time that the application starts.
The code will configure a trace listener that writes to CSV files in a subdirectory called *RollingLogs*. The subdirectory is located in the process' directory.
52
+
Enabling tracing for client libraries automatically enables tracing for [MSAL (Microsoft Authentication Library)](/azure/active-directory/develop/msal-overview).
68
53
69
-
> [!NOTE]
70
-
> Any .NET-compatible trace listener class may be used as well.
Once the tracing system is initialized, trace files are written to the specified folder or its subfolders. They're formatted as CSV files with the `.csv` extension. Files that are currently being written use the extension `.csv.in-progress` and are automatically renamed once they're completed.
57
+
58
+
Each trace file record includes the following fields:
73
59
74
-
Once tracing for client libraries is enabled, tracing for [MSAL (Microsoft Authentication Library)](/azure/active-directory/develop/msal-overview) is enabled automatically.
60
+
***Trace record identifier:** Uniquely identifies each trace record.
61
+
***Timestamp:** The timestamp of the trace record.
62
+
***Trace source name:** The name of the trace source.
63
+
***Trace level:** The verbosity level of the trace.
1. Right-click on the downloaded package. From the menu, select your archiving tool and extract the package contents. If the archiving tool isn't visible from the menu, select **Show more options**. The extraction results in multiple folders, one of which is named *tools*.
24
-
1. Inside the *tools* folder, there are different subfolders catering to different PowerShell versions. For PowerShell version 5.1, use the *net472* folder. For PowerShell version 7 or later, use any of the version folders. Copy the path of the relevant folder.
24
+
1. Inside the *tools* folder, there are different subfolders catering to different PowerShell versions. For PowerShell version 5.1, use the *net472* folder. For PowerShell version 7 or later, use any of the version folders except the *net472* folder. Copy the path of the relevant folder.
25
25
1. From PowerShell, load the libraries, replacing `<path>` with the copied folder path:
26
26
27
27
```powershell
@@ -36,13 +36,16 @@ To use the Kusto .NET client libraries in PowerShell:
36
36
37
37
Once loaded, you can use the libraries to [connect to a cluster and database](#connect-to-a-cluster-and-database).
38
38
39
+
> [!NOTE]
40
+
> The tools in the *net472* folder aren't supported on Linux.
41
+
39
42
## Connect to a cluster and database
40
43
41
44
Authenticate to a cluster and database with one of the following methods:
42
45
43
46
* **User authentication:** Prompt the user to verify their identity in a web browser.
44
47
* **Application authentication:** [Create a Microsoft Entra app](../../access-control/provision-entra-id-app.md) and use the credentials for authentication.
45
-
* **Azure CLI authentication:** Sign-in to the Azure CLI on your machine, and Kusto will retrieve the token from Azure CLI.
48
+
* **Azure CLI authentication:** Sign-in to the Azure CLI on your machine, and Kusto retrieves the token from Azure CLI.
[Create an MS Entra app](../../access-control/provision-entra-id-app.md) and grant it access to your database. Then, provide the app credentials in place of the `$applicationId`, `$applicationKey`, and `$authority`.
65
+
[Create a Microsoft Entra application](../../access-control/provision-entra-id-app.md) and grant it access to your database. Then, provide the app credentials in place of the `$applicationId`, `$applicationKey`, and `$authority`.
Create a query provider and run [Kusto Query Language](../../query/index.md) queries.
92
95
93
-
The following example defines a simple [take](../../query/take-operator.md) query to sample the data. To run the query, replace `<TableName>` with the name of a table in your database. Before running the query, the [ClientRequestProperties class](../netfx/client-request-properties.md) is used to set a client request ID and a server timeout. Then, the query is run and the result set is formatted and sorted.
96
+
The following example defines a simple [take](../../query/take-operator.md) query to sample the data. To run the query, replace `<TableName>` with the name of a table in your database. The [ClientRequestProperties class](../netfx/client-request-properties.md) is used to set a client request ID and a server time-out before running the query. Then, the query is run and the result set is formatted and sorted.
Copy file name to clipboardExpand all lines: data-explorer/kusto/query/join-rightouter.md
+3-1Lines changed: 3 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,7 +3,7 @@ title: rightouter join
3
3
description: Learn how to use the rightouter join flavor to merge the rows of two tables.
4
4
ms.reviewer: alexans
5
5
ms.topic: reference
6
-
ms.date: 08/11/2024
6
+
ms.date: 01/21/2025
7
7
---
8
8
9
9
# rightouter join
@@ -29,6 +29,8 @@ The `rightouter` join flavor returns all the records from the right side and onl
29
29
30
30
## Example
31
31
32
+
This query returns all rows from table Y and any matching rows from table X, filling in NULL values where there is no match from X.
33
+
32
34
:::moniker range="azure-data-explorer"
33
35
> [!div class="nextstepaction"]
34
36
> <ahref="https://dataexplorer.azure.com/clusters/help/databases/Samples?query=H4sIAAAAAAAAA8tJLVGIULBVSEksAcKknFQN79RKq+KSosy8dB2FsMSc0lRDq5z8vHRNrmguBSBQT1TXMdSBMJPUdYwQTGMoM1ldx4Qr1porB2h0JH6jjVCNBhpiaIAwxQiJbQxjpwBNNwAZH6FQo5CVn5mnkJ2Zl2JblJmeUZJfWpJaBLQzP08BaBUAPvRgAtsAAAA="target="_blank">Run the query</a>
Copy file name to clipboardExpand all lines: data-explorer/kusto/query/join-rightsemi.md
+7-1Lines changed: 7 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,7 +3,7 @@ title: rightsemi join
3
3
description: Learn how to use the rightsemi join flavor to merge the rows of two tables.
4
4
ms.reviewer: alexans
5
5
ms.topic: reference
6
-
ms.date: 08/11/2024
6
+
ms.date: 01/21/2025
7
7
---
8
8
9
9
# rightsemi join
@@ -29,6 +29,8 @@ The `rightsemi` join flavor returns all records from the right side that match a
29
29
30
30
## Example
31
31
32
+
This query filters and returns only those rows from table Y that have a matching key in table X.
33
+
32
34
:::moniker range="azure-data-explorer"
33
35
> [!div class="nextstepaction"]
34
36
> <ahref="https://dataexplorer.azure.com/clusters/help/databases/Samples?query=H4sIAAAAAAAAA8tJLVGIULBVSEksAcKknFQN79RKq+KSosy8dB2FsMSc0lRDq5z8vHRNrmguBSBQT1TXMdSBMJPUdYwQTGMoM1ldx4Qr1porB2h0JH6jjVCNBhpiaIAwxQiJbQxjpwBNNwAZH6FQo5CVn5mnkJ2Zl2JblJmeUVKcmpsJtDI/TwFoEwCXFUWa2gAAAA=="target="_blank">Run the query</a>
@@ -59,3 +61,7 @@ X | join kind=rightsemi Y on Key
59
61
| b | 10 |
60
62
| c | 20 |
61
63
| c | 30 |
64
+
65
+
## Related content
66
+
67
+
* Learn about other [join flavors](join-operator.md#returns)
It's often useful to join between two large datasets on some high-cardinality key, such as an operation ID or a session ID, and further limit the right-hand-side ($right) records that need to match up with each left-hand-side ($left) record by adding a restriction on the "time-distance" between `datetime` columns on the left and on the right.
13
13
14
-
The above operation differs from the usual Kusto join operation, since for the `equi-join` part of matching the high-cardinality key between the left and right datasets, the system can also apply a distance function and use it to considerably speed up the join.
14
+
The above operation differs from the usual join operation, since for the `equi-join` part of matching the high-cardinality key between the left and right datasets, the system can also apply a distance function and use it to considerably speed up the join.
15
15
16
16
> [!NOTE]
17
-
> A distance function doesn't behave like equality (that is, when both dist(x,y) and dist(y,z) are true it doesn't follow that dist(x,z) is also true.) Internally, we sometimes refer to this as "diagonal join".
17
+
> A distance function doesn't behave like equality (that is, when both dist(x,y) and dist(y,z) are true it doesn't follow that dist(x,z) is also true.) This is sometimes referred to as a "diagonal join".
18
18
19
-
For example, if you want to identify event sequences within a relatively small time window, assume that you have a table `T` with the following schema:
19
+
## Example to identify event sequences without time window
20
+
21
+
To identify event sequences within a relatively small time window, this example uses a table `T` with the following schema:
20
22
21
23
*`SessionId`: A column of type `string` with correlation IDs.
22
24
*`EventType`: A column of type `string` that identifies the event type of the record.
23
25
*`Timestamp`: A column of type `datetime` indicates when the event described by the record happened.
24
26
27
+
| SessionId | EventType | Timestamp |
28
+
|--|--|--|
29
+
| 0 | A | 2017-10-01T00:00:00Z |
30
+
| 0 | B | 2017-10-01T00:01:00Z |
31
+
| 1 | B | 2017-10-01T00:02:00Z |
32
+
| 1 | A | 2017-10-01T00:03:00Z |
33
+
| 3 | A | 2017-10-01T00:04:00Z |
34
+
| 3 | B | 2017-10-01T00:10:00Z |
35
+
36
+
The following query creates the dataset and then identifies all the session IDs in which event type `A` was followed by an event type `B` within a `1min` time window.
37
+
25
38
:::moniker range="azure-data-explorer"
26
39
> [!div class="nextstepaction"]
27
-
> <ahref="https://dataexplorer.azure.com/clusters/help/databases/Samples?query=H4sIAAAAAAAAA8tJLVEIUbBVSEksAcKknFSN4NTi4sz8PM8Uq+KSosy8dB0F17LUvJKQyoJUuEhIZm5qcUliboEVUF9qCZCnycsVzculAATqBuo6CuqOQAImp2FkYGiua2iga2CoYGBgBUaaOsiqnfCoNkRWbUhItRGGanwuMUZWbUxItQmGajwuMYT5MtaalysEAKb/JupnAQAA"target="_blank">Run the query</a>
40
+
> <ahref="https://dataexplorer.azure.com/clusters/help/databases/Samples?query=H4sIAAAAAAAAA4WQTWvDMAyG74H8B91iQ1LsdjDI8GGFHnZubmOHdBGdu8YJjlgZ7MdPbsgHtKS2sbD12O8rnZGgAANVSTwPZxR77DrbuLcq78hbd0xh94OOit8Wx5vC1thRWbc5v0Pik4yj9zgCHolKUkheeRtyYq30c6ZVpjQolV+XTOf0doHWc1o/otc39JKTzZzePKKfbugFJ3qo8uMljgqIoz+4fKHHqZtgTNALmdY3J/wkGHufwp5KT2ZsdKBOjXXwbV1lrHPoeyOiD0EhxPsq22TI3lHa8YczncBJaNyETN4Fs5D13iQckC6IDoSq2dhqBZqjXKrnKvYPNlcRxHMCAAA="target="_blank">Run the query</a>
28
41
::: moniker-end
29
42
30
43
```kusto
@@ -38,38 +51,6 @@ let T = datatable(SessionId:string, EventType:string, Timestamp:datetime)
38
51
'3', 'B', datetime(2017-10-01 00:10:00),
39
52
];
40
53
T
41
-
```
42
-
43
-
**Output**
44
-
45
-
|SessionId|EventType|Timestamp|
46
-
|---|---|---|
47
-
|0|A|2017-10-01 00:00:00.0000000|
48
-
|0|B|2017-10-01 00:01:00.0000000|
49
-
|1|B|2017-10-01 00:02:00.0000000|
50
-
|1|A|2017-10-01 00:03:00.0000000|
51
-
|3|A|2017-10-01 00:04:00.0000000|
52
-
|3|B|2017-10-01 00:10:00.0000000|
53
-
54
-
**Problem statement**
55
-
56
-
Our query should answer the following question:
57
-
58
-
Find all the session IDs in which event type `A` was followed by an
59
-
event type `B` within a `1min` time window.
60
-
61
-
> [!NOTE]
62
-
> In the sample data above, the only such session ID is `0`.
63
-
64
-
Semantically, the following query answers this question, albeit inefficiently.
65
-
66
-
:::moniker range="azure-data-explorer"
67
-
> [!div class="nextstepaction"]
68
-
> <ahref="https://dataexplorer.azure.com/clusters/help/databases/Samples?query=H4sIAAAAAAAAA4WQTWvDMAyG74H8B91iQ1LsdjDI8GGFHnZubmOHdBGdu8YJjlgZ7MdPbsgHtKS2sbD12O8rnZGgAANVSTwPZxR77DrbuLcq78hbd0xh94OOit8Wx5vC1thRWbc5v0Pik4yj9zgCHolKUkheeRtyYq30c6ZVpjQolV+XTOf0doHWc1o/otc39JKTzZzePKKfbugFJ3qo8uMljgqIoz+4fKHHqZtgTNALmdY3J/wkGHufwp5KT2ZsdKBOjXXwbV1lrHPoeyOiD0EhxPsq22TI3lHa8YczncBJaNyETN4Fs5D13iQckC6IDoSq2dhqBZqjXKrnKvYPNlcRxHMCAAA="target="_blank">Run the query</a>
To optimize this query, we can rewrite it as described below
92
-
so that the time window is expressed as a join key.
72
+
## Example optimized with time window
93
73
94
-
**Rewrite the queryto account for the time window**
74
+
To optimize this query, we can rewrite it to account for the time window. THe time window is expressed as a join key. Rewrite the query so that the `datetime` values are "discretized" into buckets whose size is half the size of the time window. Use *`equi-join`* to compare the bucket IDs.
95
75
96
-
Rewrite the query so that the `datetime` values are "discretized" into buckets whose size is half the size of the time window. Use Kusto's *`equi-join`* to compare those bucket IDs.
97
-
98
-
```kusto
99
-
let lookupWindow = 1min;
100
-
let lookupBin = lookupWindow / 2.0; // lookup bin = equal to 1/2 of the lookup window
101
-
T
102
-
| where EventType == 'A'
103
-
| project SessionId, Start=Timestamp,
104
-
// TimeKey on the left side of the join is mapped to a discrete time axis for the join purpose
105
-
TimeKey = bin(Timestamp, lookupBin)
106
-
| join kind=inner
107
-
(
108
-
T
109
-
| where EventType == 'B'
110
-
| project SessionId, End=Timestamp,
111
-
// TimeKey on the right side of the join - emulates event 'B' appearing several times
// 'mv-expand' translates the TimeKey array range into a column
117
-
| mv-expand TimeKey to typeof(datetime)
118
-
) on SessionId, TimeKey
119
-
| where (End - Start) between (0min .. lookupWindow)
120
-
| project SessionId, Start, End
121
-
```
122
-
123
-
**Runnable query reference (with table inlined)**
76
+
The query finds pairs of events within the same session (*SessionId*) where an 'A' event is followed by a 'B' event within 1 minute. It projects the session ID, the start time of the 'A' event, and the end time of the 'B' event.
![learn-build-service-prod[bot]](https://avatars.githubusercontent.com/in/237442?v=4&size=40){kind=avatar}
0 commit comments