Repo sync for protected branch

data-explorer/kusto/api/netfx/controlling-tracing.md

@@ -1,74 +1,64 @@
 ---
-title:  Control and suppress SDK client side tracing
+title:  Control and suppress SDK client side-tracing
 description: This article describes controlling and suppressing SDK client-side tracing.
 ms.reviewer: orspodek
 ms.topic: reference
 ms.custom: has-adal-ref
-ms.date: 08/11/2024
+ms.date: 02/03/2025
 ---
 # Controlling and suppressing Kusto SDK client-side tracing
 
 > [!INCLUDE [applies](../../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../../includes/applies-to-version/azure-data-explorer.md)]
 
-The Kusto client libraries use a common platform for tracing. The platform uses a large number of trace sources (`System.Diagnostics.TraceSource`), and each is connected to the default set of trace listeners (`System.Diagnostics.Trace.Listeners`) during its construction.
+The Kusto client libraries are instrumented to write traces to local files. By default, the tracing mechanism is disabled but can be enabled programmatically.
 
-If an application has trace listeners associated with the default `System.Diagnostics.Trace` instance
-(for example, through its `app.config` file), then the Kusto client libraries will emit traces to those listeners.
+## Enable tracing
 
-The tracing can be suppressed or controlled programmatically or through a config file.
-
-## Suppress tracing programmatically
-
-To suppress tracing from the Kusto client libraries programmatically, invoke this piece of code when loading the relevant library:
+To enable tracing, make sure you have the `Microsoft.Azure.Kusto.Cloud.Platform` [NuGet package](https://www.nuget.org/packages/Microsoft.Azure.Kusto.Cloud.Platform/) installed. Then run the following code:
 
 ```csharp
-TraceSourceManager.SetTraceVerbosityForAll(TraceVerbosity.Fatal);
+using Kusto.Cloud.Platform.Utils; // Requires the NuGet package, Microsoft.Azure.Kusto.Cloud.Platform.
+
+var manifest = new RollingCsvTraceListener2Manifest
+{
+  TracesLocalRootPath=@"c:\temp" // The folder where trace files will be written.
+};
+RollingCsvTraceListener2.CreateAndInitialize(manifest);
+TraceSourceManager.StartupDone();
 ```
 
-## Use a config file to suppress tracing
+## Control trace level
 
-To suppress tracing from the client libraries through a config file, modify the file `Kusto.Cloud.Platform.dll.tweaks` (which is included with the `Kusto.Data` library).
+Each trace source in the library can have its own default verbosity level. A trace source only writes to file traces whose verbosity is equal to or above its own verbosity level. You can control the verbosity of the trace sources. The following example sets the verbosity level for all trace sources to `Verbose`, ensuring that all traces are written to files:
 
-```xml
-    <!--Overrides the default trace verbosity level-->
-    <add key="Kusto.Cloud.Platform.Utils.Tracing.OverrideTraceVerbosityLevel" value="0" />
+```csharp
+using Kusto.Cloud.Platform.Utils; // Requires the NuGet package, Microsoft.Azure.Kusto.Cloud.Platform.
+
+TraceSourceManager.SetOverrideTraceVerbosityLevel(TraceVerbosity.Verbose);
 ```
 
-> [!NOTE]
-> For the tweak to take effect, there must not be a minus sign in the value of `key`
+Use the `TraceVerbosity.Fatal` argument to trace only the most severe events.
+
+## Flush all pending traces
 
-An alternative, is:
+Flushing pending trace is recommended when the application hosting the trace system is closed to ensure unwritten traces are saved. It can be done safely even if the tracing system isn't initialized. The following code forces all pending traces to flush to files and recycle all files:
 
 ```csharp
-Anchor.Tweaks.SetProgrammaticAppSwitch(
-    "Kusto.Cloud.Platform.Utils.Tracing.OverrideTraceVerbosityLevel",
-    "0"
-);
+TraceSourceManager.SuperFlush(SuperFlushMode.Emergency);
 ```
 
-## Enable client libraries tracing
-
-To enable tracing out of the client libraries, enable .NET tracing in your application's *app.config file*. For example, assume that the application `MyApp.exe` uses the Kusto.Data client library. Changing file *MyApp.exe.config* to include the following, will enable `Kusto.Data` tracing the next time that the application starts.
-
-```xml
-<?xml version="1.0" encoding="utf-8" ?>
-<configuration>
-  <system.diagnostics>
-    <trace indentsize="4">
-      <listeners>
-        <add type="Kusto.Cloud.Platform.Utils.RollingCsvTraceListener2, Kusto.Cloud.Platform" name="RollingCsvTraceListener" initializeData="RollingLogs" />
-        <remove name="Default" />
-      </listeners>
-    </trace>
-  </system.diagnostics>
-</configuration>
-```
+## Enable MSAL (Microsoft Authentication Library) tracing
 
-The code will configure a trace listener that writes to CSV files in a subdirectory called *RollingLogs*. The subdirectory is located in the process' directory.
+ Enabling tracing for client libraries automatically enables tracing for [MSAL (Microsoft Authentication Library)](/azure/active-directory/develop/msal-overview).
 
-> [!NOTE]
-> Any .NET-compatible trace listener class may be used as well.
+## Read trace files
 
-## Enable MSAL (Microsoft Authentication Library) tracing
+Once the tracing system is initialized,  trace files are written to the specified folder or its subfolders. They're formatted as CSV files with the `.csv` extension. Files that are currently being written use the extension `.csv.in-progress` and are automatically renamed once they're completed.
+
+Each trace file record includes the following fields:
 
-Once tracing for client libraries is enabled, tracing for [MSAL (Microsoft Authentication Library)](/azure/active-directory/develop/msal-overview) is enabled automatically.
+* **Trace record identifier:** Uniquely identifies each trace record.
+* **Timestamp:** The timestamp of the trace record.
+* **Trace source name:** The name of the trace source.
+* **Trace level:** The verbosity level of the trace.
+* **Textual content:** The trace record content.

data-explorer/kusto/api/powershell/powershell.md

@@ -3,7 +3,7 @@ title:  Kusto .NET Client Libraries from PowerShell
 description: This article describes how to use Kusto .NET Client Libraries from PowerShell.
 ms.reviewer: salevy
 ms.topic: reference
-ms.date: 08/11/2024
+ms.date: 02/02/2025
 ---
 # Use Kusto .NET client libraries from PowerShell
 
@@ -21,7 +21,7 @@ To use the Kusto .NET client libraries in PowerShell:
 
 1. Download [`Microsoft.Azure.Kusto.Tools`](https://www.nuget.org/packages/Microsoft.Azure.Kusto.Tools/).
 1. Right-click on the downloaded package. From the menu, select your archiving tool and extract the package contents. If the archiving tool isn't visible from the menu, select **Show more options**. The extraction results in multiple folders, one of which is named *tools*.
-1. Inside the *tools* folder, there are different subfolders catering to different PowerShell versions. For PowerShell version 5.1, use the *net472* folder. For PowerShell version 7 or later, use any of the version folders. Copy the path of the relevant folder.
+1. Inside the *tools* folder, there are different subfolders catering to different PowerShell versions. For PowerShell version 5.1, use the *net472* folder. For PowerShell version 7 or later, use any of the version folders except the *net472* folder. Copy the path of the relevant folder.
 1. From PowerShell, load the libraries, replacing `<path>` with the copied folder path:
 
     ```powershell
@@ -36,13 +36,16 @@ To use the Kusto .NET client libraries in PowerShell:
 
 Once loaded, you can use the libraries to [connect to a cluster and database](#connect-to-a-cluster-and-database).
 
+> [!NOTE]
+> The tools in the *net472* folder aren't supported on Linux.
+
 ## Connect to a cluster and database
 
 Authenticate to a cluster and database with one of the following methods:
 
 * **User authentication:** Prompt the user to verify their identity in a web browser.
 * **Application authentication:** [Create a Microsoft Entra app](../../access-control/provision-entra-id-app.md) and use the credentials for authentication.
-* **Azure CLI authentication:** Sign-in to the Azure CLI on your machine, and Kusto will retrieve the token from Azure CLI.
+* **Azure CLI authentication:** Sign-in to the Azure CLI on your machine, and Kusto retrieves the token from Azure CLI.
 
 Select the relevant tab.
 
@@ -59,7 +62,7 @@ $kcsb = New-Object Kusto.Data.KustoConnectionStringBuilder($clusterUrl, $databas
 
 ### [Application](#tab/app)
 
-[Create an MS Entra app](../../access-control/provision-entra-id-app.md) and grant it access to your database. Then, provide the app credentials in place of the `$applicationId`, `$applicationKey`, and `$authority`.
+[Create a Microsoft Entra application](../../access-control/provision-entra-id-app.md) and grant it access to your database. Then, provide the app credentials in place of the `$applicationId`, `$applicationKey`, and `$authority`.
 
 ```powershell
 $clusterUrl = "<Your cluster URI>"
@@ -90,7 +93,7 @@ $kcsb = $kcsb.WithAadAzCliAuthentication()
 
 Create a query provider and run [Kusto Query Language](../../query/index.md) queries.
 
-The following example defines a simple [take](../../query/take-operator.md) query to sample the data. To run the query, replace `<TableName>` with the name of a table in your database. Before running the query, the [ClientRequestProperties class](../netfx/client-request-properties.md) is used to set a client request ID and a server timeout. Then, the query is run and the result set is formatted and sorted.
+The following example defines a simple [take](../../query/take-operator.md) query to sample the data. To run the query, replace `<TableName>` with the name of a table in your database. The [ClientRequestProperties class](../netfx/client-request-properties.md) is used to set a client request ID and a server time-out before running the query. Then, the query is run and the result set is formatted and sorted.
 
 ```powershell
 $queryProvider = [Kusto.Data.Net.Client.KustoClientFactory]::CreateCslQueryProvider($kcsb)

data-explorer/kusto/query/join-rightouter.md

@@ -3,7 +3,7 @@ title:  rightouter join
 description: Learn how to use the rightouter join flavor to merge the rows of two tables. 
 ms.reviewer: alexans
 ms.topic: reference
-ms.date: 08/11/2024
+ms.date: 01/21/2025
 ---
 
 # rightouter join
@@ -29,6 +29,8 @@ The `rightouter` join flavor returns all the records from the right side and onl
 
 ## Example
 
+This query returns all rows from table Y and any matching rows from table X, filling in NULL values where there is no match from X.
+
 :::moniker range="azure-data-explorer"
 > [!div class="nextstepaction"]
 > <a href="https://dataexplorer.azure.com/clusters/help/databases/Samples?query=H4sIAAAAAAAAA8tJLVGIULBVSEksAcKknFQN79RKq+KSosy8dB2FsMSc0lRDq5z8vHRNrmguBSBQT1TXMdSBMJPUdYwQTGMoM1ldx4Qr1porB2h0JH6jjVCNBhpiaIAwxQiJbQxjpwBNNwAZH6FQo5CVn5mnkJ2Zl2JblJmeUZJfWpJaBLQzP08BaBUAPvRgAtsAAAA=" target="_blank">Run the query</a>

data-explorer/kusto/query/join-rightsemi.md

@@ -3,7 +3,7 @@ title:  rightsemi join
 description: Learn how to use the rightsemi join flavor to merge the rows of two tables. 
 ms.reviewer: alexans
 ms.topic: reference
-ms.date: 08/11/2024
+ms.date: 01/21/2025
 ---
 
 # rightsemi join
@@ -29,6 +29,8 @@ The `rightsemi` join flavor returns all records from the right side that match a
 
 ## Example
 
+This query filters and returns only those rows from table Y that have a matching key in table X.
+
 :::moniker range="azure-data-explorer"
 > [!div class="nextstepaction"]
 > <a href="https://dataexplorer.azure.com/clusters/help/databases/Samples?query=H4sIAAAAAAAAA8tJLVGIULBVSEksAcKknFQN79RKq+KSosy8dB2FsMSc0lRDq5z8vHRNrmguBSBQT1TXMdSBMJPUdYwQTGMoM1ldx4Qr1porB2h0JH6jjVCNBhpiaIAwxQiJbQxjpwBNNwAZH6FQo5CVn5mnkJ2Zl2JblJmeUVKcmpsJtDI/TwFoEwCXFUWa2gAAAA==" target="_blank">Run the query</a>
@@ -59,3 +61,7 @@ X | join kind=rightsemi Y on Key
 | b | 10 |
 | c | 20 |
 | c | 30 |
+
+## Related content
+
+* Learn about other [join flavors](join-operator.md#returns)

data-explorer/kusto/query/join-time-window.md

@@ -3,28 +3,41 @@ title:  Joining within time window
 description: Learn how to perform a time window join operation to match between two large datasets.
 ms.reviewer: alexans
 ms.topic: reference
-ms.date: 08/11/2024
+ms.date: 01/28/2025
 ---
 # Time window join
 
 > [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] [!INCLUDE [monitor](../includes/applies-to-version/monitor.md)] [!INCLUDE [sentinel](../includes/applies-to-version/sentinel.md)]
 
 It's often useful to join between two large datasets on some high-cardinality key, such as an operation ID or a session ID, and further limit the right-hand-side ($right) records that need to match up with each left-hand-side ($left) record by adding a restriction on the "time-distance" between `datetime` columns on the left and on the right.
 
-The above operation differs from the usual Kusto join operation, since for the `equi-join` part of matching the high-cardinality key between the left and right datasets, the system can also apply a distance function and use it to considerably speed up the join.
+The above operation differs from the usual join operation, since for the `equi-join` part of matching the high-cardinality key between the left and right datasets, the system can also apply a distance function and use it to considerably speed up the join.
 
 > [!NOTE]
-> A distance function doesn't behave like equality (that is, when both dist(x,y) and dist(y,z) are true it doesn't follow that dist(x,z) is also true.) Internally, we sometimes refer to this as "diagonal join".
+> A distance function doesn't behave like equality (that is, when both dist(x,y) and dist(y,z) are true it doesn't follow that dist(x,z) is also true.) This is sometimes referred to as a "diagonal join".
 
-For example, if you want to identify event sequences within a relatively small time window, assume that you have a table `T` with the following schema:
+## Example to identify event sequences without time window
+
+To identify event sequences within a relatively small time window, this example uses a table `T` with the following schema:
 
 * `SessionId`: A column of type `string` with correlation IDs.
 * `EventType`: A column of type `string` that identifies the event type of the record.
 * `Timestamp`: A column of type `datetime` indicates when the event described by the record happened.
 
+| SessionId | EventType | Timestamp |
+|--|--|--|
+| 0 | A | 2017-10-01T00:00:00Z |
+| 0 | B | 2017-10-01T00:01:00Z |
+| 1 | B | 2017-10-01T00:02:00Z |
+| 1 | A | 2017-10-01T00:03:00Z |
+| 3 | A | 2017-10-01T00:04:00Z |
+| 3 | B | 2017-10-01T00:10:00Z |
+
+The following query creates the dataset and then identifies all the session IDs in which event type `A` was followed by an event type `B` within a `1min` time window.
+
 :::moniker range="azure-data-explorer"
 > [!div class="nextstepaction"]
-> <a href="https://dataexplorer.azure.com/clusters/help/databases/Samples?query=H4sIAAAAAAAAA8tJLVEIUbBVSEksAcKknFSN4NTi4sz8PM8Uq+KSosy8dB0F17LUvJKQyoJUuEhIZm5qcUliboEVUF9qCZCnycsVzculAATqBuo6CuqOQAImp2FkYGiua2iga2CoYGBgBUaaOsiqnfCoNkRWbUhItRGGanwuMUZWbUxItQmGajwuMYT5MtaalysEAKb/JupnAQAA" target="_blank">Run the query</a>
+> <a href="https://dataexplorer.azure.com/clusters/help/databases/Samples?query=H4sIAAAAAAAAA4WQTWvDMAyG74H8B91iQ1LsdjDI8GGFHnZubmOHdBGdu8YJjlgZ7MdPbsgHtKS2sbD12O8rnZGgAANVSTwPZxR77DrbuLcq78hbd0xh94OOit8Wx5vC1thRWbc5v0Pik4yj9zgCHolKUkheeRtyYq30c6ZVpjQolV+XTOf0doHWc1o/otc39JKTzZzePKKfbugFJ3qo8uMljgqIoz+4fKHHqZtgTNALmdY3J/wkGHufwp5KT2ZsdKBOjXXwbV1lrHPoeyOiD0EhxPsq22TI3lHa8YczncBJaNyETN4Fs5D13iQckC6IDoSq2dhqBZqjXKrnKvYPNlcRxHMCAAA=" target="_blank">Run the query</a>
 ::: moniker-end
 
 ```kusto
@@ -38,38 +51,6 @@ let T = datatable(SessionId:string, EventType:string, Timestamp:datetime)
     '3', 'B', datetime(2017-10-01 00:10:00),
 ];
 T
-```
-
-**Output**
-
-|SessionId|EventType|Timestamp|
-|---|---|---|
-|0|A|2017-10-01 00:00:00.0000000|
-|0|B|2017-10-01 00:01:00.0000000|
-|1|B|2017-10-01 00:02:00.0000000|
-|1|A|2017-10-01 00:03:00.0000000|
-|3|A|2017-10-01 00:04:00.0000000|
-|3|B|2017-10-01 00:10:00.0000000|
-
-**Problem statement**
-
-Our query should answer the following question:
-
-   Find all the session IDs in which event type `A` was followed by an
-   event type `B` within a `1min` time window.
-
-> [!NOTE]
-> In the sample data above, the only such session ID is `0`.
-
-Semantically, the following query answers this question, albeit inefficiently.
-
-:::moniker range="azure-data-explorer"
-> [!div class="nextstepaction"]
-> <a href="https://dataexplorer.azure.com/clusters/help/databases/Samples?query=H4sIAAAAAAAAA4WQTWvDMAyG74H8B91iQ1LsdjDI8GGFHnZubmOHdBGdu8YJjlgZ7MdPbsgHtKS2sbD12O8rnZGgAANVSTwPZxR77DrbuLcq78hbd0xh94OOit8Wx5vC1thRWbc5v0Pik4yj9zgCHolKUkheeRtyYq30c6ZVpjQolV+XTOf0doHWc1o/otc39JKTzZzePKKfbugFJ3qo8uMljgqIoz+4fKHHqZtgTNALmdY3J/wkGHufwp5KT2ZsdKBOjXXwbV1lrHPoeyOiD0EhxPsq22TI3lHa8YczncBJaNyETN4Fs5D13iQckC6IDoSq2dhqBZqjXKrnKvYPNlcRxHMCAAA=" target="_blank">Run the query</a>
-::: moniker-end
-
-```kusto
-T 
 | where EventType == 'A'
 | project SessionId, Start=Timestamp
 | join kind=inner
@@ -84,43 +65,15 @@ T
 
 **Output**
 
-|SessionId|Start|End|
-|---|---|---|
-|0|2017-10-01 00:00:00.0000000|2017-10-01 00:01:00.0000000|
+| SessionId | Start | End |
+|--|--|--|
+| 0 | 2017-10-01 00:00:00.0000000 | 2017-10-01 00:01:00.0000000 |
 
-To optimize this query, we can rewrite it as described below
-so that the time window is expressed as a join key.
+## Example optimized with time window
 
-**Rewrite the query to account for the time window**
+To optimize this query, we can rewrite it to account for the time window. THe time window is expressed as a join key. Rewrite the query so that the `datetime` values are "discretized" into buckets whose size is half the size of the time window. Use *`equi-join`* to compare the bucket IDs.
 
-Rewrite the query so that the `datetime` values are "discretized" into buckets whose size is half the size of the time window. Use Kusto's *`equi-join`* to compare those bucket IDs.
-
-```kusto
-let lookupWindow = 1min;
-let lookupBin = lookupWindow / 2.0; // lookup bin = equal to 1/2 of the lookup window
-T 
-| where EventType == 'A'
-| project SessionId, Start=Timestamp,
-          // TimeKey on the left side of the join is mapped to a discrete time axis for the join purpose
-          TimeKey = bin(Timestamp, lookupBin)
-| join kind=inner
-    (
-    T 
-    | where EventType == 'B'
-    | project SessionId, End=Timestamp,
-              // TimeKey on the right side of the join - emulates event 'B' appearing several times
-              // as if it was 'replicated'
-              TimeKey = range(bin(Timestamp-lookupWindow, lookupBin),
-                              bin(Timestamp, lookupBin),
-                              lookupBin)
-    // 'mv-expand' translates the TimeKey array range into a column
-    | mv-expand TimeKey to typeof(datetime)
-    ) on SessionId, TimeKey 
-| where (End - Start) between (0min .. lookupWindow)
-| project SessionId, Start, End 
-```
-
-**Runnable query reference (with table inlined)**
+The query finds pairs of events within the same session (*SessionId*) where an 'A' event is followed by a 'B' event within 1 minute. It projects the session ID, the start time of the 'A' event, and the end time of the 'B' event.
 
 :::moniker range="azure-data-explorer"
 > [!div class="nextstepaction"]
@@ -158,13 +111,13 @@ T
 
 **Output**
 
-|SessionId|Start|End|
-|---|---|---|
-|0|2017-10-01 00:00:00.0000000|2017-10-01 00:01:00.0000000|
+| SessionId | Start | End |
+|--|--|--|
+| 0 | 2017-10-01 00:00:00.0000000 | 2017-10-01 00:01:00.0000000 |
 
-**5M data query**
+## 5 million data query
 
-The next query emulates a dataset of 5M records and ~1M IDs and runs the query with the technique described above.
+The next query emulates an extensive dataset of 5M records and approximately 1M Session IDs and runs the query with the time window technique.
 
 :::moniker range="azure-data-explorer"
 > [!div class="nextstepaction"]
@@ -199,6 +152,10 @@ T
 
 **Output**
 
-|Count|
-|---|
-|3344|
+| Count |
+|--|
+| 3344 |
+
+## Related content
+
+* [join operator](join-operator.md)