From d160a0266ee9a5067e0a93ccb723542d922b7439 Mon Sep 17 00:00:00 2001 From: hadastern <39373775+hadastern@users.noreply.github.com> Date: Wed, 28 May 2025 17:04:20 +0300 Subject: [PATCH 1/6] Update scan-operator.md Add an example for calculating session length --- data-explorer/kusto/query/scan-operator.md | 38 ++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/data-explorer/kusto/query/scan-operator.md b/data-explorer/kusto/query/scan-operator.md index 20f570c182..0b34f55ecd 100644 --- a/data-explorer/kusto/query/scan-operator.md +++ b/data-explorer/kusto/query/scan-operator.md @@ -230,6 +230,44 @@ Events |00:41:00|E|00:32:00|1| |01:15:00|A|01:15:00|2| +### Calculating Session Length per User + +Calculate the session start time, end time, and duration for each user's session using the `scan` operator. A session is defined as a period between a user's login and the subsequent logout. By combining `partition` and `scan` with `output=none` and `output=all`, this pattern ensures that a **single row is returned per session** (i.e., per login/logout pair), rather than a row per event. + +The logic works by: +- In step s1: Capturing the login timestamp using a scan step with `output=none` +- In step s2: Emitting a row only when a matching logout is found using `output=all` + +```kusto +let LogsEvents = datatable(Timestamp:datetime, userID:int, EventType:string) +[ + datetime(2024-05-28 08:15:23), 1, "login", + datetime(2024-05-28 08:30:15), 2, "login", + datetime(2024-05-28 09:10:27), 3, "login", + datetime(2024-05-28 12:30:45), 1, "logout", + datetime(2024-05-28 11:45:32), 2, "logout", + datetime(2024-05-28 13:25:19), 3, "logout" +]; +LogsEvents +| sort by userID, Timestamp +| partition hint.strategy=native by userID ( + sort by Timestamp asc + | scan declare (start: datetime, end: datetime, sessionDuration: timespan) with ( + step s1 output=none: EventType == "login" => start = Timestamp; + step s2 output=all: EventType == "logout" => start = s1.start, end = Timestamp, sessionDuration = Timestamp - s1.start; + ) +) +| project start, end, userID, sessionDuration +``` + +**Output** +|start |end |userID|sessionDuration| +|---------------------|---------------------|------|----------------| +|2024-05-28 08:15:23.0000000|2024-05-28 12:30:45.0000000|1 |04:15:22 | +|2024-05-28 09:10:27.0000000|2024-05-28 13:25:19.0000000|3 |04:14:52 | +|2024-05-28 08:30:15.0000000|2024-05-28 11:45:32.0000000|2 |03:15:17 | + + ### Events between Start and Stop Find all sequences of events between the event `Start` and the event `Stop` that occur within 5 minutes. Assign a match ID for each sequence. From 036095628e3343719d9dcb9a673dc3483c553295 Mon Sep 17 00:00:00 2001 From: hadastern <39373775+hadastern@users.noreply.github.com> Date: Fri, 30 May 2025 10:31:55 +0300 Subject: [PATCH 2/6] Update data-explorer/kusto/query/scan-operator.md Co-authored-by: ktalmor <193799742+ktalmor@users.noreply.github.com> --- data-explorer/kusto/query/scan-operator.md | 1 + 1 file changed, 1 insertion(+) diff --git a/data-explorer/kusto/query/scan-operator.md b/data-explorer/kusto/query/scan-operator.md index 0b34f55ecd..a988884a90 100644 --- a/data-explorer/kusto/query/scan-operator.md +++ b/data-explorer/kusto/query/scan-operator.md @@ -261,6 +261,7 @@ LogsEvents ``` **Output** + |start |end |userID|sessionDuration| |---------------------|---------------------|------|----------------| |2024-05-28 08:15:23.0000000|2024-05-28 12:30:45.0000000|1 |04:15:22 | From a11f676729dd6b57147ea38bade4b80f7c25b040 Mon Sep 17 00:00:00 2001 From: hadastern <39373775+hadastern@users.noreply.github.com> Date: Fri, 30 May 2025 17:17:24 +0300 Subject: [PATCH 3/6] Update scan-operator.md Fixing PR --- data-explorer/kusto/query/scan-operator.md | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/data-explorer/kusto/query/scan-operator.md b/data-explorer/kusto/query/scan-operator.md index a988884a90..34db7e4b12 100644 --- a/data-explorer/kusto/query/scan-operator.md +++ b/data-explorer/kusto/query/scan-operator.md @@ -237,7 +237,13 @@ Calculate the session start time, end time, and duration for each user's session The logic works by: - In step s1: Capturing the login timestamp using a scan step with `output=none` - In step s2: Emitting a row only when a matching logout is found using `output=all` +- + +:::moniker range="azure-data-explorer" +> [!div class="nextstepaction"] +> Run the query +::: moniker-end ```kusto let LogsEvents = datatable(Timestamp:datetime, userID:int, EventType:string) [ @@ -262,11 +268,11 @@ LogsEvents **Output** -|start |end |userID|sessionDuration| -|---------------------|---------------------|------|----------------| -|2024-05-28 08:15:23.0000000|2024-05-28 12:30:45.0000000|1 |04:15:22 | -|2024-05-28 09:10:27.0000000|2024-05-28 13:25:19.0000000|3 |04:14:52 | -|2024-05-28 08:30:15.0000000|2024-05-28 11:45:32.0000000|2 |03:15:17 | +| userID | start | end | sessionDuration | +|--------|--------------------------|---------------------------|-----------------| +| 1 | 2024-05-28 08:15:23.0000 | 2024-05-28 12:30:45.0000 | 04:15:22 | +| 3 | 2024-05-28 09:10:27.0000 | 2024-05-28 13:25:19.0000 | 04:14:52 | +| 2 | 2024-05-28 08:30:15.0000 | 2024-05-28 11:45:32.0000 | 03:15:17 | ### Events between Start and Stop From c2ba5db50d0aeded189779767e83e2c47679e33c Mon Sep 17 00:00:00 2001 From: hadastern <39373775+hadastern@users.noreply.github.com> Date: Fri, 30 May 2025 18:05:46 +0300 Subject: [PATCH 4/6] Update and rename scan-operator.md to ms.date update ms.date --- data-explorer/kusto/query/{scan-operator.md => ms.date} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename data-explorer/kusto/query/{scan-operator.md => ms.date} (99%) diff --git a/data-explorer/kusto/query/scan-operator.md b/data-explorer/kusto/query/ms.date similarity index 99% rename from data-explorer/kusto/query/scan-operator.md rename to data-explorer/kusto/query/ms.date index 34db7e4b12..5a8ae7396d 100644 --- a/data-explorer/kusto/query/scan-operator.md +++ b/data-explorer/kusto/query/ms.date @@ -3,7 +3,7 @@ title: scan operator description: Learn how to use the scan operator to scan data, match, and build sequences based on the predicates. ms.reviewer: alexans ms.topic: reference -ms.date: 01/22/2025 +ms.date: 05/30/2025 --- # scan operator From 2866fd702758820a6a7bc286081ad5eff83efb01 Mon Sep 17 00:00:00 2001 From: hadastern <39373775+hadastern@users.noreply.github.com> Date: Wed, 4 Jun 2025 09:53:26 +0300 Subject: [PATCH 5/6] Rename ms.date to scan-operator.md Fixing file name --- data-explorer/kusto/query/{ms.date => scan-operator.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename data-explorer/kusto/query/{ms.date => scan-operator.md} (100%) diff --git a/data-explorer/kusto/query/ms.date b/data-explorer/kusto/query/scan-operator.md similarity index 100% rename from data-explorer/kusto/query/ms.date rename to data-explorer/kusto/query/scan-operator.md From 8d3cf39ee8301e0fb018e36e9a0a592a147d525c Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Thu, 5 Jun 2025 17:58:07 +0300 Subject: [PATCH 6/6] KUS: scan operator example - replaces PR#6881 --- data-explorer/kusto/query/scan-operator.md | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/data-explorer/kusto/query/scan-operator.md b/data-explorer/kusto/query/scan-operator.md index 5a8ae7396d..40d986462e 100644 --- a/data-explorer/kusto/query/scan-operator.md +++ b/data-explorer/kusto/query/scan-operator.md @@ -71,7 +71,7 @@ Each input record is evaluated against all of the steps in reverse order, from t * **Check 2:** If the state of *s_k* has an active sequence or *s_k* is the first step, and *r* meets the *Condition* of *s_k*, then a match occurs. The match leads to the following actions: 1. The assignments of *s_k* are calculated and extend *r*. - 2. The values that represent *s_k* in the state of *s_k* are replaced with the values of the extended *r*. + 1. The values that represent *s_k* in the state of *s_k* are replaced with the values of the extended *r*. 1. If *s_k* is defined as `output=all`, the extended *r* is added to the output. 1. If *s_k* is the first step, a new sequence begins and the match ID increases by `1`. This only affects the output when `with_match_id` is used. @@ -82,7 +82,7 @@ For a detailed example of this logic, see the [scan logic walkthrough](#scan-log ## Examples The example in this section shows how to use the syntax to help you get started. - + [!INCLUDE [help-cluster](../includes/help-cluster-note.md)] ### Cumulative sum @@ -235,15 +235,15 @@ Events Calculate the session start time, end time, and duration for each user's session using the `scan` operator. A session is defined as a period between a user's login and the subsequent logout. By combining `partition` and `scan` with `output=none` and `output=all`, this pattern ensures that a **single row is returned per session** (i.e., per login/logout pair), rather than a row per event. The logic works by: -- In step s1: Capturing the login timestamp using a scan step with `output=none` -- In step s2: Emitting a row only when a matching logout is found using `output=all` -- +* In step s1: Capturing the login timestamp using a scan step with `output=none` +* In step s2: Emitting a row only when a matching logout is found using `output=all` :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] > Run the query ::: moniker-end + ```kusto let LogsEvents = datatable(Timestamp:datetime, userID:int, EventType:string) [ @@ -274,7 +274,6 @@ LogsEvents | 3 | 2024-05-28 09:10:27.0000 | 2024-05-28 13:25:19.0000 | 04:14:52 | | 2 | 2024-05-28 08:30:15.0000 | 2024-05-28 11:45:32.0000 | 03:15:17 | - ### Events between Start and Stop Find all sequences of events between the event `Start` and the event `Stop` that occur within 5 minutes. Assign a match ID for each sequence. @@ -414,7 +413,7 @@ The "X" indicates that a specific field is irrelevant for that step. This section follows the [matching logic](#matching-logic) through each record of the `Events` table, explaining the transformation of the state and output at each step. > [!NOTE] -> An input record is evaluated against the steps in reverse order, from the last step (`s3`) to the first step (`s1`). +> An input record is evaluated against the steps in reverse order, from the last step (`s3`) to the first step (`s1`). #### Record 1 @@ -476,7 +475,7 @@ This section follows the [matching logic](#matching-logic) through each record o |s2|0|00:01:00|"Start"|00:02:00|"B"|X|X| |s3|||||||| -#### Record 4 +#### Record 4 |Ts|Event| |---|---| @@ -508,7 +507,6 @@ This section follows the [matching logic](#matching-logic) through each record o * `s2`: **Check 1** isn't passed because the state of `s1` is empty, and **Check 2** isn't passed because `s2` lacks an active sequence. * `s1`: **Check 1** is irrelevant because there's no previous step. **Check 2** isn't passed because the record doesn't meet the condition of `Event == "Start"`. - **State:** |step|m_id|s1.Ts|s1.Event|s2.Ts|s2.Event|s3.Ts|s3.Event| @@ -547,7 +545,7 @@ This section follows the [matching logic](#matching-logic) through each record o * `s3`: **Check 1** isn't passed because the state of `s2` is empty, and **Check 2** isn't passed because it doesn't meet the condition of `Event == "Stop"`. * `s2`: **Check 1** isn't passed because the state of `s1` is empty, and **Check 2** isn't passed because `s2` lacks an active sequence. -* `s1`: **Check 1** isn't passed because there's no previous step. it passes **Check 2** because it meets the condition of `Event == "Start"`. This match initiates a new sequence in `s1` with a new `m_id`. **Record 7** and its `m_id` (`1`) are added to the state and the output. +* `s1`: **Check 1** isn't passed because there's no previous step. it passes **Check 2** because it meets the condition of `Event == "Start"`. This match initiates a new sequence in `s1` with a new `m_id`. **Record 7** and its `m_id` (`1`) are added to the state and the output. **State:**