diff --git a/data-explorer/hot-windows.md b/data-explorer/hot-windows.md index 7d35530c1c..cff5164ee9 100644 --- a/data-explorer/hot-windows.md +++ b/data-explorer/hot-windows.md @@ -39,52 +39,7 @@ After changing the cache policy, the cluster automatically caches the relevant d Now you can expect optimal performance during the use of hot windows. -## Set hot windows - -Hot windows are part of the [cache policy commands syntax](/kusto/management/show-table-cache-policy-command?view=azure-data-explorer&preserve-view=true) and are set with the [`.alter policy caching` command](/kusto/management/cache-policy?view=azure-data-explorer&preserve-view=true). - -> [!NOTE] -> It can take up to an hour to fully update the cluster disk cache based on the updated cache policy definition. - -1. Take note of the initial caching policy by using the `.show policy caching` command. - - ```kusto - .show table MyDatabase.MyTable policy caching - ``` - -1. Alter the cache policy using the following syntax. Several hot windows may be defined for a single database or table. - - ```kusto - .alter policy caching - hot = - [, hot_window = datetime(*from*) .. datetime(*to*)] - [, hot_window = datetime(*from*) .. datetime(*to*)] - ... - ``` - - Where: - * `from`: Start time of the hot window (datetime) - * `to`: End time of the hot window (datetime) - - For example, queries run under the following settings will examine the last 14 days of data, on data that is kept for three years. - - ```kusto - .alter table MyTable policy caching - hot = 14d, - hot_window = datetime(2021-01-01) .. datetime(2021-02-01), - hot_window = datetime(2021-04-01) .. datetime(2021-05-01) - ``` - -## Run query - -Run the query or queries you want over the time period specified in the hot windows. - -## Revert settings - -1. Use the original cache settings retrieved above in [Set hot windows](#set-hot-windows). -1. Revert the cache policy to the original settings with the [`.alter policy caching` command](/kusto/management/show-table-cache-policy-command?view=azure-data-explorer&preserve-view=true). - -Since you've configured optimized autoscale for that cluster, the cluster will shrink to its original size. +[!INCLUDE [set-hot-windows](includes/cross-repo/set-hot-windows.md)] ## Related content diff --git a/data-explorer/includes/cross-repo/set-hot-windows.md b/data-explorer/includes/cross-repo/set-hot-windows.md new file mode 100644 index 0000000000..f52c1979c0 --- /dev/null +++ b/data-explorer/includes/cross-repo/set-hot-windows.md @@ -0,0 +1,51 @@ +--- +ms.topic: include +ms.date: 07/02/2023 +--- + +## Set hot windows + +Hot windows are part of the [cache policy commands syntax](/kusto/management/show-table-cache-policy-command?view=azure-data-explorer&preserve-view=true) and are set with the [`.alter policy caching` command](/kusto/management/cache-policy?view=azure-data-explorer&preserve-view=true). + +> [!NOTE] +> It can take up to an hour to fully update the cluster disk cache based on the updated cache policy definition. + +1. Take note of the initial caching policy by using the `.show policy caching` command. + + ```kusto + .show table MyDatabase.MyTable policy caching + ``` + +1. Alter the cache policy using the following syntax. Several hot windows may be defined for a single database or table. + + ```kusto + .alter policy caching + hot = + [, hot_window = datetime(*from*) .. datetime(*to*)] + [, hot_window = datetime(*from*) .. datetime(*to*)] + ... + ``` + + Where: + * `from`: Start time of the hot window (datetime) + * `to`: End time of the hot window (datetime) + + For example, queries run under the following settings will examine the last 14 days of data, on data that is kept for three years. + + ```kusto + .alter table MyTable policy caching + hot = 14d, + hot_window = datetime(2021-01-01) .. datetime(2021-02-01), + hot_window = datetime(2021-04-01) .. datetime(2021-05-01) + ``` + +## Run query + +Run the query or queries you want over the time period specified in the hot windows. + +## Revert settings + +1. Use the original cache settings retrieved above in [Set hot windows](#set-hot-windows). +1. Revert the cache policy to the original settings with the [`.alter policy caching` command](/kusto/management/show-table-cache-policy-command?view=azure-data-explorer&preserve-view=true). + +Since you've configured optimized autoscale for that cluster, the cluster will shrink to its original size. diff --git a/data-explorer/kusto/query/make-graph-operator.md b/data-explorer/kusto/query/make-graph-operator.md index 90d3e27aff..628b3e2cb2 100644 --- a/data-explorer/kusto/query/make-graph-operator.md +++ b/data-explorer/kusto/query/make-graph-operator.md @@ -17,6 +17,10 @@ The `make-graph` operator builds a graph structure from tabular inputs of edges *Edges* `|` `make-graph` *SourceNodeId* `-->` *TargetNodeId* [ `with_node_id=` *NodeIdPropertyName* ] +*Edges* `|` `make-graph` *SourceNodeId* `-->` *TargetNodeId* [ `with` *Nodes1* `on` *NodeId1* [`,` *Nodes2* `on` *NodeId2* ]] `partitioned-by` *PartitionColumn* `(` *GraphOperator* `)` + +*Edges* `|` `make-graph` *SourceNodeId* `-->` *TargetNodeId* [ `with_node_id=` *NodeIdPropertyName* ] `partitioned-by` *PartitionColumn* `(` *GraphOperator* `)` + ## Parameters | Name | Type | Required | Description | @@ -27,11 +31,15 @@ The `make-graph` operator builds a graph structure from tabular inputs of edges | *Nodes1*, *Nodes2* | `string` | | The tabular expressions containing the properties of the nodes in the graph. | | *NodesId1*, *NodesId2* | `string` | | The corresponding columns with the node IDs in *Nodes1*, *Nodes2* respectively. | | *NodeIdPropertyName* | `string` | | The name of the property for node ID on the nodes of the graph. | +| *PartitionColumn* | `string` | | The column to partition the graph by. Creates separate graphs for each unique value in this column. | +| *GraphOperator* | `string` | | The graph operator to apply to each partitioned graph. | ## Returns The `make-graph` operator returns a graph expression and must be followed by a [graph operator](graph-operators.md#supported-graph-operators). Each row in the source *Edges* expression becomes an edge in the graph with properties that are the column values of the row. Each row in the *Nodes* tabular expression becomes a node in the graph with properties that are the column values of the row. Nodes that appear in the *Edges* table but don't have a corresponding row in the *Nodes* table are created as nodes with the corresponding node ID and empty properties. +When using the `partitioned-by` clause, separate graphs are created for each unique value in the specified *PartitionColumn*. The specified *GraphOperator* is then applied to each partitioned graph independently, and the results are combined into a single output. This is particularly useful for multitenant scenarios where you want to analyze each tenant's data separately while maintaining the same graph structure and analysis logic. + > [!NOTE] > Each node has a unique identifier. If the same node ID appears in both the *Nodes1* and *Nodes2* tables, a single node is created by merging their properties. If there are conflicting property values for the same node, one of the values is arbitrarily chosen. @@ -115,6 +123,78 @@ edges |---|---|---| |Mallory|Bob|Trent| +### Multitenant partitioned graph + +This example demonstrates using the `partitioned-by` clause to analyze a multitenant social network. The `partitioned-by` clause creates separate graphs for each unique value in the partition column (in this case, `tenantId`), applies the graph operator to each partition independently, and combines the results. + +:::moniker range="azure-data-explorer" +> [!div class="nextstepaction"] +> Run the query +::: moniker-end + +```kusto +// Nodes table representing users across multiple tenants (organizations) +let nodes = datatable(userId:string, tenantId:string, name:string, department:string, role:string, location:dynamic) +[ + // Tenant: CompanyA - San Francisco Bay Area + "u001", "CompanyA", "Alice Johnson", "Engineering", "Senior Developer", dynamic({"type": "Point", "coordinates": [-122.4194, 37.7749]}), + "u002", "CompanyA", "Bob Smith", "Engineering", "Team Lead", dynamic({"type": "Point", "coordinates": [-122.4094, 37.7849]}), + "u003", "CompanyA", "Charlie Brown", "Marketing", "Manager", dynamic({"type": "Point", "coordinates": [-122.4294, 37.7649]}), + "u004", "CompanyA", "Diana Prince", "HR", "Director", dynamic({"type": "Point", "coordinates": [-122.3994, 37.7949]}), + "u005", "CompanyA", "Eve Wilson", "Engineering", "Junior Developer", dynamic({"type": "Point", "coordinates": [-122.4394, 37.7549]}), + // Tenant: CompanyB - New York Area + "u006", "CompanyB", "Frank Miller", "Sales", "Account Manager", dynamic({"type": "Point", "coordinates": [-74.0060, 40.7128]}), + "u007", "CompanyB", "Grace Lee", "Engineering", "Senior Developer", dynamic({"type": "Point", "coordinates": [-74.0160, 40.7228]}), + "u008", "CompanyB", "Henry Davis", "Marketing", "Specialist", dynamic({"type": "Point", "coordinates": [-73.9960, 40.7028]}), + "u009", "CompanyB", "Ivy Chen", "Engineering", "Team Lead", dynamic({"type": "Point", "coordinates": [-74.0260, 40.7328]}), + "u010", "CompanyB", "Jack Thompson", "Operations", "Manager", dynamic({"type": "Point", "coordinates": [-73.9860, 40.6928]}), + // Tenant: CompanyC - Austin Area + "u011", "CompanyC", "Kate Anderson", "Finance", "Analyst", dynamic({"type": "Point", "coordinates": [-97.7431, 30.2672]}), + "u012", "CompanyC", "Liam Murphy", "Engineering", "Architect", dynamic({"type": "Point", "coordinates": [-97.7331, 30.2772]}), + "u013", "CompanyC", "Maya Patel", "Product", "Manager", dynamic({"type": "Point", "coordinates": [-97.7531, 30.2572]}), + "u014", "CompanyC", "Noah Garcia", "Engineering", "Developer", dynamic({"type": "Point", "coordinates": [-97.7631, 30.2472]}), + "u015", "CompanyC", "Olivia Rodriguez", "Marketing", "Director", dynamic({"type": "Point", "coordinates": [-97.7231, 30.2872]}) +]; +// Edges table representing relationships/interactions between users +let edges = datatable(sourceUserId:string, targetUserId:string, tenantId:string, relationshipType:string, strength:int) +[ + // CompanyA relationships + "u001", "u002", "CompanyA", "reportsTo", 9, + "u005", "u002", "CompanyA", "reportsTo", 8, + "u002", "u003", "CompanyA", "collaborates", 6, + "u001", "u005", "CompanyA", "mentors", 7, + "u003", "u004", "CompanyA", "collaborates", 5, + "u001", "u003", "CompanyA", "communicates", 4, + // CompanyB relationships + "u007", "u009", "CompanyB", "reportsTo", 9, + "u006", "u010", "CompanyB", "reportsTo", 8, + "u008", "u006", "CompanyB", "collaborates", 6, + "u009", "u010", "CompanyB", "communicates", 5, + "u007", "u008", "CompanyB", "mentors", 7, + "u006", "u007", "CompanyB", "collaborates", 6, + // CompanyC relationships + "u014", "u012", "CompanyC", "reportsTo", 9, + "u012", "u013", "CompanyC", "collaborates", 7, + "u011", "u013", "CompanyC", "collaborates", 6, + "u013", "u015", "CompanyC", "reportsTo", 8, + "u012", "u015", "CompanyC", "communicates", 5, + "u011", "u014", "CompanyC", "mentors", 6 +]; +edges +| make-graph sourceUserId --> targetUserId with nodes on userId partitioned-by tenantId ( + graph-match cycles=none (n1)-[e*2..4]->(n2) + where n1.userId != n2.userId and all(e, relationshipType == "collaborates") and + geo_distance_2points(todouble(n1.location.coordinates[0]), todouble(n1.location.coordinates[1]), + todouble(n2.location.coordinates[0]), todouble(n2.location.coordinates[1])) < 10000 + project Start = strcat(n1.name, " (", n1.tenantId, ")"), Tenants = map(e, tenantId), End = strcat(n2.name, " (", n2.tenantId, ")") +) +``` + +|Start|Tenants|End| +|---|---|---| +|Bob Smith (CompanyA)|[
"CompanyA",
"CompanyA"
]|Diana Prince (CompanyA)| +|Henry Davis (CompanyB)|[
"CompanyB",
"CompanyB"
]|Grace Lee (CompanyB)| + ## Related content * [Graph operators](graph-operators.md) diff --git a/data-explorer/kusto/query/toc.yml b/data-explorer/kusto/query/toc.yml index b2fe62c484..a08684079b 100644 --- a/data-explorer/kusto/query/toc.yml +++ b/data-explorer/kusto/query/toc.yml @@ -1278,52 +1278,56 @@ items: href: variancepif-aggregation-function.md - name: Graph items: - - name: Graph semantics overview - href: graph-semantics-overview.md - - name: Graph sample data - href: graph-sample-data.md - - name: Graph exploration basics - href: graph-exploration-basics.md - - name: Graph operators + - name: Start here items: - - name: Graph operators overview - href: graph-operators.md - - name: make-graph - href: make-graph-operator.md - - name: graph-match - href: graph-match-operator.md - - name: graph-shortest-paths - href: graph-shortest-paths-operator.md - - name: graph-mark-components - href: graph-mark-components-operator.md - - name: graph-to-table - href: graph-to-table-operator.md - - name: Graph functions + - name: Graph semantics overview + href: graph-semantics-overview.md + - name: Graph sample data + href: graph-sample-data.md + - name: Graph exploration basics + href: graph-exploration-basics.md + - name: Reference items: - - name: all() - displayName: all() graph function - href: all-graph-function.md - - name: any() - displayName: any() graph function - href: any-graph-function.md - - name: map() - displayName: map() graph function - href: map-graph-function.md - - name: inner_nodes() - displayName: inner nodes() graph function, inner nodes - href: inner-nodes-graph-function.md - - name: labels() - displayName: labels() graph function, labels - href: labels-graph-function.md - - name: node_degree_in() - displayName: graph, node, Node degree in node_degree_in, indegree node_degree_in - href: node-degree-in.md - - name: node_degree_out() - displayName: graph, node, Node degree out node_degree_out, outdegree node_degree_out - href: node-degree-out.md - - name: Graph scenarios + - name: Operators + items: + - name: Graph operators overview + href: graph-operators.md + - name: make-graph + href: make-graph-operator.md + - name: graph-match + href: graph-match-operator.md + - name: graph-to-table + href: graph-to-table-operator.md + - name: graph-shortest-paths + href: graph-shortest-paths-operator.md + - name: graph-mark-components + href: graph-mark-components-operator.md + - name: Functions + items: + - name: all() + displayName: all() graph function + href: all-graph-function.md + - name: any() + displayName: any() graph function + href: any-graph-function.md + - name: map() + displayName: map() graph function + href: map-graph-function.md + - name: inner_nodes() + displayName: inner nodes() graph function, inner nodes + href: inner-nodes-graph-function.md + - name: labels() + displayName: labels() graph function, labels + href: labels-graph-function.md + - name: node_degree_in() + displayName: graph, node, Node degree in node_degree_in, indegree node_degree_in + href: node-degree-in.md + - name: node_degree_out() + displayName: graph, node, Node degree out node_degree_out, outdegree node_degree_out + href: node-degree-out.md + - name: Scenarios href: graph-scenarios.md - - name: Graph best practices + - name: Best practices href: graph-best-practices.md - name: Geospatial items: