Skip to content

Commit b71448c

Browse files
denreadenrea
authored
Merge pull request #299653 from guywi-ms/summary-rule-templates-2
Summary rule templates 2
2 parents cf5ffdf + ce2a9a8 commit b71448c

11 files changed

+260
-105
lines changed

‎articles/sentinel/TOC.yml

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -348,8 +348,12 @@
348348
href: normalization-manage-parsers.md
349349
- name: Modify content to use ASIM
350350
href: normalization-modify-content.md
351-
- name: Aggregate data with summary rules
352-
href: summary-rules.md
351+
- name: Aggregate data
352+
items:
353+
- name: Aggregate data with summary rules
354+
href: summary-rules.md
355+
- name: Aggregate insights from raw data into an Auxiliary table
356+
href: summary-rules-tutorial.md
353357
- name: Integrate threat intelligence
354358
items:
355359
- name: Overview

‎articles/sentinel/media/summary-rules/diagram.mmd

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
---
2+
config:
3+
look: neo
4+
theme: default
5+
sequence:
6+
actorFontSize: 22
7+
messageFontSize: 20
8+
noteFontSize: 20
9+
---
10+
sequenceDiagram
11+
actor User as User
12+
participant Entra ID as Entra ID
13+
participant Azure Monitor as Azure Monitor
14+
participant Log Analytics Workspace as Log Analytics Workspace
15+
participant VM as VM
16+
User ->> Entra ID: 1. Register Microsoft Entra application
17+
Entra ID ->> User: Provide Application ID and secret
18+
User ->> Azure Monitor: 2. Create DCE, DCR, and custom Auxiliary table using ARM template
19+
Azure Monitor ->> User: Provide DCE URI and DCR Immutable ID
20+
User ->> Azure Monitor: 3. Grant your application permission to send data to the DCE
21+
User ->> VM: 4. Update Logstash configuration file
22+
VM ->> Azure Monitor: Send data to DCE
23+
Azure Monitor ->> Log Analytics Workspace: Ingest raw data into CommonSecurityLog_CL table
24+
User ->> Log Analytics Workspace: 5. Query Auxiliary table to verify data ingestion
25+
User ->> Azure Monitor: 6. Create summary rule
26+
VM ->> Azure Monitor: Send data to DCE
27+
Azure Monitor ->> Log Analytics Workspace: Ingest raw data into CommonSecurityLog_CL table
28+
Azure Monitor ->> Log Analytics Workspace: Ingest aggregated data into custom Analytics table
29+
User ->> Log Analytics Workspace: 7. Query Analytics table to view aggregated data

‎articles/sentinel/media/summary-rules/diagram.svg

Lines changed: 1 addition & 0 deletions
Loading

‎articles/sentinel/media/summary-rules/summary-rule-auxiliary-logs-overview.png

354 KB
Loading

‎articles/sentinel/media/summary-rules/summary-rule-template-create.png

196 KB
Loading

‎articles/sentinel/media/summary-rules/summary-rule-template-details.png

265 KB
Loading

‎articles/sentinel/media/summary-rules/summary-rule-templates-content-hub.png

160 KB
Loading

‎articles/sentinel/sentinel-solutions.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ Content is available in the Microsoft Sentinel **Content hub**, and includes the
3434
| **[Playbooks and Azure Logic Apps custom connectors](automate-responses-with-playbooks.md)** | Automate investigation, remediation, and response scenarios in Microsoft Sentinel. |
3535
| **[Watchlists](watchlists.md)** | Ingest specific data for better threat detection and less alert fatigue. |
3636
| **[Workbooks](get-visibility.md)** | Monitor, visualize, and interact with data in Microsoft Sentinel to see meaningful insights. |
37+
| **[Summary rule templates](summary-rules.md#deploy-pre-built-summary-rule-templates)** | Deploy tested, prebuilt rules that optimize costs and improve query performance by aggregating insights from incoming verbose logs. |
3738

3839
The **Content hub** delivers these content types as *solutions* and *standalone* items. *Solutions* are packages of Microsoft Sentinel content or Microsoft Sentinel API integrations that support an end-to-end product, domain, or industry vertical scenario in Microsoft Sentinel.
3940

@@ -58,7 +59,7 @@ The Microsoft Sentinel **Content hub** lets you find content in the product, dep
5859

5960
### Why use Microsoft Sentinel solutions?
6061

61-
Microsoft Sentinel solutions are packaged integrations that deliver end-to-end product value for one or more domain or vertical scenario in the **Content hub**.
62+
Microsoft Sentinel solutions are packaged integrations that deliver end-to-end product value for one or more domains or vertical scenarios in the **Content hub**.
6263

6364
The solutions experience, powered by [Azure Marketplace](https://azuremarketplace.microsoft.com/marketplace), helps you find and deploy the content you want. For more information about authoring and publishing solutions in the Azure Marketplace, see the [Microsoft Sentinel Solutions Build Guide](https://aka.ms/sentinelsolutionsbuildguide).
6465

‎articles/sentinel/summary-rules-tutorial.md

Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,166 @@
1+
---
2+
title: Summarize insights from raw data in an Auxiliary table to an Analytics table in Microsoft Sentinel (Preview)
3+
description: This article walks you through a sample process of using summary rules to extract actionable analytics from verbose logs ingested into low-cost storage.
4+
author: guywi-ms
5+
ms.author: guywild
6+
ms.topic: how-to #Don't change
7+
ms.date: 05/25/2025
8+
appliesto:
9+
- Microsoft Sentinel in the Microsoft Defender portal
10+
- Microsoft Sentinel in the Azure portal
11+
ms.collection: usx-security
12+
13+
#customer intent: As a SOC engineer, I want to understand how to use summary rules extract actionable analytics from verbose logs ingested into low-cost storage.
14+
15+
---
16+
17+
# Tutorial: Send logs to low-cost storage and extract actionable analytics using summary rules in Microsoft Sentinel (Preview)
18+
19+
This article provides an example of how to use summary rules to aggregate insights from an [auxiliary logs table](basic-logs-use-cases.md) to an Analytics table. In this example, you ingest Common Event Format (CEF) data from Logstash by deploying a custom connector using an ARM template.
20+
21+
> [!IMPORTANT]
22+
> Summary rules are currently in PREVIEW. See the [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/) for legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.
23+
>
24+
> [!INCLUDE [unified-soc-preview-without-alert](includes/unified-soc-preview-without-alert.md)]
25+
>
26+
27+
## Prerequisites
28+
29+
To complete this tutorial, you need:
30+
31+
- A Microsoft Sentinel-enabled workspace.
32+
- A virtual machine (VM) with Logstash installed. For more information, see [Install Logstash](https://www.elastic.co/guide/en/logstash/current/installing-logstash.html).
33+
- Access to Microsoft Sentinel with [**Microsoft Sentinel Contributor**](../role-based-access-control/built-in-roles.md#microsoft-sentinel-contributor) permissions. For more information, see [Roles and permissions in Microsoft Sentinel](roles.md).
34+
- [Monitoring Contributor](/azure/role-based-access-control/built-in-roles#monitoring-contributor) permissions to create a data collection rule (DCR) and a data collection endpoint (DCE). For more information, see [Data collection rules](/azure/azure-monitor/data-collection/data-collection-rule-overview).
35+
- To create summary rules in the Microsoft Defender portal, you must first onboard your workspace to the Defender portal. For more information, see [Connect Microsoft Sentinel to the Microsoft Defender portal](/microsoft-365/security/defender/microsoft-sentinel-onboard).
36+
37+
## Process overview
38+
39+
This diagram shows the process described in this tutorial:
40+
41+
:::image type="content" source="media/summary-rules/summary-rule-auxiliary-logs-overview.png" alt-text="Screenshot of the Content Hub page in Microsoft Sentinel showing summary rule templates." lightbox="media/summary-rules/summary-rule-auxiliary-logs-overview.png":::
42+
43+
## Use summary rules with auxiliary logs
44+
45+
1. **Register a Microsoft Entra application.**
46+
47+
Create a Microsoft Entra application, and note the application's **Client ID** and **Secret**. For more information, see [Tutorial: Send data to Azure Monitor Logs with Logs ingestion API (Azure portal)](/azure/azure-monitor/logs/tutorial-logs-ingestion-portal).
48+
49+
The Microsoft Entra application authenticates the Logstash output plugin, which sends logs to your Log Analytics workspace.
50+
51+
52+
1. **Create a data collection endpoint (DCE), data collection rule (DCR), and a custom Auxiliary table.**
53+
54+
Deploy this ARM template to create the required resources:
55+
56+
[![Deploy to Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2FAzure-Sentinel%2Fmaster%2FDataConnectors%2Fmicrosoft-sentinel-log-analytics-logstash-output-plugin%2Fexamples%2Fauxiliry-logs%2Farm-template%2Fdeploy-dcr-dce-cef-table.json)
57+
58+
Note the following details from the ARM template output:
59+
60+
- `tenant_id`
61+
- `data_collection_endpoint`
62+
- `dcr_immutable_id`
63+
- `dcr_stream_name`
64+
65+
The data collection endpoint is the endpoint to which your Logstash instance sends logs. The data collection rule (DCR) defines which data to send to which table and how to process that data. For more information, see [Data collection endpoints](/azure/azure-monitor/data-collection/data-collection-endpoint-overview) and [Data collection rules](/azure/azure-monitor/data-collection/data-collection-rule-overview).
66+
67+
1. **Grant your application permission to send data to your data collection endpoint.**
68+
69+
Navigate to your data collection endpoint, and assign the **Log Analytics Data Contributor** role to your Microsoft Entra application.
70+
71+
For more information, see [Assign Azure roles using the Azure portal](/azure/role-based-access-control/role-assignments-portal).
72+
73+
1. **Update the Logstash configuration file on your VM.**
74+
75+
Copy our [sample Logstash configuration](https://github.com/Azure/Azure-Sentinel/blob/master/DataConnectors/microsoft-sentinel-log-analytics-logstash-output-plugin/examples/auxiliry-logs/config/bronze.conf) to your VM. Make sure to replace placeholder values with your own values for the custom table and Microsoft Entra app you created earlier.
76+
77+
This file configures Logstash to send CEF logs to the custom table created by the ARM template, transforming JSON data to the format used in your destination table schema.
78+
79+
After you update the configuration file and restart Logstash, CEF data that your VM logs will be sent to your Log Analytics workspace.
80+
81+
1. **Query your Auxiliary table to verify that data is being ingested.**
82+
83+
In Microsoft Sentinel, go to the **Logs** page and run a query. For example:
84+
85+
```kusto
86+
CommonSecurityLog_CL
87+
| take 10
88+
```
89+
90+
1. **Create a summary rule.**
91+
92+
Create a summary rule to aggregate insights from the Auxiliary table to an Analytics table.
93+
For more information about creating summary rules in Microsoft Sentinel, see [Create a new summary rule](./summary-rules.md#create-a-new-summary-rule).
94+
95+
Here are a couple of examples of summary rules to aggregate your CEF data:
96+
97+
- **Lookup indicator of compromise (IoC) data**: Hunt for specific IoCs by running aggregated summary queries to bring unique occurrences, and then query only those occurrences for faster results. The `Message` field is device-specific and in JSON format, so you need to parse this field to extract relevant data. This summary rule is an example of how to bring a unique `SourceIP` feed along with other metadata, which you can then use against IoC lookups:
98+
99+
```kusto
100+
// Daily Network traffic trend Per Destination IP along with Data transfer stats
101+
// Frequency - Daily - Maintain 30 day or 60 Day History.
102+
CommonSecurityLog_CL
103+
| extend j=parse_json(Message)
104+
| extend DestinationIP=tostring(j.destinationAddress)
105+
| extend SourceIP=tostring(j.sourceAddress)
106+
| extend SentBytes=toint(j.bytesOut)
107+
| extend ReceivedBytes=toint(j.bytesOut)
108+
| extend Day = format_datetime(TimeGenerated, "yyyy-MM-dd")
109+
| summarize Count= count(), DistinctSourceIps = dcount(SourceIP), NoofBytesTransferred = sum(SentBytes), NoofBytesReceived = sum(ReceivedBytes) by Day,DestinationIP, DeviceVendor
110+
```
111+
112+
- **Query a summary baseline for anomaly detections**. Instead of running your queries against large historical periods, such as 30 or 60 days, we recommend that you ingest data into an Auxiliary table, and then only query summary baseline data, such as for time series anomaly detections. For example:
113+
114+
```kusto
115+
// Time series data for Firewall traffic logs
116+
let starttime = 14d;
117+
let endtime = 1d;
118+
let timeframe = 1h;
119+
CommonSecurityLog_CL
120+
| extend j=parse_json(Message)
121+
| extend DestinationIP=tostring(j.destinationAddress)
122+
| extend SourceIP=tostring(j.sourceAddress)
123+
| extend SentBytes=toint(j.bytesOut)
124+
| where isnotempty(DestinationIP) and isnotempty(SourceIP)
125+
| where ipv4_is_private(DestinationIP) == false
126+
| project TimeGenerated, SentBytes, DeviceVendor
127+
| make-series TotalBytesSent=sum(SentBytes) on TimeGenerated from startofday(ago(starttime)) to startofday(ago(endtime)) step timeframe by DeviceVendor
128+
```
129+
130+
1. **Query the destination Analytics table.**
131+
132+
To view the aggregated data, run a query against the Analytics table you specified in the summary rule.
133+
134+
135+
See more information on the following items used in the preceding examples, in the Kusto documentation:
136+
- [***let*** statement](/kusto/query/let-statement?view=microsoft-sentinel&preserve-view=true)
137+
- [***where*** operator](/kusto/query/where-operator?view=microsoft-sentinel&preserve-view=true)
138+
- [***extend*** operator](/kusto/query/extend-operator?view=microsoft-sentinel&preserve-view=true)
139+
- [***project*** operator](/kusto/query/project-operator?view=microsoft-sentinel&preserve-view=true)
140+
- [***summarize*** operator](/kusto/query/summarize-operator?view=microsoft-sentinel&preserve-view=true)
141+
- [***lookup*** operator](/kusto/query/lookup-operator?view=microsoft-sentinel&preserve-view=true)
142+
- [***union*** operator](/kusto/query/union-operator?view=microsoft-sentinel&preserve-view=true)
143+
- [***make-series*** operator](/kusto/query/make-series-operator?view=microsoft-sentinel&preserve-view=true)
144+
- [***isnotempty()*** function](/kusto/query/isnotempty-function?view=microsoft-sentinel&preserve-view=true)
145+
- [***format_datetime()*** function](/kusto/query/format-datetime-function?view=microsoft-sentinel&preserve-view=true)
146+
- [***column_ifexists()*** function](/kusto/query/column-ifexists-function?view=microsoft-sentinel&preserve-view=true)
147+
- [***iff()*** function](/kusto/query/iff-function?view=microsoft-sentinel&preserve-view=true)
148+
- [***ipv4_is_private()*** function](/kusto/query/ipv4-is-private-function?view=microsoft-sentinel&preserve-view=true)
149+
- [***min()*** function](/kusto/query/min-aggregation-function?view=microsoft-sentinel&preserve-view=true)
150+
- [***tostring()*** function](/kusto/query/tostring-function?view=microsoft-sentinel&preserve-view=true)
151+
- [***ago()*** function](/kusto/query/ago-function?view=microsoft-sentinel&preserve-view=true)
152+
- [***startofday()*** function](/kusto/query/startofday-function?view=microsoft-sentinel&preserve-view=true)
153+
- [***parse_json()*** function](/kusto/query/parse-json-function?view=microsoft-sentinel&preserve-view=true)
154+
- [***count()*** aggregation function](/kusto/query/count-aggregation-function?view=microsoft-sentinel&preserve-view=true)
155+
- [***make_set()*** aggregation function](/kusto/query/make-set-aggregation-function?view=microsoft-sentinel&preserve-view=true)
156+
- [***dcount()*** aggregation function](/kusto/query/dcount-aggregation-function?view=microsoft-sentinel&preserve-view=true)
157+
- [***sum()*** aggregation function](/kusto/query/sum-aggregation-function?view=microsoft-sentinel&preserve-view=true)
158+
159+
[!INCLUDE [kusto-reference-general-no-alert](includes/kusto-reference-general-no-alert.md)]
160+
161+
## Related content
162+
163+
- [Aggregate data in Log Analytics workspace with Summary rules](/azure/azure-monitor/logs/summary-rules)
164+
- [Plan costs and understand Microsoft Sentinel pricing and billing](billing.md)
165+
- [Log sources to use for Auxiliary Logs ingestion](basic-logs-use-cases.md)
166+
- [Summary rules restrictions and limitations](/azure/azure-monitor/logs/summary-rules)

0 commit comments

Comments
 (0)