|
| 1 | +--- |
| 2 | +title: 'Create analytics rules for Microsoft Sentinel solutions' |
| 3 | +description: This article guides you through the process of creating and publishing analytics rules to Microsoft Sentinel solutions. |
| 4 | +author: anilgodavarthy |
| 5 | +ms.author: angodavarthy |
| 6 | +ms.service: microsoft-sentinel |
| 7 | +ms.topic: conceptual |
| 8 | +ms.date: 1/27/2025 |
| 9 | + |
| 10 | +#CustomerIntent: As a ISV partner, I want to create and publish analytics rules to my Microsoft Sentinel solution so that I can provide inbuilt detection use cases to my customers. |
| 11 | +--- |
| 12 | + |
| 13 | +# Creating and publishing analytics rules for Microsoft Sentinel solutions |
| 14 | + |
| 15 | +Microsoft Sentinel analytics Rules are sets of criteria that define how data should be monitored, what should be detected, and what actions should be taken when specific conditions are met. These rules help identify suspicious behavior, anomalies, and potential security threats by analyzing logs and signals from various data sources. Microsoft Sentinel analytics Rules are a powerful tool for enhancing an organization's security posture by proactively detecting and responding to potential threats. By following a structured approach to creating and managing these rules, organizations can use Microsoft Sentinel's capabilities to protect their digital assets and maintain a robust security infrastructure. For more information, see [Threat detection in Microsoft Sentinel | Microsoft Learn](/azure/sentinel/threat-detection) |
| 16 | + |
| 17 | +This article walks you through the process of creating and publishing analytics rules to Microsoft Sentinel solutions. |
| 18 | + |
| 19 | +## Use cases for Microsoft Sentinel analytics rules |
| 20 | +Microsoft Sentinel analytics Rules can be applied to a wide range of scenarios to enhance security monitoring and threat detection. Some common use cases include: |
| 21 | + |
| 22 | +- **Intrusion Detection:** Identifying unauthorized access attempts or suspicious sign in activities that could indicate a potential breach. |
| 23 | +- **Malware Detection:** Monitoring for known malware signatures or unusual behavior that might suggest the presence of malicious software. |
| 24 | +- **Data Exfiltration:** Detecting large or unusual data transfers that could signify data is being exfiltrated from the network. |
| 25 | +- **Insider Threats:** Identifying anomalous behavior from internal users, such as accessing sensitive data outside of normal hours or patterns. |
| 26 | +- **Compliance Monitoring:** Ensuring adherence to regulatory requirements by monitoring for specific activities or access patterns. |
| 27 | +- **Threat Hunting:** Proactively searching for indicators of compromise or other signs of malicious activity within the network. |
| 28 | +- **Account Compromise:** Detecting signs that user accounts might be compromised, such as unusual geographic sign in patterns or multiple failed sign in attempts. |
| 29 | +- **Network Anomalies:** Identifying unusual network traffic patterns that could indicate the presence of a threat or misconfiguration. |
| 30 | +- **Privilege Escalation:** Monitoring for attempts to gain elevated privileges within the network, which can be a precursor to further malicious activity. |
| 31 | +- **Privilege Escalation:** Monitoring for attempts to gain elevated privileges within the network, which can be a precursor to further malicious activity. |
| 32 | +- **Endpoint Security:** Ensuring that endpoints are secure by detecting deviations from normal behavior or the presence of unauthorized software. |
| 33 | + |
| 34 | +## Creating and publishing analytics rules |
| 35 | + |
| 36 | +Analytics rules should be created in [YAML](https://yaml.org/) format. You can use this analytics rule as reference for creating your own detections - [Sample analytics rule in GitHub](https://github.com/Azure/Azure-Sentinel/blob/master/Solutions/Microsoft%20Entra%20ID/Analytic%20Rules/FailedLogonToAzurePortal.yaml). In this section, we provide a detailed walkthrough of various attributes in the analytics rule. |
| 37 | + |
| 38 | +- **ID** - ID is a standard GUID. Generate it using any development tool, online generator, or PowerShell's [New-GUID cmdlet](/powershell/module/microsoft.powershell.utility/new-guid?view=powershell-6&preserve-view=true). It must be unique among other GUIDs. **This field is mandatory**. |
| 39 | +- **kind** - Represents the type of rule. **This field is mandatory**. Accepted values are: |
| 40 | + - "scheduled" - requires defining other properties - queryFrequency, queryPeriod, triggerThreshold, and triggerOperator |
| 41 | + - "NRT" - Near Real Time |
| 42 | +- **Name** - Provide a brief label that summarizes the detection which is clear and concise to help users understand the purpose of the rule. Use alertDetailsOverride for dynamic names to aid analysts in understanding the alert. **This field is mandatory**. |
| 43 | + - Use Sentence case capitalization |
| 44 | + - Do NOT end with a period |
| 45 | + - Length SHOULD NOT exceed 50 chars whenever possible |
| 46 | +- **Description** - Provide a detailed description of the detection. Description should include information about the behavior being detected, the potential effect, and any recommended actions. **This field is mandatory**. |
| 47 | + - Use Sentence case capitalization |
| 48 | + - Start with - "This query searches for" or "Identifies" |
| 49 | + - Isn't a copy of the name field, it needs to be more descriptive. |
| 50 | + - 'Description' for 'Hunting Query' should NOT exceed 255 characters. |
| 51 | + - Restrict to five sentences or less. |
| 52 | + - Do NOT Describe the Data source (connector or datatype). |
| 53 | + - Do NOT provide a Technical explanation for the query language used. **This field is mandatory**. |
| 54 | +- **Severity** - Define the severity level of the detection. Severity should reflect the potential effect of the behavior being detected and the urgency of the response. |
| 55 | + - Informational: The incident might not represent a security threat directly but might be of interest for follow-up investigation, or to add context or situational awareness to an analyst. |
| 56 | + - Low: Immediate effect is minimal, and a threat actor would need to conduct multiple steps before achieving affect on an environment. |
| 57 | + - Medium: The threat actor could perform some effect on the environment with this activity, but it would be limited in scope or require extra activity. |
| 58 | + - High: The activity identified provides the threat actor with wide ranging access to conduct actions on the environment. |
| 59 | + > [!NOTE] |
| 60 | +> Severity level defaults aren't a guarantee of current or environment impact level. Severity level applies only to Microsoft Sentinel analytics templates. Severity in the Alerts table is otherwise controlled by the security service for which the alert came from. You can use alertDetailsOverride to provide a dynamic severity that depends on the actual outcome of the query. |
| 61 | +- **requiredDataConnectors** - Represents the list of data connectors that are required for the rule to function correctly. This should include the data sources that the rule query against. If there's no current data connector mapping, then an open brace must be used - requiredDataConnectors: [] |
| 62 | + - **connectorId** - Specifies the ID of data connector that is required for the rule to function correctly. If your detection query is dependent on the data fetched from a specific connector, you must specify the connector ID here. For instance, if your analytics rule query depends on the data from this [connector](https://github.com/Azure/Azure-Sentinel/blob/master/Solutions/1Password/Data%20Connectors/1Password_ccpv2/1Password_DataConnectorDefinition.json), you must specify the connector ID as "1PasswordCCPDefinition" |
| 63 | + - **dataTypes** - Data types that the analytics rule is dependent on. This should mention the name of the data type that is mentioned in the "dataTypes" section of the connector. For instance, if your analytics rule query depends on the data from this [connector](https://github.com/Azure/Azure-Sentinel/blob/master/Solutions/1Password/Data%20Connectors/1Password_ccpv2/1Password_DataConnectorDefinition.json), you must specify the data type as "OnePasswordEventLogs_CL." If the analytics rule operates on a Kusto Function/ Parser instead of the table (like Syslog, CommonEventFormat, _CL), dataTypes should be the Kusto Function name/Parser name and not the table name. |
| 64 | +- **queryPeriod** - The query runs across a specified period, such as the last 3 days. **This field is mandatory for scheduled analytics rules**. |
| 65 | + - Use Kusto Query Language (KQL) TimeSpan Format (for example, three days is 3d, 2 hours is 2h). |
| 66 | + - Ensure any learning or reference period is within this time frame. |
| 67 | + - Maximum supported value: 14d |
| 68 | +- **queryFrequency** - The frequency at which the query runs. **This field is mandatory for scheduled analytics rules**. |
| 69 | + - Use Kusto Query Language (KQL) TimeSpan Format (for example, three days is 3d, 2 hours is 2h). |
| 70 | + - QueryFrequency must be less than, or equal to, the QueryPeriod. |
| 71 | + - If the QueryPeriod is greater than or equal to two days (2d), the QueryFrequency value MUST NOT be less than 1 hour (1h) and is only used for High severity detections |
| 72 | +- **triggerOperator** - Indicates the mechanism that triggers the alert, such as greater than a count of 6 (in this case, an alert is triggered if the number of results returned from the query is higher than 6). **This field is mandatory for scheduled analytics rules**. |
| 73 | + - gt – Greater Than |
| 74 | + - lt – Less Than |
| 75 | + - eq – Equal To |
| 76 | +- **triggerThreshold** - The threshold that triggers the alert. Threshold is the value that the triggerOperator compares against. Supported Values: Any integer between 0 and 10000. **This field is mandatory for scheduled analytics rules**. |
| 77 | + - The alert triggers when AlertTriggerOperator is set to Greater Than and AlertTriggerThreshold is above 1. |
| 78 | +- **tactics** - Define the [MITRE ATT&CK tactics](https://attack.mitre.org/versions/v13/matrices/enterprise/) that the detection is related to. This should help users understand the context of the detection and how it fits into the overall threat landscape. **This field is mandatory**. |
| 79 | + - ATT&CK Framework v13 Supported |
| 80 | + - Names MUST NOT have any spaces. Example – InitialAccess or LateralMovement |
| 81 | +- **relevantTechniques** - Define the [MITRE ATT&CK techniques](https://attack.mitre.org/versions/v13/matrices/enterprise/) that the detection is related to. This should help users understand the context of the detection and how it fits into the overall threat landscape. *This field is mandatory**. |
| 82 | + - ATT&CK Framework v13 Supported |
| 83 | + 2. MUST match MITRE Tactics |
| 84 | + 3. Names MUST NOT have any spaces. Example – T1078 or T1078.001 |
| 85 | +- **query** - This is the Kusto query that defines the detection logic. It should be written in Kusto Query Language (KQL) and should be well-structured and easy to understand. The query should be efficient and optimized for performance to ensure it can be run against large datasets without impacting performance. **This field is mandatory**. |
| 86 | + - The query is limited to 10,000 characters. If the query section exceeds this limit, consider reducing the number of characters. This is typically due to including a static list of items used for comparison within the query body. It's recommended to move these lists to use a [Watchlist function](/azure/sentinel/watchlists), [custom JSON/CSV](https://github.com/Azure/Azure-Sentinel/blob/master/Detections/MultipleDataSources/ExchangeServerVulnerabilitiesMarch2021IoCs.yaml) with your list, or a [custom function](https://techcommunity.microsoft.com/t5/azure-sentinel/using-kql-functions-to-speed-up-analysis-in-azure-sentinel/ba-p/712381) with your list. |
| 87 | + - Each line in the query body must have at least one space at the beginning; we standardized on two spaces for readability. |
| 88 | + - If submitting a query for a datatype not present in the Detections or Hunting Queries folder, name the subfolder containing the YAML files after the table being queried. |
| 89 | + - For instance, if your query pertains to the AzureDevOpsAuditing table, create a folder named AzureDevOpsAuditing. |
| 90 | + - Define human-readable names for explicit constants: |
| 91 | + - let FailedLoginEventID = 4625; |
| 92 | + - let countThreshold = 6; |
| 93 | + - Using comments to clarify the query is highly recommended. |
| 94 | + - Comments should be on a separate line rather than at the end of a query statement line: |
| 95 | + - // Removing noisy processes for an environment, adjust as needed |
| 96 | + - If referencing a parser instead of a table name, ensure clarity in the description and include a comment next to the parser function reference. The parser must be imported into the workspace first; otherwise, these queries won't recognize it as valid. |
| 97 | + - Ensure that every available entity field is returned for mapping purposes. Refer to the Entity Mapping - |
| 98 | + - Sanitize the returned table so that it provides only the necessary properties for further investigation. |
| 99 | + - No TimeGenerated filter is required when a simple lookback is used across the entire query; this will be controlled by the queryPeriod value in the YAML. |
| 100 | + - For baselining or historical comparisons, such as comparing today to the previous seven days, include a time-bounded filter such as "*| where TimeGenerated >= ago(lookback)*", as the YAML template doesn't currently support multiple queryPeriod values. |
| 101 | + - Avoid using timeframes shorter than one day unless there's a specific reason. |
| 102 | + - Timeframes longer than 14 days aren't recommended due to potential performance impacts. |
| 103 | + - Summarize when necessary, ensuring to include the time field (usually TimeGenerated) as it's needed in the Entity part. |
| 104 | + - Include both the min() and max() values as follows: "*| summarize StartTime = max(TimeGenerated), EndTime = min(TimeGenerated)*" |
| 105 | + - Use the terms StartTime and EndTime exclusively; do NOT assign the fields the names StartTimeUtc or EndTimeUtc, as this can conflict with user experience preferences. |
| 106 | + - Additionally, include as many fields as possible to help the user understand the context of the alert. It's recommended to include at least one of the primary entities: Host, Account, or IP. |
| 107 | +- **eventGroupingSettings** - An alert rule can generate a separate alert for each query result. For instance, a rule identifying third-party alerts in the event stream could create a Microsoft Sentinel alert for each source alert. |
| 108 | + - To produce a single alert for all query results (the default), use: |
| 109 | + ```json |
| 110 | + eventGroupingSettings: |
| 111 | + aggregationKind: SingleAlert |
| 112 | + ``` |
| 113 | + - To produce a separate alert for each query result, use: |
| 114 | + ```json |
| 115 | + eventGroupingSettings: |
| 116 | + aggregationKind: AlertPerResult |
| 117 | + ``` |
| 118 | +- **entityMappings** - Entity mapping is an integral part of the configuration of scheduled analytics rules. It enriches the rules' output (alerts and incidents) with essential information that serves as the building blocks of any investigative processes and remedial actions that follow. **This field is mandatory**. |
| 119 | + - **entityType** - Represents the standard list of entities recognized by Microsoft Sentinel. See allowed values under the Entity type column value at [Entity Mapping table](/azure/sentinel/entities-reference#entity-types-and-identifiers) |
| 120 | +- **fieldMappings** - Represents the identifier of the field in the query output that corresponds to the entity type. See allowed values under the identifiers column value at [Entity Mapping table](/azure/sentinel/entities-reference#entity-types-and-identifiers). |
| 121 | + - Each template can have up to 10 entity mappings. |
| 122 | + - Each entity mapping can have up to three field mappings (that is, identifiers). |
| 123 | + ```json |
| 124 | + entityMappings: |
| 125 | + - entityType: Account |
| 126 | + fieldMappings: |
| 127 | + - identifier: FullName |
| 128 | + columnName: AccountCustomEntity |
| 129 | + - entityType: Host |
| 130 | + fieldMappings: |
| 131 | + - identifier: FullName |
| 132 | + columnName: HostCustomEntity |
| 133 | + - entityType: IP |
| 134 | + fieldMappings: |
| 135 | + - identifier: Address |
| 136 | + columnName: ClientIP |
| 137 | + - entityType: DNS |
| 138 | + fieldMappings: |
| 139 | + - identifier: DomainName |
| 140 | + columnName: Name |
| 141 | + |
| 142 | + ``` |
| 143 | +- **customDetails** - Custom Details integrate event data into alerts, making it visible in security incidents for faster triaging, investigation, and response. Defined as key-value pairs of property and column names, more information on Custom Details is available [here](/azure/sentinel/surface-custom-details-in-alerts). Up to 20 custom details (that is, key-value pairs) can be defined per template. |
| 144 | + ```json |
| 145 | + customDetails: |
| 146 | + Computers: Computer |
| 147 | + IPs: ComputerIP |
| 148 | + ``` |
| 149 | +- **alertDetailsOverride** - This is a dynamic field that can be used to override the alert details. This can be used to provide more context or information to the analyst when the alert is triggered. Using this feature ensures that analysts receive pertinent information, including relevant entity names to facilitate a quicker and more accurate understanding of the incident. Limitations - |
| 150 | + - A maximum of three parameters can be included in either the Name or Description. |
| 151 | + - The Name must not exceed 256 characters, while the Description is limited to 5,000 characters. |
| 152 | + - The column name within the curly braces must precisely match the expected column name, without any leading or trailing whitespace (for example, {{columnName}}, not {{ columnName }}). See example (columnName1, columnName2, dynamicTactic, and dynamicSeverity are output fields of the scheduled alert query). |
| 153 | + |
| 154 | + ```json |
| 155 | + alertDetailsOverride: |
| 156 | + alertDisplayNameFormat: free text with field names embedded using the format {{columnName}} # Up to 256 chars and 3 placeholders |
| 157 | + alertDescriptionFormat: free text with field names embedded using the format {{columnName}} # Up to 5000 chars and 3 placeholders |
| 158 | + alertTacticsColumnName: dynamicTacticColumnName |
| 159 | + alertSeverityColumnName: dynamicSeverityColumnName |
| 160 | + |
| 161 | + ``` |
| 162 | + |
| 163 | + ```json |
| 164 | + alertDetailsOverride: |
| 165 | + alertDisplayNameFormat: rule {{columnName1}} display name |
| 166 | + alertDescriptionFormat: rule {{columnName2}} display name |
| 167 | + alertTacticsColumnName: dynamicTactic |
| 168 | + alertSeverityColumnName: dynamicSeverity |
| 169 | + ``` |
| 170 | + |
| 171 | +- **Version** - This template version is saved when a customer creates a rule from it. If a new template version is published, customers are notified in the UX. Versions follow the format a.b.c, where a is the major version, b is the minor version, and c is the patch. The version field should be the template's last line. **This field is mandatory**. |
0 commit comments