Merge pull request #276234 from yelevin/yelevin/analytics-rules-refactoring

Analytics rule refactoring

Author: JamesJBarnett

articles/azure-vmware/azure-security-integration.md

@@ -159,7 +159,7 @@ After connecting data sources to Microsoft Sentinel, you can create rules to gen
 
 6. On the **Incident settings** tab, enable **Create incidents from alerts triggered by this analytics rule** and select **Next: Automated response**.
  
-    :::image type="content" source="../sentinel/media/detect-threats-custom/general-tab.png" alt-text="Screenshot showing the Analytic rule wizard for creating a new rule in Microsoft Sentinel.":::
+    :::image type="content" source="../sentinel/media/create-analytics-rules/general-tab.png" alt-text="Screenshot showing the Analytic rule wizard for creating a new rule in Microsoft Sentinel.":::
 
 7. Select **Next: Review**.
 

articles/sentinel/TOC.yml

@@ -952,6 +952,8 @@
       href: threat-detection.md
     - name: Scheduled analytics rules
       items:
+      - name: Overview
+        href: scheduled-rules-overview.md
       - name: Create a scheduled rule from a template
         href: create-analytics-rule-from-template.md
       - name: Create a scheduled rule from scratch

articles/sentinel/automate-incident-handling-with-automation-rules.md

@@ -52,7 +52,7 @@ Automation rules are made up of several components:
 
 ### Triggers
 
-Automation rules are triggered **when an incident is created or updated** or **when an alert is created**. Recall that incidents include alerts, and that both alerts and incidents can be created by analytics rules, of which there are several types, as explained in [Detect threats with built-in analytics rules in Microsoft Sentinel](detect-threats-built-in.md).
+Automation rules are triggered **when an incident is created or updated** or **when an alert is created**. Recall that incidents include alerts, and that both alerts and incidents can be created by analytics rules, of which there are several types, as explained in [Threat detection in Microsoft Sentinel](threat-detection.md).
 
 The following table shows the different possible scenarios that will cause an automation rule to run.
 
@@ -85,7 +85,7 @@ Even without being onboarded to the unified portal, you might anyway decide to u
 - A playbook can be triggered by an alert and send the alert to an external ticketing system for incident creation and management, creating a new ticket for each alert.
 
 > [!NOTE]
-> - Alert-triggered automation is available only for alerts created by [**Scheduled**, **NRT**, and **Microsoft security** analytics rules](detect-threats-built-in.md).
+> - Alert-triggered automation is available only for alerts created by [**Scheduled**, **NRT**, and **Microsoft security** analytics rules](threat-detection.md).
 >
 > - Alert-triggered automation for alerts created by Microsoft Defender XDR is not available in the unified security operations platform. For more information, see [Automation with the unified security operations platform](automation.md#automation-with-the-unified-security-operations-platform).
 

articles/sentinel/configure-content.md

@@ -14,14 +14,14 @@ In the previous deployment step, you enabled Microsoft Sentinel, health monitori
 
 ## Configure your security content
 
-|Step  |Description  |
-|---------|---------|
-|**Set up data connectors** |Based on the [data sources you selected when you planned your deployment](prioritize-data-connectors.md), and after [enabling the relevant solutions](enable-sentinel-features-content.md), you can now install or set up your data connectors.<br><br>- If you're using an existing connector, [find your connector](data-connectors-reference.md) from this full list of data connectors.<br>- If you're creating a custom connector, use [these resources](create-custom-connector.md).<br>- If you're setting up a connector to ingest CEF or Syslog logs, review these [options](connect-cef-syslog-options.md). |
-|**Set up analytics rules**     |After you've set up Microsoft Sentinel to collect data from all over your organization, you can begin using threat detection rules or [analytics rules](detect-threats-built-in.md). Select the steps you need to set up and configure your analytics rules:<br><br>- [Create a scheduled query rule](detect-threats-custom.md): Create custom analytics rules to help discover threats and anomalous behaviors in your environment.<br>- [Map data fields to entities](map-data-fields-to-entities.md): Add or change entity mappings in an existing analytics rule.<br>- [Surface custom details in alerts](surface-custom-details-in-alerts.md): Add or change custom details in an existing analytics rule.<br>- [Customize alert details](customize-alert-details.md): Override the default properties of alerts with content from the underlying query results.<br>- [Export and import analytics rules](import-export-analytics-rules.md): Export your analytics rules to Azure Resource Manager (ARM) template files, and import rules from these files. The export action creates a JSON file in your browser's downloads location, that you can then rename, move, and otherwise handle like any other file.<br>- [Create near-real-time (NRT) detection analytics rules](create-nrt-rules.md): Create near-time analytics rules for up-to-the-minute threat detection out-of-the-box. This type of rule was designed to be highly responsive by running its query at intervals just one minute apart.<br>- [Work with anomaly detection analytics rules](work-with-anomaly-rules.md): Work with built-in anomaly templates that use thousands of data sources and millions of events, or change thresholds and parameters for the anomalies within the user interface.<br>- [Manage template versions for your scheduled analytics rules](manage-analytics-rule-templates.md): Track the versions of your analytics rule templates, and either revert active rules to existing template versions, or update them to new ones.<br>- [Handle ingestion delay in scheduled analytics rules](ingestion-delay.md): Learn how ingestion delay might impact your scheduled analytics rules and how you can fix them to cover these gaps.          |
-|**Set up automation rules**     |[Create automation rules](create-manage-use-automation-rules.md). Define the triggers and conditions that determine when your [automation rule](automate-incident-handling-with-automation-rules.md) runs, the various actions that you can have the rule perform, and the remaining features and functionalities.    |
-|**Set up playbooks**     |A [playbook](automate-responses-with-playbooks.md) is a collection of remediation actions that you run from Microsoft Sentinel as a routine, to help automate and orchestrate your threat response. To set up playbooks:<br><br>- Review [recommended playbooks](automate-responses-with-playbooks.md#recommended-playbooks)<br>- [Create playbooks from templates](use-playbook-templates.md): A playbook template is a prebuilt, tested, and ready-to-use workflow that can be customized to meet your needs. Templates can also serve as a reference for best practices when developing playbooks from scratch, or as inspiration for new automation scenarios.<br>- Review these [steps for creating a playbook](automate-responses-with-playbooks.md#steps-for-creating-a-playbook)          |
-|**Set up workbooks**     |[Workbooks](monitor-your-data.md) provide a flexible canvas for data analysis and the creation of rich visual reports within Microsoft Sentinel. Workbook templates allow you to quickly gain insights across your data as soon as you connect a data source. To set up workbooks:<br><br>- Review [commonly used Microsoft Sentinel workbooks](top-workbooks.md)<br>- [Use existing workbook templates available with packaged solutions](monitor-your-data.md)<br>- [Create custom workbooks across your data](monitor-your-data.md#create-new-workbook)         |
-|**Set up watchlists**     |[Watchlists](watchlists.md) allow you to correlate data from a data source you provide with the events in your Microsoft Sentinel environment. To set up watchlists:<br><br>- [Create watchlists](watchlists-create.md)<br>- [Build queries or detection rules with watchlists](watchlists-queries.md): Query data in any table against data from a watchlist by treating the watchlist as a table for joins and lookups. When you create a watchlist, you define the SearchKey. The search key is the name of a column in your watchlist that you expect to use as a join with other data or as a frequent object of searches.           |
+| Step | Description |
+| ---- | ----------- |
+| **Set up data connectors** | Based on the [data sources you selected when you planned your deployment](prioritize-data-connectors.md), and after [enabling the relevant solutions](enable-sentinel-features-content.md), you can now install or set up your data connectors.<br><br>- If you're using an existing connector, [find your connector](data-connectors-reference.md) from this full list of data connectors.<br>- If you're creating a custom connector, use [these resources](create-custom-connector.md).<br>- If you're setting up a connector to ingest CEF or Syslog logs, review these [options](connect-cef-syslog-options.md). |
+| **Set up analytics rules** | After you've set up Microsoft Sentinel to collect data from all over your organization, you can begin using [analytics rules](threat-detection.md) to detect threats. Select the steps you need to set up and configure your analytics rules:<br><br>- Create scheduled rules [from templates](create-analytics-rule-from-template.md) or [from scratch](create-analytics-rules.md): Create analytics rules to help discover threats and anomalous behaviors in your environment.<br>- [Map data fields to entities](map-data-fields-to-entities.md): Add or change entity mappings in an analytics rule.<br>- [Surface custom details in alerts](surface-custom-details-in-alerts.md): Add or change custom details in an analytics rule.<br>- [Customize alert details](customize-alert-details.md): Override the default properties of alerts with content from the underlying query results.<br>- [Export and import analytics rules](import-export-analytics-rules.md): Export your analytics rules to Azure Resource Manager (ARM) template files, and import rules from these files. The export action creates a JSON file in your browser's downloads location, that you can then rename, move, and otherwise handle like any other file.<br>- [Create near-real-time (NRT) detection analytics rules](create-nrt-rules.md): Create near-time analytics rules for up-to-the-minute threat detection out-of-the-box. This type of rule was designed to be highly responsive by running its query at intervals just one minute apart.<br>- [Work with anomaly detection analytics rules](work-with-anomaly-rules.md): Work with built-in anomaly templates that use thousands of data sources and millions of events, or change thresholds and parameters for the anomalies within the user interface.<br>- [Manage template versions for your scheduled analytics rules](manage-analytics-rule-templates.md): Track the versions of your analytics rule templates, and either revert active rules to existing template versions, or update them to new ones.<br>- [Handle ingestion delay in scheduled analytics rules](ingestion-delay.md): Learn how ingestion delay might impact your scheduled analytics rules and how you can fix them to cover these gaps.          |
+| **Set up automation rules** | [Create automation rules](create-manage-use-automation-rules.md). Define the triggers and conditions that determine when your [automation rule](automate-incident-handling-with-automation-rules.md) runs, the various actions that you can have the rule perform, and the remaining features and functionalities.    |
+| **Set up playbooks** | A [playbook](automate-responses-with-playbooks.md) is a collection of remediation actions that you run from Microsoft Sentinel as a routine, to help automate and orchestrate your threat response. To set up playbooks:<br><br>- Review [recommended playbooks](automate-responses-with-playbooks.md#recommended-playbooks)<br>- [Create playbooks from templates](use-playbook-templates.md): A playbook template is a prebuilt, tested, and ready-to-use workflow that can be customized to meet your needs. Templates can also serve as a reference for best practices when developing playbooks from scratch, or as inspiration for new automation scenarios.<br>- Review these [steps for creating a playbook](automate-responses-with-playbooks.md#steps-for-creating-a-playbook) |
+| **Set up workbooks** | [Workbooks](monitor-your-data.md) provide a flexible canvas for data analysis and the creation of rich visual reports within Microsoft Sentinel. Workbook templates allow you to quickly gain insights across your data as soon as you connect a data source. To set up workbooks:<br><br>- Review [commonly used Microsoft Sentinel workbooks](top-workbooks.md)<br>- [Use existing workbook templates available with packaged solutions](monitor-your-data.md)<br>- [Create custom workbooks across your data](monitor-your-data.md#create-new-workbook) |
+| **Set up watchlists** | [Watchlists](watchlists.md) allow you to correlate data from a data source you provide with the events in your Microsoft Sentinel environment. To set up watchlists:<br><br>- [Create watchlists](watchlists-create.md)<br>- [Build queries or detection rules with watchlists](watchlists-queries.md): Query data in any table against data from a watchlist by treating the watchlist as a table for joins and lookups. When you create a watchlist, you define the SearchKey. The search key is the name of a column in your watchlist that you expect to use as a join with other data or as a frequent object of searches. |
 
 ## Next steps
 

articles/sentinel/configure-fusion-rules.md

@@ -13,7 +13,7 @@ ms.author: yelevin
 
 [!INCLUDE [reference-to-feature-availability](includes/reference-to-feature-availability.md)]
 
-Microsoft Sentinel uses Fusion, a correlation engine based on scalable machine learning algorithms, to automatically detect multistage attacks by identifying combinations of anomalous behaviors and suspicious activities that are observed at various stages of the kill chain. On the basis of these discoveries, Microsoft Sentinel generates incidents that would otherwise be difficult to catch. These incidents comprise two or more alerts or activities. By design, these incidents are low-volume, high-fidelity, and high-severity.
+Microsoft Sentinel uses Fusion, a correlation engine based on scalable machine learning algorithms, to automatically detect multistage attacks by identifying combinations of anomalous behaviors and suspicious activities that are observed at various stages of the kill chain. Based on these discoveries, Microsoft Sentinel generates incidents that would otherwise be difficult to catch. These incidents comprise two or more alerts or activities. By design, these incidents are low-volume, high-fidelity, and high-severity.
 
 Customized for your environment, this detection technology not only reduces [false positive](false-positives.md) rates but can also detect attacks with limited or missing information.
 
@@ -54,7 +54,7 @@ This detection is enabled by default in Microsoft Sentinel. To check or change i
 
 
 
-    - **Exclude specific detection patterns from Fusion detection**. Certain Fusion detections may not be applicable to your environment, or may be prone to generating false positives. If you’d like to exclude a specific Fusion detection pattern, follow the instructions below:
+    - **Exclude specific detection patterns from Fusion detection**. Certain Fusion detections might not be applicable to your environment, or might be prone to generating false positives. If you’d like to exclude a specific Fusion detection pattern, follow the instructions below:
 
         1. Locate and open a Fusion incident of the kind you want to exclude.
 
@@ -64,9 +64,9 @@ This detection is enabled by default in Microsoft Sentinel. To check or change i
 
             :::image type="content" source="media/configure-fusion-rules/exclude-fusion-incident.png" alt-text="Screenshot of Fusion incident. Select the exclusion link.":::
 
-        On the **Configure Fusion** tab, you'll see the detection pattern - a combination of alerts and anomalies in a Fusion incident - has been added to the exclusion list, along with the time when the detection pattern was added.
+        On the **Configure Fusion** tab, you see that the detection pattern—a combination of alerts and anomalies in a Fusion incident—has been added to the exclusion list, along with the time when the detection pattern was added.
 
-        You can remove an excluded detection pattern any time by selecting the trashcan icon on that detection pattern.
+        You can remove an excluded detection pattern at any time by selecting the trashcan icon on that detection pattern.
 
         :::image type="content" source="media/configure-fusion-rules/exclusion-patterns-list.png" alt-text="Screenshot of list of excluded detection patterns.":::
 
@@ -95,15 +95,15 @@ This detection is enabled by default in Microsoft Sentinel. To check or change i
 
 **Fusion** can detect scenario-based multi-stage attacks and emerging threats using alerts generated by [scheduled analytics rules](detect-threats-custom.md). We recommend you take the following steps to configure and enable these rules, so that you can get the most out of Microsoft Sentinel's Fusion capabilities.
 
-1. Fusion for emerging threats can use alerts generated by any scheduled analytics rules, both [built-in](detect-threats-built-in.md#scheduled) and those [created by your security analysts](detect-threats-custom.md), that contain kill-chain (tactics) and entity mapping information. To ensure that an analytics rule's output can be used by Fusion to detect emerging threats:
+1. Fusion for emerging threats can use alerts generated by any [scheduled analytics rules](scheduled-rules-overview.md) that contain kill-chain (tactics) and entity mapping information. To ensure that an analytics rule's output can be used by Fusion to detect emerging threats:
 
     - Review **entity mapping** for these scheduled rules. Use the [entity mapping configuration section](map-data-fields-to-entities.md) to map parameters from your query results to Microsoft Sentinel-recognized entities. Because Fusion correlates alerts based on entities (such as *user account* or *IP address*), its ML algorithms cannot perform alert matching without the entity information.
 
-    - Review the **tactics and techniques** in your analytics rule details. The Fusion ML algorithm uses [MITRE ATT&CK](https://attack.mitre.org/) information for detecting multi-stage attacks, and the tactics and techniques you label the analytics rules with will show up in the resulting incidents. Fusion calculations may be affected if incoming alerts are missing tactic information.
+    - Review the **tactics and techniques** in your analytics rule details. The Fusion ML algorithm uses [MITRE ATT&CK](https://attack.mitre.org/) information for detecting multi-stage attacks, and the tactics and techniques you label the analytics rules with will show up in the resulting incidents. Fusion calculations might be affected if incoming alerts are missing tactic information.
 
 1. Fusion can also detect scenario-based threats using rules based on the following **scheduled analytics rule templates**.
 
-    To enable the queries available as templates in the **Analytics** blade, go to the **Rule templates** tab, select the rule name in the templates gallery, and click **Create rule** in the details pane. 
+    To enable the queries available as templates in the **Analytics** page, go to the **Rule templates** tab, select the rule name in the templates gallery, and select **Create rule** in the details pane. 
 
     - [Cisco - firewall block but success logon to Microsoft Entra ID](https://github.com/Azure/Azure-Sentinel/blob/60e7aa065b196a6ed113c748a6e7ae3566f8c89c/Detections/MultipleDataSources/SigninFirewallCorrelation.yaml)
     - [Fortinet - Beacon pattern detected](https://github.com/Azure/Azure-Sentinel/blob/83c6d8c7f65a5f209f39f3e06eb2f7374fd8439c/Detections/CommonSecurityLog/Fortinet-NetworkBeaconPattern.yaml)