Merge pull request #293836 from austinmccollum/austinmc-preingest

add ingestion rules

Author: denrea

articles/sentinel/media/understand-threat-intelligence/ingestion-rules-overview.png

@@ -40,7 +40,7 @@ The following table outlines the activities required to make the most of threat
 | Action | Description|
 |---|---|
 | **Store threat intelligence in Microsoft Sentinel's workspace** | <ul><li>Import threat intelligence into Microsoft Sentinel by enabling data connectors to various threat intelligence platforms and feeds.</li><li>Connect threat intelligence to Microsoft Sentinel by using the upload API to connect various TI platforms or custom applications.</li><li>Create threat intelligence with a streamlined management interface.</li>|
-| **Manage threat intelligence** | <ul><li>View imported threat intelligence using queries or advanced search.</li><li>Curate threat intelligence with relationships or tags</li><li>Visualize key information about your TI with workbooks.</li>|
+| **Manage threat intelligence** | <ul><li>View imported threat intelligence using queries or advanced search.</li><li>Curate threat intelligence with relationships, ingestion rules or tags</li><li>Visualize key information about your TI with workbooks.</li>|
 | **Use threat intelligence** | <ul><li>Detect threats and generate security alerts and incidents with built-in analytics rule templates based on your threat intelligence.</li><li>Hunt for threats using your threat intel to ask the right questions about the signals captured for your organization.</li>|
 
 Threat intelligence also provides useful context within other Microsoft Sentinel experiences, such as notebooks. For more information, see [Get started with notebooks and MSTICPy](/azure/sentinel/notebook-get-started). 
@@ -49,7 +49,7 @@ Threat intelligence also provides useful context within other Microsoft Sentinel
 
 ## Import and connect threat intelligence
 
-Most threat intelligence is imported using data connectors or an API. Here are the solutions available for Microsoft Sentinel.
+Most threat intelligence is imported using data connectors or an API. Configure ingestion rules to reduce noise and ensure your intelligence feeds are optimized. Here are the solutions available for Microsoft Sentinel.
 
 - **Microsoft Defender Threat Intelligence** data connector to ingest Microsoft's threat intelligence 
 - **Threat Intelligence - TAXII** data connector for industry-standard STIX/TAXII feeds
@@ -132,7 +132,8 @@ Threat intelligence powered by Microsoft Sentinel is managed next to Microsoft D
 >[!NOTE]
 > Threat intelligence in the Azure portal is still accessed from **Microsoft Sentinel** > **Threat management** > **Threat intelligence**.
 
-Two of the most common threat intelligence tasks are creating new threat intelligence related to security investigations and adding tags. The management interface streamlines the manual process of creating individual threat intel with a few key features.
+Two of the most common threat intelligence tasks are creating new threat intelligence related to security investigations and adding tags. The management interface streamlines the manual process of curating individual threat intel with a few key features.
+- Configure ingestion rules to optimize threat intel from incoming sources.
 - Define relationships as you create new STIX objects.
 - Curate existing TI with the relationship builder.
 - Copy common metadata from a new or existing TI object with the duplicate feature.
@@ -149,11 +150,30 @@ The following STIX objects are available in Microsoft Sentinel:
 | **Identity** | Describe victims, organizations, and other groups or individuals along with the business sectors most closely associated with them. |
 | **Relationship** | The threads that connect threat intelligence, helping to make connections across disparate signals and data points are described with relationships. |
 
+### Configure ingestion rules
+
+Optimize threat intelligence feeds by filtering and enhancing objects before they're delivered to your workspace. Ingestion rules update attributes, or filter objects out all together. The following table lists some use cases:
+
+| Ingestion rule use case | Description |
+|---|---|
+| Reduce noise | Filter out old threat intelligence not updated for 6 months that also has low confidence. |
+| Extend validity date | Promote high fidelity IOCs from trusted sources by extending their `Valid until` by 30 days. |
+| Remember the old days | The new threat actor taxonomy is great, but some of the analysts want to be sure to tag the old names. |
+
+:::image type="content" source="media/understand-threat-intelligence/ingestion-rules-overview.png" alt-text="Screenshot shows four ingestion rules matching the use cases.":::
+
+Keep in mind the following tips for using ingestion rules:
+- All rules apply in order. Threat intelligence objects being ingested will get processed by each rule until a `Delete` action is taken. If no action is taken on an object, it is ingested from the source as is.
+- The `Delete` action means the threat intelligence object is skipped for ingestion, meaning it's removed from the pipeline. Any previous versions of the object already ingested aren't affected.
+- New and edited rules take up to 15 minutes to take effect.
+
+For more information, see [Work with threat intelligence ingestion rules](work-with-threat-indicators.md#optimize-threat-intelligence-feeds-with-ingestion-rules).
+
 ### Create relationships
 
-Enhance threat detection and response by establishing connections between objects with the relationship builder. The following table lists some of its use cases.
+Enhance threat detection and response by establishing connections between objects with the relationship builder. The following table lists some of its use cases:
 
-| Use case | Description |
+| Relationship use case | Description |
 |---|---|
 | Connect a threat actor to an attack pattern | The threat actor `APT29` *Uses* the attack pattern `Phishing via Email` to gain initial access.|
 | Link an indicator to a threat actor|  A domain indicator `allyourbase.contoso.com` is *Attributed to* the threat actor `APT29`. |

articles/sentinel/media/use-matching-analytics-to-detect-threats/matching-analytics-template-ga.png

@@ -4,7 +4,7 @@ titleSuffix: Microsoft Sentinel
 description: This article explains how to detect threats with Microsoft-generated threat intelligence in Microsoft Sentinel.
 author: austinmccollum
 ms.topic: how-to
-ms.date: 3/14/2024
+ms.date: 01/28/2025
 ms.author: austinmc
 appliesto:
     - Microsoft Sentinel in the Azure portal
@@ -20,23 +20,18 @@ ms.collection: usx-security
 
 Take advantage of threat intelligence produced by Microsoft to generate high-fidelity alerts and incidents with the **Microsoft Defender Threat Intelligence Analytics** rule. This built-in rule in Microsoft Sentinel matches indicators with Common Event Format (CEF) logs, Windows DNS events with domain and IPv4 threat indicators, syslog data, and more.
 
-> [!IMPORTANT]
-> Matching analytics is currently in preview. See the [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/) for more legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.
->
-
 ## Prerequisites
 
 You must install one or more of the supported data connectors to produce high-fidelity alerts and incidents. A premium Microsoft Defender Threat Intelligence license isn't required. Install the appropriate solutions from the **Content hub** to connect these data sources:
 
-  - Common Event Format
-  - DNS (preview)
-  - Syslog
-  - Office activity logs
+  - Common Event Format (CEF) via Legacy Agent
+  - Syslog via Legacy Agent
+  - Microsoft 365 (formerly, Office 365)
   - Azure activity logs
-  - ASIM DNS logs
+  - Windows DNS via AMA
   - ASIM Network sessions
 
-  :::image type="content" source="media/use-matching-analytics-to-detect-threats/data-sources.png" alt-text="A screenshot that shows the Microsoft Defender Threat Intelligence Analytics rule data source connections."::: 
+  :::image type="content" source="media/use-matching-analytics-to-detect-threats/matching-analytics-template-ga.png" alt-text="A screenshot that shows the Microsoft Defender Threat Intelligence Analytics rule data source connections."::: 
 
   For example, depending on your data source, you might use the following solutions and data connectors:
 
@@ -71,7 +66,6 @@ Matching analytics is configured when you enable the **Microsoft Defender Threat
 Microsoft Defender Threat Intelligence Analytics matches your logs with domain, IP, and URL indicators in the following ways:
 
 - **CEF logs** ingested into the Log Analytics `CommonSecurityLog` table match URL and domain indicators if populated in the `RequestURL` field, and IPv4 indicators in the `DestinationIP` field.
-- **Windows DNS logs**, where `SubType == "LookupQuery"` ingested into the `DnsEvents` table matches domain indicators populated in the `Name` field, and IPv4 indicators in the `IPAddresses` field.
 - **Syslog events**, where `Facility == "cron"` ingested into the `Syslog` table matches domain and IPv4 indicators directly from the `SyslogMessage` field.
 - **Office activity logs** ingested into the `OfficeActivity` table match IPv4 indicators directly from the `ClientIP` field.
 - **Azure activity logs** ingested into the `AzureActivity` table match IPv4 indicators directly from the `CallerIpAddress` field.
@@ -84,7 +78,7 @@ If Microsoft's analytics finds a match, any alerts generated are grouped into in
 
 Use the following steps to triage through the incidents generated by the **Microsoft Defender Threat Intelligence Analytics** rule:
 
-1. In the Microsoft Sentinel workspace where you enabled the **Microsoft Defender Threat Intelligence Analytics** rule, select **Incidents**, and search for **Microsoft Defender Threat Intelligence Analytics**.
+1. In the Microsoft Sentinel workspace where you enabled the **Microsoft Defender Threat Intelligence Analytics** rule, select **Incidents**, and search for **Microsoft Threat Intelligence Analytics**.
 
     Any incidents that are found appear in the grid.
 
@@ -98,7 +92,7 @@ Use the following steps to triage through the incidents generated by the **Micro
 
     Alerts are then grouped on a per-observable basis of the indicator. For example, all alerts generated in a 24-hour time period that match the `contoso.com` domain are grouped into a single incident with a severity assigned based on the highest alert severity.
 
-1. Observe the indicator information. When a match is found, the indicator is published to the Log Analytics `ThreatIntelligenceIndicators` table, and it appears on the **Threat Intelligence** page. For any indicators published from this rule, the source is defined as **Microsoft Defender Threat Intelligence Analytics**.
+1. Observe the indicator information. When a match is found, the indicator is published to the Log Analytics `ThreatIntelligenceIndicators` table, and it appears on the **Threat Intelligence** page. For any indicators published from this rule, the source is defined as **Microsoft Threat Intelligence Analytics**.
 
 Here's an example of the `ThreatIntelligenceIndicators` table.
 

articles/sentinel/media/work-with-threat-indicators/new-ingestion-rule.png

@@ -23,13 +23,25 @@ Get notified when this page is updated by copying and pasting the following URL
 
 ## January 2025
 
+- [Optimize threat intelligence feeds with ingestion rules](#optimize-threat-intelligence-feeds-with-ingestion-rules)
+- [Matching analytics rule now generally available (GA)](#matching-analytics-rule-now-generally-available-ga)
 - [Threat intelligence management interface updated](#threat-intelligence-management-interface-has-moved)
 - [Unlock advanced hunting with new STIX objects by opting in to new threat intelligence tables](#unlock-advanced-hunting-with-new-stix-objects-by-opting-in-to-new-threat-intelligence-tables)
 - [Threat intelligence upload API now supports more STIX objects](#threat-intelligence-upload-api-now-supports-more-stix-objects)
 - [Microsoft Defender Threat Intelligence data connectors now generally available (GA)](#microsoft-defender-threat-intelligence-data-connectors-now-generally-available-ga)
 - [Bicep template support for repositories (Preview)](#bicep-template-support-for-repositories-preview)
 - [View granular solution content in the Microsoft Sentinel content hub](#view-granular-solution-content-in-the-microsoft-sentinel-content-hub)
 
+### Optimize threat intelligence feeds with ingestion rules
+
+Optimize threat intelligence feeds by filtering and enhancing objects before they're delivered to your workspace. Ingestion rules update threat intel object attributes, or filter objects out all together.
+
+For more information, see [Understand threat intelligence ingestion rules](understand-threat-intelligence.md#configure-ingestion-rules).
+
+### Matching analytics rule now generally available (GA)
+
+Microsoft provides access to its premium threat intelligence through the Defender Threat Intelligence analytics rule which is now generally available (GA). For more information on how to take advantage of this rule, which generates high-fidelity alerts and incidents, see [Use matching analytics to detect threats](use-matching-analytics-to-detect-threats.md).
+
 ### Threat intelligence management interface has moved
 
 Threat intelligence for Microsoft Sentinel in the Defender portal has changed! We've renamed the page **Intel management** and moved it with other threat intelligence workflows. There's no change for customers using Microsoft Sentinel in the Azure experience.

articles/sentinel/media/work-with-threat-indicators/select-ingestion-rules.png

@@ -63,13 +63,40 @@ For more information on supported STIX objects, see [Understand threat intellige
 
 ## Manage threat intelligence 
 
-Curate existing TI with the relationship builder. Use the management interface to search, filter and sort, then add tags to your threat intelligence.
+Optimize TI from your sources with ingestion rules. Curate existing TI with the relationship builder. Use the management interface to search, filter and sort, then add tags to your threat intelligence.
+
+### Optimize threat intelligence feeds with ingestion rules
+
+Reduce noise from your TI feeds, extend the validity of high value indicators, and add meaningful tags to incoming objects. These are just some of the use cases for ingestion rules. Here are the steps for extending the validity date on high value indicators.
+
+1. Select **Ingestion rules** to open a whole new page to view existing rules and construct new rule logic.
+
+   :::image type="content" source="media/work-with-threat-indicators/select-ingestion-rules.png" alt-text="Screenshot showing threat intelligence management menu hovering on ingestion rules.":::
+
+1. Enter a descriptive name for your rule. The ingestion rules page has ample rule for the name, but it's the only text description available to differentiate your rules without editing them.
+
+1. Select the **Object type**. This use case is based on extending the `Valid from` property which is only available for `Indicator` object types.
+
+1. **Add condition** for `Source` `Equals` and select your high value `Source`.
+1. **Add condition** for `Confidence` `Greater than or equal` and enter a `Confidence` score.
+
+1. Select the **Action**. Since we want to modify this indicator, select `Edit`.
+1. Select the **Add action** for `Valid until`, `Extend by`, and select a time span in days.
+1. Consider adding a tag to indicate the high value placed on these indicators, like `Extended`. The modified date is not updated by ingestion rules.
+1. Select the **Order** you want the rule to run. Rules run from lowest order number to highest. Each rule evaluates every object ingested.
+1. If the rule is ready to be enabled, toggle **Status** to on.
+1. Select **Add** to create the ingestion rule.
+
+:::image type="content" source="media/work-with-threat-indicators/new-ingestion-rule.png" alt-text="Screenshot showing new ingestion rule creation for extending valid until date.":::
+
+For more information, see [Understand threat intelligence ingestion rules](understand-threat-intelligence.md#configure-ingestion-rules).
 
 ### Curate threat intelligence with the relationship builder
 
 Connect threat intelligence objects with the relationship builder. There's a maximum of 20 relationships in the builder at once, but more connections can be created through multiple iterations and by adding relationship target references for new objects.
 
 1. Start with an object like a threat actor or attack pattern where the single object connects to one or more objects, like indicators.
+
 1. Add the relationship type according to the best practices outlined in the following table and in the [STIX 2.1 reference relationship summary table](https://docs.oasis-open.org/cti/stix/v2.1/os/stix-v2.1-os.html#_6n2czpjuie3v):
 
 | Relationship type | Description |
@@ -101,6 +128,7 @@ In the following image, multiple sources were used to search by placing them in
 
 :::image type="content" source="media/work-with-threat-indicators/advanced-search.png" alt-text="Screenshot shows an OR operator combined with multiple AND conditions to search threat intelligence." lightbox="media/work-with-threat-indicators/advanced-search.png":::
 
+
 Microsoft Sentinel only displays the most current version of your threat intel in this view. For more information on how objects are updated, see [Understand threat intelligence](understand-threat-intelligence.md#view-your-threat-intelligence).
 
 IP and domain name indicators are enriched with extra `GeoLocation` and `WhoIs` data so you can provide more context for any investigations where indicator is found.