Merge pull request #5595 from limwainstein/custom-data-collection-mde

Custom data collection

Author: limwainstein

defender-endpoint/TOC.yml

@@ -625,6 +625,12 @@
         href: exclude-devices.md
       - name: Identifying transient devices
         href: transient-device-tagging.md
+      - name: Collect custom device data
+        items:
+        - name: Overview
+          href: custom-data-collection.md
+        - name: Create custom data collection rules
+          href: create-custom-data-collection-rules.md
       - name: Internet facing devices
         href: internet-facing-devices.md
       - name: Device timeline
@@ -1098,10 +1104,7 @@
           href: live-response-command-examples.md
 
       - name: Use sensitivity labels to prioritize incident response
-        href: information-protection-investigation.md
-
-    - name: Advanced hunting
-      href: /defender-xdr/advanced-hunting-overview?toc=/defender-endpoint/toc.json&bc=/defender-endpoint/breadcrumb/toc.json
+        href: information-protection-investigation.md    
 
     - name: Threat analytics
       items:

defender-endpoint/create-custom-data-collection-rules.md

@@ -0,0 +1,115 @@
+---
+title: Create and manage custom data collection rules in Microsoft Defender for Endpoint
+description: Learn how to create and manage custom data collection rules in Microsoft Defender for Endpoint to enhance your threat hunting capabilities.
+ms.service: defender-endpoint
+f1.keywords: 
+  - NOCSH
+ms.author: lwainstein
+author: limwainstein
+ms.localizationpriority: medium
+manager: bagol
+audience: ITPro
+ms.collection: 
+  - m365-security
+  - tier1
+  - usx-security
+ms.topic: how-to
+search.appverid: 
+  - MOE150
+  - MET150
+ms.date: 11/12/2025
+appliesto:
+  - Microsoft Defender for Endpoint
+---
+
+# Create and manage custom data collection rules in Microsoft Defender for Endpoint (Preview)
+
+[!INCLUDE [Prerelease information](../includes/prerelease.md)]
+
+[Custom data collection (Preview)](custom-data-collection.md) enables organizations to expand and customize telemetry collection beyond default configurations to support specialized threat hunting and security monitoring needs.
+
+Custom data collection rules allow you to define specific events and analyze the data to enhance your security visibility and threat hunting operations. Custom data collection rules are based on tailored filters for event properties such as folder paths, process names, and network connections.
+
+This article shows you how to create and manage custom data collection rules in the Microsoft Defender portal.
+
+## Create custom data collection rules
+
+### Prerequisites
+
+To use custom data collection, check that you have the following prerequisites:
+
+- A Microsoft Defender for Endpoint P2 license.
+- A connected [Microsoft Sentinel workspace](/azure/sentinel/quickstart-onboard): required for custom data storage and querying. You can currently only connect one Sentinel workspace per Defender for Endpoint tenant for custom data collection.
+- Dynamic tags configured in [Asset Rule Management](/defender-xdr/configure-asset-rules) for device targeting. To use a tag for custom data collection, the tag should be run at least once.
+
+### Supported operating systems
+
+- **Windows 10 and 11** with a minimum Defender for Endpoint client version of 10.8805.
+    - Windows 10 requires enrollment in [Extended Security Updates (ESU) program](/windows/whats-new/extended-security-updates).
+
+### Performance and limits
+
+- Each collection rule can capture up to 25,000 events per device within a 24-hour rolling window. Once the device reaches the limit, telemetry for the specific rule on the specific device stops until the window resets.
+    - If the device reaches the threshold early in the cycle, it can take up to 24 hours for telemetry to resume. For example, if the device reaches the limit one hour after the window resets, telemetry resumes after 23 hours.
+    - If the device reaches the threshold near the end of the window, the delay is shorter. For example, if the device reaches the limit two hours before the window resets, telemetry resumes after two hours.
+- Rule deployment typically takes 20 minutes to one hour.
+- Custom collection operates alongside default Defender for Endpoint configuration without interference.
+
+### Data costs
+
+Custom data collection is included with Microsoft Defender for Endpoint P2 licensing. However, data ingestion into Microsoft Sentinel workspaces incurs charges based on your Sentinel billing arrangement.
+
+### Create rules
+
+1. In the Microsoft Defender portal, navigate to **Settings** > **Endpoints** > **Rules** > **Custom Data Collection**.
+
+    :::image type="content" source="media/custom-data-collection/custom-data-collection-main-view.png" alt-text="Screenshot of the main Custom Data Collection page." lightbox="media/custom-data-collection/custom-data-collection-main-view.png":::
+
+1. To switch your Microsoft Sentinel workspace, select the workspace name on the top right, and select the workspace.
+1. Select **Create rule**. In the **General Information** section, type a rule name and description, and select **Next**.
+
+    :::image type="content" source="media/create-custom-data-collection-rules/create-custom-data-collection-rule-general.png" alt-text="Screenshot of creating a rule: General Information page." lightbox="media/create-custom-data-collection-rules/create-custom-data-collection-rule-general.png":::
+
+1. In the **Create rule** section:
+
+    1. Select which table you want to collect data from. For more information, see [Supported event tables](custom-data-collection.md#supported-event-tables).
+    1. Select the action for which you want to collect data.
+    1. Add rule conditions to filter the data even further. You can add multiple conditions to refine the data collection. Rule conditions are based on the selected table. For more information, see the respective table link under [Supported event tables](custom-data-collection.md#supported-event-tables).
+
+    :::image type="content" source="media/create-custom-data-collection-rules/create-custom-data-collection-rule.png" alt-text="Screenshot of creating a rule: Create rule page." lightbox="media/create-custom-data-collection-rules/create-custom-data-collection-rule.png":::
+   
+1. Select **Next**.
+
+1. In the **Define rule scope** section, select whether you want to collect data from all applicable client devices or from specific devices that include dynamic tags. For more information, see [Create dynamic rules for devices in asset rule management](/defender-xdr/configure-asset-rules).
+
+    :::image type="content" source="media/create-custom-data-collection-rules/create-custom-data-collection-rule-define-scope.png" alt-text="Screenshot of creating a rule: Define scope page." lightbox="media/create-custom-data-collection-rules/create-custom-data-collection-rule-define-scope.png":::
+
+    > [!NOTE]
+    > Custom data collection only supports dynamic tags.
+
+1. In the **Review and finish** section, review your rule settings, and select **Submit**.
+
+    :::image type="content" source="media/create-custom-data-collection-rules/create-custom-data-collection-rule-review.png" alt-text="Screenshot of creating a rule: Review and finish page." lightbox="media/create-custom-data-collection-rules/create-custom-data-collection-rule-review.png":::
+
+It can take up to an hour for the rule to be deployed to the targeted devices.
+
+## Monitor and troubleshoot
+
+If rules aren't working as expected:
+
+- Create a broad rule to collect events in an unexpected use case. For example, create a rule that collects all network events where `port not equals 0`.
+- Apply individual filters and tags to isolate issues.
+- If a device isn't responding after you enable the feature, reboot the device.
+
+Review these considerations when monitoring and troubleshooting custom data collection rules:
+
+- Endpoint detection and response (EDR) exclusions may override custom collection rules.
+- Dynamic tags update approximately every hour. Check the **Custom collection** > **Last run time** column for the status.
+
+## Edit, delete, and enable or disable custom data collection rules
+
+- To edit a rule, navigate to **Settings** > **Endpoints** > **Rules** > **Custom Collection**, select the rule you want to edit, and select **Edit**.
+- To disable or enable a rule, select the rule you want to modify, and select or clear the **Enable** check-box under the rule description. When you disable a rule,data collection for that rule stops on all targeted devices.
+- To delete a rule, select the rule you want to delete, and select **Delete**. When you delete a rule, the rule is permanently removed from the system.
+
+[!INCLUDE [Microsoft Defender XDR rebranding](../includes/defender-m3d-techcommunity.md)]

defender-endpoint/custom-data-collection.md

@@ -0,0 +1,105 @@
+---
+title: Custom data collection in Microsoft Defender for Endpoint
+description: Custom data collection allows organizations to tailor telemetry collection to their specific threat hunting needs with customizable filters and enhanced visibility.
+ms.service: defender-endpoint
+f1.keywords: 
+  - NOCSH
+ms.author: lwainstein
+author: limwainstein
+ms.localizationpriority: medium
+manager: bagol
+audience: ITPro
+ms.collection: 
+  - m365-security
+  - tier1
+  - usx-security
+ms.topic: concept-article
+search.appverid: 
+  - MOE150
+  - MET150
+ms.date: 11/12/2025
+appliesto:
+  - Microsoft Defender for Endpoint
+---
+
+# Custom data collection in Microsoft Defender for Endpoint (Preview)
+
+[!INCLUDE [Prerelease information](../includes/prerelease.md)]
+
+Custom data collection (Preview) enables organizations to expand and customize telemetry collection beyond default configurations to support specialized threat hunting and security monitoring needs. This feature allows security teams to define specific collection rules with tailored filters for event properties such as folder paths, process names, and network connections.
+
+This article provides an overview of custom data collection so that you can understand the feature's capabilities and how it enhances your security visibility and threat hunting operations.
+
+## How custom data collection works
+
+Custom data collection uses rule-based filtering to capture specific events from endpoint devices and route them to your Microsoft Sentinel workspace for analysis and threat hunting.
+
+:::image type="content" source="media/custom-data-collection/custom-data-collection-main-view.png" alt-text="Screenshot of the main Custom Data Collection page." lightbox="media/custom-data-collection/custom-data-collection-main-view.png":::
+
+Custom collection rules allow you to define the specific events you want to capture and the conditions under which they should be collected.
+
+To create custom data collection rules, see [Create custom data collection rules](create-custom-data-collection-rules.md).
+
+## Supported event tables
+
+Custom data collection supports the following event tables.
+
+| Table name | Description | Learn more |
+|------------|-------------|------------|
+| **DeviceCustomProcessEvents** | Stores data on process creation, termination, and other process-related activities. | [In-portal schema reference](/defender-xdr/advanced-hunting-schema-tables?#get-schema-information-in-the-security-center) or [DeviceProcessEvents](/defender-xdr/advanced-hunting-deviceprocessevents-table) table reference |
+| **DeviceCustomImageLoadEvents** | Stores data on image loading events, including details about the loaded images and their origins. | [In-portal schema reference](/defender-xdr/advanced-hunting-schema-tables?#get-schema-information-in-the-security-center) or [DeviceImageLoadEvents](/defender-xdr/advanced-hunting-deviceimageloadevents-table) table reference |
+| **DeviceCustomFileEvents** | Stores data on file creation, modification, deletion, and access activities. | [In-portal schema reference](/defender-xdr/advanced-hunting-schema-tables?#get-schema-information-in-the-security-center) or [DeviceFileEvents](/defender-xdr/advanced-hunting-devicefileevents-table) table reference |
+| **DeviceCustomNetworkEvents** | Stores data on network connection events, including IP addresses, ports, and protocols. | [In-portal schema reference](/defender-xdr/advanced-hunting-schema-tables?#get-schema-information-in-the-security-center) or [DeviceNetworkEvents](/defender-xdr/advanced-hunting-devicenetworkevents-table) table reference |
+| **DeviceCustomScriptEvents** | Stores data on script execution and process details related to any explicit customer request for collection. This table is a new addition and does not have a reference in the default event tables. | [In-portal schema reference](/defender-xdr/advanced-hunting-schema-tables?#get-schema-information-in-the-security-center) |
+
+## Data flow and integration
+
+This is the typical data flow for custom data collection:
+
+1. Define collection rules in the Microsoft Defender portal with specific filters and device targets.
+2. Rules are transmitted to targeted endpoints, typically within 20 minutes to one hour.
+3. Endpoints collect events matching your rule criteria alongside default telemetry.
+4. Custom event data flows to your connected Microsoft Sentinel workspace.
+5. Query custom data using the supported event tables to learn about specific activities on your endpoints.
+
+## Frequently asked questions
+
+### Does custom data collection affect the default Defender for Endpoint configuration?
+
+No, custom data collection rules live side-by-side with the Defender for Endpoint out-of-the-box configuration.
+
+### Is a Microsoft Sentinel workspace required?
+
+Yes, you need a connected Microsoft Sentinel workspace to create custom data collection rules. For more information, see the [prerequisites](create-custom-data-collection-rules.md#prerequisites).
+
+### How can I know if a rule has reached the endpoint?
+
+You can query for events collected by the relevant rule, for the specific endpoint. For example, the following query returns all effective rules on the endpoint (now and in the past), counting the rules' collected events.
+
+```kusto
+search in (DeviceCustomFileEvents, DeviceCustomScriptEvents, DeviceCustomNetworkEvents) "your_device_id"
+| where DeviceId == "your_device_id"
+| summarize count() by RuleName, RuleLastModificationTime, $table
+```
+
+### Does custom data collection incur additional costs?
+
+See [data costs](create-custom-data-collection-rules.md#data-costs).
+
+### What client versions and operating systems are currently supported?
+
+See [supported operating systems](create-custom-data-collection-rules.md#supported-operating-systems). To query your client version, in [advanced hunting](/defender-xdr/advanced-hunting-overview), use the **ClientVersion** column in the **DeviceInfo** table.
+
+### Are manual (static) tags supported?
+
+No, we currently only support dynamic tags. However, you can create dynamic tags out of manual tags in **Settings > Microsoft Defender XDR > Asset rule management**. For more information, see [Configure dynamic rules for devices in asset rule management](/defender-xdr/configure-asset-rules).
+
+### How can I collect all events for a specific event type?
+
+See [Monitor and troubleshoot](create-custom-data-collection-rules.md#monitor-and-troubleshoot).
+
+## Next steps
+
+- Learn how to [create and manage custom data collection rules](create-custom-data-collection-rules.md)
+
+[!INCLUDE [Microsoft Defender XDR rebranding](../includes/defender-m3d-techcommunity.md)]

defender-endpoint/media/create-custom-data-collection-rules/create-custom-data-collection-rule-define-scope.png

@@ -22,14 +22,19 @@ appliesto:
 ---
 # What's new in Microsoft Defender for Endpoint
 
-
 This article describes Microsoft Defender for Endpoint features that are in preview or generally available (GA) in the latest release.
 
 Learn more:
 
 - [What's new in Microsoft Defender for Endpoint on other operating systems and services](#whats-new-in-defender-for-endpoint-on-other-operating-systems-and-services)
 - [Preview features](/defender-xdr/preview)
 
+## November 2025
+
+|Feature  |Preview/GA  |Description  |
+|---------|------------|-------------|
+|[Custom data collection](custom-data-collection.md) |Preview |Custom data collection enables organizations to expand and customize telemetry collection beyond default configurations to support specialized threat hunting and security monitoring needs. |
+
 ## October 2025
 
 |Feature  |Preview/GA  |Description  |