Merge pull request #199821 from yoninalmsft/micro-agent-twin-config

event aggregation changes

Author: PMEds28

articles/defender-for-iot/device-builders/concept-event-aggregation.md

@@ -1,39 +1,35 @@
 ---
 title: Micro agent event collection (Preview)
-description: Defender for IoT security agents collects data and system events from your local device, and sends the data to the Azure cloud for processing, and analytics.
+description: Defender for IoT security agents collect data and system events from your local device, and send the data to the Azure cloud for processing, and analytics.
 ms.date: 04/26/2022
 ms.topic: conceptual
 ---
 
 # Micro agent event collection (Preview)
 
-Defender for IoT security agents collects data, and system events from your local device, and sends the data to the Azure cloud for processing.
+Defender for IoT security agents collect data and system events from your local device, and send the data to the Azure cloud for processing.
 
 If you've configured and connected a Log Analytics workspace, you'll see these events in Log Analytics. For more information, see [Tutorial: Investigate security alerts](tutorial-investigate-security-alerts.md).
 
-The Defender for IoT micro agent collects many types of device events including new processes, and all new connection events. Both the new process, and new connection events may occur frequently on a device. This capability is important for comprehensive security, however, the number of messages the security agents send may quickly meet, or exceed your IoT Hub quota, and cost limits. These messages, and events contain highly valuable security information that is crucial to protecting your device.
+The Defender for IoT micro agent collects many types of device events including new processes, and all new connection events. Both the new process and new connection events may occur frequently on a device. This capability is important for comprehensive security, however, the number of messages the security agents send may quickly meet, or exceed your IoT Hub quota, and cost limits. These messages and events contain highly valuable security information that is crucial to protecting your device.
 
-To reduce the number of messages, and costs while maintaining your device's security, Defender for IoT agents aggregate the following types of events:
+To reduce the number of messages and costs while maintaining your device's security, Defender for IoT agents aggregate the following types of events:
 
-- ProcessCreate (Linux only)
+- Process events (Linux only)
 
-- Network ConnectionCreate
+- Network Activity events
 
-Event-based collectors are collectors that are triggered based on corresponding activity from within the device. For example, ``a process was started in the device``.
-
-Triggered based collectors are collectors that are triggered in a scheduled manner based on the customer's configurations.
+For more information, see [event aggregation for process and network collectors](#event-aggregation-for-process-and-network-collectors).
 
-## How does event aggregation work?
-
-Defender for IoT agents aggregate events for the interval period, or time window. Once the interval period has passed, the agent sends the aggregated events to the Azure cloud for further analysis. The aggregated events are stored in memory until being sent to the Azure cloud.
+Event-based collectors are collectors that are triggered based on corresponding activity from within the device. For example, ``a process was started in the device``.
 
-The agent collects identical events to the ones that are already stored in memory. This collection causes the agent to increase the hit count of this specific event to reduce the memory footprint of the agent. When the aggregation time window passes, the agent sends the hit count of each type of event that occurred. Event aggregation is simply the aggregation of the hit counts of each collected type of event.
+Trigger-based collectors are collectors that are triggered in a scheduled manner based on the customer's configurations.
 
-## Process events (event based)
+## Process events (event-based collector)
 
 Process events are supported on Linux operating systems.
 
-Process events are considered identical when the *command line*, and *userid* are identical.
+Process events are considered identical when the *command line* and *userid* are identical.
 
 The default buffer for process events is 256 processes. When this limit is met, the buffer will cycle, and the oldest process event is discarded in order to make room for the newest processed event. A warning to increase the cache size will be logged.
 
@@ -48,11 +44,11 @@ The data collected for each event is:
 | **Type** | Can be either `fork`, or `exec`. |
 | **hit_count** | The aggregate count. The number of executions of the same process, during the same time frame, until the events are sent to the cloud. |
 
-## Network Connection events (event-based collector)
+## Network Activity events (event-based collector)
 
-Network Connection events are considered identical when the local port, remote port, transport protocol, local address, and remote address are identical.
+Network activity events are considered identical when the local port, remote port, transport protocol, local address, and remote address are identical.
 
-The default buffer for a network connection event is 256. For situations where the cache is full:
+The default buffer for a network activity event is 256. For situations where the cache is full:
 
 - **Azure RTOS devices**: No new network events will be cached until the next collection cycle starts.  
 
@@ -75,10 +71,9 @@ The data collected for each event is:
 | **Extended properties** | The Additional details of the connection. For example, `host name`. |
 | **DNS hit count** | Total hit count of DNS requests |
 
-
 ## Login collector (event-based collector)
 
-The Login collector, collects user sign-ins, sign-outs, and failed sign-in attempts.
+The Login collector collects user sign-ins, sign-outs, and failed sign-in attempts.
 
 The Login collector supports the following types of collection methods:
 
@@ -97,8 +92,7 @@ The following data is collected:
 | **remote_address** | The source of connection, either a remote IP address in IPv6 or IPv4 format, or `127.0.0.1/0.0.0.0` to indicate local connection. |
 | **Login_UsePAM** | Boolean: <br>- **True**: Only the PAM Login collector is used <br>- **False**: The UTMP Login collector is used, with SYSLOG if SYSLOG is enabled |
 
-
-## System information (trigger based collector)
+## System Information (trigger-based collector)
 
 The data collected for each event is:
 
@@ -116,17 +110,15 @@ The **nics** properties are composed of the following;
 
 | Parameter | Description|
 |--|--|
-|**type** | one of the following values: `UNKNOWN`, `ETH`, `WIFI`, `MOBILE`, or `SATELLITE`. |
+|**type** | One of the following values: `UNKNOWN`, `ETH`, `WIFI`, `MOBILE`, or `SATELLITE`. |
 | **vlans** | The virtual lan associated with the network interface. |
 | **vendor** | The vendor of the network controller. |
 | **info** | IPS, and MACs associated with the network controller. This Includes the following fields; <br> - **ipv4_address**: The IPv4 address. <br> - **ipv6_address**: The IPv6 address. <br> - **mac**: The MAC address.|
 
-## Baseline (trigger based)
+## Baseline (trigger-based collector)
 
 The baseline collector performs periodic CIS checks, and *failed*, *pass*, and *skip* check results are sent to the Defender for IoT cloud service. Defender for IoT aggregates the results and provides recommendations based on any failures.
 
-### Data collection
-
 The data collected for each event is:
 
 | Parameter | Description|
@@ -138,22 +130,32 @@ The data collected for each event is:
 | **Remediation** | The recommendation for remediation from CIS. |
 | **Severity** | The severity level. |
 
-## SBoM (trigger based)
+## SBoM (trigger-based collector)
 
 The SBoM (Software Bill of Materials) collector collects the packages installed on the device periodically.
 
 The data collected on each package includes:
 
 |Parameter  |Description  |
 |---------|---------|
-|**Name**     |   The package name      |
-|**Version**     |  The package version       |
-|**Vendor**     |    The package's vendor, which is the **Maintainer** field in deb packages     |
-
+|**Name**     |   The package name.      |
+|**Version**     |  The package version.       |
+|**Vendor**     |    The package's vendor, which is the **Maintainer** field in deb packages.     |
 
 > [!NOTE]
 > The SBoM collector currently only collects the first 500 packages ingested.
 
+## Event aggregation for Process and Network collectors
+
+How event aggregation works for the [Process events](#process-events-event-based-collector) and [Network Activity events](#network-activity-events-event-based-collector):
+
+Defender for IoT agents aggregate events during the send interval defined in the message frequency configuration for each collector, such as [**Process_MessageFrequency**](concept-micro-agent-configuration.md#process-collector-specific-settings) or [**NetworkActivity_MessageFrequency**](concept-micro-agent-configuration.md#network-activity-collector-specific-settings). Once the send interval period has passed, the agent sends the aggregated events to the Azure cloud for further analysis. The aggregated events are stored in memory until being sent to the Azure cloud.
+
+When the agent collects similar events to the ones that are already stored in memory, the agent will increase the hit count of this specific event to reduce the memory footprint of the agent. When the aggregation time window passes, the agent sends the hit count of each type of event that occurred. Event aggregation is the aggregation of the hit counts of similar events. For example, network activity with the same remote host and on the same port, is aggregated as one event, instead of as a separate event for each packet.
+
 ## Next steps
 
-Check your [Defender for IoT security alerts](concept-security-alerts.md).
+For more information, see:
+
+- [Micro agent configurations (Preview)](concept-micro-agent-configuration.md)
+- Check your [Defender for IoT security alerts](concept-security-alerts.md).

articles/defender-for-iot/device-builders/concept-micro-agent-configuration.md

@@ -9,9 +9,9 @@ ms.topic: conceptual
 
 This article describes the different types of configurations that the micro agent supports. Customers can configure the micro agent to fit the needs of their devices, and network environments.  
 
-The micro agent's behavior is configured by a set of module twin properties. You can configure the micro agent to best suit your needs. For example, you can exclude events automatically, minimize power consumption, and reduce network bandwidth.
+The micro agent's behavior is configured by a set of module twin properties. You can configure the micro agent to best suit your needs. For example, you can turn off certain events to minimize power consumption, and reduce other resource usage.
 
-After any change in configuration, the collector will immediately send all unsent event data. After the data is sent, the changes will be applied, and all the collectors will restart.
+After any change in configuration, the collector will immediately send all unsent event data. After the data is sent, the changes will be applied, and collectors will be restarted as needed.
 
 ## General configuration
 
@@ -25,7 +25,7 @@ Default values are as follows:
 | **Medium** | 120 (2 hours) |
 | **High** | 30 (.5 hours) |
 
-To reduce the number of messages sent to cloud, each priority should be set as a multiple of the one below it. For example, High: 60 minutes, Medium: 120 minutes, Low: 480 minutes.
+To reduce resource consumption on the device, each priority should be set as a multiple of the one below it. For example, High: 60 minutes, Medium: 120 minutes, Low: 480 minutes.
 
 The syntax for configuring the frequencies is as follows:
 
@@ -35,55 +35,75 @@ For example:
 
 `"CollectorsCore_PriorityIntervals"` : `"30,120,1440"`
 
-## Baseline collector-specific settings
+## Collector types and properties
+
+Configure the micro agent using the following collector-specific properties and settings:
+
+### Baseline collector-specific settings
 
 | Setting Name | Setting options | Description | Default |
 |--|--|--|--|
-| **Baseline_GroupsDisabled** | A list of Baseline group names, separated by a comma. <br><br>For example: `Time Synchronization, Network Parameters Host` | Defines the full list of Baseline group names that should be disabled. | Null |
-| **Baseline_ChecksDisabled** |A list of Baseline check IDs, separated by a comma. <br><br>For example: `3.3.5,2.2.1.1` | Defines the full list of Baseline check IDs that should be disabled. | Null |
+| **Baseline_Disabled** | `True`/`False` | Disables the Baseline collector. | `False` |
+| **Baseline_MessageFrequency** | `Low`/`Medium`/`High` | Defines the frequency in which to send Baseline events. | `Low` |
+| **Baseline_GroupsDisabled** | A list of Baseline group names, separated by a comma. <br><br>For example: `Time Synchronization, Network Parameters Host` | Defines the full list of Baseline group names that should be disabled. | `Null` |
+| **Baseline_ChecksDisabled** |A list of Baseline check IDs, separated by a comma. <br><br>For example: `3.3.5,2.2.1.1` | Defines the full list of Baseline check IDs that should be disabled. | `Null` |
 
+### System Information collector-specific settings
 
-## Event-based collector configurations
+| Setting Name | Setting options | Description | Default |
+|--|--|--|--|
+| **SystemInformation_Disabled** | `True`/`False` | Disables the System Information collector. | `False` |
+| **SystemInformation_MessageFrequency** | `Low`/`Medium`/`High` | Defines the frequency in which to send System Information events. | `Low` |
 
-These configurations include process, and network activity collectors.
+### SBoM collector-specific settings
 
 | Setting Name | Setting options | Description | Default |
 |--|--|--|--|
-| **Interval** | `High` <br>`Medium`<br>`Low` | Determines the sending frequency.  | `Medium` |
-| **Aggregation mode** | `True` <br>`False` | Determines whether to process event aggregation for an identical event.  | `True` |
-| **Cache size** | cycle FIFO | Defines the number of events collected in between the times that data is sent. | `256` |
-| **Disable collector** | `True` <br> `False` | Determines whether or not the collector is operational. | `False` |
+| **SBoM_Disabled** | `True`/`False` | Disables the SBoM collector. | `False` |
+| **SBoM_MessageFrequency** | `Low`/`Medium`/`High` | Defines the frequency in which to send SBoM events. | `Low` |
 
-## IoT Hub Module-specific settings
+### Heartbeat collector-specific settings
 
 | Setting Name | Setting options | Description | Default |
 |--|--|--|--|
-| **IothubModule_MessageTimeout** | Positive integer, including limits | Defines the number of minutes to retain messages in the outbound queue to the IoT Hub, after which point the messages are dropped. | `2880` (=2 days) |
-## Network activity collector-specific settings
+| **Heartbeat_Disabled** | `True`/`False` | Disables sending the Heartbeat event. | `False` |
+| **Heartbeat_MessageFrequency** | `Low`/`Medium`/`High` | Defines the frequency in which to send Heartbeat events. | `Low` |
+
+### Login collector-specific settings
 
 | Setting Name | Setting options | Description | Default |
 |--|--|--|--|
-| **Devices** | A list of the network devices separated by a comma. <br><br>For example `eth0,eth1` | Defines the list of network devices (interfaces) that the agent will use to monitor the traffic. <br><br>If a network device isn't listed, the Network Raw events won't be recorded for the missing device.| `eth0` |
-| | | | |
-
-## Process collector specific-settings
+| **Login_Disabled** | `True`/`False` | Disables the Login collector. | `False` |
+| **Login_MessageFrequency** | `Low`/`Medium`/`High` | Defines the frequency in which to send Login events. | `Medium` |
+| **Login_UsePAM** | `True`/`False` | Use a PAM module to gather login events. Without PAM, the agent uses a combination of reading UTMP and Syslog to gather login events. If the system doesn't have UTMP or Syslog enabled, using PAM is an option, but will require additional configuration to work properly. For more information, see [Configure Pluggable Authentication Modules (PAM) to audit sign-in events](configure-pam-to-audit-sign-in-events.md) | `False` |
 
+### IoT Hub module-specific settings
 
 | Setting Name | Setting options | Description | Default |
 |--|--|--|--|
-| **Process_Mode** | `1` = Auto <br>`2` = Netlink <br>`3`= Polling | Determines the process collector mode. In `Auto` mode, the agent first tries to enable the Netlink mode. <br><br>If that fails, it will automatically fall back / switch to the Polling mode.| `1` |
-|**Process_PollingInterval** |Integer |Defines the polling interval in microseconds. This value is used when the **Process_Mode** is in `Polling` mode. | `100000` (=0.1 second) |
-
-## Trigger-based collector configurations
+| **IothubModule_MessageTimeout** | Positive integer, including limits | Defines the number of minutes to retain messages in the outbound queue to the IoT Hub, after which point the messages are dropped. | `2880` (=2 days) |
 
-These configurations include system information, and baseline collectors.
+### Network Activity collector-specific settings
 
 | Setting Name | Setting options | Description | Default |
 |--|--|--|--|
-| **Interval** | `High` <br>`Medium`<br>`Low`  | The frequency in which data is sent. | `Low` |
-| **Disable collector** | `True` <br> `False` | Whether or not the collector is operational. | `False` |
+| **NetworkActivity_Disabled** | `True`/`False` | Disables the Network Activity collector. | `False` |
+| **NetworkActivity_MessageFrequency** | `Low`/`Medium`/`High` | Defines the frequency in which to send Network Activity events. | `Medium` |
+| **NetworkActivity_Devices** | A list of the network devices separated by a comma. <br><br>For example `eth0,eth1` | Defines the list of network devices (interfaces) that the agent will use to monitor the traffic. <br><br>If a network device isn't listed, the network raw events won't be recorded for the missing device.| `eth0` |
+| **NetworkActivity_CacheSize** | Positive integer | The number of Network Activity events (after aggregation) to keep in the cache between send intervals. Beyond that number, older events will be dropped (lost).| `256` |
 
+### Process collector-specific settings
+
+| Setting Name | Setting options | Description | Default |
+|--|--|--|--|
+| **Process_Disabled** | `True`/`False` | Disables the Process collector. | `False` |
+| **Process_MessageFrequency** | `Low`/`Medium`/`High` | Defines the frequency in which to send Process events. | `Medium` |
+|**Process_PollingInterval** | Positive Integer | Defines the polling interval in microseconds. This value is used when the **Process_Mode** is in `Polling` mode. | `100000` (=0.1 second) |
+| **Process_Mode** | `1` = Auto <br>`2` = Netlink <br>`3`= Polling | Determines the Process collector mode. In `Auto` mode, the agent first tries to enable the Netlink mode. <br><br>If that fails, it will automatically fall back / switch to the Polling mode.| `1` |
+| **Process_CacheSize** | Positive integer | The number of Process events (after aggregation) to keep in the cache between send intervals. Beyond that number, older events will be dropped (lost).| `256` |
 
 ## Next steps
 
-For more information, see [Configure a micro agent twin](how-to-configure-micro-agent-twin.md).
+For more information, see:
+- [Configure a micro agent twin](how-to-configure-micro-agent-twin.md).
+- [Micro agent event collection (Preview)](concept-event-aggregation.md)