You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/sentinel/data-transformation.md
+14-16Lines changed: 14 additions & 16 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -14,22 +14,22 @@ ms.date: 09/25/2024
14
14
15
15
[Azure Monitor Logs](/azure/azure-monitor/logs/data-platform-logs) serves as the data platform for Microsoft Sentinel. All logs ingested into Microsoft Sentinel are stored in a [Log Analytics workspace](/azure/azure-monitor/logs/log-analytics-workspace-overview), and [log queries](/azure/azure-monitor/logs/log-query-overview) written in [Kusto Query Language (KQL)](/kusto/query/kusto-sentinel-overview?view=microsoft-sentinel&toc=%2Fazure%2Fsentinel%2FTOC.json&bc=%2Fazure%2Fsentinel%2Fbreadcrumb%2Ftoc.json) are used to detect threats and monitor your network activity.
16
16
17
-
Log Analytics gives you a high level of control over the data that gets ingested to your workspace with custom data ingestion and [*data collection rules (DCRs)*](/azure/azure-monitor/essentials/data-collection-rule-overview). DCRs allow you to both collect and manipulate your data before it's stored in your workspace. DCRs format and send data to both standard Log Analytics tables and customizable tables for data sources that produce unique log formats.
17
+
Log Analytics gives you a high level of control over the data that gets ingested to your workspace with custom data ingestion and [data collection rules (DCRs)](/azure/azure-monitor/essentials/data-collection-rule-overview). DCRs allow you to both collect and manipulate your data before it's stored in your workspace. DCRs both format and send data to both standard Log Analytics tables and customizable tables for data sources that produce unique log formats.
18
18
19
19
## Azure Monitor tools for custom data ingestion in Microsoft Sentinel
20
20
21
21
Microsoft Sentinel uses the following Azure Montitor tools to control custom data ingestion:
22
22
23
23
-[**Transformations**](/azure/azure-monitor/essentials/data-collection-transformations) are defined in DCRs and apply KQL queries to incoming data before it's stored in your workspace. These transformations can filter out irrelevant data, enrich existing data with analytics or external data, or mask sensitive or personal information.
24
24
25
-
-[**Logs ingestion API**](/azure/azure-monitor/logs/logs-ingestion-api-overview) allows you to send custom-format logs from any data source to your Log Analytics workspace, and store those logs either in certain standard tables, or in custom-formatted tables that you create. You have full control over the creation of these custom tables, down to specifying the column names and types. The API uses [**DCRs**](/azure/azure-monitor/essentials/data-collection-rule-overview) to define, configure, and apply transformations to these data flows.
25
+
-The[**Logs ingestion API**](/azure/azure-monitor/logs/logs-ingestion-api-overview) allows you to send custom-format logs from any data source to your Log Analytics workspace, and store those logs either in certain standard tables, or in custom-formatted tables that you create. You have full control over the creation of these custom tables, down to specifying the column names and types. The API uses [**DCRs**](/azure/azure-monitor/essentials/data-collection-rule-overview) to define, configure, and apply transformations to these data flows.
26
26
27
27
> [!NOTE]
28
-
> Log Analytics workspaces enabled for Microsoft Sentinel aren't subject to the [filtering ingestion charge](/azure/azure-monitor/essentials/data-collection-transformations#cost-for-transformations), regardless of how much data the transformation filters. However, transformations in Microsoft Sentinel have the same limitations as Azure Monitor. For more information, see [Limitations and considerations](/azure/azure-monitor/essentials/data-collection-transformations-create#limitations-and-considerations).
28
+
> Log Analytics workspaces enabled for Microsoft Sentinel aren't subject to Azure Monitor's [filtering ingestion charge](/azure/azure-monitor/essentials/data-collection-transformations#cost-for-transformations), regardless of how much data the transformation filters. However, transformations in Microsoft Sentinel otherwise have the same limitations as Azure Monitor. For more information, see [Limitations and considerations](/azure/azure-monitor/essentials/data-collection-transformations-create#limitations-and-considerations).
29
29
30
30
31
31
### DCR support in Microsoft Sentinel
32
-
Ingestion-time transformations are defined in [data collection rules (DCRs)](/azure/azure-monitor/essentials/data-collection-rule-overview), which control the data flow in Azure Monitor. DCRs are currently used by AMA-based Sentinel connectors and workflows using the [Logs ingestion API](/azure/azure-monitor/logs/logs-ingestion-api-overview). Each DCR contains the configuration for a particular data collection scenario, and multiple connectors or sources can share different DCRs.
32
+
Ingestion-time transformations are defined in [data collection rules (DCRs)](/azure/azure-monitor/essentials/data-collection-rule-overview), which control the data flow in Azure Monitor. DCRs are used by AMA-based Sentinel connectors and workflows using the [Logs ingestion API](/azure/azure-monitor/logs/logs-ingestion-api-overview). Each DCR contains the configuration for a particular data collection scenario, and multiple connectors or sources can share a single DCR.
33
33
34
34
[Workspace transformation DCRs](/azure/azure-monitor/essentials/data-collection-transformations#workspace-transformation-dcr) support workflows that don't otherwise use DCRs. Workspace transformation DCRs contain transformations for any [supported tables](/azure/azure-monitor/logs/tables-feature-support) and are applied to all traffic sent to that table.
35
35
@@ -41,41 +41,39 @@ For more information, see:
41
41
42
42
## Use cases and sample scenarios
43
43
44
-
[Sample transformations in Azure Monitor](/azure/azure-monitor/essentials/data-collection-transformations-samples) provides description and sample queries for common scenarios using ingestion-time transformations in Azure Monitor. Scenarios that are particularly useful for Microsoft Sentinel include:
44
+
The article [Sample transformations in Azure Monitor](/azure/azure-monitor/essentials/data-collection-transformations-samples) provides description and sample queries for common scenarios using ingestion-time transformations in Azure Monitor. Scenarios that are particularly useful for Microsoft Sentinel include:
45
45
46
-
-[Reduce data costs.](/azure/azure-monitor/essentials/data-collection-transformations-samples#reduce-data-costs) Filter data collection by either rows or columns to reduce ingestion and storage costs.
46
+
-[Reduce data costs](/azure/azure-monitor/essentials/data-collection-transformations-samples#reduce-data-costs). Filter data collection by either rows or columns to reduce ingestion and storage costs.
47
47
48
-
-[Normalization](/azure/azure-monitor/essentials/data-collection-transformations-samples#normalize-data). Normalize logs with the [Advanced Security Information Model (ASIM)](normalization.md) to improve the performance of normalized queries. For more information, see [Ingest-time normalization](normalization-ingest-time.md).
48
+
-[Normalize data](/azure/azure-monitor/essentials/data-collection-transformations-samples#normalize-data). Normalize logs with the [Advanced Security Information Model (ASIM)](normalization.md) to improve the performance of normalized queries. For more information, see [Ingest-time normalization](normalization-ingest-time.md).
49
49
50
50
-[Enrich data](/azure/azure-monitor/essentials/data-collection-transformations-samples#enrich-data). Ingestion-time transformations let you improve analytics by enriching your data with extra columns added to the configured KQL transformation. Extra columns might include parsed or calculated data from existing columns.
51
51
52
52
-[Remove sensitive data](/azure/azure-monitor/essentials/data-collection-transformations-samples#remove-sensitive-data). Ingestion-time transformations can be used to mask or remove personal information such as masking all but the last digits of a social security number or credit card number.
53
53
54
54
## Data ingestion flow in Microsoft Sentinel
55
55
56
-
The following image shows where ingestion-time data transformation enters the data ingestion flow in Microsoft Sentinel. This data can be supported standard tables or in a specific set of custom tables.
57
-
58
-
> [!NOTE]
59
-
> This image shows the cloud pipeline, which represents the data collection component of Azure Monitor. It's automatically available in your Azure subscription doesn't appear in the Azure portal. You can learn more about it along with other data collection scenarios in [Data collection rules (DCRs) in Azure Monitor](/azure/azure-monitor/essentials/data-collection-rule-overview#azure-monitor-pipeline).
56
+
The following image shows where ingestion-time data transformation enters the data ingestion flow in Microsoft Sentinel. This data can be supported standard tables or in a [specific set of custom tables](/azure/azure-monitor/logs/tables-feature-support).
60
57
61
58
:::image type="content" source="media/data-transformation/data-transformation-architecture.png" alt-text="Diagram of the Microsoft Sentinel data transformation architecture." lightbox="media/data-transformation/data-transformation-architecture.png" border="false":::
62
59
60
+
This image shows the cloud pipeline, which represents the data collection component of Azure Monitor. You can learn more about it along with other data collection scenarios in [Data collection rules (DCRs) in Azure Monitor](/azure/azure-monitor/essentials/data-collection-rule-overview#azure-monitor-pipeline).
61
+
63
62
Microsoft Sentinel collects data in the Log Analytics workspace from multiple sources.
64
63
65
64
-**Data collected from the Logs ingestion API endpoint or Azure Monitor agent (AMA)** is processed by a specific DCR that may include an ingestion-time transformation.
66
-
-**Data from built-in data connectors** is processed in Log Analytics using some combination of hardcoded workflows and ingestion-time transformations in the workspace DCR.
65
+
-**Data from built-in data connectors** is processed in Log Analytics using a combination of hardcoded workflows and ingestion-time transformations in the workspace DCR.
67
66
68
67
The following table describes DCR support for Microsoft Sentinel data connector types:
69
68
70
69
| Data connector type | DCR support |
71
70
| ------------------- | ----------- |
71
+
|[**Azure Monitor agent (AMA) logs**](connect-services-windows-based.md), such as: <li>[Windows Security Events via AMA](./data-connectors/windows-security-events-via-ama.md)<li>[Windows Forwarded Events](./data-connectors/windows-forwarded-events.md)<li>[CEF data](connect-cef-ama.md)<li>[Syslog data](connect-cef-syslog.md)| One or more DCRs associated with the agent |
72
72
|**Direct ingestion via [Logs ingestion API](/azure/azure-monitor/logs/logs-ingestion-api-overview)**| DCR specified in API call |
73
-
|[**Azure Monitor agent (AMA) logs**](connect-services-windows-based.md), such as: <li>[Windows Security Events via AMA](./data-connectors/windows-security-events-via-ama.md)<li>[Windows Forwarded Events](./data-connectors/windows-forwarded-events.md)<li>[CEF data](connect-cef-ama.md)<li>[Syslog data](connect-cef-syslog.md)| One or more DCRs associated with agent |
74
-
|[**Diagnostic settings-based connections**](connect-services-diagnostic-setting-based.md)| Workspace transformation DCR with [supported output tables](/azure/azure-monitor/logs/tables-feature-support)|
75
-
|**Built-in, service-to-service data connectors**, such as:<li>[Microsoft Office 365](connect-services-api-based.md)<li>[Microsoft Entra ID](connect-azure-active-directory.md)<li>[Amazon S3](connect-aws.md)| Workspace transformation DCR with [supported output tables](/azure/azure-monitor/logs/tables-feature-support)|
76
73
|**Built-in, API-based data connector**, such as: <li>[Codeless data connectors](create-codeless-connector.md)| DCR created for connector |
74
+
|[**Diagnostic settings-based connections**](connect-services-diagnostic-setting-based.md)| Workspace transformation DCR with [supported output tables](/azure/azure-monitor/logs/tables-feature-support)|
77
75
|**Built-in, API-based data connectors**, such as: <li>[Legacy codeless data connectors](create-codeless-connector-legacy.md)<li>[Azure Functions-based data connectors](connect-azure-functions-template.md)| Not currently supported |
78
-
76
+
|**Built-in, service-to-service data connectors**, such as:<li>[Microsoft Office 365](connect-services-api-based.md)<li>[Microsoft Entra ID](connect-azure-active-directory.md)<li>[Amazon S3](connect-aws.md)| Workspace transformation DCR for [tables that support transformations](/azure/azure-monitor/logs/tables-feature-support)|
0 commit comments