You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/sentinel/data-transformation.md
+8-9Lines changed: 8 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,7 +4,7 @@ description: Learn about how Azure Monitor's custom log ingestion and data trans
4
4
author: yelevin
5
5
ms.author: yelevin
6
6
ms.topic: conceptual
7
-
ms.date: 02/27/2022
7
+
ms.date: 09/25/2024
8
8
---
9
9
10
10
# Custom data ingestion and transformation in Microsoft Sentinel
@@ -15,7 +15,7 @@ Log Analytics' custom data ingestion process gives you a high level of control o
15
15
16
16
Microsoft Sentinel gives you two tools to control this process:
17
17
18
-
- The [**Logs ingestion API**](/azure/azure-monitor/logs/logs-ingestion-api-overview) allows you to send custom-format logs from any data source to your Log Analytics workspace, and store those logs either in certain specific standard tables, or in custom-formatted tables that you create. You have full control over the creation of these custom tables, down to specifying the column names and types. You create [**Data collection rules (DCRs)**](/azure/azure-monitor/essentials/data-collection-rule-overview) to define, configure, and apply transformations to these data flows.
18
+
- The [**Logs ingestion API**](/azure/azure-monitor/logs/logs-ingestion-api-overview) allows you to send custom-format logs from any data source to your Log Analytics workspace, and store those logs either in certain specific standard tables, or in custom-formatted tables that you create. You have full control over the creation of these custom tables, down to specifying the column names and types. You create [**DCRs**](/azure/azure-monitor/essentials/data-collection-rule-overview) to define, configure, and apply transformations to these data flows.
19
19
20
20
-[**Data collection transformation**](/azure/azure-monitor/essentials/data-collection-transformations) uses DCRs to apply basic KQL queries to incoming standard logs (and certain types of custom logs) before they're stored in your workspace. These transformations can filter out irrelevant data, enrich existing data with analytics or external data, or mask sensitive or personal information.
21
21
@@ -36,9 +36,9 @@ Ingestion-time data transformation supports [multiple-workspace scenarios](exten
36
36
37
37
### Normalization
38
38
39
-
Ingest-time transformation also allows you to normalize logs when ingested into built-in or customer ASIM normalized tables. Using ingest-time normalization improves normalized queries performance.
39
+
Ingest-time transformation also allows you to normalize logs when they're ingested into built-in or customer-normalized tables with [Advanced Security Information Model (ASIM)](normalization.md). Using ingest-time normalization improves the performance of normalized queries.
40
40
41
-
For more information on ingest-time normalization using transformations, refer to[Ingest-time normalization](normalization-ingest-time.md).
41
+
For more information, see[Ingest-time normalization](normalization-ingest-time.md).
42
42
43
43
### Enrichment and tagging
44
44
@@ -58,23 +58,23 @@ Microsoft Sentinel collects data into the Log Analytics workspace from multiple
58
58
- Data from built-in data connectors is processed in Log Analytics using some combination of hardcoded workflows and ingestion-time transformations in the workspace DCR. This data can be stored in standard tables or in a specific set of custom tables.
59
59
- Data ingested directly into the Logs ingestion API endpoint is processed by a standard DCR that may include an ingestion-time transformation. This data can then be stored in either standard or custom tables of any kind.
60
60
61
-
:::image type="content" source="media/data-transformation/data-transformation-architecture.png" alt-text="Diagram of the Microsoft Sentinel data transformation architecture.":::
61
+
:::image type="content" source="media/data-transformation/data-transformation-architecture.png" alt-text="Diagram of the Microsoft Sentinel data transformation architecture." border="false":::
62
62
63
63
## DCR support in Microsoft Sentinel
64
64
65
65
In Log Analytics, data collection rules (DCRs) determine the data flow for different input streams. A data flow includes: the data stream to be transformed (standard or custom), the destination workspace, the KQL transformation, and the output table. For standard input streams, the output table is the same as the input stream.
66
66
67
67
Support for DCRs in Microsoft Sentinel includes:
68
68
69
-
-*Standard DCRs*, currently supported only for AMA-based connectors and workflows using the new [Logs ingestion API](/azure/azure-monitor/logs/logs-ingestion-api-overview).
69
+
-*Standard DCRs*, currently supported only for AMA-based connectors and workflows using the [Logs ingestion API](/azure/azure-monitor/logs/logs-ingestion-api-overview). <!--"currently". is this still true?-->
70
70
71
71
Each connector or log source workflow can have its own dedicated *standard DCR*, though multiple connectors or sources can share a common *standard DCR* as well.
72
72
73
73
-*Workspace transformation DCRs*, for workflows that don't currently support standard DCRs.
74
74
75
75
A single *workspace transformation DCR* serves all the supported workflows in a workspace that aren't served by standard DCRs. A workspace can have only one *workspace transformation DCR*, but that DCR contains separate transformations for each input stream. Also, *workspace transformation DCR*s are supported only for a [specific set of tables](/azure/azure-monitor/logs/tables-feature-support).
76
76
77
-
Microsoft Sentinel's support for ingestion-time transformation depends on the type of data connector you're using. For more in-depth information on custom logs, ingestion-time transformation, and data collection rules, see the articles linked in the [Next steps](#next-steps) section at the end of this article.
77
+
Microsoft Sentinel's support for ingestion-time transformation depends on the type of data connector you're using. For more in-depth information on custom logs, ingestion-time transformation, and data collection rules, see the articles linked in the [Related content](#related-content) section at the end of this article.
78
78
79
79
### DCR support for Microsoft Sentinel data connectors
80
80
@@ -84,7 +84,6 @@ The following table describes DCR support for Microsoft Sentinel data connector
84
84
| ------------------- | ----------- |
85
85
|**Direct ingestion via [Logs ingestion API](/azure/azure-monitor/logs/logs-ingestion-api-overview)**| Standard DCRs |
86
86
|[**AMA standard logs**](connect-services-windows-based.md), such as: <li>[Windows Security Events via AMA](./data-connectors/windows-security-events-via-ama.md)<li>[Windows Forwarded Events](./data-connectors/windows-forwarded-events.md)<li>[CEF data](connect-cef-ama.md)<li>[Syslog data](connect-cef-syslog.md)| Standard DCRs |
87
-
|[**MMA standard logs**](connect-services-windows-based.md), such as <li>[Syslog data](connect-syslog.md)<li>[CommonSecurityLog](connect-azure-windows-microsoft-services.md)| Workspace transformation DCRs |
88
87
|[**Diagnostic settings-based connections**](connect-services-diagnostic-setting-based.md)| Workspace transformation DCRs, based on the [supported output tables](/azure/azure-monitor/logs/tables-feature-support) for specific data connectors |
89
88
|**Built-in, service-to-service data connectors**, such as:<li>[Microsoft Office 365](connect-services-api-based.md)<li>[Microsoft Entra ID](connect-azure-active-directory.md)<li>[Amazon S3](connect-aws.md)| Workspace transformation DCRs, based on the [supported output tables](/azure/azure-monitor/logs/tables-feature-support) for specific data connectors |
90
89
|**Built-in, API-based data connector**, such as: <li>[Codeless data connectors](create-codeless-connector.md)| Standard DCRs |
@@ -129,7 +128,7 @@ Ingestion-time data transformation currently has the following known issues for
129
128
130
129
- You can only send logs from one specific data source to one workspace. To send data from a single data source to multiple workspaces (destinations) with a standard DCR, please create one DCR per workspace.
131
130
132
-
## Next steps
131
+
## Related content
133
132
134
133
[Get started configuring ingestion-time data transformation in Microsoft Sentinel](configure-data-transformation.md).
0 commit comments