Skip to content

Commit 1264c83

Browse files
committed
starting freshness
1 parent 55e5496 commit 1264c83

File tree

2 files changed

+8
-9
lines changed

2 files changed

+8
-9
lines changed

articles/sentinel/data-transformation.md

Lines changed: 8 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ description: Learn about how Azure Monitor's custom log ingestion and data trans
44
author: yelevin
55
ms.author: yelevin
66
ms.topic: conceptual
7-
ms.date: 02/27/2022
7+
ms.date: 09/25/2024
88
---
99

1010
# Custom data ingestion and transformation in Microsoft Sentinel
@@ -15,7 +15,7 @@ Log Analytics' custom data ingestion process gives you a high level of control o
1515

1616
Microsoft Sentinel gives you two tools to control this process:
1717

18-
- The [**Logs ingestion API**](/azure/azure-monitor/logs/logs-ingestion-api-overview) allows you to send custom-format logs from any data source to your Log Analytics workspace, and store those logs either in certain specific standard tables, or in custom-formatted tables that you create. You have full control over the creation of these custom tables, down to specifying the column names and types. You create [**Data collection rules (DCRs)**](/azure/azure-monitor/essentials/data-collection-rule-overview) to define, configure, and apply transformations to these data flows.
18+
- The [**Logs ingestion API**](/azure/azure-monitor/logs/logs-ingestion-api-overview) allows you to send custom-format logs from any data source to your Log Analytics workspace, and store those logs either in certain specific standard tables, or in custom-formatted tables that you create. You have full control over the creation of these custom tables, down to specifying the column names and types. You create [**DCRs**](/azure/azure-monitor/essentials/data-collection-rule-overview) to define, configure, and apply transformations to these data flows.
1919

2020
- [**Data collection transformation**](/azure/azure-monitor/essentials/data-collection-transformations) uses DCRs to apply basic KQL queries to incoming standard logs (and certain types of custom logs) before they're stored in your workspace. These transformations can filter out irrelevant data, enrich existing data with analytics or external data, or mask sensitive or personal information.
2121

@@ -36,9 +36,9 @@ Ingestion-time data transformation supports [multiple-workspace scenarios](exten
3636

3737
### Normalization
3838

39-
Ingest-time transformation also allows you to normalize logs when ingested into built-in or customer ASIM normalized tables. Using ingest-time normalization improves normalized queries performance.
39+
Ingest-time transformation also allows you to normalize logs when they're ingested into built-in or customer-normalized tables with [Advanced Security Information Model (ASIM)](normalization.md). Using ingest-time normalization improves the performance of normalized queries.
4040

41-
For more information on ingest-time normalization using transformations, refer to [Ingest-time normalization](normalization-ingest-time.md).
41+
For more information, see [Ingest-time normalization](normalization-ingest-time.md).
4242

4343
### Enrichment and tagging
4444

@@ -58,23 +58,23 @@ Microsoft Sentinel collects data into the Log Analytics workspace from multiple
5858
- Data from built-in data connectors is processed in Log Analytics using some combination of hardcoded workflows and ingestion-time transformations in the workspace DCR. This data can be stored in standard tables or in a specific set of custom tables.
5959
- Data ingested directly into the Logs ingestion API endpoint is processed by a standard DCR that may include an ingestion-time transformation. This data can then be stored in either standard or custom tables of any kind.
6060

61-
:::image type="content" source="media/data-transformation/data-transformation-architecture.png" alt-text="Diagram of the Microsoft Sentinel data transformation architecture.":::
61+
:::image type="content" source="media/data-transformation/data-transformation-architecture.png" alt-text="Diagram of the Microsoft Sentinel data transformation architecture." border="false":::
6262

6363
## DCR support in Microsoft Sentinel
6464

6565
In Log Analytics, data collection rules (DCRs) determine the data flow for different input streams. A data flow includes: the data stream to be transformed (standard or custom), the destination workspace, the KQL transformation, and the output table. For standard input streams, the output table is the same as the input stream.
6666

6767
Support for DCRs in Microsoft Sentinel includes:
6868

69-
- *Standard DCRs*, currently supported only for AMA-based connectors and workflows using the new [Logs ingestion API](/azure/azure-monitor/logs/logs-ingestion-api-overview).
69+
- *Standard DCRs*, currently supported only for AMA-based connectors and workflows using the [Logs ingestion API](/azure/azure-monitor/logs/logs-ingestion-api-overview). <!--"currently". is this still true?-->
7070

7171
Each connector or log source workflow can have its own dedicated *standard DCR*, though multiple connectors or sources can share a common *standard DCR* as well.
7272

7373
- *Workspace transformation DCRs*, for workflows that don't currently support standard DCRs.
7474

7575
A single *workspace transformation DCR* serves all the supported workflows in a workspace that aren't served by standard DCRs. A workspace can have only one *workspace transformation DCR*, but that DCR contains separate transformations for each input stream. Also, *workspace transformation DCR*s are supported only for a [specific set of tables](/azure/azure-monitor/logs/tables-feature-support).
7676

77-
Microsoft Sentinel's support for ingestion-time transformation depends on the type of data connector you're using. For more in-depth information on custom logs, ingestion-time transformation, and data collection rules, see the articles linked in the [Next steps](#next-steps) section at the end of this article.
77+
Microsoft Sentinel's support for ingestion-time transformation depends on the type of data connector you're using. For more in-depth information on custom logs, ingestion-time transformation, and data collection rules, see the articles linked in the [Related content](#related-content) section at the end of this article.
7878

7979
### DCR support for Microsoft Sentinel data connectors
8080

@@ -84,7 +84,6 @@ The following table describes DCR support for Microsoft Sentinel data connector
8484
| ------------------- | ----------- |
8585
| **Direct ingestion via [Logs ingestion API](/azure/azure-monitor/logs/logs-ingestion-api-overview)** | Standard DCRs |
8686
| [**AMA standard logs**](connect-services-windows-based.md), such as: <li>[Windows Security Events via AMA](./data-connectors/windows-security-events-via-ama.md)<li>[Windows Forwarded Events](./data-connectors/windows-forwarded-events.md)<li>[CEF data](connect-cef-ama.md)<li>[Syslog data](connect-cef-syslog.md) | Standard DCRs |
87-
| [**MMA standard logs**](connect-services-windows-based.md), such as <li>[Syslog data](connect-syslog.md)<li>[CommonSecurityLog](connect-azure-windows-microsoft-services.md) | Workspace transformation DCRs |
8887
| [**Diagnostic settings-based connections**](connect-services-diagnostic-setting-based.md) | Workspace transformation DCRs, based on the [supported output tables](/azure/azure-monitor/logs/tables-feature-support) for specific data connectors |
8988
| **Built-in, service-to-service data connectors**, such as:<li>[Microsoft Office 365](connect-services-api-based.md)<li>[Microsoft Entra ID](connect-azure-active-directory.md)<li>[Amazon S3](connect-aws.md) | Workspace transformation DCRs, based on the [supported output tables](/azure/azure-monitor/logs/tables-feature-support) for specific data connectors |
9089
| **Built-in, API-based data connector**, such as: <li>[Codeless data connectors](create-codeless-connector.md) | Standard DCRs |
@@ -129,7 +128,7 @@ Ingestion-time data transformation currently has the following known issues for
129128

130129
- You can only send logs from one specific data source to one workspace. To send data from a single data source to multiple workspaces (destinations) with a standard DCR, please create one DCR per workspace.
131130

132-
## Next steps
131+
## Related content
133132

134133
[Get started configuring ingestion-time data transformation in Microsoft Sentinel](configure-data-transformation.md).
135134

1.18 KB
Loading

0 commit comments

Comments
 (0)