You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/azure-monitor/essentials/data-collection-transformations.md
+234-3Lines changed: 234 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -15,9 +15,9 @@ The following table describes the different goals that transformations can be us
15
15
16
16
| Category | Details |
17
17
|:---|:---|
18
-
| Remove sensitive data | You may have a data source that sends information you don't want stored for privacy or compliancy reasons.<br><br>**Filter sensitive information.** Filter out entire rows or just particular columns that contain sensitive information.<br><br>**Obfuscate sensitive information**. For example, you might replace digits with a common character in an IP address or telephone number.|
18
+
| Remove sensitive data | You may have a data source that sends information you don't want stored for privacy or compliancy reasons.<br><br>**Filter sensitive information.** Filter out entire rows or just particular columns that contain sensitive information.<br><br>**Obfuscate sensitive information**. Replace information such as digits in an IP address or telephone number with a common character.<br><br>**Send to alternate table.** Send sensitive records to an alternate table with different RBAC configuration. |
19
19
| Enrich data with additional or calculated information | Use a transformation to add information to data that provides business context or simplifies querying the data later.<br><br>**Add a column with additional information.** For example, you might add a column identifying whether an IP address in another column is internal or external.<br><br>**Add business specific information.** For example, you might add a column indicating a company division based on location information in other columns. |
20
-
| Reduce data costs | Since you're charged ingestion cost for any data sent to a Log Analytics workspace, you want to filter out any data that you don't require to reduce your costs.<br><br>**Remove entire rows.** For example, you might have a diagnostic setting to collect resource logs from a particular resource but not require all of the log entries that it generates. Create a transformation that filters out records that match a certain criteria.<br><br>**Remove a column from each row.** For example, your data may include columns with data that's redundant or has minimal value. Create a transformation that filters out columns that aren't required.<br><br>**Parse important data from a column.** You may have a table with valuable data buried in a particular column. Use a transformation to parse the valuable data into a new column and remove the original. |
20
+
| Reduce data costs | Since you're charged ingestion cost for any data sent to a Log Analytics workspace, you want to filter out any data that you don't require to reduce your costs.<br><br>**Remove entire rows.** For example, you might have a diagnostic setting to collect resource logs from a particular resource but not require all of the log entries that it generates. Create a transformation that filters out records that match a certain criteria.<br><br>**Remove a column from each row.** For example, your data may include columns with data that's redundant or has minimal value. Create a transformation that filters out columns that aren't required.<br><br>**Parse important data from a column.** You may have a table with valuable data buried in a particular column. Use a transformation to parse the valuable data into a new column and remove the original.<br><br>**Send certain rows to basic logs.** Send rows in your data that require on basic query capabilities to basic logs tables for a lower ingestion cost. |
21
21
22
22
23
23
@@ -50,20 +50,36 @@ A common use of the workspace transformation DCR is collection of [resource logs
50
50
51
51
:::image type="content" source="media/data-collection-transformations/transformation-diagnostic-settings.png" lightbox="media/data-collection-transformations/transformation-diagnostic-settings.png" alt-text="Diagram of workspace transformation for resource logs configured with diagnostic settings." border="false":::
52
52
53
+
## Multiple destinations
54
+
55
+
Transformations allow you to send data to multiple destinations in a Log Analytics workspace using a single DCR. You provide a KQL query for each destination, and the results of each query are sent to their corresponding location. You can send different sets of data to different tables, or use multiple queries to send different sets of data to the same table.
56
+
57
+
For example, you may send event data into Azure Monitor using the Logs ingestion API. Most of the events should be sent an analytics table where it could be queried regularly, while audit events should be sent to a custom table configured for [basic logs](../logs/basic-logs-configure.md) to reduce your cost.
58
+
59
+
To use multiple destinations, you must currently either manually create a new DCR or [edit an existing one](data-collection-rule-edit.md). See the [Samples](#samples) section for examples of DCRs using multiple destinations.
60
+
61
+
> [!IMPORTANT]
62
+
> Currently, the tables in the DCR must be in the same Log Analytics workspace. To send to multiple workspaces from a single data source, use multiple DCRs and configure your application to send the data to each.
63
+
64
+
:::image type="content" source="media/data-collection-transformations/transformation-multiple-destinations.png" lightbox="media/data-collection-transformations/transformation-multiple-destinations.png" alt-text="Diagram of transformation sending data to multiple tables." border="false":::
65
+
66
+
67
+
53
68
## Creating a transformation
54
69
There are multiple methods to create transformations depending on the data collection method. The following table lists guidance for different methods for creating transformations.
55
70
56
71
| Type | Reference |
57
72
|:---|:---|
58
73
| Logs ingestion API with transformation |[Send data to Azure Monitor Logs using REST API (Azure portal)](../logs/tutorial-logs-ingestion-portal.md)<br>[Send data to Azure Monitor Logs using REST API (Resource Manager templates)](../logs/tutorial-logs-ingestion-api.md)|
59
-
| Transformation in workspace DCR | [Add workspace transformation to Azure Monitor Logs using the Azure portal](../logs/tutorial-workspace-transformations-portal.md)<br>[Add workspace transformation to Azure Monitor Logs using resource manager templates](../logs/tutorial-workspace-transformations-api.md)
74
+
| Transformation in workspace DCR | [Add workspace transformation to Azure Monitor Logs using the Azure portal](../logs/tutorial-workspace-transformations-portal.md)<br>[Add workspace transformation to Azure Monitor Logs using Resource Manager templates](../logs/tutorial-workspace-transformations-api.md)
60
75
61
76
## Cost for transformations
62
77
There is no direct cost for transformations, but you may incur charges for the following:
63
78
64
79
- If your transformation increases the size of the incoming data, adding a calculated column for example, then you're charged at the normal rate for ingestion of that additional data.
65
80
- If your transformation reduces the incoming data by more than 50%, then you're charged for ingestion of the amount of filtered data above 50%.
66
81
82
+
67
83
The formula to determine the filter ingestion charge from transformations is `[GB filtered out by transformations] - ( [Total GB ingested] / 2 )`. For example, suppose that you ingest 100 GB on a particular day, and transformations remove 70 GB. You would be charged for 70 GB - (100 GB / 2) or 20 GB. To avoid this charge, you should use other methods to filter incoming data before the transformation is applied.
68
84
69
85
See [Azure Monitor pricing](https://azure.microsoft.com/pricing/details/monitor) for current charges for ingestion and retention of log data in Azure Monitor.
@@ -72,6 +88,221 @@ See [Azure Monitor pricing](https://azure.microsoft.com/pricing/details/monitor)
72
88
> If Azure Sentinel is enabled for the Log Analytics workspace, then there is no filtering ingestion charge regardless of how much data the transformation filters.
73
89
74
90
91
+
## Samples
92
+
Following are Resource Manager templates of sample DCRs with different patterns. You can use these templates as a starting point to creating DCRs with transformations for your own scenarios.
93
+
94
+
### Single destination
95
+
96
+
The following example is a DCR for Azure Monitor agent that sends data to the `Syslog` table. In this example, the transformation filters the data for records with *error* in the message.
"transformKql": "source | where message contains 'error'",
152
+
"destinations": [
153
+
"centralWorkspace"
154
+
]
155
+
}
156
+
]
157
+
}
158
+
}
159
+
]
160
+
}
161
+
```
162
+
163
+
### Multiple Azure tables
164
+
165
+
The following example is a DCR for data from Logs Ingestion API that sends data to both the `Syslog` and `SecurityEvent` table. This requires a separate `dataFlow` for each with a different `transformKql` and `OutputStream` for each. In this example, all incoming data is sent to the `Syslog` table while malicious data is also sent to the `SecurityEvent` table. If you didn't want to replicate the malicious data in both tables, you could add a `where` statement to first query to remove those records.
"transformKql": "source | where (AdditionalContext contains 'malicious traffic!' | project TimeGenerated = Time, Computer, Subject = AdditionalContext",
224
+
"outputStream": "Microsoft-SecurityEvent"
225
+
}
226
+
]
227
+
}
228
+
}
229
+
]
230
+
}
231
+
```
232
+
233
+
### Combination of Azure and custom tables
234
+
235
+
The following example is a DCR for data from Logs Ingestion API that sends data to both the `Syslog` table and a custom table with the data in a different format. This requires a separate `dataFlow` for each with a different `transformKql` and `OutputStream` for each.
-[Create a data collection rule](../agents/data-collection-rule-azure-monitor-agent.md) and an association to it from a virtual machine using the Azure Monitor agent.
0 commit comments