Skip to content

Commit 9714a40

Browse files
bwrenbwren
committed
final updates
1 parent 94f7279 commit 9714a40

File tree

1 file changed

+179
-157
lines changed

1 file changed

+179
-157
lines changed

‎articles/azure-monitor/essentials/data-collection-transformations.md

Lines changed: 179 additions & 157 deletions
Original file line numberDiff line numberDiff line change
@@ -15,9 +15,9 @@ The following table describes the different goals that transformations can be us
1515

1616
| Category | Details |
1717
|:---|:---|
18-
| Remove sensitive data | You may have a data source that sends information you don't want stored for privacy or compliancy reasons.<br><br>**Filter sensitive information.** Filter out entire rows or just particular columns that contain sensitive information.<br><br>**Obfuscate sensitive information**. For example, you might replace digits with a common character in an IP address or telephone number. |
18+
| Remove sensitive data | You may have a data source that sends information you don't want stored for privacy or compliancy reasons.<br><br>**Filter sensitive information.** Filter out entire rows or just particular columns that contain sensitive information.<br><br>**Obfuscate sensitive information**. Replace information such as digits in an IP address or telephone number with a common character.<br><br>**Send to alternate table.** Send sensitive records to an alternate table with different RBAC configuration. |
1919
| Enrich data with additional or calculated information | Use a transformation to add information to data that provides business context or simplifies querying the data later.<br><br>**Add a column with additional information.** For example, you might add a column identifying whether an IP address in another column is internal or external.<br><br>**Add business specific information.** For example, you might add a column indicating a company division based on location information in other columns. |
20-
| Reduce data costs | Since you're charged ingestion cost for any data sent to a Log Analytics workspace, you want to filter out any data that you don't require to reduce your costs.<br><br>**Remove entire rows.** For example, you might have a diagnostic setting to collect resource logs from a particular resource but not require all of the log entries that it generates. Create a transformation that filters out records that match a certain criteria.<br><br>**Remove a column from each row.** For example, your data may include columns with data that's redundant or has minimal value. Create a transformation that filters out columns that aren't required.<br><br>**Parse important data from a column.** You may have a table with valuable data buried in a particular column. Use a transformation to parse the valuable data into a new column and remove the original. |
20+
| Reduce data costs | Since you're charged ingestion cost for any data sent to a Log Analytics workspace, you want to filter out any data that you don't require to reduce your costs.<br><br>**Remove entire rows.** For example, you might have a diagnostic setting to collect resource logs from a particular resource but not require all of the log entries that it generates. Create a transformation that filters out records that match a certain criteria.<br><br>**Remove a column from each row.** For example, your data may include columns with data that's redundant or has minimal value. Create a transformation that filters out columns that aren't required.<br><br>**Parse important data from a column.** You may have a table with valuable data buried in a particular column. Use a transformation to parse the valuable data into a new column and remove the original.<br><br>**Send certain rows to basic logs.** Send rows in your data that require on basic query capabilities to basic logs tables for a lower ingestion cost. |
2121

2222

2323

@@ -52,12 +52,14 @@ A common use of the workspace transformation DCR is collection of [resource logs
5252

5353
## Multiple destinations
5454

55-
Transformations allow you to send data to multiple tables in a Log Analytics workspace in a single DCR. For example, you may send data into Azure Monitor using the Logs ingestion API that should be separated between two different tables depending on particular criteria. You provide a KQL query for each destination, and the results of each query are sent to their corresponding tables. For example, you might want to send audit events from a particular data source to a custom table configured for [basic logs](../logs/basic-logs-configure.md) to reduce your cost. Other events would be sent to an analytics table where it could be queried regularly.
55+
Transformations allow you to send data to multiple destinations in a Log Analytics workspace using a single DCR. You provide a KQL query for each destination, and the results of each query are sent to their corresponding location. You can send different sets of data to different tables, or use multiple queries to send different sets of data to the same table.
56+
57+
For example, you may send event data into Azure Monitor using the Logs ingestion API. Most of the events should be sent an analytics table where it could be queried regularly, while audit events should be sent to a custom table configured for [basic logs](../logs/basic-logs-configure.md) to reduce your cost.
5658

5759
To use multiple destinations, you must currently either manually create a new DCR or [edit an existing one](data-collection-rule-edit.md). See the [Samples](#samples) section for examples of DCRs using multiple destinations.
5860

5961
> [!IMPORTANT]
60-
> Currently, the tables in the DCR must be in the same Log Analytics workspace. To send to multiple workspaces from a single data source, use multiple DCRs.
62+
> Currently, the tables in the DCR must be in the same Log Analytics workspace. To send to multiple workspaces from a single data source, use multiple DCRs and configure your application to send the data to each.
6163
6264
:::image type="content" source="media/data-collection-transformations/transformation-multiple-destinations.png" lightbox="media/data-collection-transformations/transformation-multiple-destinations.png" alt-text="Diagram of transformation sending data to multiple tables." border="false":::
6365

@@ -87,66 +89,74 @@ See [Azure Monitor pricing](https://azure.microsoft.com/pricing/details/monitor)
8789
8890

8991
## Samples
92+
Following are Resource Manager templates of sample DCRs with different patterns. You can use these templates as a starting point to creating DCRs with transformations for your own scenarios.
9093

9194
### Single destination
9295

9396
The following example is a DCR for Azure Monitor agent that sends data to the `Syslog` table. In this example, the transformation filters the data for records with *error* in the message.
9497

98+
9599
```json
96100
{
97-
"type": "Microsoft.Insights/dataCollectionRules",
98-
"name": "singleDestinationDCR",
99-
"apiVersion": "2021-09-01-preview",
100-
"location": "eastus",
101-
"properties": {
102-
"dataSources": {
103-
"syslog": [
104-
{
105-
"name": "sysLogsDataSource",
106-
"streams": [
107-
"Microsoft-Syslog"
108-
],
109-
"facilityNames": [
110-
"auth",
111-
"authpriv",
112-
"cron",
113-
"daemon",
114-
"mark",
115-
"kern",
116-
"mail",
117-
"news",
118-
"syslog",
119-
"user",
120-
"uucp"
121-
],
122-
"logLevels": [
123-
"Debug",
124-
"Critical",
125-
"Emergency"
126-
]
127-
}
128-
]
129-
},
130-
"destinations": {
131-
"logAnalytics": [
132-
{
133-
"workspaceResourceId": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/my-resource-group/providers/Microsoft.OperationalInsights/workspaces/my-workspace",
134-
"name": "centralWorkspace"
135-
}
136-
]
137-
},
138-
"dataFlows": [
139-
{
140-
"streams": [
141-
"Microsoft-Syslog"
142-
],
143-
"transformKql": "source | where message contains 'error'",
144-
"destinations": [
145-
"centralWorkspace"
146-
]
147-
}
148-
]
149-
}
101+
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
102+
"contentVersion": "1.0.0.0",
103+
"resources" : [
104+
{
105+
"type": "Microsoft.Insights/dataCollectionRules",
106+
"name": "singleDestinationDCR",
107+
"apiVersion": "2021-09-01-preview",
108+
"location": "eastus",
109+
"properties": {
110+
"dataSources": {
111+
"syslog": [
112+
{
113+
"name": "sysLogsDataSource",
114+
"streams": [
115+
"Microsoft-Syslog"
116+
],
117+
"facilityNames": [
118+
"auth",
119+
"authpriv",
120+
"cron",
121+
"daemon",
122+
"mark",
123+
"kern",
124+
"mail",
125+
"news",
126+
"syslog",
127+
"user",
128+
"uucp"
129+
],
130+
"logLevels": [
131+
"Debug",
132+
"Critical",
133+
"Emergency"
134+
]
135+
}
136+
]
137+
},
138+
"destinations": {
139+
"logAnalytics": [
140+
{
141+
"workspaceResourceId": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/my-resource-group/providers/Microsoft.OperationalInsights/workspaces/my-workspace",
142+
"name": "centralWorkspace"
143+
}
144+
]
145+
},
146+
"dataFlows": [
147+
{
148+
"streams": [
149+
"Microsoft-Syslog"
150+
],
151+
"transformKql": "source | where message contains 'error'",
152+
"destinations": [
153+
"centralWorkspace"
154+
]
155+
}
156+
]
157+
}
158+
}
159+
]
150160
}
151161
```
152162

@@ -156,62 +166,68 @@ The following example is a DCR for data from Logs Ingestion API that sends data
156166

157167
```json
158168
{
159-
"type": "Microsoft.Insights/dataCollectionRules",
160-
"name": "multiDestinationDCR",
161-
"location": "eastus",
162-
"apiVersion": "2021-09-01-preview",
163-
"properties": {
164-
"dataCollectionEndpointId": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/my-resource-group/providers//Microsoft.Insights/dataCollectionEndpoints/my-dce",
165-
"streamDeclarations": {
166-
"Custom-MyTableRawData": {
167-
"columns": [
168-
{
169-
"name": "Time",
170-
"type": "datetime"
171-
},
169+
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
170+
"contentVersion": "1.0.0.0",
171+
"resources" : [
172+
{
173+
"type": "Microsoft.Insights/dataCollectionRules",
174+
"name": "multiDestinationDCR",
175+
"location": "eastus",
176+
"apiVersion": "2021-09-01-preview",
177+
"properties": {
178+
"dataCollectionEndpointId": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/my-resource-group/providers//Microsoft.Insights/dataCollectionEndpoints/my-dce",
179+
"streamDeclarations": {
180+
"Custom-MyTableRawData": {
181+
"columns": [
182+
{
183+
"name": "Time",
184+
"type": "datetime"
185+
},
186+
{
187+
"name": "Computer",
188+
"type": "string"
189+
},
190+
{
191+
"name": "AdditionalContext",
192+
"type": "string"
193+
}
194+
]
195+
}
196+
},
197+
"destinations": {
198+
"logAnalytics": [
199+
{
200+
"workspaceResourceId": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/my-resource-group/providers/Microsoft.OperationalInsights/workspaces/my-workspace",
201+
"name": "clv2ws1"
202+
},
203+
]
204+
},
205+
"dataFlows": [
172206
{
173-
"name": "Computer",
174-
"type": "string"
207+
"streams": [
208+
"Custom-MyTableRawData"
209+
],
210+
"destinations": [
211+
"clv2ws1"
212+
],
213+
"transformKql": "source | project TimeGenerated = Time, Computer, Message = AdditionalContext",
214+
"outputStream": "Microsoft-Syslog"
175215
},
176216
{
177-
"name": "AdditionalContext",
178-
"type": "string"
217+
"streams": [
218+
"Custom-MyTableRawData"
219+
],
220+
"destinations": [
221+
"clv2ws1"
222+
],
223+
"transformKql": "source | where (AdditionalContext contains 'malicious traffic!' | project TimeGenerated = Time, Computer, Subject = AdditionalContext",
224+
"outputStream": "Microsoft-SecurityEvent"
179225
}
180226
]
181227
}
182-
},
183-
"destinations": {
184-
"logAnalytics": [
185-
{
186-
"workspaceResourceId": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/my-resource-group/providers/Microsoft.OperationalInsights/workspaces/my-workspace",
187-
"name": "clv2ws1"
188-
},
189-
]
190-
},
191-
"dataFlows": [
192-
{
193-
"streams": [
194-
"Custom-MyTableRawData"
195-
],
196-
"destinations": [
197-
"clv2ws1"
198-
],
199-
"transformKql": "source | project TimeGenerated = Time, Computer, Message = AdditionalContext",
200-
"outputStream": "Microsoft-Syslog"
201-
},
202-
{
203-
"streams": [
204-
"Custom-MyTableRawData"
205-
],
206-
"destinations": [
207-
"clv2ws1"
208-
],
209-
"transformKql": "source | where (AdditionalContext contains 'malicious traffic!' | project TimeGenerated = Time, Computer, Subject = AdditionalContext",
210-
"outputStream": "Microsoft-SecurityEvent"
211-
}
212-
]
213-
}
214-
}
228+
}
229+
]
230+
}
215231
```
216232

217233
### Combination of Azure and custom tables
@@ -221,62 +237,68 @@ The following example is a DCR for data from Logs Ingestion API that sends data
221237

222238
```json
223239
{
224-
"type": "Microsoft.Insights/dataCollectionRules",
225-
"name": "multiDestinationDCR",
226-
"location": "eastus",
227-
"apiVersion": "2021-09-01-preview",
228-
"properties": {
229-
"dataCollectionEndpointId": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/my-resource-group/providers//Microsoft.Insights/dataCollectionEndpoints/my-dce",
230-
"streamDeclarations": {
231-
"Custom-MyTableRawData": {
232-
"columns": [
233-
{
234-
"name": "Time",
235-
"type": "datetime"
236-
},
240+
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
241+
"contentVersion": "1.0.0.0",
242+
"resources" : [
243+
{
244+
"type": "Microsoft.Insights/dataCollectionRules",
245+
"name": "multiDestinationDCR",
246+
"location": "eastus",
247+
"apiVersion": "2021-09-01-preview",
248+
"properties": {
249+
"dataCollectionEndpointId": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/my-resource-group/providers//Microsoft.Insights/dataCollectionEndpoints/my-dce",
250+
"streamDeclarations": {
251+
"Custom-MyTableRawData": {
252+
"columns": [
253+
{
254+
"name": "Time",
255+
"type": "datetime"
256+
},
257+
{
258+
"name": "Computer",
259+
"type": "string"
260+
},
261+
{
262+
"name": "AdditionalContext",
263+
"type": "string"
264+
}
265+
]
266+
}
267+
},
268+
"destinations": {
269+
"logAnalytics": [
270+
{
271+
"workspaceResourceId": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/my-resource-group/providers/Microsoft.OperationalInsights/workspaces/my-workspace",
272+
"name": "clv2ws1"
273+
},
274+
]
275+
},
276+
"dataFlows": [
237277
{
238-
"name": "Computer",
239-
"type": "string"
278+
"streams": [
279+
"Custom-MyTableRawData"
280+
],
281+
"destinations": [
282+
"clv2ws1"
283+
],
284+
"transformKql": "source | project TimeGenerated = Time, Computer, SyslogMessage = AdditionalContext",
285+
"outputStream": "Microsoft-Syslog"
240286
},
241287
{
242-
"name": "AdditionalContext",
243-
"type": "string"
288+
"streams": [
289+
"Custom-MyTableRawData"
290+
],
291+
"destinations": [
292+
"clv2ws1"
293+
],
294+
"transformKql": "source | extend jsonContext = parse_json(AdditionalContext) | project TimeGenerated = Time, Computer, AdditionalContext = jsonContext, ExtendedColumn=tostring(jsonContext.CounterName)",
295+
"outputStream": "Custom-MyTable_CL"
244296
}
245297
]
246298
}
247-
},
248-
"destinations": {
249-
"logAnalytics": [
250-
{
251-
"workspaceResourceId": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/my-resource-group/providers/Microsoft.OperationalInsights/workspaces/my-workspace",
252-
"name": "clv2ws1"
253-
},
254-
]
255-
},
256-
"dataFlows": [
257-
{
258-
"streams": [
259-
"Custom-MyTableRawData"
260-
],
261-
"destinations": [
262-
"clv2ws1"
263-
],
264-
"transformKql": "source | project TimeGenerated = Time, Computer, SyslogMessage = AdditionalContext",
265-
"outputStream": "Microsoft-Syslog"
266-
},
267-
{
268-
"streams": [
269-
"Custom-MyTableRawData"
270-
],
271-
"destinations": [
272-
"clv2ws1"
273-
],
274-
"transformKql": "source | extend jsonContext = parse_json(AdditionalContext) | project TimeGenerated = Time, Computer, AdditionalContext = jsonContext, ExtendedColumn=tostring(jsonContext.CounterName)",
275-
"outputStream": "Custom-MyTable_CL"
276-
}
277-
]
278-
}
279-
}
299+
}
300+
]
301+
}
280302
```
281303

282304

0 commit comments

Comments
 (0)