Skip to content

Commit e0f65c9

Browse files
bwrenbwren
committed
Edit data collection rules
1 parent 36c2b17 commit e0f65c9

File tree

2 files changed

+123
-0
lines changed

2 files changed

+123
-0
lines changed

‎articles/azure-monitor/essentials/data-collection-rule-edit.md

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
---
2+
title: Tutorial - Editing Data Collection Rules
3+
description: This article describes how to make changes in Data Collection Rule definition using command line tools and simple API calls.
4+
ms.topic: tutorial
5+
ms.date: 05/31/2022
6+
---
7+
8+
# Tutorial: Editing Data Collection Rules
9+
This tutorial will describe how to edit the definition of Data Collection Rule (DCR) that has been already provisioned using command line tools.
10+
11+
In this tutorial, you learn how to:
12+
> [!div class="checklist"]
13+
> * Leverage existing portal functionality to pre-create DCRs
14+
> * Get the content of a Data Collection Rule using ARM API call
15+
> * Apply changes to a Data Collection Rule using ARM API call
16+
> * Automate the process of DCR update using PowerShell scripts
17+
18+
## Prerequisites
19+
To complete this tutorial you need the following:
20+
- Log Analytics workspace where you have at least [contributor rights](manage-access.md#manage-access-using-azure-permissions) .
21+
- [Permissions to create Data Collection Rule objects](/azure/azure-monitor/essentials/data-collection-rule-overview#permissions) in the workspace.
22+
- Up to date version of PowerShell. Using Azure Cloud Shell is recommended.
23+
24+
## Overview of tutorial
25+
While going through the wizard on the portal is the simplest way to set up the ingestion of your custom data to Log Analytics, in some cases you might want to update your Data Collection Rule later to:
26+
- Change data collection settings (e.g. Data Collection Endpoint, associated with the DCR)
27+
- Update data parsing or filtering logic for your data stream
28+
- Change data destination (e.g. send data to an Azure table, as this option is not directly offered as part of the DCR-based custom log wizard)
29+
30+
In this tutorial, you will, first, set up ingestion of a custom log, then. you will modify the KQL transformation for your custom log to include additional filtering and apply the changes to your DCR. Finally, we are going to combine all editing operations into a single PowerShell script, which can be used to edit any DCR for any of the above mentioned reasons.
31+
32+
## Set up new custom log
33+
Start by setting up a new custom log. Follow [Tutorial: Send custom logs to Azure Monitor Logs using the Azure portal (preview)]( /azure/azure-monitor/logs/tutorial-custom-logs). Note the resource ID of the DCR created.
34+
35+
## Retrieve DCR content
36+
In order to update DCR, we are going to retrieve its content and save it as a file, which can be further edited.
37+
1. Click the **Cloud Shell** button in the Azure portal and ensure the environment is set to **PowerShell**.
38+
39+
:::image type="content" source="media/tutorial-ingestion-time-transformations-api/open-cloud-shell.png" lightbox="media/tutorial-ingestion-time-transformations-api/open-cloud-shell.png" alt-text="Screenshot of opening cloud shell":::
40+
41+
2. Execute the following commands to retrieve DCR content and save it to a file. Replace `<ResourceId>` with DCR ResourceID and `<FilePath>` with the name of the file to store DCR.
42+
43+
```PowerShell
44+
$ResourceId = “<ResourceId>” # Resource ID of the DCR to edit
45+
$FilePath = “<FilePath>” # Store DCR content in this file
46+
$DCR = Invoke-AzRestMethod -Path ("$ResourceId"+"?api-version=2021-09-01-preview") -Method GET
47+
$DCR.Content | ConvertFrom-Json | ConvertTo-Json -Depth 20 | Out-File -FilePath $FilePath
48+
```
49+
## Edit DCR
50+
Now, when DCR content is stored as a JSON file, you can use an editor of your choice to make changes in the DCR. You may [prefer to download the file from the Cloud Shell environment](azure/cloud-shell/using-the-shell-window#upload-and-download-files), if you are using one.
51+
52+
Alternatively you can use code editors supplied with the environment. For example, if you saved your DCR in a file named `temp.dcr` on your Cloud Drive, you could use the following command to open DCR for editing right in the Cloud Shell window:
53+
```PowerShell
54+
code "temp.dcr"
55+
```
56+
57+
Let’s modify the KQL transformation within DCR to drop rows where RequestType is anything, but “GET”.
58+
1. Open the file created in the previous part for editing using an editor of your choice.
59+
2. Locate the line containing `”transformKql”` attribute, which, if you followed the tutorial for custom log creation, should look similar to this:
60+
``` JSON
61+
"transformKql": " source\n | extend TimeGenerated = todatetime(Time)\n | parse RawData with \n ClientIP:string\n ' ' *\n ' ' *\n ' [' * '] \"' RequestType:string\n \" \" Resource:string\n \" \" *\n '\" ' ResponseCode:int\n \" \" *\n | where ResponseCode != 200\n | project-away Time, RawData\n"
62+
```
63+
3. Modify KQL transformation to include additional filter by RequestType
64+
``` JSON
65+
"transformKql": " source\n | where RawData contains \"GET\"\n | extend TimeGenerated = todatetime(Time)\n | parse RawData with \n ClientIP:string\n ' ' *\n ' ' *\n ' [' * '] \"' RequestType:string\n \" \" Resource:string\n \" \" *\n '\" ' ResponseCode:int\n \" \" *\n | where ResponseCode != 200\n | project-away Time, RawData\n"
66+
```
67+
4. Save the file with modified DCR content.
68+
69+
## Apply changes
70+
Our final step is to update DCR back in the system. This is accomplished by “PUT” HTTP call to ARM API, with updated DCR content sent in the HTTP request body.
71+
1. If you are using Azure Cloud Shell, save the file and close the embedded editor, or [upload modified DCR file back to the Cloud Shell environment](azure/cloud-shell/using-the-shell-window#upload-and-download-files).
72+
2. Execute the following commands to load DCR content from the file and place HTTP call to update the DCR in the system. Replace `<ResourceId>` with DCR ResourceID and `<FilePath>` with the name of the file modified in the previous part of the tutorial. You can omit first two lines if you read and write to the DCR within the same PowerShell session.
73+
```PowerShell
74+
$ResourceId = “<ResourceId>” # Resource ID of the DCR to edit
75+
$FilePath = “<FilePath>” # Store DCR content in this file
76+
$DCRContent = Get-Content $FilePath -Raw
77+
Invoke-AzRestMethod -Path ("$ResourceId"+"?api-version=2021-09-01-preview") -Method PUT -Payload $DCRContent
78+
```
79+
3. Upon successful call, you should get the response with status code “200”, indicating that your DCR is now updated.
80+
4. You can now navigate to your DCR and examine its content on the portal via “JSON View” function, or you could repeat the first part of the tutorial to retrieve DCR content into a file.
81+
82+
## Putting everything together
83+
Now, when we know how to read and update the content of a DCR, let’s put everything together into utility script, which can be used to perform both operations together.
84+
85+
```PowerShell
86+
param ([Parameter(Mandatory=$true)] $ResourceId)
87+
88+
# get DCR content and put into a file
89+
$FilePath = "temp.dcr"
90+
$DCR = Invoke-AzRestMethod -Path ("$ResourceId"+"?api-version=2021-09-01-preview") -Method GET
91+
$DCR.Content | ConvertFrom-Json | ConvertTo-Json -Depth 20 | Out-File $FilePath
92+
93+
# Open DCR in code editor
94+
code $FilePath | Wait-Process
95+
96+
#Wait for confirmation to apply changes
97+
$Output = Read-Host "Apply changes to DCR (Y/N)? "
98+
if ("Y" -eq $Output.toupper())
99+
{
100+
#write DCR content back from the file
101+
$DCRContent = Get-Content $FilePath -Raw
102+
Invoke-AzRestMethod -Path ("$ResourceId"+"?api-version=2021-09-01-preview") -Method PUT -Payload $DCRContent
103+
}
104+
105+
#Delete temporary file
106+
Remove-Item $FilePath
107+
```
108+
### How to use this utility
109+
110+
Assuming you saved the script as a file, named `DCREditor.ps1` and need to modify a Data Collection Rule with resource ID of `/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/foo/providers/Microsoft.Insights/dataCollectionRules/bar`, this could be accomplished by running the following command:
111+
112+
```PowerShell
113+
.\DCREditor.ps1 "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/foo/providers/Microsoft.Insights/dataCollectionRules/bar"
114+
```
115+
116+
DCR content will open in embedded code editor. Once editing is complete, entering "Y" on script prompt will apply changes back to the DCR.
117+
118+
## Next steps
119+
120+
- [Read more about data collection rules and options for creationg them.](data-collection-rule-overview.md)

‎articles/azure-monitor/toc.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -431,6 +431,9 @@ items:
431431
- name: Overview
432432
displayName: Data collection rules
433433
href: essentials/data-collection-rule-overview.md
434+
- name: Edit
435+
displayName: Data collection rules
436+
href: essentials/data-collection-rule-edit.md
434437
- name: Structure
435438
displayName: Data collection rules
436439
href: essentials/data-collection-rule-structure.md

0 commit comments

Comments
 (0)