Skip to content

Commit b6ce770

Browse files
authored
Added Approach 2 section
Added Approach 2 section
1 parent e7a21f5 commit b6ce770

File tree

1 file changed

+171
-2
lines changed

1 file changed

+171
-2
lines changed

articles/hdinsight/azure-monitor-agent.md

Lines changed: 171 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ title: Azure Monitor Agent (AMA) migration guide for Azure HDInsight clusters
33
description: Learn how to migrate to Azure Monitor Agent (AMA) in Azure HDInsight clusters.
44
ms.service: azure-hdinsight
55
ms.topic: how-to
6-
ms.date: 08/29/2024
6+
ms.date: 09/03/2024
77
---
88

99
# Azure Monitor Agent (AMA) migration guide for Azure HDInsight clusters
@@ -75,7 +75,7 @@ The following sections describe how customers can use the new Azure Monitor Agen
7575
>
7676
> For more information about how to create a Log Analytics workspace, see [Create a Log Analytics workspace in the Azure portal](/azure/azure-monitor/logs/quick-create-workspace).
7777
78-
### Enable Azure monitor agent using Portal
78+
### Approach 1: Enable Azure monitor agent using Portal
7979

8080
Activate the new integration by going to your cluster's portal page and scrolling down the menu on the left until you reach the Monitoring section.
8181

@@ -92,6 +92,175 @@ Activate the new integration by going to your cluster's portal page and scrollin
9292

9393
1. Select Save once precondition steps are complete.
9494

95+
### Approach 2: Enable Azure monitor agent using Azure CLI
96+
97+
1. Enable system-assigned MSI
98+
99+
1. First get cluster information to check the MSI of cluster
100+
101+
102+
```
103+
az hdinsight show –-resource-group $resourceGroup –name $cluster
104+
105+
#get access token if needed
106+
107+
accessToken=$(az account get-access-token --query accessToken -o tsv)
108+
109+
url="https://management.azure.com/subscriptions/${subscriptionId}/resourcegroups/${resourceGroupName}/providers/Microsoft.HDInsight/clusters/${clusterName}?api-version=2024-08-01-preview"
110+
```
111+
112+
1. If this cluster has no MSI, directly enable system assigned MSI via rest API
113+
114+
```
115+
body="{\"identity\": {\"type\": \"SystemAssigned\"}}"
116+
117+
az rest --method patch --url "$url" --body "$body" --headers "Authorization=Bearer $accessToken"
118+
```
119+
1. If this cluster only has user assigned MSI, add system assigned MSI to identity.
120+
```
121+
body="{\"identity\": {\"type\": \"SystemAssigned,UserAssigned\", \"userAssignedIdentities\": {$userAssignedIdentityResourceId:{}}}}"
122+
123+
az rest --method patch --url "$url" --body "$body" --headers "Authorization=Bearer $accessToken"
124+
```
125+
126+
127+
1. If this cluster already system assigned MSI, no need to anything .
128+
129+
130+
1. Creation of DCR
131+
132+
For more insormation, see [Create and edit data collection rules (DCRs)](/azure/azure-monitor/essentials/data-collection-rule-create-edit?tabs=CLI#create-a-dcr)
133+
134+
```
135+
# The URL of the DCR template file, change {HDIClusterType} to your cluster type.
136+
137+
# The valid types are: hadoop, hbase, interactivehive, kafka, llap, spark
138+
139+
$dcrTemplatejsonUrl = "https://hdiconfigactions.blob.core.windows.net/azuremonitoriningagent/DCR/{HDIClusterType}_dcr_template.json?api-version=2020-08-01"
140+
141+
142+
143+
# Download dcr template to local
144+
145+
$dcrTemplateLocalFile = "dcrTemplateFileName.json"
146+
147+
azcopy copy $dcrTemplatejsonUrl $dcrTemplateLocalFile
148+
149+
150+
151+
# Set subscription
152+
153+
az account set --subscription "{yourSubscription}"
154+
155+
156+
157+
# Get details of your Log Analytics workspace
158+
159+
$workspaceResourceGroupName = "{yourWorkspaceResourceGroup}"
160+
161+
$workspaceName = "{yourWorkspaceName}"
162+
163+
$workspace = az monitor log-analytics workspace show --resource-group $workspaceResourceGroupName --workspace-name $workspaceName
164+
165+
166+
167+
# Customize the DCR content. Below script depends on jq, you need to install it if it’s not available in your environment.
168+
169+
$workspaceResourceId = $workspace | jq -r '.id'
170+
171+
$workspaceId = $workspace | jq -r '.customerId'
172+
173+
$location = $workspace | jq -r '.location'
174+
175+
176+
177+
# Read the JSON file
178+
179+
$templateJsonData=cat $dcrTemplateLocalFile
180+
181+
182+
183+
# Update the JSON fields using jq
184+
185+
$templateJsonData=echo $templateJsonData | jq --arg workspaceResourceId $workspaceResourceId '.properties.destinations.logAnalytics[0].workspaceResourceId = $workspaceResourceId'
186+
187+
$templateJsonData=echo $templateJsonData | jq --arg workspaceId $workspaceId '.properties.destinations.logAnalytics[0].workspaceId = $workspaceId'
188+
189+
$templateJsonData=echo $templateJsonData | jq --arg location $location '.location = $location'
190+
191+
192+
193+
# Save the updated JSON back to the file
194+
195+
echo $templateJsonData > $dcrTemplateLocalFile
196+
197+
198+
199+
# Print the updated JSON
200+
201+
cat $dcrTemplateLocalFile
202+
203+
204+
205+
# Create the DCR using the customized JSON (DCR needs to be in the same location as Log Analytics workspace)
206+
207+
# If your HDInsight cluster is in another subscription, you need to set subscription to your cluster’s subscription
208+
209+
$dcrName = "{yourDcrName}"
210+
211+
$resourceGroupName = "{YourDcrResourceGroup}" # Suggest to put DCR in the same resource group as your HDInsight cluster
212+
213+
$dcr = az monitor data-collection rule create --name $dcrName --location $location --resource-group $resourceGroupName --rule-file $dcrTemplateLocalFile
214+
```
215+
216+
217+
1. Association of DCR
218+
219+
```
220+
# Associate DCR to HDInsight cluster
221+
222+
$hdinsightClusterResourceId = "{YourHDInsightClusterResourceId}"
223+
224+
$dcrAssociationName = "{yourDcrAssociation}"
225+
226+
$dcrId = $dcr | jq -r '.id'
227+
228+
az monitor data-collection rule association create --association-name $dcrAssociationName --resource $hdinsightClusterResourceId --data-collection-rule-id $dcrId
229+
```
230+
231+
232+
1. Enabling Azure Monitor Agent
233+
234+
```
235+
# set variables
236+
237+
export resourceGroup=RESOURCEGROUPNAME
238+
239+
export cluster=CLUSTERNAME
240+
241+
export LAW=LOGANALYTICSWORKSPACENAME
242+
243+
244+
245+
# Enable the Azure Monitor Agent logs integration on an HDInsight cluster.
246+
247+
az hdinsight azure-monitor-agent enable --name $cluster --resource-group $resourceGroup --workspace $LAW
248+
249+
250+
251+
# Get the status of Azure Monitor Agent logs integration on an HDInsight cluster.
252+
253+
az hdinsight azure-monitor-agent show --name $cluster --resource-group $resourceGroup
254+
```
255+
256+
257+
1. (Optional) disabling Azure Monitor Agent
258+
259+
```
260+
az hdinsight azure-monitor-agent disable --name $cluster --resource-group $resourceGroup
261+
```
262+
263+
95264
### Enable Azure Monitor Agent logging for Spark cluster
96265
97266
Azure HDInsight Spark clusters control AMA integration using a Spark configuration `spark.hdi.ama.enabled`, by default the value is set to false. This configuration controls whether the Spark specific logs will come up in the Log Analytics workspace. If you want to enable AMA in your Spark clusters and retrieve the Spark event logs in their LA workspaces, you need to perform an additional step to enable AMA for spark specific logs.

0 commit comments

Comments
 (0)