@@ -15,30 +15,63 @@ This article provides information on troubleshooting and resolving issues with A
15
15
16
16
To help troubleshoot issues with extension-based Hybrid Runbook Workers:
17
17
18
- - Check the OS is supported and the prerequisites have been met. See [ Prerequisites] ( ../extension-based-hybrid-runbook-worker-install.md#prerequisites ) .
18
+ - Check the OS is supported, and the prerequisites have been met. See [ Prerequisites] ( ../extension-based-hybrid-runbook-worker-install.md#prerequisites ) .
19
19
20
20
- Check whether the system-assigned managed identity is enabled on the VM. Azure VMs and Arc enabled Azure Machines should be enabled with a system-assigned managed identity.
21
21
22
22
- Check whether the extension is enabled with the right settings. Setting file should have right ` AutomationAccountURL ` . Cross-check the URL with Automation account property - ` AutomationHybridServiceUrl ` .
23
- - For windows: you can find the settings file at ` C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\<version>\RuntimeSettings ` .
24
- - For Linux: you can find the settings file at ` /var/lib/waagent/Microsoft.Azure.Automation.HybridWorker.HybridWorkerForLinux/ ` .
23
+ - For Windows, you can find the settings file here:
24
+ > [ !TIP]
25
+ > Replace ` * ` in the below path with the specific version that is installed if you know it.
26
+ ```
27
+ C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\*\RuntimeSettings
28
+ ```
29
+ - For Linux, you can find the settings file here:
30
+ ```
31
+ /var/lib/waagent/Microsoft.Azure.Automation.HybridWorker.HybridWorkerForLinux/
32
+ ```
25
33
26
34
- Check the error message shown in the Hybrid worker extension status/Detailed Status. It contains error message(s) and respective recommendation(s) to fix the issue.
27
35
28
36
- Run the troubleshooter tool on the VM and it generates an output file. Open the output file and verify the errors identified by the troubleshooter tool.
29
- - For windows: you can find the troubleshooter at ` C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\<version>\bin\troubleshooter\TroubleShootWindowsExtension.ps1 `
30
- - For Linux: you can find the troubleshooter at ` /var/lib/waagent/Microsoft.Azure.Automation.HybridWorker.HybridWorkerForLinux-<version>/Troubleshooter/LinuxTroubleshooter.py `
37
+ - For Windows, you can find the troubleshooter here:
38
+ > [!TIP]
39
+ > Replace `*` in the below path with the specific version that is installed if you know it.
40
+ ```
41
+ C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\*\bin\troubleshooter\TroubleShootWindowsExtension.ps1
42
+ ```
43
+ - For Linux, you can find the troubleshooter here:
44
+ > [!TIP]
45
+ > Replace `*` in the below path with the specific version that is installed if you know it.
46
+ ```
47
+ /var/lib/waagent/Microsoft.Azure.Automation.HybridWorker.HybridWorkerForLinux-*/Troubleshooter/LinuxTroubleshooter.py
48
+ ```
31
49
32
50
- For Linux machines, the Hybrid worker extension creates a `hweautomation` user and starts the Hybrid worker under the user. Check whether the user `hweautomation` is set up with the correct permissions. If your runbook is trying to access any local resources, ensure that the `hweautomation` has the correct permissions to the local resources.
33
51
34
52
- Check whether the hybrid worker process is running.
35
- - For Windows: check the ` Hybrid Worker Service ` service.
36
- - For Linux: check the ` hwd ` service.
53
+ - For Windows, check the `Hybrid Worker Service` (***HybridWorkerService***) service.
54
+ - For Linux, check the `hwd` service.
37
55
38
56
- Collect logs:
39
- - For Windows: Run the log collector tool in </br >` C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\<version>\bin\troubleshooter\PullLogs.ps1 ` </br >
40
- Logs are in ` C:\HybridWorkerExtensionLogs ` .
41
- - For Linux: Logs are in folders </br >` /var/log/azure/Microsoft.Azure.Automation.HybridWorker.HybridWorkerForLinux ` and ` /home/hweautomation ` .
57
+ - For Windows, run the log collector tool located here:
58
+ > [!TIP]
59
+ > Replace `*` in the below path with the specific version that is installed if you know it.
60
+ ```
61
+ C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\*\bin\troubleshooter\PullLogs.ps1
62
+ ```
63
+ Logs will be located here:
64
+ ```
65
+ C:\HybridWorkerExtensionLogs
66
+ ```
67
+ - For Linux: Logs are in the following folders:
68
+ ```
69
+ /var/log/azure/Microsoft.Azure.Automation.HybridWorker.HybridWorkerForLinux
70
+ ```
71
+ and
72
+ ```
73
+ /home/hweautomation
74
+ ```
42
75
43
76
44
77
### Unable to update Az modules while using the Hybrid Worker
@@ -51,9 +84,16 @@ The Hybrid Runbook Worker jobs failed as it was unable to import Az modules.
51
84
52
85
As a workaround, you can follow these steps:
53
86
54
- 1 . Go to the folder: C:\Program Files\Microsoft Monitoring Agent\Agent\AzureAutomation\7.3.1722.0\HybridAgent
55
- 1 . Edit the file with the name * Orchestrator.Sandbox.exe.config*
56
- 1 . Add the following lines inside the ` <assemblyBinding> ` tags:
87
+ 1. Navigate to the folder:
88
+ > [!TIP]
89
+ > Replace `*` in the below path with the specific version that is installed if you know it.
90
+ ```
91
+ C:\Program Files\Microsoft Monitoring Agent\Agent\AzureAutomation\* \HybridAgent
92
+ ```
93
+
94
+ 1. Edit the file with the name `Orchestrator.Sandbox.exe.config`
95
+
96
+ 1. Add the following lines inside the `<assemblyBinding>` tag:
57
97
```xml
58
98
<dependentAssembly>
59
99
<assemblyIdentity name="Newtonsoft.Json" publicKeyToken="30ad4fe6b2a6aeed" culture="neutral" />
@@ -75,20 +115,20 @@ For Custom user on the Hybrid Runbook Worker, update the permissions in the foll
75
115
76
116
| Folder | Permissions |
77
117
| --- | --- |
78
- | C:\ProgramData\AzureConnectedMachineAgent\Tokens | Read |
79
- | C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows | Read and Execute |
118
+ | ` C:\ProgramData\AzureConnectedMachineAgent\Tokens ` | Read |
119
+ | ` C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows ` | Read and Execute |
80
120
81
121
82
122
### Scenario: Job failed to start as the Hybrid Worker wasn't available when the scheduled job started
83
123
84
124
#### Issue
85
- Job fails to start on a Hybrid Worker and you see the following error:
125
+ Job fails to start on a Hybrid Worker, and you see the following error:
86
126
87
127
* Failed to start, as hybrid worker wasn't available when scheduled job started, the hybrid worker was last active at mm/dd/yyyy* .
88
128
89
129
#### Cause
90
130
This error can occur due to the following reasons:
91
- - The machines doesn 't exist anymore.
131
+ - The machines don 't exist anymore.
92
132
- The machine is turned off and is unreachable.
93
133
- The machine has a network connectivity issue.
94
134
- The Hybrid Runbook Worker extension has been uninstalled from the machine.
@@ -178,13 +218,40 @@ Sometimes the installation process might get stuck.
178
218
### Resolution
179
219
Follow the steps mentioned below to install Hybrid Worker extension again:
180
220
181
- 1 . Open PowerShell console
182
- 1 . Remove registry entry, if present: * HKLM:/Software/Microsoft/Azure/HybridWorker*
183
- 1 . Remove the registry entry, if present: * HKLM:/Software/Microsoft/HybridRunbookWorkerV2*
184
- 1 . Go to Hybrid Worker extension installation folder
185
- Cd "C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\< version>"
186
- 1 . Install Hybrid Worker extension: ` .\bin\install.ps1 `
187
- 1 . Enable Hybrid Worker extension: ` .\bin\enable.ps1 `
221
+ 1 . Open PowerShell console.
222
+
223
+ 1 . ** Remove the registry key** , if present: ` HKLM:\Software\Microsoft\Azure\HybridWorker `
224
+
225
+ 1 . PowerShell code to remove the registry key along with any subkeys and values under it.:
226
+
227
+ ``` powershell
228
+ Get-Item HKLM:\Software\Microsoft\Azure\HybridWorker | Remove-Item -Recurse
229
+ ```
230
+
231
+ 1. **Remove the registry key**, if present: `HKLM:\Software\Microsoft\HybridRunbookWorkerV2`
232
+
233
+ 1. PowerShell code to remove the registry key along with any subkeys and values under it.:
234
+
235
+ ```powershell
236
+ Get-Item HKLM:\Software\Microsoft\HybridRunbookWorkerV2 | Remove-Item -Recurse
237
+ ```
238
+ 1. Navigate to the Hybrid Worker extension installation folder:
239
+
240
+ > [!TIP]
241
+ > Replace `*` in the below command with the specific version that is installed if you know it.
242
+ ```powershell
243
+ cd "C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\*"
244
+ ```
245
+ 1 . ** Install** the Hybrid Worker extension:
246
+
247
+ ``` powershell
248
+ .\bin\install.ps1
249
+ ```
250
+ 1 . ** Enable** the Hybrid Worker extension:
251
+
252
+ ``` powershell
253
+ .\bin\enable.ps1
254
+ ```
188
255
189
256
### Scenario: Uninstallation process of Hybrid Worker extension on Windows VM gets stuck
190
257
@@ -195,13 +262,40 @@ You have installed a Hybrid Worker extension on a Windows VM from the portal, bu
195
262
Sometimes the uninstallation process might get stuck.
196
263
197
264
#### Resolution
198
- 1 . Open PowerShell console
199
- 1 . Go to Hybrid Worker extension installation folder
200
- Cd "C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\< version\> "
201
- 1 . Disable Hybrid Worker extension: ` .\bin\disable.cmd `
202
- 1 . Uninstall Hybrid Worker extension: ` .\bin\uninstall.ps1 `
203
- 1 . Remove registry entry, if present: * HKLM:/Software/Microsoft/Azure/HybridWorker*
204
- 1 . Remove the registry entry, if present: * HKLM:/Software/Microsoft/HybridRunbookWorkerV2*
265
+ 1 . Open PowerShell console.
266
+
267
+ 1 . Navigate to the Hybrid Worker extension installation folder:
268
+
269
+ > [ !TIP]
270
+ > Replace ` * ` in the below command with the specific version that is installed if you know it.
271
+ ``` powershell
272
+ cd "C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\*"
273
+ ```
274
+ 1 . ** Disable** the Hybrid Worker extension:
275
+
276
+ ``` powershell
277
+ .\bin\disable.cmd
278
+ ```
279
+ 1 . ** Uninstall** the Hybrid Worker extension:
280
+
281
+ ``` powershell
282
+ .\bin\uninstall.ps1
283
+ ```
284
+ 1 . ** Remove registry key** , if present: ` HKLM:\Software\Microsoft\Azure\HybridWorker `
285
+
286
+ 1 . PowerShell code to remove the registry key along with any subkeys and values under it.:
287
+
288
+ ``` powershell
289
+ Get-Item HKLM:\Software\Microsoft\Azure\HybridWorker | Remove-Item -Recurse
290
+ ```
291
+
292
+ 1. **Remove the registry key**, if present: `HKLM:\Software\Microsoft\HybridRunbookWorkerV2`
293
+
294
+ 1. PowerShell code to remove the registry key along with any subkeys and values under it.:
295
+
296
+ ```powershell
297
+ Get-Item HKLM:\Software\Microsoft\HybridRunbookWorkerV2 | Remove-Item -Recurse
298
+ ```
205
299
206
300
207
301
### Scenario: Installation process of Hybrid Worker extension on Linux VM gets stuck
@@ -213,11 +307,30 @@ You have installed a Hybrid Worker extension on a Linux VM from the portal, but
213
307
Sometimes the uninstallation process might get stuck.
214
308
215
309
#### Resolution
216
- 1 . Go to folder: ` rm -r /home/hweautomation/state `
217
- 1 . Go to Hybrid Worker extension installation folder * /var/lib/waagent/Microsoft.Azure.Automation.HybridWorker.HybridWorkerForLinux-\< version\> /*
218
- 1 . Go to above folder and run command ` rm mrseq `
219
- 1 . Install Hybrid Worker Extension: * "installCommand": "./extension_shim.sh -c ./HWExtensionHandlers.py -i"*
220
- 1 . Enable Hybrid Worker extension: * "enableCommand": "./extension_shim.sh -c ./HWExtensionHandlers.py -e"*
310
+ 1. **Delete** the `state` folder:
311
+ ```bash
312
+ rm -r /home/hweautomation/state
313
+ ```
314
+ 1 . Navigate to the Hybrid Worker extension installation folder:
315
+ > [ !TIP]
316
+ > Replace ` * ` in the below command with the specific version that is installed if you know it.
317
+ ``` bash
318
+ cd /var/lib/waagent/Microsoft.Azure.Automation.HybridWorker.HybridWorkerForLinux-* /
319
+ ```
320
+
321
+ 1 . ** Delete** the mrseq file:
322
+ ``` bash
323
+ rm mrseq
324
+ ```
325
+ 1 . ** Install** the Hybrid Worker Extension:
326
+ ``` bash
327
+ ./extension_shim.sh -c ./HWExtensionHandlers.py -i
328
+ ```
329
+
330
+ 1 . ** Enable** the Hybrid Worker extension:
331
+ ``` bash
332
+ ./extension_shim.sh -c ./HWExtensionHandlers.py -e
333
+ ```
221
334
222
335
### Scenario: Uninstallation process of Hybrid Worker extension on Linux VM gets stuck
223
336
@@ -230,10 +343,20 @@ Sometimes the uninstallation process might get stuck.
230
343
#### Resolution
231
344
Follow the steps mentioned below to completely uninstall Hybrid Worker extension:
232
345
233
- 1 . Go to Hybrid Worker Extension installation folder:
234
- * /var/lib/waagent/Microsoft.Azure.Automation.HybridWorker.HybridWorkerForLinux-\< version\> /*
235
- 1 . Disable the extension: ` "disableCommand": "./extension_shim.sh -c ./HWExtensionHandlers.py -d" `
236
- 1 . Uninstall the extension: ` "uninstallCommand": "./extension_shim.sh -c ./HWExtensionHandlers.py -u" `
346
+ 1 . Navigate to the Hybrid Worker Extension installation folder:
347
+ > [ !TIP]
348
+ > Replace ` * ` in the below command with the specific version that is installed if you know it.
349
+ ``` bash
350
+ cd /var/lib/waagent/Microsoft.Azure.Automation.HybridWorker.HybridWorkerForLinux-* /
351
+ ```
352
+ 1 . ** Disable** the Hybrid Worker extension:
353
+ ``` bash
354
+ ./extension_shim.sh -c ./HWExtensionHandlers.py -d
355
+ ```
356
+ 1 . ** Uninstall** the Hybrid Worker extension:
357
+ ``` bash
358
+ ./extension_shim.sh -c ./HWExtensionHandlers.py -u
359
+ ```
237
360
238
361
### Scenario: Runbook execution fails
239
362
@@ -270,12 +393,14 @@ Check the **Microsoft-SMA** event log for a corresponding event with the descrip
270
393
271
394
A runbook running on a Hybrid Runbook Worker fails with the following error message:
272
395
273
- ` Connect-AzAccount : No certificate was found in the certificate store with thumbprint 0000000000000000000000000000000000000000 `
274
- ` At line:3 char:1 `
275
- ` + Connect-AzAccount -ServicePrincipal -Tenant $Conn.TenantID -Appl ... `
276
- ` + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ `
277
- ` + CategoryInfo : CloseError: (:) [Connect-AzAccount],ArgumentException `
278
- ` + FullyQualifiedErrorId : Microsoft.Azure.Commands.Profile.ConnectAzAccountCommand `
396
+ ```
397
+ Connect-AzAccount : No certificate was found in the certificate store with thumbprint 0000000000000000000000000000000000000000
398
+ At line:3 char:1
399
+ + Connect-AzAccount -ServicePrincipal -Tenant $Conn.TenantID -Appl ...
400
+ + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
401
+ + CategoryInfo : CloseError: (:) [Connect-AzAccount],ArgumentException
402
+ + FullyQualifiedErrorId : Microsoft.Azure.Commands.Profile.ConnectAzAccountCommand
403
+ ```
279
404
280
405
#### Cause
281
406
@@ -310,9 +435,12 @@ As a workaround, you can create a configuration file named `OrchestratorSandbox.
310
435
</configuration>
311
436
```
312
437
313
- Place this file in the same folder as the executable file ` OrchestratorSandbox.exe ` . For example,
314
-
315
- ` %ProgramFiles%\Microsoft Monitoring Agent\Agent\AzureAutomation\7.3.702.0\HybridAgent `
438
+ Place this file in the same folder as the executable file ` OrchestratorSandbox.exe ` . For example:
439
+ > [ !TIP]
440
+ > Replace ` * ` in the below path with the specific version that is installed if you know it.
441
+ ```
442
+ %ProgramFiles%\Microsoft Monitoring Agent\Agent\AzureAutomation\*\HybridAgent
443
+ ```
316
444
317
445
318
446
### Scenario: Microsoft Azure VMs automatically dropped from a hybrid worker group
0 commit comments