Skip to content

Commit 039ab93

Browse files
denreadenrea
authored
Merge pull request #15531 from LouisBerner/v-loberner-ado-287017-arc-vm-remediation-using-collected-files
PR for new troubleshooting article.
2 parents d1920b5 + 718ed84 commit 039ab93

File tree

3 files changed

+98
-36
lines changed

3 files changed

+98
-36
lines changed

‎azure-stack/hci/manage/collect-log-files-arc-enabled-vms.md

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
---
2+
title: Collect log files for Azure Arc VM on Azure Stack HCI
3+
description: Learn how to collect log files for an Azure Arc VM on your Azure Stack HCI system.
4+
author: alkohli
5+
ms.topic: how-to
6+
ms.date: 08/16/2024
7+
ms.author: alkohli
8+
ms.reviewer: vlakshmanan
9+
---
10+
11+
# Collect log files for Azure Arc VM on your Azure Stack HCI system
12+
13+
[!INCLUDE [hci-applies-to-23h2](../../includes/hci-applies-to-23h2.md)]
14+
15+
Collect logs and other files to identify and troubleshoot issues with Arc virtual machines (VMs) in your Azure Stack HCI system.
16+
17+
## Collect files when VM provisioning fails
18+
19+
Use these files to gather key information about a VM provisioning failure before you contact Microsoft Support for more help.
20+
21+
### Windows VMs
22+
23+
The Windows Panther folder contains Windows setup, installation, and upgrade log files. This folder also includes logs for all scripts used to customize the VM.
24+
25+
| File | Directory | Description |
26+
|-------------------|-----------------|-------------|
27+
| agent-log-0 | C:\ProgramData\mocguestagent\log\ | Operational logs |
28+
| SetupAct.log | C:\Windows\panther\ | Contains information about setup actions during the installation. |
29+
| SetupErr.log | C:\Windows\panther\ | Contains information about setup errors during the installation. |
30+
| SetupComplete.log | C:\Windows\panther\ | Contains custom scripts that run during or after the Windows Setup process. For HCI, includes enabling WinRM, enabling ssh, and installing Microsoft On-premises Cloud (MOC) guest agent. |
31+
| Script files | C:\Windows\setup\scripts\ | Scripts from ISO |
32+
| System.evtx | C:\Windows\system32\winevt\logs\ | Windows event logs |
33+
34+
### Linux VMs
35+
36+
Examine these log files to investigate a VM provisioning failure:
37+
38+
| File | Directory | Description |
39+
|-------------------|-----------------|-------------|
40+
| cloud-init-output.log | /var/log/ | Captures the output from each stage of cloud-init when it runs. |
41+
| cloud-init.log | /var/log/ | A detailed log with debugging output, detailing each action taken. |
42+
| log files | /run/cloud-init/ | Contains logs about how cloud-init decided to enable or disable itself, and what platforms/datasources were detected. These logs are most useful when trying to determine what cloud-init ran or didn't run. |
43+
44+
## Collect guest logs
45+
46+
Collect guest logs to gather information on Arc VM issues before you contact Microsoft Support.
47+
48+
### Logs inside the VM
49+
50+
Windows VM domain join and extension logs:
51+
52+
| File | Directory | Description |
53+
|-------------------|-----------------|-------------|
54+
| Netsetup.log | C:\Windows\debug\ | Netlogon logs are used for domain join failure. If you don't see a domain join error, this log is optional. |
55+
| Extension logs | C:\ProgramData\GuestConfig\extension_logs\ | Extension logs |
56+
57+
For more information, see [Active Directory domain join troubleshooting guidance](/troubleshoot/windows-server/active-directory/active-directory-domain-join-troubleshooting-guidance).
58+
59+
### MOC guest agent logs
60+
61+
MOC guest agent logs are useful when Arc VM provisioning fails with the following error:
62+
63+
`Could not establish HyperV connection for VM ID...`
64+
65+
Example error:
66+
67+
`{"code":"moc-operator virtualmachine serviceClient returned an error while reconciling: rpc error: code = Unknown desc = Could not establish HyperV connection for VM ID [E5BF0FC3-DB6D-40AA-BB46-DD94E4E0719A] within [900] seconds, error: [<nil>]","message":"moc-operator virtualmachine serviceClient returned an error while reconciling: rpc error: code = Unknown desc = Could not establish HyperV connection for VM ID [E5BF0FC3-DB6D-40AA-BB46-DD94E4E0719A] within [900] seconds, error: [<nil>]","additionalInfo":[{"type":"ErrorInfo","info":{"category":"Uncategorized","recommendedAction":"","troubleshootingURL":""}}]}`
68+
69+
#### Windows VMs
70+
71+
Examine these MOC guest agent log files:
72+
73+
| File | Directory | Description |
74+
|-------------------|-----------------|-------------|
75+
| mocguestagent.log | C:\ProgramData\mocguestagent\log\ | Critical logs |
76+
| agent-log-0 | C:\ProgramData\mocguestagent\log\ | Operational logs |
77+
78+
#### Linux VMs
79+
80+
Examine these MOC guest agent log files:
81+
82+
| File | Directory | Description |
83+
|-------------------|-----------------|-------------|
84+
| mocguestagent.service | sudo journalctl -u | Critical logs |
85+
| agent-log-0 | /opt/mocguestagent/log/ | Operational logs |
86+
87+
## Next steps
88+
89+
- [Collect logs](./collect-logs.md).

‎azure-stack/hci/manage/troubleshoot-arc-enabled-vms.md

Lines changed: 7 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ title: Troubleshoot Azure Arc VM management
33
description: Learn how to troubleshoot Azure Arc VM management
44
author: alkohli
55
ms.topic: how-to
6-
ms.date: 07/10/2024
6+
ms.date: 07/26/2024
77
ms.author: alkohli
88
ms.reviewer: vlakshmanan
99
---
@@ -14,46 +14,17 @@ ms.reviewer: vlakshmanan
1414

1515
This article provides guidance on how to collect logs and troubleshoot issues with Azure Arc virtual machines (VMs) in your Azure Stack HCI cluster. It also lists the limitations and known issues that currently exist with Azure Arc VM management.
1616

17-
18-
## Collect logs
19-
20-
You can collect logs to identify and troubleshoot issues with Arc VMs in your Azure Stack HCI system. Use these logs to gather key information before you contact Microsoft support for additional help.
21-
22-
Make sure you have the latest PowerShell module for log collection. To update the PowerShell module, run the following command:
23-
24-
```PowerShell
25-
#Update the PowerShell module
26-
Install-Module -Name ArcHci -Force -Confirm:$false -SkipPublisherCheck -AcceptLicense
27-
```
28-
29-
To collect logs for Arc VMs in your Azure Stack HCI cluster, run the following command:
30-
31-
```PowerShell
32-
$csv_path="<input-from-admin>"
33-
$VMIP_1="<input-from-admin>"
34-
az login --use-device-code
35-
Get-ArcHCILogs -workDirectory $csv_path\ResourceBridge -kvaTokenPath $csv_path\ResourceBridge\kvatoken.tok -ip $VMIP_1
36-
```
37-
38-
Where:
39-
40-
- **$csv_path** is the full path of the cluster shared volume provided for creating Arc Resource Bridge.
41-
42-
- **$VMIP_1** is the IP address of the Arc Resource Bridge VM.
43-
44-
- Optionally, set the `-logDir` parameter to specify the path to the directory where the generated logs are stored. If you don't specify the path or the parameter, by default the logs are stored in your current working directory.
45-
4617
## Troubleshoot Azure Arc VMs
4718

4819
This section describes the errors related to Azure Arc VM management and their recommended resolutions.
4920

50-
### Failure when trying to enable guest management
21+
## Failure when trying to enable guest management
5122

5223
When trying to run the command to enable guest management, you see the following error:
5324

5425
**Error:** `Deployment failed. Correlation ID: 5d0c4921-78e0-4493-af16-dffee5cbf9d8. VM Spec validation failed for guest agent provisioning: Invalid managed identity. A system-assigned managed identity must be enabled in parent resource: Invalid Configuration`
5526

56-
The above failure is because the managed identity wasn't created for this VM. System-assigned Managed Identity is required to enable guest management.
27+
This failure is because the managed identity wasn't created for this VM. System-assigned Managed Identity is required to enable guest management.
5728

5829
**Resolution:**
5930

@@ -104,13 +75,13 @@ Follow these steps to verify that the Managed Identity isn't created for this VM
10475
10576
:::image type="content" source="./media/troubleshoot-arc-enabled-vms/managed-identity-missing-3.png" alt-text="Screenshot of JSON view when Managed Identity is enabled." lightbox="./media/troubleshoot-arc-enabled-vms/managed-identity-missing-3.png":::
10677
107-
### Failure deploying an Arc VM
78+
## Failure deploying an Arc VM
10879
10980
You see the following error when trying to deploy an Arc VM on your Azure Stack HCI cluster:
11081
11182
**Error:** `{"code":"ConflictingOperation","message":"Unable to process request 'Microsoft.AzureStackHCI/virtualMachineInstances'. There is already a previous running operation for resource '/subscriptions/<subscription ID>/resourceGroups/<Resource group name>/providers/Microsoft.HybridCompute/machines/<VM name>/providers/Microsoft.AzureStackHCI/virtualMachineInstances/default'. Please wait for the previous operation to complete."}`
11283
113-
The above failure is because the `SystemAssigned` managed identity object isn't under the `Microsoft.HybridCompute/machines` resource type.
84+
This failure is because the `SystemAssigned` managed identity object isn't under the `Microsoft.HybridCompute/machines` resource type.
11485
11586
**Resolution:**
11687
@@ -120,7 +91,7 @@ The `SystemAssigned` managed identity object is under `Microsoft.HybridCompute/m
12091
12192
The deployment template should match the provided sample template. For more information, see the sample template in [Create Arc virtual machines on Azure Stack HCI](./create-arc-virtual-machines.md).
12293
123-
### Failure deleting storage path
94+
## Failure deleting storage path
12495
12596
When trying to delete a storage path on your Azure Stack HCI cluster, you might see an error similar to the following message. Resource numbers and versions may vary in your scenario.
12697
@@ -140,7 +111,7 @@ Follow these steps before trying to remove a storage path:
140111
1. Remove the associated workloads and the images present on the storage path you want to delete. Look for the following prefixes on the image names: `linux-cblmariner`, `windows-windows2019`, `windows-windows2022`, `windows_k8s`, `aks-image-merged`, `linux-K8s`.
141112
1. File a [support ticket in the Azure portal](/azure/azure-portal/supportability/how-to-create-azure-support-request).
142113
143-
### Azure CLI installation isn't recognized
114+
## Azure CLI installation isn't recognized
144115
145116
If your environment fails to recognize Azure CLI after installing it, run the following code block to add the Azure CLI installation path to the environment path.
146117

‎azure-stack/hci/toc.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -291,6 +291,8 @@ items:
291291
- name: Extended Security Updates (ESUs)
292292
href: manage/azure-benefits-esu.md
293293

294+
- name: Collect log files for Azure Arc VM
295+
href: manage/collect-log-files-arc-enabled-vms.md
294296
- name: Troubleshoot
295297
href: manage/troubleshoot-arc-enabled-vms.md
296298
- name: FAQs

0 commit comments

Comments
 (0)