You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
If you see the following error message, you either didn't enable the required azure-arc custom locations feature, or you enabled the custom locations feature with an incorrect custom locations RP OID.
29
+
30
+
```ouput
29
31
Message: Microsoft.ExtendedLocation resource provider does not have the required permissions to create a namespace on the cluster.
30
32
```
31
33
32
-
This error is commonly caused by either not enabling the required azure-arc custom locations feature, or enabling the custom locations feature with an incorrect custom locations RP OID. To resolve, follow [this guidance](/azure-arc/kubernetes/custom-locations#enable-custom-locations-on-your-cluster) for enabling the custom locations feature with the correct OID.
34
+
To resolve, follow [this guidance](/azure-arc/kubernetes/custom-locations#enable-custom-locations-on-your-cluster) for enabling the custom locations feature with the correct OID.
If you see the following error message, your custom location resource associated with the deployment isn't properly configured with the API version(s) of resources attempting to be projected to the cluster.
39
+
40
+
```output
37
41
Message: The resource {resource Id} extended location {custom location resource Id} does not support the resource type {IoT Operations resource type} or api version {IoT Operations ARM API}. Please check with the owner of the extended location to ensure the host has the CRD {custom resource name} with group {api group name}.iotoperations.azure.com, plural {custom resource plural name}, and versions [{api group version}] installed.
38
42
```
39
43
40
-
This error happens when the custom location resource associated with the deployment isn't properly configured with the API version(s) of resources attempting to be projected to the cluster. To resolve, delete any provisioned resources associated with prior deployment(s) including custom locations. You can use `az iot ops delete` or alternative mechanism. Due to a potential caching issue, waiting a few minutes after deletion before re-deploying AIO or choosing a custom location name via `az iot ops create --custom-location` is recommended.
44
+
To resolve, delete any provisioned resources associated with prior deployment(s) including custom locations. You can use `az iot ops delete` or alternative mechanism. Due to a potential caching issue, waiting a few minutes after deletion before re-deploying AIO or choosing a custom location name via `az iot ops create --custom-location` is recommended.
If you see the following error message, the logged-in principal doesn't have the required permissions to deploy resources to the resource group specified in the resource sync resource ID.
49
+
50
+
```output
45
51
Message: The client {principal Id} with object id {principal object Id} has permission to perform action Microsoft.ExtendedLocation/customLocations/resourceSyncRules/write on scope {resource sync resource Id}; however, it does not have permission to perform action(s) Microsoft.Authorization/roleAssignments/write on the linked scope(s) {resource sync resource group} (respectively) or the linked scope(s) are invalid.
46
52
```
47
53
48
54
Deployment of resource sync rules require the logged-in principal to have the `Microsoft.Authorization/roleAssignments/write` permission against the resource group that resources are being deployed to. This is a necessary security constraint as edge to cloud resource hydration will create new resources in the target resource group.
49
55
50
56
To resolve, either elevate principal permissions, or don't deploy resource sync rules. Current AIO CLI has an opt-in mechanism to deploy resource sync rules via `--enable-rsync`. Simply omit this flag. Legacy AIO CLIs had an opt-out mechanism via `--disable-rsync-rules`.
51
57
52
-
## Secret management
58
+
## Troubleshoot Azure Key Vault secret management
53
59
54
60
If you see the following error message related to secret management, you need to update your Azure Key Vault contents:
55
61
@@ -63,7 +69,7 @@ For help resolving this issue, please see https://go.microsoft.com/fwlink/?linki
63
69
64
70
This error occurs when Azure IoT Operations tries to synchronize a secret from Azure Key Vault that doesn't exist. To resolve this issue, you need to add the secret in Azure Key Vault before you create resources such as a secret provider class.
65
71
66
-
## Connector for OPC UA
72
+
## Troubleshoot asset connector for OPC UA
67
73
68
74
An OPC UA server connection fails with a `BadSecurityModeRejected` error if the connector tries to connect to a server that only exposes endpoints with no security. There are two options to resolve this issue:
69
75
@@ -76,11 +82,11 @@ An OPC UA server connection fails with a `BadSecurityModeRejected` error if the
76
82
77
83
- Add a secure endpoint to the OPC UA server and set up the certificate mutual trust to establish the connection.
The troubleshooting guidance in this section is specific to Azure IoT Operations when using the Layered Network Management component. For more information, see [How does Azure IoT Operations work in layered network?](../manage-layered-network/concept-iot-operations-in-layered-network.md).
82
88
83
-
### Can't install Layered Network Management on the parent level
89
+
### You can't install Layered Network Management on the parent level
84
90
85
91
If the Layered Network Management operator install fails or you can't apply the custom resource for a Layered Network Management instance:
86
92
@@ -89,7 +95,7 @@ If the Layered Network Management operator install fails or you can't apply the
89
95
1. Verify the Layered Network Management operator is in the *Running and Ready* state.
90
96
1. If applying the custom resource `kubectl apply -f cr.yaml` fails, the output of this command lists the reason for error. For example, CRD version mismatch or wrong entry in CRD.
91
97
92
-
### Can't Arc-enable the cluster through the parent level Layered Network Management
98
+
### You can't Arc-enable the cluster through the parent level Layered Network Management
93
99
94
100
If you repeatedly remove and onboard a cluster with the same machine, you might get an error while Arc-enabling the cluster on nested layers. For example:
95
101
@@ -107,12 +113,12 @@ If your cluster is behind an outbound proxy server, please ensure that you have
107
113
108
114
1. Reboot the host machine.
109
115
110
-
### Other types of Arc-enablement failures
116
+
If you still see the error, check the following:
111
117
112
118
1. Add the `--debug` parameter when running the `connectedk8s` command.
113
119
1. Capture and investigate a network packet trace. For more information, see [capture Layered Network Management packet trace](#capture-layered-network-management-packet-trace).
114
120
115
-
### Can't install Azure IoT Operations on the isolated cluster
121
+
### You can't install Azure IoT Operations on the isolated cluster
116
122
117
123
You can't install Azure IoT Operations components on nested layers. For example, Layered Network Management on level 4 is running but can't install Azure IoT Operations on level 3.
118
124
@@ -128,7 +134,7 @@ You can't install Azure IoT Operations components on nested layers. For example,
128
134
1. If the domain is being resolved correctly, verify the domain is added to the allowlist. For more information, see [Check the allowlist of Layered Network Management](#check-the-allowlist-of-layered-network-management).
129
135
1. Capture and investigate a network packet trace. For more information, see [capture Layered Network Management packet trace](#capture-layered-network-management-packet-trace).
130
136
131
-
### A pod fails when installing Azure IoT Operations on an isolated cluster
137
+
### You can install Azure IoT Operations on the isolated cluster but the pods fail to start
132
138
133
139
When installing the Azure IoT Operations components to a cluster, the installation starts and proceeds. However, initialization of one or few of the components (pods) fails.
134
140
@@ -150,9 +156,9 @@ When installing the Azure IoT Operations components to a cluster, the installati
150
156
Warning Failed 3m14s kubelet Failed to pull image "…
151
157
```
152
158
153
-
### Check the allowlist of Layered Network Management
159
+
### You can't connect to the Azure IoT Operations service from the child level Layered Network Management
154
160
155
-
Layered Network Management blocks traffic if the destination domain isn't on the allowlist.
161
+
Layered Network Management blocks traffic if the destination domain isn't on the allowlist. The allowlist is a list of domains that are allowed to be accessed from the child level Layered Network Management. Check the allowlist of Layered Network Management to verify if the domain is included. If the domain isn't on the allowlist, you can add it to the allowlist.
156
162
157
163
1. Run the following command to list the config maps.
158
164
@@ -176,18 +182,18 @@ Layered Network Management blocks traffic if the destination domain isn't on the
176
182
177
183
1. All the allowed domains are listed in the output.
### You want to capture Layered Network Management to a packet trace
180
186
181
187
In some cases, you might suspect that Layered Network Management instance at the parent level isn't forwarding network traffic to a particular endpoint. Connection to a required endpoint is causing an issue for the service running on your node. It's possible that the service you enabled is trying to connect to a new endpoint after an update. Or you're trying to install a new Arc extension or service that requires connection to endpoints that aren't on the default allowlist. Usually there would be information in the error message to notify the connection failure. However, if there's no clear information about the missing endpoint, you can capture the network traffic on the child node for detailed debugging.
182
188
183
-
#### Windows host
189
+
#### [Windows host](#tab/tabid-windows)
184
190
185
191
1. Install Wireshark network traffic analyzer on the host.
186
192
1. Run Wireshark and start capturing.
187
193
1. Reproduce the installation or connection failure.
188
194
1. Stop capturing.
189
195
190
-
#### Linux host
196
+
#### [Linux host](#tab/tabid-linux)
191
197
192
198
1. Run the following command to start capturing:
193
199
@@ -198,14 +204,14 @@ In some cases, you might suspect that Layered Network Management instance at the
198
204
1. Reproduce the installation or connection failure.
199
205
1. Stop capturing.
200
206
201
-
#### Analyze the packet trace
207
+
***
202
208
203
209
Use Wireshark to open the trace file. Look for connection failures or unresponsive connections.
204
210
205
211
1. Filter the packets with the *ip.addr == [IP address]* parameter. Input the IP address of your custom DNS service address.
206
212
1. Review the DNS query and response, check if there's a domain name that isn't on the allowlist of Layered Network Management.
207
213
208
-
## Operations experience
214
+
## IoT Operations experience
209
215
210
216
To sign in to the [operations experience](https://iotoperations.azure.com) web UI, you need a Microsoft Entra ID account with at least contributor permissions for the resource group that contains your **Kubernetes - Azure Arc** instance.
0 commit comments