Skip to content

Commit 9f30409

Browse files
kgrembankgremban
committed
Restructure troubleshooting article and add new content
1 parent 8c6e528 commit 9f30409

File tree

3 files changed

+384
-274
lines changed

3 files changed

+384
-274
lines changed

‎articles/iot-edge/TOC.yml

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -206,7 +206,11 @@
206206
- name: IoT Edge plugin for Jenkins
207207
href: how-to-devops-plugins.md
208208
- name: Troubleshoot
209-
href: troubleshoot.md
209+
items:
210+
- name: Standard diagnostic steps
211+
href: troubleshoot.md
212+
- name: Common error resolutions
213+
href: troubleshoot-common-errors.md
210214
- name: Resources
211215
items:
212216
- name: Support and help options

‎articles/iot-edge/troubleshoot-common-errors.md

Lines changed: 315 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,315 @@
1+
---
2+
title: Troubleshoot - Azure IoT Edge | Microsoft Docs
3+
description: Use this article to learn standard diagnostic skills for Azure IoT Edge, like retrieving component status and logs, and resolve common issues
4+
author: kgremban
5+
manager: philmea
6+
ms.author: kgremban
7+
ms.date: 04/27/2020
8+
ms.topic: conceptual
9+
ms.service: iot-edge
10+
services: iot-edge
11+
ms.custom: [amqp, mqtt]
12+
---
13+
14+
# Common issues and resolutions for Azure IoT Edge
15+
16+
Use this article to find steps to resolve common issues encountered when deploying IoT Edge solutions. If you need to learn how to find logs and errors from your IoT Edge device, see [Troubleshoot your IoT Edge device](troubleshoot.md).
17+
18+
## IoT Edge agent stops after about a minute
19+
20+
**Observed behavior:**
21+
22+
The edgeAgent module starts and runs successfully for about a minute, then stops. The logs indicate that the IoT Edge agent attempts to connect to IoT Hub over AMQP, and then attempts to connect using AMQP over WebSocket. When that fails, the IoT Edge agent exits.
23+
24+
Example edgeAgent logs:
25+
26+
```output
27+
2017-11-28 18:46:19 [INF] - Starting module management agent.
28+
2017-11-28 18:46:19 [INF] - Version - 1.0.7516610 (03c94f85d0833a861a43c669842f0817924911d5)
29+
2017-11-28 18:46:19 [INF] - Edge agent attempting to connect to IoT Hub via AMQP...
30+
2017-11-28 18:46:49 [INF] - Edge agent attempting to connect to IoT Hub via AMQP over WebSocket...
31+
```
32+
33+
**Root cause:**
34+
35+
A networking configuration on the host network is preventing the IoT Edge agent from reaching the network. The agent attempts to connect over AMQP (port 5671) first. If the connection fails, it tries WebSockets (port 443).
36+
37+
The IoT Edge runtime sets up a network for each of the modules to communicate on. On Linux, this network is a bridge network. On Windows, it uses NAT. This issue is more common on Windows devices using Windows containers that use the NAT network.
38+
39+
**Resolution:**
40+
41+
Ensure that there is a route to the internet for the IP addresses assigned to this bridge/NAT network. Sometimes a VPN configuration on the host overrides the IoT Edge network.
42+
43+
## IoT Edge hub fails to start
44+
45+
**Observed behavior:**
46+
47+
The edgeHub module fails to start. You may see a message like one of the following errors in the logs:
48+
49+
```output
50+
One or more errors occurred.
51+
(Docker API responded with status code=InternalServerError, response=
52+
{\"message\":\"driver failed programming external connectivity on endpoint edgeHub (6a82e5e994bab5187939049684fb64efe07606d2bb8a4cc5655b2a9bad5f8c80):
53+
Error starting userland proxy: Bind for 0.0.0.0:443 failed: port is already allocated\"}\n)
54+
```
55+
56+
Or
57+
58+
```output
59+
info: edgelet_docker::runtime -- Starting module edgeHub...
60+
warn: edgelet_utils::logging -- Could not start module edgeHub
61+
warn: edgelet_utils::logging -- caused by: failed to create endpoint edgeHub on network nat: hnsCall failed in Win32:
62+
The process cannot access the file because it is being used by another process. (0x20)
63+
```
64+
65+
**Root cause:**
66+
67+
Some other process on the host machine has bound a port that the edgeHub module is trying to bind. The IoT Edge hub maps ports 443, 5671, and 8883 for use in gateway scenarios. The module fails to start if another process has already bound one of those ports.
68+
69+
**Resolution:**
70+
71+
You can resolve this issue two ways:
72+
73+
If the IoT Edge device is functioning as a gateway device, then you need to find and stop the process that is using port 443, 5671, or 8883. If the error specifies port 443, then the other process is usually a web server.
74+
75+
If you don't need to use the IoT Edge device as a gateway, then you can remove the port bindings from edgeHub's module create options. You can change the create options in the Azure portal or directly in the deployment.json file.
76+
77+
In the Azure portal:
78+
79+
1. Navigate to your IoT hub and select **IoT Edge**.
80+
81+
2. Select the IoT Edge device that you want to update.
82+
83+
3. Select **Set Modules**.
84+
85+
4. Select **Runtime Settings**.
86+
87+
5. In the **Edge Hub** module settings, delete everything from the **Create Options** text box.
88+
89+
6. Save your changes and create the deployment.
90+
91+
In the deployment.json file:
92+
93+
1. Open the deployment.json file that you applied to your IoT Edge device.
94+
95+
2. Find the `edgeHub` settings in the edgeAgent desired properties section:
96+
97+
```json
98+
"edgeHub": {
99+
"settings": {
100+
"image": "mcr.microsoft.com/azureiotedge-hub:1.0",
101+
"createOptions": "{\"HostConfig\":{\"PortBindings\":{\"8883/tcp\":[{\"HostPort\":\"8883\"}],\"443/tcp\":[{\"HostPort\":\"443\"}]}}}"
102+
},
103+
"type": "docker",
104+
"status": "running",
105+
"restartPolicy": "always"
106+
}
107+
```
108+
109+
3. Remove the `createOptions` line, and the trailing comma at the end of the `image` line before it:
110+
111+
```json
112+
"edgeHub": {
113+
"settings": {
114+
"image": "mcr.microsoft.com/azureiotedge-hub:1.0"
115+
},
116+
"type": "docker",
117+
"status": "running",
118+
"restartPolicy": "always"
119+
}
120+
```
121+
122+
4. Save the file and apply it to your IoT Edge device again.
123+
124+
## IoT Edge agent can't access a module's image (403)
125+
126+
A container fails to run, and the edgeAgent logs show a 403 error.
127+
128+
**Root cause:**
129+
130+
The IoT Edge agent doesn't have permissions to access a module's image.
131+
132+
**Resolution:**
133+
134+
Make sure that your registry credentials are correctly specified in your deployment manifest
135+
136+
## IoT Edge security daemon fails with an invalid hostname
137+
138+
**Observed behavior:**
139+
140+
The command `sudo journalctl -u iotedge` fails and prints the following message:
141+
142+
```output
143+
Error parsing user input data: invalid hostname. Hostname cannot be empty or greater than 64 characters
144+
```
145+
146+
**Root cause:**
147+
148+
The IoT Edge runtime can only support hostnames that are shorter than 64 characters. Physical machines usually don't have long hostnames, but the issue is more common on a virtual machine. The automatically generated hostnames for Windows virtual machines hosted in Azure, in particular, tend to be long.
149+
150+
**Resolution:**
151+
152+
When you see this error, you can resolve it by configuring the DNS name of your virtual machine, and then setting the DNS name as the hostname in the setup command.
153+
154+
1. In the Azure portal, navigate to the overview page of your virtual machine.
155+
2. Select **configure** under DNS name. If your virtual machine already has a DNS name configured, you don't need to configure a new one.
156+
157+
![Configure DNS name of virtual machine](./media/troubleshoot/configure-dns.png)
158+
159+
3. Provide a value for **DNS name label** and select **Save**.
160+
4. Copy the new DNS name, which should be in the format **\<DNSnamelabel\>.\<vmlocation\>.cloudapp.azure.com**.
161+
5. Inside the virtual machine, use the following command to set up the IoT Edge runtime with your DNS name:
162+
163+
* On Linux:
164+
165+
```bash
166+
sudo nano /etc/iotedge/config.yaml
167+
```
168+
169+
* On Windows:
170+
171+
```cmd
172+
notepad C:\ProgramData\iotedge\config.yaml
173+
```
174+
175+
## Stability issues on smaller devices
176+
177+
**Observed behavior:**
178+
179+
You may encounter stability problems on constrained devices like the Raspberry Pi, especially when used as a gateway. Symptoms include out of memory exceptions in the edge hub module, downstream devices cannot connect or the device stops sending telemetry messages after a few hours.
180+
181+
**Root cause:**
182+
183+
The IoT Edge hub, which is part of the IoT Edge runtime, is optimized for performance by default and attempts to allocate large chunks of memory. This optimization is not ideal for constrained edge devices and can cause stability problems.
184+
185+
**Resolution:**
186+
187+
For the IoT Edge hub, set an environment variable **OptimizeForPerformance** to **false**. There are two ways to set environment variables:
188+
189+
In the Azure portal:
190+
191+
In your IoT Hub, select your IoT Edge device and from the device details page and select **Set Modules** > **Runtime Settings**. Create an environment variable for the Edge Hub module called *OptimizeForPerformance* that is set to *false*.
192+
193+
![OptimizeForPerformance set to false](./media/troubleshoot/optimizeforperformance-false.png)
194+
195+
In the deployment manifest:
196+
197+
```json
198+
"edgeHub": {
199+
"type": "docker",
200+
"settings": {
201+
"image": "mcr.microsoft.com/azureiotedge-hub:1.0",
202+
"createOptions": <snipped>
203+
},
204+
"env": {
205+
"OptimizeForPerformance": {
206+
"value": "false"
207+
}
208+
},
209+
...
210+
```
211+
212+
## Can't get the IoT Edge daemon logs on Windows
213+
214+
**Observed behavior:**
215+
216+
If you get an EventLogException when using `Get-WinEvent` on Windows, check your registry entries.
217+
218+
**Root cause:**
219+
220+
The `Get-WinEvent` PowerShell command relies on a registry entry to be present to find logs by a specific `ProviderName`.
221+
222+
**Resolution:**
223+
224+
Set a registry entry for the IoT Edge daemon. Create a **iotedge.reg** file with the following content, and import in to the Windows Registry by double-clicking it or using the `reg import iotedge.reg` command:
225+
226+
```reg
227+
Windows Registry Editor Version 5.00
228+
229+
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\EventLog\Application\iotedged]
230+
"CustomSource"=dword:00000001
231+
"EventMessageFile"="C:\\ProgramData\\iotedge\\iotedged.exe"
232+
"TypesSupported"=dword:00000007
233+
```
234+
235+
## IoT Edge module fails to send a message to the edgeHub with 404 error
236+
237+
**Observed behavior:**
238+
239+
A custom IoT Edge module fails to send a message to the edgeHub with a 404 `Module not found` error. The IoT Edge daemon prints the following message to the logs:
240+
241+
```output
242+
Error: Time:Thu Jun 4 19:44:58 2018 File:/usr/sdk/src/c/provisioning_client/adapters/hsm_client_http_edge.c Func:on_edge_hsm_http_recv Line:364 executing HTTP request fails, status=404, response_buffer={"message":"Module not found"}u, 04 )
243+
```
244+
245+
**Root cause:**
246+
247+
The IoT Edge daemon enforces process identification for all modules connecting to the edgeHub for security reasons. It verifies that all messages being sent by a module come from the main process ID of the module. If a message is being sent by a module from a different process ID than initially established, it will reject the message with a 404 error message.
248+
249+
**Resolution:**
250+
251+
As of version 1.0.7, all module processes are authorized to connect. If upgrading to 1.0.7 isn't possible, complete the following steps. For more information, see the [1.0.7 release changelog](https://github.com/Azure/iotedge/blob/master/CHANGELOG.md#iotedged-1).
252+
253+
Make sure that the same process ID is always used by the custom IoT Edge module to send messages to the edgeHub. For instance, make sure to `ENTRYPOINT` instead of `CMD` command in your Docker file, since `CMD` will lead to one process ID for the module and another process ID for the bash command running the main program whereas `ENTRYPOINT` will lead to a single process ID.
254+
255+
## Edge Agent module continually reports 'empty config file' and no modules start on the device
256+
257+
**Observed behavior:**
258+
259+
The device has trouble starting modules defined in the deployment. Only the edgeAgent is running but continually reporting 'empty config file...'.
260+
261+
**Root cause:**
262+
263+
By default, IoT Edge starts modules in their own isolated container network. The device may be having trouble with DNS name resolution within this private network.
264+
265+
**Resolution:**
266+
267+
**Option 1: Set DNS server in container engine settings**
268+
269+
Specify the DNS server for your environment in the container engine settings, which will apply to all container modules started by the engine. Create a file named `daemon.json` specifying the DNS server to use. For example:
270+
271+
```json
272+
{
273+
"dns": ["1.1.1.1"]
274+
}
275+
```
276+
277+
The above example sets the DNS server to a publicly accessible DNS service. If the edge device cannot access this IP from its environment, replace it with DNS server address that is accessible.
278+
279+
Place `daemon.json` in the right location for your platform:
280+
281+
| Platform | Location |
282+
| --------- | -------- |
283+
| Linux | `/etc/docker` |
284+
| Windows host with Windows containers | `C:\ProgramData\iotedge-moby\config` |
285+
286+
If the location already contains `daemon.json` file, add the **dns** key to it and save the file.
287+
288+
Restart the container engine for the updates to take effect.
289+
290+
| Platform | Command |
291+
| --------- | -------- |
292+
| Linux | `sudo systemctl restart docker` |
293+
| Windows (Admin Powershell) | `Restart-Service iotedge-moby -Force` |
294+
295+
**Option 2: Set DNS server in IoT Edge deployment per module**
296+
297+
You can set DNS server for each module's *createOptions* in the IoT Edge deployment. For example:
298+
299+
```json
300+
"createOptions": {
301+
"HostConfig": {
302+
"Dns": [
303+
"x.x.x.x"
304+
]
305+
}
306+
}
307+
```
308+
309+
Be sure to set this configuration for the *edgeAgent* and *edgeHub* modules as well.
310+
311+
## Next steps
312+
313+
Do you think that you found a bug in the IoT Edge platform? [Submit an issue](https://github.com/Azure/iotedge/issues) so that we can continue to improve.
314+
315+
If you have more questions, create a [Support request](https://portal.azure.com/#create/Microsoft.Support) for help.

0 commit comments

Comments
 (0)