Skip to content

Commit 49ebf71

Browse files
authored
Merge pull request #94227 from JimacoMS3/add-troubleshoot-errors-content
Provide troubleshooting for common errors using new troubleshooting f…
2 parents a7a0f2c + 98f04e9 commit 49ebf71

19 files changed

+643
-33
lines changed

articles/iot-hub/TOC.yml

Lines changed: 35 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -369,11 +369,43 @@
369369
- name: Device management with IoT extension for Azure CLI
370370
href: iot-hub-device-management-iot-extension-azure-cli-2-0.md
371371
- name: Remote monitoring and notifications with ​Logic ​Apps
372-
href: iot-hub-monitoring-notifications-with-azure-logic-apps.md
373-
- name: Troubleshoot
372+
href: iot-hub-monitoring-notifications-with-azure-logic-apps.md
373+
- name: Troubleshoot and problem solution
374374
items:
375-
- name: Device disconnections
375+
- name: Device disconnects
376376
href: iot-hub-troubleshoot-connectivity.md
377+
- name: Problem resolution
378+
items:
379+
- name: 400027 ConnectionForcefullyClosedOnNewConnection
380+
href: iot-hub-troubleshoot-error-400027-connectionforcefullyclosedonnewconnection.md
381+
- name: 401003 IoTHubUnauthorized
382+
href: iot-hub-troubleshoot-error-401003-iothubunauthorized.md
383+
- name: 403002 IoTHubQuotaExceeded
384+
href: iot-hub-troubleshoot-error-403002-iothubquotaexceeded.md
385+
- name: 403004 DeviceMaximumQueueDepthExceeded
386+
href: iot-hub-troubleshoot-error-403004-devicemaximumqueuedepthexceeded.md
387+
- name: 403006 DeviceMaximumActiveFileUploadLimitExceeded
388+
href: iot-hub-troubleshoot-error-403006-devicemaximumactivefileuploadlimitexceeded.md
389+
- name: 404001 DeviceNotFound
390+
href: iot-hub-troubleshoot-error-404001-devicenotfound.md
391+
- name: 404003 DeviceNotOnline
392+
href: iot-hub-troubleshoot-error-404103-devicenotonline.md
393+
- name: 404104 DeviceConnectionClosedRemotely
394+
href: iot-hub-troubleshoot-error-404104-deviceconnectionclosedremotely.md
395+
- name: 409001 DeviceAlreadyExists
396+
href: iot-hub-troubleshoot-error-409001-devicealreadyexists.md
397+
- name: 409002 LinkCreationConflict
398+
href: iot-hub-troubleshoot-error-409002-linkcreationconflict.md
399+
- name: 412002 DeviceMessageLockLost
400+
href: iot-hub-troubleshoot-error-412002-devicemessagelocklost.md
401+
- name: 429001 ThrottlingException
402+
href: iot-hub-troubleshoot-error-429001-throttlingexception.md
403+
- name: 500xxx Internal errors
404+
href: iot-hub-troubleshoot-error-500xxx-internal-errors.md
405+
- name: 503003 PartitionNotFound
406+
href: iot-hub-troubleshoot-error-503003-partitionnotfound.md
407+
- name: 504101 GatewayTimeout
408+
href: iot-hub-troubleshoot-error-504101-gatewaytimeout.md
377409
- name: Reference
378410
items:
379411
- name: Azure CLI

articles/iot-hub/iot-hub-mqtt-support.md

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,24 @@ The following table contains links to code samples for each supported language a
4444
| [C#](https://github.com/Azure/azure-iot-sdk-csharp/tree/master/iothub/device/samples) |TransportType.Mqtt |
4545
| [Python](https://github.com/Azure/azure-iot-sdk-python/tree/master/azure-iot-device/samples) |Always supports MQTT by default |
4646

47+
### Default keep-alive timeout
48+
49+
In order to ensure a client/IoT Hub connection stays alive, both the service and the client regularly send a *keep-alive* ping to each other. The client using IoT SDK sends a keep-alive at the interval defined in this table below:
50+
51+
|Language |Default keep-alive interval |Configurable |
52+
|---------|---------|---------|
53+
|Node.js | 180 seconds | No |
54+
|Java | 230 seconds | No |
55+
|C | 240 seconds | [Yes](https://github.com/Azure/azure-iot-sdk-c/blob/master/doc/Iothub_sdk_options.md#mqtt-transport) |
56+
|C# | 300 seconds | [Yes](https://github.com/Azure/azure-iot-sdk-csharp/blob/master/iothub/device/src/Transport/Mqtt/MqttTransportSettings.cs#L89) |
57+
|Python (V2) | 60 seconds | No |
58+
59+
Following [MQTT spec](http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718081), IoT Hub's keep-alive ping interval is 1.5 times the client keep-alive value. However, IoT Hub limits the maximum server-side timeout to 29.45 minutes (1767 seconds) because all Azure services are bound to the Azure load balancer TCP idle timeout, which is 29.45 minutes.
60+
61+
For example, a device using the Java SDK sends the keep-alive ping then loses network connectivity. 230 seconds later, the device misses the keep-alive ping because it's offline. However, IoT Hub doesn't close the connection immediately - it waits another `(230 * 1.5) - 230 = 115` seconds before disconnecting the device with the error [404104 DeviceConnectionClosedRemotely](iot-hub-troubleshoot-error-404104-deviceconnectionclosedremotely.md).
62+
63+
The maximum client keep-alive value you can set is `1767 / 1.5 = 1177` seconds. Any traffic will reset the keep-alive. For example, a successful SAS token refresh resets the keep-alive.
64+
4765
### Migrating a device app from AMQP to MQTT
4866

4967
If you are using the [device SDKs](https://github.com/Azure/azure-iot-sdks), switching from using AMQP to MQTT requires changing the protocol parameter in the client initialization as stated previously.
@@ -147,7 +165,7 @@ Mosquitto_subscribe: to see events occurring in an Azure IoT hub.
147165

148166
Connecting to IoT Hub over MQTT using a module identity is similar to the device (described [above](#using-the-mqtt-protocol-directly-as-a-device)) but you need to use the following:
149167

150-
* Set the client id to `{device_id}/{module_id}`.
168+
* Set the client ID to `{device_id}/{module_id}`.
151169

152170
* If authenticating with username and password, set the username to `<hubname>.azure-devices.net/{device_id}/{module_id}/?api-version=2018-06-30` and use the SAS token associated with the module identity as your password.
153171

@@ -313,7 +331,7 @@ For more information, see [Device twins developer's guide](iot-hub-devguide-devi
313331

314332
## Update device twin's reported properties
315333

316-
To update reported properties, the device issues a request to IoT Hub via a publication over a designated MQTT topic. After processing the request, IoT Hub responds the success or failure status of the update operation via a publication to another topic. This topic can be subscribed by the device in order to notify it about the result of its twin update request. To implement this type of request/response interaction in MQTT, we leverage the notion of request id (`$rid`) provided initially by the device in its update request. This request id is also included in the response from IoT Hub to allow the device to correlate the response to its particular earlier request.
334+
To update reported properties, the device issues a request to IoT Hub via a publication over a designated MQTT topic. After processing the request, IoT Hub responds the success or failure status of the update operation via a publication to another topic. This topic can be subscribed by the device in order to notify it about the result of its twin update request. To implement this type of request/response interaction in MQTT, we leverage the notion of request ID (`$rid`) provided initially by the device in its update request. This request ID is also included in the response from IoT Hub to allow the device to correlate the response to its particular earlier request.
317335

318336
The following sequence describes how a device updates the reported properties in the device twin in IoT Hub:
319337

articles/iot-hub/iot-hub-troubleshoot-connectivity.md

Lines changed: 31 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -1,26 +1,26 @@
11
---
2-
title: Diagnose and troubleshoot disconnects with Azure IoT Hub
3-
description: Learn to diagnose and troubleshoot common errors with device connectivity for Azure IoT Hub
2+
title: Monitor and troubleshoot disconnects with Azure IoT Hub
3+
description: Learn to monitor and troubleshoot common errors with device connectivity for Azure IoT Hub
44
author: jlian
55
manager: briz
66
ms.service: iot-hub
77
services: iot-hub
88
ms.topic: conceptual
9-
ms.date: 07/19/2018
9+
ms.date: 01/30/2020
1010
ms.author: jlian
1111
# As an operator for Azure IoT Hub, I need to know how to find out when devices are disconnecting unexpectedly and troubleshoot resolve those issues right away
1212
---
13-
# Detect and troubleshoot disconnects with Azure IoT Hub
13+
# Monitor, diagnose, and troubleshoot disconnects with Azure IoT Hub
1414

15-
Connectivity issues for IoT devices can be difficult to troubleshoot because there are many possible points of failure. Device-side application logic, physical networks, protocols, hardware, and Azure IoT Hub can all cause problems. This article provides recommendations on how to detect and troubleshoot device connectivity issues from the cloud side (as opposed to device side).
15+
Connectivity issues for IoT devices can be difficult to troubleshoot because there are many possible points of failure. Application logic, physical networks, protocols, hardware, IoT Hub, and other cloud services can all cause problems. The ability to detect and pinpoint the source of an issue is critical. However, an IoT solution at scale could have thousands of devices, so it's not practical to check individual devices manually. To help you detect, diagnose, and troubleshoot these issues at scale, use the monitoring capabilities IoT Hub provides through Azure Monitor. These capabilities are limited to what IoT Hub can observe, so we also recommend that you follow monitoring best practices for your devices and other Azure services.
1616

1717
## Get alerts and error logs
1818

19-
Use Azure Monitor to get alerts and write logs when device connections drop.
19+
Use Azure Monitor to get alerts and write logs when devices disconnect.
2020

2121
### Turn on diagnostic logs
2222

23-
To log device connection events and errors, turn on diagnostics for IoT Hub.
23+
To log device connection events and errors, turn on diagnostics for IoT Hub. We recommend turning on these logs as early as possible, because if diagnostic logs aren't enabled, when device disconnects occur, you won't have any information to troubleshoot the problem with.
2424

2525
1. Sign in to the [Azure portal](https://portal.azure.com).
2626

@@ -38,9 +38,9 @@ To log device connection events and errors, turn on diagnostics for IoT Hub.
3838

3939
To learn more, see [Monitor the health of Azure IoT Hub and diagnose problems quickly](iot-hub-monitor-resource-health.md).
4040

41-
### Set up alerts for the _connected devices_ count metric
41+
### Set up alerts for device disconnect at scale
4242

43-
To get alerts when devices disconnect, configure alerts on the **connected devices (preview)** metric.
43+
To get alerts when devices disconnect, configure alerts on the **Connected devices (preview)** metric.
4444

4545
1. Sign in to the [Azure portal](https://portal.azure.com).
4646

@@ -52,43 +52,46 @@ To get alerts when devices disconnect, configure alerts on the **connected devic
5252

5353
5. Select **Add condition**, then select "Connected devices (preview)".
5454

55-
6. Finish setting up your desired thresholds and alerting options by following prompts.
55+
6. Set up threshold and alerting by following prompts.
5656

57-
To learn more, see [What are classic alerts in Microsoft Azure?](../azure-monitor/platform/alerts-overview.md).
57+
To learn more, see [What are alerts in Microsoft Azure?](../azure-monitor/platform/alerts-overview.md).
58+
59+
#### Detecting individual device disconnects
60+
61+
To detect *per-device* disconnects, such as when you need to know a factory just went offline, [configure device disconnect events with Event Grid](iot-hub-event-grid.md).
5862

5963
## Resolve connectivity errors
6064

61-
When you turn on diagnostic logs and alerts for connected devices, you get alerts when errors occur. This section describes how to resolve common issues when you receive an alert. The steps below assume you've set up Azure Monitor logs for your diagnostic logs.
65+
When you turn on diagnostic logs and alerts for connected devices, you get alerts when errors occur. This section describes how to look for common issues when you receive an alert. The steps below assume you've set up Azure Monitor logs for your diagnostic logs.
66+
67+
1. Sign in to the [Azure portal](https://portal.azure.com).
6268

63-
1. Go your workspace for **Log Analytics** in the Azure portal.
69+
1. Browse to your IoT hub.
6470

65-
2. Select **Log Search**.
71+
1. Select **Logs**.
6672

67-
3. To isolate connectivity error logs for IoT Hub, enter the following query and then select **Run**:
73+
1. To isolate connectivity error logs for IoT Hub, enter the following query and then select **Run**:
6874

6975
```kusto
70-
search *
71-
| where ( Type == "AzureDiagnostics" and ResourceType == "IOTHUBS")
72-
| where ( Category == "Connections" and Level == "Error")
76+
AzureDiagnostics
77+
| where ( ResourceType == "IOTHUBS" and Category == "Connections" and Level == "Error")
7378
```
7479
7580
1. If there are results, look for `OperationName`, `ResultType` (error code), and `ResultDescription` (error message) to get more detail on the error.
7681
7782
![Example of error log](./media/iot-hub-troubleshoot-connectivity/diag-logs.png)
7883
79-
2. Use this table to understand and resolve common errors.
84+
1. Follow the problem resolution guides for the most common errors:
8085
81-
| Error | Root cause | Resolution |
82-
|-------|------------|------------|
83-
| 404104 DeviceConnectionClosedRemotely | The connection was closed by the device, but IoT Hub doesn't know why. Common causes include MQTT/AMQP timeout and internet connectivity loss. | Make sure the device can connect to IoT Hub by [testing the connection](tutorial-connectivity.md). If the connection is fine, but the device disconnects intermittently, make sure to implement proper keep alive device logic for your choice of protocol (MQTT/AMPQ). |
84-
| 401003 IoTHubUnauthorized | IoT Hub couldn't authenticate the connection. | Make sure that the SAS or other security token you use isn't expired. [Azure IoT SDKs](iot-hub-devguide-sdks.md) automatically generate tokens without requiring special configuration. |
85-
| 409002 LinkCreationConflict | A device has more than one connection. When a new connection request comes for a device, IoT Hub closes the previous one with this error. | In the most common case, a device detects a disconnect and tries to reestablish the connection, but IoT Hub still considers the device connected. IoT Hub closes the previous connection and logs this error. This error usually appears as a side effect of a different, transient issue, so look for other errors in the logs to troubleshoot further. Otherwise, make sure to issue a new connection request only if the connection drops. |
86-
| 500001 ServerError | IoT Hub ran into a server-side issue. Most likely, the issue is transient. While the IoT Hub team works hard to maintain [the SLA](https://azure.microsoft.com/support/legal/sla/iot-hub/), small subsets of IoT Hub nodes can occasionally experience transient faults. When your device tries to connect to a node that's having issues, you receive this error. | To mitigate the transient fault, issue a retry from the device. To [automatically manage retries](iot-hub-reliability-features-in-sdks.md#connection-and-retry), make sure you use the latest version of the [Azure IoT SDKs](iot-hub-devguide-sdks.md).<br><br>For best practice on transient fault handling and retries, see [Transient fault handling](/azure/architecture/best-practices/transient-faults). <br><br>If the problem persists after retries, check [Resource Health](iot-hub-monitor-resource-health.md#use-azure-resource-health) and [Azure Status](https://azure.microsoft.com/status/history/) to see if IoT Hub has a known problem. If there's no known problems and the issue continues, [contact support](https://azure.microsoft.com/support/options/) for further investigation. |
87-
| 500008 GenericTimeout | IoT Hub couldn't complete the connection request before timing out. Like a 500001 ServerError, this error is likely transient. | Follow troubleshooting steps for a 500001 ServerError to the root cause and resolve this error.|
86+
- **[404104 DeviceConnectionClosedRemotely](iot-hub-troubleshoot-error-404104-deviceconnectionclosedremotely.md)**
87+
- **[401003 IoTHubUnauthorized](iot-hub-troubleshoot-error-401003-iothubunauthorized.md)**
88+
- **[409002 LinkCreationConflict](iot-hub-troubleshoot-error-409002-linkcreationconflict.md)**
89+
- **[500001 ServerError](iot-hub-troubleshoot-error-500xxx-internal-errors.md)**
90+
- **[500008 GenericTimeout](iot-hub-troubleshoot-error-500xxx-internal-errors.md)**
8891
89-
## Other steps to try
92+
## I tried the steps, but they didn't work
9093
91-
If the previous steps didn't help, you can try:
94+
If the previous steps didn't help, try:
9295
9396
* If you have access to the problematic devices, either physically or remotely (like SSH), follow the [device-side troubleshooting guide](https://github.com/Azure/azure-iot-sdk-node/wiki/Troubleshooting-Guide-Devices) to continue troubleshooting.
9497
Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
---
2+
title: Troubleshooting Azure IoT Hub error 400027 ConnectionForcefullyClosedOnNewConnection
3+
description: Understand how to fix error 400027 ConnectionForcefullyClosedOnNewConnection
4+
author: jlian
5+
manager: briz
6+
ms.service: iot-hub
7+
services: iot-hub
8+
ms.topic: troubleshooting
9+
ms.date: 01/30/2020
10+
ms.author: jlian
11+
# As a developer or operator for Azure IoT Hub, I want to resolve 400027 ConnectionForcefullyClosedOnNewConnection errors.
12+
---
13+
14+
# 400027 ConnectionForcefullyClosedOnNewConnection
15+
16+
This article describes the causes and solutions for **400027 ConnectionForcefullyClosedOnNewConnection** errors.
17+
18+
## Symptoms
19+
20+
Your device-to-cloud twin operation (such as read or patch reported properties) or direct method invocation fails with the error code **400027**.
21+
22+
## Cause
23+
24+
Another client created a new connection to IoT Hub using the same credentials, so IoT Hub closed the previous connection. IoT Hub doesn't allow more than one client to connect using the same set of credentials.
25+
26+
## Solution
27+
28+
Ensure that each client connects to IoT Hub using its own identity.

0 commit comments

Comments
 (0)