nanoFramework.M2Mqtt bug on any NF build newer than 1.12.4.x

[JT-50]

Description:
I am experiencing a silent lock-up when using nanoFramework.M2Mqtt in a long-running application. The client eventually stops processing messages, even though the connection remains active and the broker continues to function correctly.

This issue persists across recent versions, including nanoFramework 1.14, 1.15, and the 2.0.x previews.

Environment:

Hardware: ESP32-REV3, ESP32-S3 WROOM 16r8, ESP32-C3

Last Known Working Build: 1.12.4.372 (with Mqtt library 5.1.199).
Failing M2MQTT Builds: All versions after 5.1.199.

Reproduction Case:
I have isolated the issue to a simple loop. The client successfully sends messages and receives PUBACK frames, but after a few hours of operation, the internal message processing halts entirely.

using nanoFramework.M2Mqtt;
using nanoFramework.M2Mqtt.Messages;
using System.Text;
using System.Threading;
using System;

public class Program
{
    // Point to your broker
    static MqttClient client = new MqttClient("**broker.dns or IP address**");

    public static void Main()
    {
        client.Connect("test-app");
        int count = 0;

        while (true)
        {
            try
            {
                client.Publish("test/topic", Encoding.UTF8.GetBytes($"Msg: {++count}"), null, null, MqttQoSLevel.AtLeastOnce, false);
                Console.WriteLine($"Sent: {count}");
            }
            catch (Exception ex)
            {
                Console.WriteLine($"Error: {ex.Message}");
            }
            Thread.Sleep(1000);
        }
    }
}

Additional Notes:
I have attempted various code workarounds, but the library appears to deadlock during internal inflight queue management. The fact that this works on build 1.12.4.372 but fails on all subsequent versions suggests a regression in the library's threading or socket state management.

[josesimoes]

Pinging @Ellerbach
Any ideas on this?

[alberk8]

It is directly related to this . When the network goes out (like the wifi goes away) the network layer will recover. The issue is when the server side disconnect then the hanging will occur. When the server disconnect on the low level I think it sent a FIN or RST, that is when the NF native goes into a infinite loop. This does not happen before the ESP IDF v5.4.2

[Ellerbach]

@alberk8 did you get a chance to try what's in my PR? nanoframework/nf-interpreter#3261

[JT-50]

I don't think this is the same issue.

The bug described by alberk8 involves the network dropping — WiFi going away, the server sending FIN/RST, an ESP IDF v5.4.2 regression in the native layer.

My setup is a stable LAN with the broker always up. I just logged 1500+ consecutive successful publishes with no disconnections. The network is not the problem. It just suddenly hangs, no further CPU activity.

The hang I'm seeing occurs under normal operating conditions with no connection loss involved. PR #3261 and the native socket fix don't appear to apply to my situation.

Please run the sample app and let it go for hours, it should fail within 30 mins to 3 hours.

[alberk8]

is the MQTT server under your control? IF so, can you check if there is a server disconnect or restart when the hang happens? if the hang occur, can you still ping the IP of the ESP32?

[JT-50]

there is no disconnect, no congestion no wifi drop.

the sample code provided will run 'forever' on 1.12.4.372, but not more than a few hours any of the build in the 1.14.x or 1.15.x or 2.0.x

Whether the mqtt publishes are set to 1 minute or 1 sec has no effect on how soon it crashes.

---

I know AI can make lots of mistakes, but this is it's explanation of what's wrong:

Inside MqttClient.cs, look for the event handler that processes the incoming PUBACK message. It typically looks like this:

C#
private void HandlePubAck(MqttMessage msg)
{
    // 1. The code receives the PUBACK packet
    // 2. It attempts to remove the message from the inflight queue
    
    lock (this._inflightQueue) // <--- THIS LOCK
    {
        // ... logic to find and remove message ...
        
        // 3. It tries to signal the waiting thread that the message is sent
        _inflightQueue.Remove(messageId);
        _inflightQueueSignal.Set(); 
    }
}

[alberk8]

if you are compiling you own firmware, just to confirm. Put CLR_Debug::Printf("Outer Loop Helper\n"); in line 788 and 791 of this function. The message will show up in VS Debug or serial console.

[JT-50]

My current development environment isn't set up to compile the C++ firmware from source.

If you can provide a pre-compiled 1.15.x binary with those Printf statements included, I can flash it immediately and report back with the serial output.

[Ellerbach]

Note: we have dev containers ready to build any firmware ;-) it makes things easier as well!

[JT-50]

Update — PR #3261 confirmed to fix native freeze, but exposes managed code bug
After testing AlbertK's custom 1.15 firmware with PR #3261 applied:
Good news: The hard freeze is gone. The device no longer locks up silently. PR #3261 works.
New finding: Instead of freezing, two exceptions are now thrown from internal M2Mqtt threads:

System.Net.Sockets.SocketException - CLR_E_FAIL (7)
nanoFramework.M2Mqtt.MqttClient::ProcessInflightThread

System.Net.Sockets.SocketException - CLR_E_FAIL (5)
nanoFramework.M2Mqtt.MqttClient::KeepAliveThread

The problem: Both ProcessInflightThread and KeepAliveThread have no exception handling. When the socket throws, both threads die silently. IsConnected remains true even though the connection is dead. Client code has no way to detect the failure and reconnect.
Observed sequence:

Internal threads throw and die
IsConnected stays true
TCP connection drops almost instantly, ~50ms — broker logs NotClean disconnect
Device goes silent

Fix needed in M2Mqtt managed code: ProcessInflightThread and KeepAliveThread need try/catch blocks that set IsConnected = false on socket exception, so client code can detect and recover.
No workaround in the meantime.

[JT-50]

The fix needed in M2Mqtt:

catch (Exception)  // catch ALL exceptions, not just MqttCommunicationException
{
    IsConnected = false;  // set immediately, don't wait for DispatchEventThread
    OnConnectionClosing();
}

but the socket shouldnt be dying to begin with, but at least we catch it.

[josesimoes]

@JT-50 wow! Nice detective work you've been doing. From your description it makes perfect sense (I haven't checked the code yet). Can you please go ahead and submit a PR with those fixes so it can be tested? On the socket, could this be related with setting Keep Alive property? Just wondering, I haven't check it either, but I've seen issues in the past with this being (and not being set).

[JT-50]

I dont know enough of what I'm doing to do a PR; even then my suggestion is just how to handle the exception.
The real problem is the dead socket or whatever is happening at the CLR/IDF level.

[JT-50]

Worth noting that Arduino runs on the same IDF 5.x with the same closed source WiFi drivers without this issue. This suggests the problem is in how nanoFramework's native socket layer handles the responses from the IDF layer, which is what PR #3261 addresses -- but does not solve.

[JT-50]

But at least with your fix on the MQTT side, you managed to reconnect, right?

No it does not recover, the exception is revealed, but the device still locks up ... (and remains pingable)

[alberk8]

I have added the dispose and reconnect in the exception and it always reconnect back.

 while (true)
 {

     count++;
     Thread.Sleep(100);
     msg = Encoding.UTF8.GetBytes($"Count: {count} ");

     if (WifiNetworkHelper.Status == NetworkHelperStatus.NetworkIsReady)
     {
         try
         {
             if (!mqttClient.IsConnected)
             {
                 
                 var reason = mqttClient.Connect(_clientID);
                 Debug.WriteLine("Connection Reason: " + reason.ToString());
             }
             else
             {
                 var res = mqttClient.Publish("publish/topic", msg);
                 Debug.WriteLine($"Publish ID: {res}");
             }
         }
         catch (Exception ex)
         {
             Debug.WriteLine("Mqtt Error: " + ex.Message);
             mqttClient.Dispose();
             mqttClient = new MqttClient(_broker);
         }

         Thread.Sleep(100);
     }
     else
     {
         Debug.WriteLine("Not Connected: " + count);
     }

[JT-50]

We have different issues.
They may be linked, but they have different symptoms. Your problem requires your broker to go offline. Mine does not.

I cannot recover from:

System.Net.Sockets.SocketException - CLR_E_FAIL (7)
nanoFramework.M2Mqtt.MqttClient::ProcessInflightThread

System.Net.Sockets.SocketException - CLR_E_FAIL (5)
nanoFramework.M2Mqtt.MqttClient::KeepAliveThread

[alberk8]

Worth noting that Arduino runs on the same IDF 5.x with the same closed source WiFi drivers without this issue. This suggests the problem is in how nanoFramework's native socket layer handles the responses from the IDF layer, which is what PR #3261 addresses -- but does not solve.

Well you can try help to debug the native side to find out what is the issue. Also the latest Arduino v3.3,7 uses IDF is v5.5.0 nF is on IDF 5.5.3

[JT-50]

For reference: 4 devices on the same bench, same broker, same network, running 1.12.4 have been running uninterrupted for weeks. This is not an inherently unreliable network — it is a regression since 1.14+.

[alberk8]

The issue we now know is due to ESP IDF version since the native nF code from 1.12 to 1.15 of the networking has not changed. Btw the very low level of the hardware interface code for WIFI and Bluetooth is a black box (close source).