Discussion: #develop branch

[bertmelis]

I'm been doing some work on the https://github.com/marvinroger/async-mqtt-client/tree/develop branch lately.

It should solve lots of the current issues. The main improvement is that outgoing messages are queued and the memory is only freed after the TCP ack allowes to do so.

• ❗ Sending in QoS 0, 1 and 2 is supported for messages as large as memory permits.
• Receiving is supported in all three QoS levels. Messages are still directly passed to the API, in chuncks (you get an index, length and total). QoS 2 messages will be delivered "once completely", meaning that it is possible that you get parts of QoS 2 messages more than once. (OTA over MQTT is still possible)
• ❗ You don't have to wait for messages to be sent before disconnecting. The disconnect signal will be queued and the client will only disconnect once the previously queued messages are sent.

❗ : although the API remained unchanged, the exlamation mark denotes a change in behaviour.

I'm looking forward to feedback. I do like positive feedback 😄, but I'm more interested in the negative ones. I still have to go through the list of mandatory statements of the MQTT specification. TLS is also untested but there has been no change in that part of the code.

[bertmelis]

A sample sketch with the development code:

RAM:   [====      ]  36.5% (used 29936 bytes from 81920 bytes)
Flash: [===       ]  28.1% (used 292984 bytes from 1044464 bytes)

Master branch:

RAM:   [====      ]  36.7% (used 30104 bytes from 81920 bytes)
Flash: [===       ]  27.6% (used 288428 bytes from 1044464 bytes)

A similar library whose owner is quite patronizing (API differs a bit, so sketch is not 100% the same):

RAM:   [====      ]  36.5% (used 29880 bytes from 81920 bytes)
Flash: [===       ]  28.2% (used 294636 bytes from 1044464 bytes)

[luebbe]

I added https://github.com/marvinroger/async-mqtt-client.git#develop to the platformio.ini of my https://github.com/luebbe/homie-node-collection compiled and uploaded the sensor-nodes demo to a d1 mini.

This one contains a lot of nodes and many mqtt messages (84 topics) have to be sent upon startup. I'm running mosquitto on a raspberry Pi 1 and MQTT Explorer on Windows for monitoring.

The only notable difference when the device is reset is that it seems to take longer until all the initial messages have been received by MQTT explorer. I guess this is to be expected since they are queued up. Perhaps before messages were lost when the buffer was full.

I have also added "freeheap" to $stats in my fork of homie-esp2866, so I can monitor if anything runs away there.

The graph looks like this (around 15 minutes of history)

There is a significant drop directly after the start (expected) and then it is stable at around 30KB. You can also see the drop at the five minute intervals when sensor data is sent.

It is also interesting to see that when the $stats are sent, they now trickle in over the course of 1-3 seconds. With the previous version they came in the same second.

[bertmelis]

In which qos are you sending?

One notable difference in sending qos > 0 is that "the next message" is only sent when the previous has been acknowledged. I didn't want the logic to be too complicated to ensure proper message ordering (as per mqtt spec). In qos 0 this should not matter, but I might be wrong. I changed the logic a few times.

[luebbe]

Homie default. I didn't change anything. MQTT Explorer says qos 0?

[bertmelis]

Homie default. I didn't change anything. MQTT Explorer says qos 0?

qos subscribing might differ from qos publishing.

Stats are sent qos 1 (https://github.com/homieiot/homie-esp8266/blob/9cd83972f27b394eab8a5e3e2baef20eea1b5408/src/Homie/Boot/BootNormal.cpp#L180). So yes, this is slower than before.

[luebbe]

The homie convention states https://homieiot.github.io/specification/#qos-and-retained-messages that it uses qos 1, but I have to check the source to confirm. I don't know from where MQTT Explorer gets the qos 0 info.

[luebbe]

qos subscribing might differ from qos publishing.

guess so

Stats are sent qos 1 (https://github.com/homieiot/homie-esp8266/blob/9cd83972f27b394eab8a5e3e2baef20eea1b5408/src/Homie/Boot/BootNormal.cpp#L180). So yes, this is slower than before.

Fine by me. Better slow & certain than quick & dirty

I'll leave my dev board and the monitor running at least until I go to sleep, but I'm already pretty confident that there's no memory leak.

[bertmelis]

I'll anyhow add speed up to the wishlist.

[luebbe]

Just FYI, the sensor-demo has 84 subtopics. This is the second largest amount of my homie based applications.

[kleini]

For me this results in a crash. Please see attached serial output from my relays-device. I am using SSL encrypted connection.
crash.log

[bertmelis]

Do you run your ESP on 160MHz? A soft watchdog reset on the sys stack during connecting is probably caused by the handshake taking too much time. Off course I can be wrong.

Edit: for the record, I am setting up TLS on my broker to test myself.

[kleini]

It is a standard ESP12-F. This time with decoded crash:
crash.log

[bertmelis]

gotta look into that. But still, it's a watchdog reset, so something is taking too long.
I workaround could be to not send immediately after connecting but on the first onPoll. You could try with commenting out line 382.
I don't know how long a broker waits to receive a CONNECT before disconnecting.

[kleini]

I think, the connection to the MQTT broker does not work at all:

mosquitto    | 1612816096: New connection from 192.168.168.196 on port 1883.
mosquitto    | 1612816099: Client <unknown> has exceeded timeout, disconnecting.
mosquitto    | 1612816099: Socket error on client <unknown>, disconnecting.

And with some debug enabled, I see

✔ Wi-Fi connected, IP: 192.168.168.196
Triggering WIFI_CONNECTED event...
↕ Attempting to connect to MQTT...
CONNECTING
TCP conn, MQTT CONNECT
new front #1
snd #1: 165/100

--------------- CUT HERE FOR EXCEPTION DECODER ---------------

Soft WDT reset

So, is somehow the connect string not sent completely? Will try to sniff network traffic.