Adding MQTT protocol as a core feature to Pico W and Pyboard D firmware

[miketeachman]

The micropython-lib packages umqtt.simple and umqtt.robust are included in the manifests for the ESP32 and ESP8266 ports, but are missing from other WiFi enabled boards such as the Pico W and the PyBoard D series.

Is there a reason for that?

While it's possible to copy these MQTT packages to a board using tools such as rshell and the new awesome mip, it would lower the barrier to using the MQTT protocol if these packages were included in the release firmware.

Thoughts?

I'll volunteer to submit a PR if there is a favourable response to this idea.

For reference, adding the two MQTT packages adds 3640 bytes (+0.52%) to the Pico W binary and 3624 bytes (+0.34%) to the Pyboard D SF2 binary.

[jimmo]

Thanks @miketeachman -- I had this on my TODO list to investigate. A friend just got into zigbee2mqtt, and I have a DIY zigbee/HA system at home that pre-dates zigbee2mqtt but it would be nice to replace with zigbee2mqtt and so this has been on my radar. I do really hope we can encourage more tutorials to be written that involve using things like MQTT rather than DIY web servers etc.

Here's a bit of a brain dump of some things I wanted to understand:

• Why do we have a robust and simple version. Don't you always want the robust version? (This may be a very dumb question, I tried to skim the README but the answer wasn't obvious. I have exactly zero context about MicroPython+MQTT.). (I noticed that I made a mistake writing the manifest.py for the robust version, it should require() the simple version).
• Other people have implemented mqtt clients (e.g. https://github.com/peterhinch/micropython-mqtt @peterhinch . I'm sure I've seen others). What do they do better?
• Should we focus on an asyncio version instead and deprecate the one in micropython-lib? (FWIW, I think Peter's is async)

Counterpoint... it's now really easy to install micropython-lib packages again via mip / mpremote mip? Eventually we'll have .mpy-files-in-ROM too (#8381). Maybe less need to freeze things.

[peterhinch]

@jimmo In a nutshell. The mqtt_as client I developed with @kevinkk525 is async.

The official clients were designed on the assumption of guaranteed packet delivery on an ideal network. This is not reality on a radio link: WiFi suffers from outages which cause the official clients to fail. In my testing they only held up for about 30 minutes. There are numerous other issues and limitations with the official clients which are covered in the docs. In my opinion they are best viewed as demos.

[scruss]

Counterpoint... it's now really easy to install micropython-lib packages again via mip / mpremote mip? Eventually we'll have .mpy-files-in-ROM too (#8381). Maybe less need to freeze things.

Aside: are there docs / announcements about mip? It's something potentially useful that's only being mentioned in passing here. I also ask because the many new MicroPython / Pico W users are just getting on the upip train, and the docs / glossary say that it's obsolete

[jimmo]

It'll be announced as part of v1.20. I need to write docs for the top-level README in micropython-lib too.

and the docs / glossary say that it's obsolete

You're looking at the docs for latest. But I agree this is super confusing. See https://github.com/orgs/micropython/discussions/9490#discussioncomment-3802133 I think we should make it clearer that the docs are for the "next" release, not the current stable.

[peterhinch]

@jimmo To expand on this a little, a usable official MQTT library would be wonderful but there are issues to consider. I would be happy to help with defining the task. A few salient points regarding any proposed solution:

• It must be able to cope with arbitrarily bad RSSI values including outages of long duration. The client might be mounted on a mobile platform which moves in and out of radio range.
• It should be asynchronous. Firstly blocking can be for long periods, especially in the above case. Secondly the protocol requires keep-alive packets to be sent: this ideally needs to happen as a background task.
• There is a significant maintenance burden. Testing resilience is very time consuming and needs to be repeated if extending to a platform with new network architecture. I know itt shouldn't - but in practice platforms can differ in how they respond to poor RSSI.

I am not advocating for mqtt_as here which has limitations including:

• A solution should be capable of using any networking solution that supports BSD sockets. Alas mqtt_as is WiFi specific.
• qos==2 support is currently missing.
• Protocol support is 3.1.1. Version 5 exists.

[jimmo]

Thanks Peter this is really helpful!

I will need to do so experimentation and spend some time using it.

For a non-MicroPython project I briefly used https://pypi.org/project/asyncio-mqtt/ and that seems to be getting traction in the CPython world. It might be worth looking at API compatibility with that for a possible micropython-lib candidate solution (if feasible).

[karfas]

@peterhinch: While I agree with your other points, this one

* It must be able to cope with arbitrarily bad RSSI values including outages of long duration. The client might be mounted on a mobile platform which moves in and out of radio range.

is something that needs some discussion, I think.
In my opinion, global state like the connection to an AP needs to be handled by the application, not some communication/protocol library.
OK, it makes things simpler for the user, but this also reduces flexibility. An application for a mobile client might have better ways to cope with WLAN disconnects than trying to reconnect to the AP/MQTT server a few times.

[miketeachman]

Great discussion and some excellent questions and points! ... but, I was hoping to limit the scope of this discussion to the pros and cons of simply making the WiFi capable boards, such as Pico W and Pyboard D boards, consistent with the ESP boards wrt to the existing MQTT implementation (umqtt.simple and umqtt.robust in micropython-lib).

I think the topic of improving the existing MQTT implementation offered in micropython-lib is a much bigger topic, that deserves its own Discussion thread(s).

Looking at consistency across ports: Is it important in MicroPython to have consistency when feasible? e.g.

• all WiFi capable ports/boards support umqtt.simple and umqtt.robust as core features

OR

• no WiFi capable ports/boards support umqtt.simple and umqtt.robust as core features, and these packages need to be copied to a board using methods like rshell or mip/mpremote mip.

For me, consistency across ports seems preferred. Consistency makes it easier for users of MicroPython. For example, I support a tutorial repo containing examples describing how to use MQTT with the Thingspeak cloud database. For the Pyboard D, the tutorial requires special instructions on how to copy MQTT libraries to the board (soon to be updated with mip instructions). This one case is not a big deal, but inconsistencies add up, and together, they tend to denigrate the MicroPython user experience.

[jimmo]

I definitely agree on the consistency, and yes it's terrible for the user experience! (I am right this minute working on unifying the network.WLAN API across all ports... 😵 )

But I am worried that by freezing the existing umqtt libraries into the other boards we lock ourselves into that. It's basically impossible to remove something once we start freezing, and I don't want to end up in a situation where we need start freezing more than one mqtt library, or similar freeze uqmtt while "officially" recommending some other mqtt library.

I just looked up the history of adding it to the ESP32 port... #3839 some familiar faces :) I can't find the history for ESP8266 because it happened automatically in building the official releases, but I suspect it possibly goes back to the original release of the ESP8266 (i.e. kickstarter).

I think my personal preference is to make it easy to install mqtt on any board (somewhat biased because that's what I've just spent the past couple of weeks working on!). Your point about the special instructions for pybd (and now pico w) makes sense, although to be fair, if your tutorial switched from ampy to mpremote for all steps (I think you should do this anyway of course! :p ), then one additional "you might need to install mqtt, mpremote mip install umqtt.robust " is not a huge burden, and maybe a good opportunity to teach installation of lib packages anyway. (In other words, if your tutorial was oriented around using the mass storage interface to copy files, then yes this would be a whole additional burden to now have to install mpremote etc, or worse manually obtain the files and copy them, but seeing as you already have the steps to install and use a tool...).

Sorry, I realise I'm making a general point out of the specific example of your tutorial there... I think what I'm really trying to say is that the inconsistency here is ESP8266/ESP32 having it frozen, not that the other ports don't, and that the "normal" thing to do is to install needed dependencies from the library. Similarly I would argue that we should stop freezing many of the other things we freeze by default. Of course it's not that simple because of the other RAM-usage benefits of freezing...

Another option is that we could freeze umqtt.simple/robust for now with the understanding that they may be removed in the future in favour of a better alternative (i.e. MicroPython 2.0).

[miketeachman]

Thanks for taking the time to explain the direction that MicroPython will be taking for micropython-lib packages (e.g. install when needed). Given this direction and the new feature mpremote mip install I can see that adding the MQTT libs to the manifest becomes questionable.