Espnow module general discussion topic

[glenn20]

I am posting this as a place holder for general discussions or questions for new or old users of the micropython espnow module.

In particular, any questions or suggestions on documentation, usage or api are sought and welcome here.

Of course, if you have any specifc Issues to post or proposed changes, you can post Issues and Pull Requests to the micropython repo.

Resources:

• Documentation on espnow module at: https://docs.micropython.org/en/latest/library/espnow.html
• Precompiled images are available at: https://github.com/glenn20/micropython-espnow-images (until micropython 1.21 is released). The espnow module is also available in micropython nightly builds: https://micropython.org/download/.
• Prior discussions at:
  • Past and current Issues on the precompiled images and dev branch
    • https://github.com/glenn20/micropython-espnow-images/issues?q=is%3Aissue+
    • https://github.com/glenn20/micropython/issues?q=is%3Aissue+
  • Discussions on micropython espnow Pull Request esp32: ESP-NOW support for ESP32 and ESP8266 #6515
  • ESPNow topic on the now archived Micropython Forum.

[glenn20]

Missing aioespnow and WLAN.config(pm=XX)

Now that PR #6515 has been merged into micropython master, the aioespnow module has been moved to micropython-lib. Once this is incorporated into the micropython-lib submodule of micropython, it will be frozen into micropython images by default. In the meantime, you can install it onto your modules with: mpremote mip install aioespnow.

The precompiled images at https://github.com/glenn20/micropython-espnow-images included the WLAN.config(pm=WLAN.PM_NONE) option to help make espnow work reliably while the module is connected to a wifi access point. WLAN.config(pm=xxx) has been broken out into its own PR (#8993) and work is progressing on getting it merged. Till then, there are other work-arounds described at https://docs.micropython.org/en/latest/library/espnow.html#espnow-and-wifi-operation.

[glenn20]

Update: PR #8993 has now been merged into the main repo and sta.config(pm=sta.PM_NONE) can be used to disable the power saving mode after connecting to a wifi access point - improving the reliability of receiving ESP-NOW messages.

[bulletmark]

I've discovered ESPNow only since it was committed to master branch here in the last few days. I have set up a few devices with it and have the following minor(?) comments/suggestions.

1. The use with security options is not clear (although it is not particularly clear when I check the Espressif docs either). I note for the simple/unsecured case where you send a message to a receiver, then the receiver does not need to do an add_peer() for that device and the message is received regardless. For the receiver, an add_peer() is only required when you want to send messages back. However, if the sender sets an lmk option on that peer receiver (i.e. the message is now secure) then is now necessary that the receiver do an add_peer() (with the same lmk of course) before the message can be received. This best be made clear in the docs.

2. The encrypt option in the add_peer() function definition seems to me to be redundant. If no lmk=value is set in the call then encrpyt is defaults to False but if you pass lmk=value in the call then encrypt automatically is True. There are no other sensible combinations, i.e,. if you want encryption then pass a lmk value or if you don't want encryption then don't pass lmk. Thus the argument encrypt should be removed from the signature (and given it is the last argument this should be ok to do).

3. The Espressif docs use the term peer_address whereas the MP docs use the term mac. I know the address happens to be implemented as a raw mac address but I think the MP docs would read better if mac was labelled more generically as address or peer_address throughout the docs. Also, sometimes you want to determine/log your own sender address so I think it would be much more convenient and easier for users if this address was available from the module, i.e. as a function e.address() or even an attribute e.address? I know you can get this fairly trivially from other modules, e.g. machine.unique_id() or network.WLAN(network.STA_IF).config('mac') but why make the user go to another module just to get this address which is a fundamental attribute for ESPNow?

[glenn20]

1. The use with security options is not clear .... This best be made clear in the docs.

Good point. Thanks for the feedback. I have updated the docs in my dev branch to try to make this more explicit:

• Encrypted messages: To receive an encrypted message, the receiving device must first register the sender and use the same encryption keys as the sender (PMK and LMK) (see set_pmk() and add_peer().

2. The encrypt option in the add_peer() function definition seems to me to be redundant. .... Thus the argument encrypt should be removed from the signature (and given it is the last argument this should be ok to do).

I thought that too, till I found use cases where I want to set the encryption key at app startup (for key management), but only enable the encryption after receiving an unencrypted message from the peer and then be able to toggle the encryption on and off (with e.mod_peer(mac, encrypt=True/False) without having to reset the key). Also, it is simpler from a code size and conceptually to maintain symmetry between the parameters of add_peer() and mod_peer(). I also thought it simplest to just expose all the low-level parameters from esp_now_peer_info struct.

Finally, it is an optional argument and defaults to True if lmk is set, else False, so you can ignore it in add_peer(). I've updated the docs to reflect that.

3. The Espressif docs use the term peer_address whereas the MP docs use the term mac. .... as a function e.address() or even an attribute e.address? I know you can get this fairly trivially from other modules, e.g. machine.unique_id() or network.WLAN(network.STA_IF).config('mac') but why make the user go to another module just to get this address which is a fundamental attribute for ESPNow?

Yeah, I get that (e.address), however in a constrained environment like micropython, I think the lead devs would (rightly) resist methods that duplicate functionality elsewhere (eg. to get the mac address). I've resisted that in a few areas where it would be convenient to control or access some WLAN functionality from the ESPNow class.

Early drafts of the docs used language similar to the Espressif docs (peer and peer address), but I found it was less confusing to the users to remind them everywhere that this is just a MAC address. (I felt it a bit pointless to pretend the peer address is some opaque key, as the Espressif docs do in places - I also found some users had their own concepts of what a "peer" is that led to confusion). It also helps prompt people that if they change from STA_IF to AP_IF the peer (MAC) address will change as well.

Thanks again for this feedback and experience of new eyes - it's helped me improve and clarify issues in the docs.

[glenn20]

Oh - and the docs of the dev branch (espnow-g20-dev) will be updated at: https://micropython-glenn20.readthedocs.io/en/latest/library/espnow.html. Usually takes up to an hour to update.

[bulletmark]

I still don't see the point of the encrypt argument. It just seems redundant and it will initially confuse users like me that expect a succinct API. If you want to disable encryption temporarily just mod_peer(mac, lmk=''). That works as you already say in your docs. However I do see that this redundant design came from the Espressif C API so I guess it is a very fair point that you want to keep the MP API the same.

I didn't realize that the STA_IF mac address is different to the AP_IF address! (Although makes sense when I think about it now given you can have both interfaces active). Turns out I had this working by fluke since I was using machine.unique_id() which happens to be the same as the STA_IF address but I really should not use that.

Sorry but a question - I think a "professional" design of an ESPNow network should always use encryption but it seems a significant handicap that the code would require knowledge of specific mac addresses. Have you any suggestion on how best to securely "bootstrap" a network of devices using only keys?

[glenn20]

I didn't realize that the STA_IF mac address is different to the AP_IF address!

And nor did I when I started working with espnow - caused no end of confusion when devices can be setup to send messages to a peer from one interface, but receive messages on the other interface.

Sorry but a question - I think a "professional" design of an ESPNow network should always use encryption but it seems a significant handicap that the code would require knowledge of specific mac addresses.

It is also complicated by the fact you can only register at most 6 encrypted peers. I use some app logic that automatically adds and removes devices to the pool of encrypted peers.

Another limitation of the espnow security model that is often criticised is that the app on the receiving device can't distinguish if a packet it has received was encrypted by the sender or not - it just sees the plain text message in both cases. You have to account for that in your security model.

Have you any suggestion on how best to securely "bootstrap" a network of devices using only keys?

For my own use I have an espnow discovery framework where:

• A new device sends a broadcast that it is looking for a service by name (eg. "mqtt proxy").
• Any devices that are prepared to provide that service respond
• The client chooses the candidate with the strongest RSSI.
• The server and client then enable encryption using their PMK and LMK keys (either provided in app code, filesystem or NVS) and exchange messages to werify each other.
• If they have the same keys, they accept each other.
• The client saves the server identity (either to NVS or the filesystem).
• It can reinitate discovery if it loses communication with the server.

I've occasionally mentioned my intention to package that up and release it. I'll try to do that soon.

I still don't see the point of the encrypt argument. It just seems redundant and it will initially confuse users like me that expect a succinct API. If you want to disable encryption temporarily just mod_peer(mac, lmk='').

But, if you want to re-enable the encryption, that part of the app logic needs to know the key, ie. e.mod_peer(mac, lmk="xxxxxxxx"), instead of e.mod_peer(encrypt=True). If you use the encrypt parameter, you can disable/enable encryption without removing the key from the Espressif ESPNOW software stack. For some use cases that is really useful.

[davefes]

Another useful resource that you could link to is:
https://github.com/glenn20/micropython-espnow-utils
especially wifi.py

Excellent work getting ESPNow into the "main" images.

[davefes]

Seeing as ESPNow is supported for two platforms I wondered if a new category called:
Ports - ESPNow
is warranted?

[Wind-stormger]

Hi, I want to make some changes based on the routine at the beginning of the document, so that the receiver receives the message and sends a message back to the sender, the code is as follows, it can run, the receiver can receive the message, but the sender can not get the message it sends back.

sender:

import network
import espnow
import wifi

# A WLAN interface must be active to send()/recv()
sta = network.WLAN(network.STA_IF)
sta.active(True)
sta.disconnect()   # Because ESP8266 auto-connects to last Access Point
wifi.status()

e = espnow.ESPNow()
e.active(True)
peer = b'\xbb\xbb\xbb\xbb\xbb\xbb'   # MAC address of peer's wifi interface
e.add_peer(peer)      # Must add_peer() before send()

e.send(peer, "Starting...")
for i in range(100):
    e.send(peer, str(i)*20, True)
e.send(peer, b'end')
while True:
    host, msg = e.recv()
    if msg:             # msg == None if timeout in recv()
        print(host, msg)
        break

receiver:

import network
import espnow
import wifi

# A WLAN interface must be active to send()/recv()
sta = network.WLAN(network.STA_IF)
sta.active(True)
sta.disconnect()   # Because ESP8266 auto-connects to last Access Point
wifi.status()

e = espnow.ESPNow()
e.active(True)

# peer = b'\xbb\xbb\xbb\xbb\xbb\xbb'   # MAC address of peer's wifi interface
# e.add_peer(peer)      # Must add_peer() before send()
while True:
    host, msg = e.recv()
    if msg:             # msg == None if timeout in recv()
        print(host, msg)
        
        if msg == b'end':
            try:
                e.add_peer(host)
            except OSError:
                pass
            sended = e.send(host, "All received.")
            print(sended)

[Wind-stormger]

Thanks for the test, I'll try wifi.py you mentioned earlier.

[Wind-stormger]

@glenn20 I updated the code again,use wifi.status() and print(sended) after sended = e.send(host, "All received.").

The receiver gets False in the final print.

This means that the message was not sent successfully, and the sender did not receive the message.

I have tested your code and it works for me on ESP32 and ESP32S3 devices, which leads me to suspect some issue with the wifi config on the sender device.

Does this involve the configuration file before firmware compilation?

[glenn20]

I updated the code again,use wifi.status() and print(sended) after sended = e.send(host, "All received.").

And what did wifi.status() report? Were there any obvious issues?

The receiver gets False in the final print.

This means that the message was not sent successfully, and the sender did not receive the message.

This usually means the message was sent, but the receiver did not send back a packet acknowledging receipt.

I have tested your code and it works for me on ESP32 and ESP32S3 devices, which leads me to suspect some issue with the wifi config on the sender device.

Does this involve the configuration file before firmware compilation?

I'm not sure what this means.

One other obvious test is to swap the sender and receiver - does that change anything?

It is not that unusual for espnow messages to occasionally be lost. Wifi is subject to interference, but it is unusual for them to be lost in such a repeatable and reproducible manner.

[Wind-stormger]

One other obvious test is to swap the sender and receiver - does that change anything?

I found the key problem! Thanks for the reminder!

After the exchange, the original sender becomes the receiver and cannot receive any messages; it appears that it can only send messages but not receive messages.

When I take out another board, I can send and receive messages between the two boards.

[Wind-stormger]

@glenn20 Later I will try my best to find the fault point of the board that can not receive messages.

Generally speaking, it is a very strange thing that the party that can send data cannot receive data.

[bulletmark]

@glenn20 FYI, I found an odd issue using aioespnow but I think it is a MP bug so have raised a bug there.

[glenn20]

Thanks - that was useful to know.

[bulletmark]

Actually it seemed to me that you already had discovered this yourself(!). When I first read the aioespnow documentation I saw that you were passing e in all the examples there and I remember wondering to myself why that was being done as it looks unnecessary. Turns out it is actually required atm due to that MP bug, at least if you use async for to iterate over e.

[glenn20]

Actually not ;). I am just one of those who shudder every time I use a global variable (ok - slight exaggeration :). Necessary or convenient sometimes, but I usually avoid them. I violate that in a number of the examples in the docs to make them simpler for new users.

[Wind-stormger]

I express my gratitude to @glenn20 and @bulletmark for their previous assistance.
I am eager to continue discussing some issues on this page, specifically pertaining to aioespnow.
This is the code I have slightly modified based on https://docs.micropython.org/en/latest/library/espnow.html#supporting-asyncio.

import network
import aioespnow
import asyncio
import binascii

# A WLAN interface must be active to send()/recv()
sta = network.WLAN(network.STA_IF)  
sta.active(True)
mac = binascii.hexlify(sta.config('mac'),":").decode()
print("Current_MAC:", mac) # Print current device's Mac Address
sta.config(channel=3)
print("Current_Channel:", sta.config('channel'))

e = aioespnow.AIOESPNow()  # Returns AIOESPNow enhanced with async support
e.active(True)
peer = b'\xbb\xbb\xbb\xbb\xbb\xbb'
e.add_peer(peer)

# Send a periodic ping to a peer
async def heartbeat(e, peer, period=30):
    while True:
        resp = await e.asend(peer, b'ping',sync=False)
        if not resp:
            print("Heartbeat: peer not responding:", peer)
        else:
            print(resp)
            print("Heartbeat: ping", peer)
        await asyncio.sleep(period)

# Echo any received messages back to the sender
async def echo_server(e):
    i = 0
    async for mac, msg in e:
        i += 1
        print(mac)
        mac_decode = binascii.hexlify(mac,":").decode()
        print(mac_decode, "Echo:", msg.decode(), i)
        try:
            await e.asend(mac, msg)
        except OSError as err:
            if len(err.args) > 1 and err.args[1] == 'ESP_ERR_ESPNOW_NOT_FOUND':
                e.add_peer(mac)
                print("add mac:",binascii.hexlify(mac,":").decode())
                await e.asend(mac, msg)

async def main(e, peer, timeout, period):
    # asyncio.create_task(heartbeat(e, peer, period))
    # asyncio.create_task(echo_server(e))
    # await asyncio.sleep(timeout)
    tasks = [heartbeat(e, peer, period),echo_server(e)]
    await asyncio.gather(*tasks)

asyncio.run(main(e, peer, 20, 1))

I mainly made three modifications:

1. Print the decoded local MAC address and modify the channel.

mac = binascii.hexlify(sta.config('mac'),":").decode()
print("Current_MAC:", mac) # Print current device's Mac Address
sta.config(channel=3)
print("Current_Channel:", sta.config('channel'))

2. I attempted to include the variable i and decode the MAC address within the coroutine echo_server, in order to determine how many times the async for mac, msg in e loop iterates during the program's actual execution, and also to identify the MAC address from which each received message originates.

import binascii

async def echo_server(e):
    i = 0
    async for mac, msg in e:
        i += 1
        mac_decode = binascii.hexlify(mac, ":").decode()
        print(f"Iteration {i}: Received from MAC {mac_decode} - Echo: {msg.decode()}")

3. I temporarily commented out those lines of code and used asyncio.gather as an alternative.

async def main(e, peer, timeout, period):
    # asyncio.create_task(heartbeat(e, peer, period))
    # asyncio.create_task(echo_server(e))
    # await asyncio.sleep(timeout)
    tasks = [heartbeat(e, peer, period), echo_server(e)]
    await asyncio.gather(*tasks)

I have these questions I'd like to ask.

1. The for mac, msg in e works like a while loop. I have come across a term called an iterator, but I still don't fully understand it. However, with for mac, msg in e, I no longer need to use ESPNow.recv().

2. The await asyncio.gather(*tasks) achieves the same functionality as the two lines of asyncio.create_task, but it doesn't have the ability to handle the timeout as in await asyncio.sleep(timeout). I still don't grasp their fundamental differences, nor do I know how to add timeout when using asyncio.gather.

[glenn20]

Oh - and one minor tip:

• Since 1.20 (and possibly 1.19) you can replace: binascii.hexlify(mac, ":").decode() with mac.hex(":"). (You can also do the reverse (almost) with bytes.fromhex("01234556789a") - but can't handle ":"s).

[Wind-stormger]

mac.hex(":"), great!

[glenn20]

I have these questions I'd like to ask.

The for mac, msg in e works like a while loop. I have come across a term called an iterator, but I still don't fully understand it. However, with for mac, msg in e, I no longer need to use ESPNow.recv().

This is a popular idiom in python, I have defined the ESPNow.__iter__() and ESPNow.__next__() methods as a convenience, which means that the ESPNow object can act like an iterator. You can see this in the espnow module. It is the iter "dunder method" that calls irecv().

The await asyncio.gather(*tasks) achieves the same functionality as the two lines of asyncio.create_task, but it doesn't have the ability to handle the timeout as in await asyncio.sleep(timeout). I still don't grasp their fundamental differences, nor do I know how to add timeout when using asyncio.gather.

Though I haven't tried it on micropython, you should be able to use wait_for() to implement a timeout around gather(), eg:

await asyncio.wait_for(ayncio.gather(*tasks), timeout=30)

If the gather does not complete in 30 seconds, all the incomplete tasks and the gather are cancelled and an asyncio.Timeout error is raised (which you should capture and handle accordingly). Micropython also has asyncio.wait_for_ms() if that is more convenient.

[Wind-stormger]

Many thanks.

I think I will continue to study Python documentation thoroughly.
My understanding of the for loop is currently limited to iterating over lists or tuples. I am particularly interested in __next__.

[glenn20]

A search for "python iterables" will be a good start if you are interested.

[bulletmark]

@glenn20 , you are the expert who integrated ESP-Now into MP. Have you or anybody else considered integrating ESP-Wifi-Mesh? Unfortunately, from what I can tell, Espressif seems to have created ESP-Wifi-Mesh completely distinct and independent to ESP-Now.

[peterhinch]

ESPNow is logically distinct from WiFi and enables ESPx modules to communicate in the absence of any WiFi AP.

[bulletmark]

Yes, and ESP-Wifi-Mesh similarly works via it's own wifi protocol independent of TCP/IP (or UDP). E.g. see this figure which shows ESP devices outside the range of the AP and see this figure which shows how TCP/IP is only used in ESP-Wifi-Mesh on the root node only, and only for gatewaying mesh packets to the router/internet. However, as I said in my last sentence above, it seems ESP-Now and ESP-Wifi-Mesh custom protocols are entirely unrelated so it would require a "from-scratch" implementation in MP if even possible.

[peterhinch]

Nevertheless the topology of ESP-Wifi-Mesh and ESPNow are quite different: the former is tree-structured while the latter is a connectionless point-to-point or point-to-multi-point net. A key characteristic of ESPNow is that a node can wake from deep sleep (or have power applied) and communicate with any other node very quickly: vital for ultra low power applications.

ESP-Wifi-Mesh sounds like a superb design, but its architecture and use-case are very different from ESPNow.

[glenn20]

Caution: this comment is based on half remembered information from my own investigations a few years ago.

I believe ESP-WIFI-MESH is a rename of what was called ESP-MESH, and that did work over espnow (or at least it had an option to do so), in that the root nodes connected to AP with wifi, but other nodes formed a mesh over espnow. I believe it would be feasible to implement support for that in micropython, but there are other mesh options too. It just needs work done.

I should really take another look.

[anicolao]

In particular, any questions or suggestions on documentation, usage or api are sought and welcome here.

So I am not sure if this is still true 2y later, but if I'm in the wrong place please redirect me.

First a documentation suggestion: in the description of how to make a receive callback you show a problematic way of implementing it. I think an example like this might be better:

from micropython import schedule
...
RECV_BUFFER = [None, bytearray(250)]
def recv_cb(e):                                                                                                  
    global RECV_BUFFER                                                                                           
    while True:                                                                                                  
        n = e.recvinto(RECV_BUFFER, 0)                                                                           
        if n == 0:                                                                                               
            return                                                                                               
        mac, msg = RECV_BUFFER                                                                                   
        if mac is None:                                                                                          
            return                                                                                               
        schedule(handle_message, (mac, msg, n))                                                                  
        try:                                                                                                     
            schedule(recv_cb, e)                                                                                 
            return                                                                                               
        except:      
            # if the network is less important than other tasks
            # you are scheduling, return instead.                                                                                            
            pass                                                                                                 
                                                                                                                 
                                                                                                                 
e.irq(recv_cb)      

This example would ideally replace the smaller sample at https://docs.micropython.org/en/latest/library/espnow.html#espnow.ESPNow.irq ... the main caveat with it being do you really want to hog the schedule when the schedule queue is already full?

This leads to a possible API issue: irecv doesn't return n. Ideally it would. But I guess a breaking change is a very bad idea at this stage, so perhaps just a pointer from the irecv docs to the recvinto docs and perhaps also from the recv docs would be good. Because I at least could not find any good way to divine the length of the valid data in the re-used byte array and so I need n to process the message correctly.

Finally, a feature request that I have no idea how hard it is. Now that ESP.Now v2 supports up to 6 vendor specific frames, it'd be excellent if on devices that support this I could pass a 1500 byte array in RECV_BUFFER to receive these larger messages (and of course, also allow sending them). Is that on your radar at all? The per-message overhead seems very high in my mesh network code and having fewer messages would be a godsend.

Thanks for your time/consideration.

[glenn20]

First a documentation suggestion: in the description of how to make a receive callback you show a problematic way of implementing it. I think an example like this might be better:

This example would ideally replace the smaller sample at https://docs.micropython.org/en/latest/library/espnow.html#espnow.ESPNow.irq ... the main caveat with it being do you really want to hog the schedule when the schedule queue is already full?

Thanks, I agree the example in the docs is simplistic, but I was going for simple :-). My first response to the above example is that it may in fact result in more callbacks on the shedule queue, as you are using schedule to put more callbacks on the queue.

A better solution would probably be to adopt the static scheduler nodes added a few years ago. It has been on my backburner to test.

Even better, don't rely on the callback mechanism if you are likely to run into an issue with it (as hinted in the docs :-) ). I dont think it is designed for potentially high throughput usage.

This leads to a possible API issue: irecv doesn't return n. Ideally it would. But I guess a breaking change is a very bad idea at this stage, so perhaps just a pointer from the irecv docs to the recvinto docs and perhaps also from the recv docs would be good. Because I at least could not find any good way to divine the length of the valid data in the re-used byte array and so I need n to process the message correctly.

You can get the length of the message returned with the python len() builtin. eg:

mac, msg = e.irecv()
print(f"Received {len(msg)} bytes.")

Finally, a feature request that I have no idea how hard it is. Now that ESP.Now v2 supports up to 6 vendor specific frames, it'd be excellent if on devices that support this I could pass a 1500 byte array in RECV_BUFFER to receive these larger messages (and of course, also allow sending them). Is that on your radar at all? The per-message overhead seems very high in my mesh network code and having fewer messages would be a godsend.

Yes, support for v2 would be great, and @AmirHmZz has put up PR #16737 to add that. Hopefully we can merge that into micropython soon.

[anicolao]

Even better, don't rely on the callback mechanism if you are likely to run into an issue with it (as hinted in the docs :-) ). I dont think it is designed for potentially high throughput usage.

Is there something specifically wrong with the callback mechanism? It seems efficient enough to me; the problems have been more to do with the shortness of the schedule queue (4 or 8 entries) or the small size of the receive buffers. My application is hosting an AP serving a website and participating in a mesh network simultaneously and cooperative scheduling is unlikely to produce satisfying latencies.

You can get the length of the message returned with the python len() builtin. eg:

mac, msg = e.irecv()
print(f"Received {len(msg)} bytes.")

This does not work. When re-using buffers, the length is always 250 as far as micropython is concerned, while the message length is shorter. Your example may work because you're willing to reallocate buffers on every irecv call but as they say ain't no callback got time for that :)

Yes, support for v2 would be great, and @AmirHmZz has put up PR #16737 to add that. Hopefully we can merge that into micropython soon.

I will check out this PR and build it. Sorry I missed it.

[glenn20]

This does not work.

Are you saying that you have tested this and confirmed it does not work - or that you think it wont work? I am away from my devices till tomorrow, so I can't run the test to confirm, but it certainly used to work.

When re-using buffers, the length is always 250 as far as micropython is concerned, while the message length is shorter. Your example may work because you're willing to reallocate buffers on every irecv call but as they say ain't no callback got time for that :)

I don't know what that means. irecv() does not reallocate new buffers, it reuses a pre-allocated bytearray. On each return irecv() sets the length of the bytearray to the length of the returned message (well, it is supposed to do that - I will check tomorrow).

Yes, support for v2 would be great, and @AmirHmZz has put up PR #16737 to add that. Hopefully we can merge that into micropython soon.

I will check out this PR and build it. Sorry I missed it.

[anicolao]

Well now you have me doubting myself so I will have to dig in and reproduce again. My code didn't exactly match yours and I'm afraid I probably never committed the non-working state, but I'll see if I can write a specific example that exhibits the problem. I was definitely getting receive buffers of length 250 when the messages were shorter, but I was likely passing in the buffer, not calling irecv() with no parameters.

[anicolao]

The code you remember certainly does exist

    if (mp_obj_is_type(msg, &mp_type_bytearray)) {
        // Set the length of the message bytearray.
        size_t size = msg->len + msg->free;
        msg->len = msg_len;
        msg->free = size - msg_len;
    }

[anicolao]

Unfortunately I did not in fact commit the code that demonstrated the 'problem'. But inspecting the C code that implements recvinto and doing my best to recall what I had, I have a likely explanation of my observed behaviour:

• because my receive loop doesn't know when all data have arrived, it would keep receiving data until there are none;
• so if one packet is in the queue and I received data twice, the second call would reset the buffer length to 250
• code that was still processing the first call's return value would see a length of 250 instead of the proper length

In my current code I have a ring buffer of buffers so the problem of processing multiple input packets simultaneously won't exist, but in the code at that time I was probably missing any notion of doing the buffer management myself, imagining that each call to irecv() was allocating a new buffer.

Possibly it would be less error prone to add the code that sets the message length correctly in both return paths instead of setting the length to the full size when no data are read. There's a better chance the reader of the stale buffer would realize that the data were truncated than that they'd realize they're processing stale data with a length that is too long, making it easier to recognize the bug of continuing to process the shared buffer when it's been invalidated by a subsequent call.

[anicolao]

In fact, the code with the bug is probably the ancestor of the proposed documentation example above! It originally wouldn't have had the try/catch at the bottom and called irecv() instead of recvinto and so the handler would get put on the queue but not yet called and then get called after the buffer length was reset. So I feel more confident than ever that the code worked as you expect.

I still think it would be a small improvement to truncate the buffer when an empty read happens.

[glenn20]

In fact, the code with the bug is probably the ancestor of the proposed documentation example above! It originally wouldn't have had the try/catch at the bottom and called irecv() instead of recvinto and so the handler would get put on the queue but not yet called and then get called after the buffer length was reset. So I feel more confident than ever that the code worked as you expect.

Ah, I think I see now. Your example code above will also be prone to the issue of possibly overwriting the pre-allocated buffer before you get to fully process it with handle_message(). ie. if a new message arrives before handle_message() runs, it will see the next message (or the full 250 bytes if recvinto() times out). If you need to ensure newly allocated buffers for each message, use recv(), which works fine for moderate transfer rates, but may accelerate memory fragmentation for high data rates.

I still think it would be a small improvement to truncate the buffer when an empty read happens.

It is important to always check the return value of recvinto() and irecv() before assuming the data in the buffer is valid. However, you are not the first person to be caught by this, so I will re-think this - or add more explicit warnings in the docs.

[anicolao]

Instead of newly allocated buffers, I am using a ring buffer of buffers. This avoids memory allocation/fragmentation/GC issues at risk of course of overrunning the ring buffer.

It isn't enough to check the return value of irecv; the thing that makes it error prone is that when you're calling irecv() with no parameters, it isn't obvious that it is returning the same internal buffer. The documentation reads "reuse internal bytearrays" but doesn't specify how many internal byte arrays it will re-use. As the answer is in fact "a single internal byte array", the caller has to be aware that they need to finish processing that array before calling irecv again, which then means their Rx buffers in the wifi stack will overrun and drop packets.

Bottom line: I don't think the serious caller should use any of these methods to work with the API; only recvinto is a safe API to work with because it puts the onus for the buffer management on the caller, which is where it belongs. It maybe could also be emphasized how important ew.config(rxbuf=LIMIT) is to avoid dropped packets at the next level down. The docs do say 'Increase this' for 'bursty traffic' but I think especially for the common style of cooperative multitasking it surely can't be hard to send the target 3 ESP.now packets before they completely process the first one, so at present limits I'd imagine everyone would hit errors as soon as they try to build a real network on the library. Or possibly most applications are just sending intermittent sensor data so the slowness isn't an issue; in my case I want to support video streaming over the mesh and the code is a long ways away from working.

[anicolao]

Can you clarify what you meant by "static scheduler nodes added a few years ago" in a comment above? I am trying to clean up my scheduling code and can't seem to find what you mean, nor a really satisfactory way to work well with micropython to build a reasonable scheduling approach.

[glenn20]

The "static scheduler nodes" are a feature that allows micropython modules written in C to create their own "scheduler stack". Instead of pushing a callback onto the globally shared scheduler stack (length 8) from an ISR, a module can push the callback onto its own private "scheduler stack" (of length 1) and that will also be checked by the micropython scheduler code. In this way, a module's events won't be lost by other hardware overflowing the schedule stack. But the module needs to privately track how many events might have fired before the callback is invoked.

This is not something that can be utilised from the python code.

[glenn20]

Without fully understanding your use case, my recommendation for achieving high espnow throughput would be to just set a large enough input buffer (ESPnow.config(rxbuf=xx)) to cover any latency induced by your apps or other hardware activity and then just poll the espnow object to check for any data in the buffer. To poll the input buffer you can use ESPNow.any() or select.poll.poll() or asyncio (which just uses poll() under the hood).

Creating a secondary set of input buffers in your python code should not be necessary for most use cases and will be considerably less efficient.

IF you truly need to ensure the lowest possible latency response to an incoming packet, it may well be that the scheduler approach is the best, but if your device is usually idly waiting for events it may well be lower latency to just use a ESPNow.recv() loop to read the events as they come in.

[anicolao]

Due to frustration trying to get _thread to work the way I expected and a lack of understanding on my part of why the interrupt driven approach was locking up, I wound up rewriting the application to use entirely cooperative scheduling on the main thread, and just calling recvinto each time around my cyclic executive's scheduling loop.

Unfortunately for me, this didn't resolve the underlying problem which is that periodically the interpreter locks up and has to wait for the watchdog timer to expire to get reset. I am still debugging this on the assumption the problem is in my application code, though this seems less and less likely. I was just about to start testing while avoiding using the webrepl to monitor in case the problem is somehow related to it (though I need a webrepl like solution to be able to debug and monitor in the field). None of this is related to the espnow code.

wrt where the buffering is done, no doubt you're right and I should fix it to rely on the lower level buffering. Unfortunately right now the application can occasionally fall far behind, as many as 25 or 30 packets behind, before processing the incoming data and I need to fix that. I do have rxbuf = 526*32 but I imagine we'd agree that ideally I wouldn't need to allocate so much RAM for input buffering/processing of the packets needs to be faster.

[anicolao]

Well perhaps I spoke too soon. When connected over USB instead of webrepl, I get visibility into a problem that is related to the espnow usage:

E (919569) esp.emac: no mem for receive buffer
E (919569) esp.emac: no mem for receive buffer
E (919579) esp.emac: no mem for receive buffer
E (919579) esp.emac: no mem for receive buffer

I'll dig in the source to see when this happens, but I am not expecting the receive buffer to require allocation after startup?

[anicolao]

Looks like this is below the espnow code, in the esp-idf portion of the network stack.