Characterising ESPNOW

[peterhinch]

I've started testing using aioespnow on ESP32-S3. Testing was with an echo server and a single peer. These are my initial observations.

1. Range is OK, but not much better than with WiFi or other technologies such as NRF24L01. I'll test further.
2. Reliability is moderate. Sending 500 messages 28 were lost (success here meant that it was received by the peer and its response was also received).
3. Latency varied from 9ms to 63ms over 500 messages. This is the round-trip latency: in one direction these numbers can be halved (at least).
4. The asend method is pessimistic. It sometimes returns False when the peer has received the message.
5. Message integrity is perfect: I never encountered a corrupt message over thousands of exchanges.

Reliability might be an issue. I chose a channel which is a long way from any used in my house, one where a scan revealed low levels of remote WiFi. Testing was over a room-to-room distance. Testing can run for long periods with only a rare missed message, but has bouts of failing repeatedly. This can even occur when the units are a metre apart.

There are two communications scenarios that often arise:

1. Where low latency is paramount, e.g. remote control. If a message is lost it is discarded.
2. High reliability. A communications protocol sacrifices latency in return for guaranteed delivery.

In principle it seems good for low latency work and the message integrity means that devising a protocol for reliability would be easy. But the long sequences of failures over many minutes are rather troubling and I haven't seen this behaviour over short distances with other wireless technologies. Any comments?

[glenn20]

Thanks Peter. This is a very useful summary and generally accords with my own experience, although I haven't experienced the extended periods of failed messages that you have seen. Over the last few years I have run high throughput tests over weeks and not seen that behaviour. However, I haven't done those long term tests with S3 devices and could redo some of those tests with the current version of the module to check.

When the extended periods of failed transmissions occured, were you able to test if both incoming and outgoing messages were failing? I assume your tests were all with asend() and arecv().

[glenn20]

Re. latency, my experience is that usually a successful send() receives an ACK within 2ms. A message that does not receive an ACK will timeout after ~25ms (after retransmitting the message a dozen times or so). I understand that your latency measurements refer to the time to send a message and then receive a reply message from the peer.

[glenn20]

Follow-up. I've done some more tests and seen some hard-to-explain behaviour with latency. I will try digging a bit deeper this week. I have seen streams of tests with latency of <10 ms, followed by consistent latencies of 12-14ms, followed by stretches of 15-25ms, which is hard to explain.

[peterhinch]

My echo server runs this single task:

async def echo_server(e):
    async for mac, msg in e:
        print("Echo:", msg)
        try:
            await e.asend(mac, msg)
            asyncio.create_task(flash(GREEN))  # 1 second LED flash for range testing
        except OSError as err:
            if len(err.args) > 1 and err.args[1] == 'ESP_ERR_ESPNOW_NOT_FOUND':
                e.add_peer(mac)
                asyncio.create_task(flash(BLUE))  # Add peer
                await e.asend(mac, msg)
            else:
                asyncio.create_task(flash(RED))  # ???

The other peer runs two tasks, a transmitter that sends a JSON encoded message number and timestamp every 3 seconds and a receiver which runs an async for loop. The receiver measures the time difference between the transmission time and reception time (round trip delay) and also checks for missing message numbers. The shortest round-trip delay I have seen is 6ms.

The long sequences of failed messages are concerning. I changed channel from my WiFi's 3 to a quiet 10 but it had no effect (almost all of my WiFi traffic is on 5GHz). The long sequences typically occur for a few minutes after start, it then settles down with only occasional dropped messages. I live in a semi-rural area with reasonably low levels of RFI, other projects using 2.4GHz have not been affected in this way.

[EDIT]

were you able to test if both incoming and outgoing messages were failing?

The simple echo server didn't provide a way to do this. I'll do a test to establish this.

[peterhinch]

I've run a test where the echo server decodes the message, looks for missing message ID's, and appends that number to the message which it returns. The great majority of missing messages are outgoing, from the test script to the server. In 800 attempts, 170 were missed as seen by the test script. 167 were missed as seen by the server. So only three messages from server to script failed to be received.

[peterhinch]

@glenn20 I have now replaced the S3 with the original ESP32 reference board. The echo server still runs the S3. The problem has gone away. Now running >500 messages with zero loss. Round-trip latency max: 31ms min: 11ms.

The problem seems to be with the S3.

Incidentally using mip failed today and I had to copy aioespnow.py manually.

[EDIT]
This is a hardware problem. I replaced the S3 with another identical board and it's got to 1400 messages without a single error. I'll repeat the range tests now I have reliable kit :)

[glenn20]

Thanks Peter for running these tests. Those new measurements more closely match my own experience. Re. the error rate, I have never seen an error in several years of sustained echo server tests, but that is not surpising in that it is an ethernet frame (with FCS). Important to verify that with tests like this though.

Strange that the mip install failed, it is just loadfing from micropython-lib.

[glenn20]

@peterhinch : Oh, and if you are interested in range tests, this video might be interesting for you: https://youtu.be/oz0a7Ur7nko?t=368. It shows the effect of combining ESPNOW and LR_MODE (not micropython). You can enable LR_MODE on micropython with sta.config(protocol=sta.MODE_LR). A number of users have reported on the forum that setting to MODE_LR made no change to the rssi (but I would not expect it to change rssi). I haven't seen any reports of people testing range on micropython with packet loss (as in the video).

I have done no LR_MODE tests myself or range tests with espnow. My main interest has been power savings, low latency, moderate range and independence from installed wifi infra. The power savings over wifi are substantial for wake from deepsleep to send a sensor reading (see https://github.com/glenn20/upy-esp32-experiments).

[massimosala]

Hi Peter

Maybe a silly guess but are there microwave ovens nearby?

It happened to me that in the office a colleague warmed up his lunch... with ridiculous effects on the radio links.

[glenn20]

Ha yes. I have been bitten by that too - chasing mysterious lost messages due to the local microwave.

[peterhinch]

@massimosala No. It proved to be a clear case of a board with an unreliable radio. Just my luck to pick it out of about a dozen ESP's for my first test :)

@glenn20 I have now tested original ESP32, S2 and S3. All are reliable over short distances (i.e. within my house), with round-trip latencies of 5-20ms. At extremes of range, latency of successful echoes goes up to 70ms - with inevitable missed messages. I'll try the LR mode today. From the video it looks promising.

My initial interest is to look at the possibility of an MQTT gateway using mqtt-as. Clients would send data using ESPNow to a central ESP32 which would publish them, subscriptions would be handled similarly. This would make for very simple clients. In the case of publish-only clients it would facilitate micropower use, with the option also to use your optimisations. Added range would be a bonus.

I'm also interested in remote control applications. MQTT is not really fast enough.

[glenn20]

@peterhinch - I'm interested in the same kind of mqtt gateway functionality and have some works in progress that I haven't been able to get back to for some time. I'm hoping to have some time in the coming weeks and will share what I have. With the power savings using deepsleep and espnow its a really good fit for sensors sending periodic updates. (I used parts of your mqtt-as as base and inspiration).

[peterhinch]

[EDIT] I am currently working on a gateway. Prototype code is working.

There is a minor mod to mqtt-as and the gateway acts as an mqtt-as client.

ESP32 nodes send ESPNOW messages to the gateway. These specify topic, message, retain and qos - which the gateway publishes. Subsciptions are handled on the assumption that the node spends most of its time in deepsleep. Consequently the node can only receive a subs immediately after it publishes. The gateway queues any received subscribed messages, forwarding them to the relevant node immediately after that node next does a publication.

Unlike mqtt-as the design prioritises low power use over message delivery - any attempt at guaranteed delivery potentially involves long sequences of trying for a response. So you get whatever ESPNOW offers in terms of delivery. Nodes are simple to code, and it is easy to write a publish-only node for even lower power use.

Comments are welcome, and I can let you have code soon.

[peterhinch]

@glenn20 Re ensuring that the channel number is fixed, have you seen #11819?

I am using

sta.config(channel = chan)

but accoring to the PR this does not work. Issuing

sta.config(reconnects=0)

causes the application to fail to reconnect after a WiFi outage (with a WiFi timeout error). This is hard for me to test as my AP uses a fixed channel which I specify, so I'm reliant on you ESP32 gurus for advice here :)

I'd appreciate any comments.

[davefes]

Sorry, if I missed it, but are you using Glenn's wifi.py? If not, try:

 #  set up the WiFi interface
    sta, ap = wifi.reset(channel=3)

in this https://github.com/glenn20/micropython-espnow-utils/blob/main/src/wifi.py

[peterhinch]

Thanks for the useful pointer. A handy utility.

Tracing through the code, on an ESP32 .reset runs this code. IOW it's based on the assumption that sta.config works, which the PR is calling into question. I guess there will be a response to the PR in due course.

[glenn20]

@peterhinch - thanks for the pointer to that PR. Interesting and useful. Setting the channel (sta.config(channel=6)) works just fine. It is just that when you invoke sta.connect() it resets the starting channel for the connection to 1. It would be a good idea to change that (as I commented in the PR).

For setting the espnow comms channel, sta.config(channel=6) will work just fine. (In fact espressif only introduced support for changing the channel on the STA in 4.2 - and that was to support the needs of espnow - because it is not that useful for wifi client devices).

[peterhinch]

Thanks for that.

The mqtt_as driver, when used with my gateway, initiates a station interface object (self._sta_if) and sets the channel when the gateway first starts. My doubt is what happens in a WiFi outage. In this case mqtt_as issues ._sta_if.disconnect(). It then sets ._sta_if active and attempts to reconnect with SSID and password. Once reconnected, operation resumes.

My concern is whether the channel will continue to be honoured.

Is there actually a problem here? It is clearly necessary for the router/AP to be set to use a fixed channel, because the ESPNOW clients expect to operate on a fixed channel. Given that the gateway is set to connect to that AP, it will end up using that channel automatically. So is there even any need to set the channel in the gateway code?

There is no actual problem here, the gateway resumes after an outage. I just want to be clear in my mind about the channel issue.

[glenn20]

Yes, that is a real problem. In fact, when the wifi connection to the AP is dropped, the ESP32 wifi code will automatically start scanning the channels looking for the AP (in case it has change channels) which will disrupt espnow communications. You can control how long that occurs with WLAN.config(reconnects=) option.

I deal with these difficulties by recognising that the espnow-only nodes just have to take responsibility for dealing with peers that may be forced to change channels when connecting to a peer that is attached to a wifi router.

I have the small espnow_scan module in my utils repo which helps with this: https://github.com/glenn20/micropython-espnow-utils.

I also use a ESPNowRobust class which has an api that matches ESPNow, but performs all sorts of fallback options when errors are raised, including re-scanning for the peer if transmissions continue to fail (and options to control the fallback policies). I will soon have a lot more time to work on a few of these unfinished projects. I'll aim to share that this weekend.

And, yes, there is no point is setting the channel in the gateway code - it is constrained to operate on the AP channel (which may change during operation unless configured not to). The espnow infra just has to follow the gateway.

[massimosala]

For setting the espnow comms channel, sta.config(channel=6) will work just fine. (In fact espressif only introduced support for changing the channel on the STA in 4.2 - and that was to support the needs of espnow - because it is not that useful for wifi client devices).

Hi Glenn

Please can you explain because it is not that useful for wifi client devices ?

My scenario.
I am developing an app where the micro connects as a client to the user cell phone (hotspot enabled).

In my area, channel 6 is congested, so on the micro I set channel 1.
The micro connects to the phone, with no customizations on it. It only allows you to set the band, 2.4 or 5 GHz. I'll leave it on 2.4.

As far as I know, on 2.4 GHz the best channels are the lateral ones, 1 and 14, due to the less overlap with the nearby ones.
In some countries the top channels are strictly regulated, so I choose 1.

Isn't it a good solution?
Mmh is it possible that I set the channel but during the wifi "handshaking" the micropython stack goes back to channel 6?

BTW, I am using esp8266, so this is the sequence of operations:

• activate the interface AP_IF (without config() a network)
• set the channel
• de-activate AP_IF
• activate STA_IF and connect to the cell phone

[glenn20]

@massimosala - Sorry for the delayed response. I typed a reply and forgot to hit Send.

Mmh is it possible that I set the channel but during the wifi "handshaking" the micropython stack goes back to channel 6?

Exactly - while connecting to the phone's wifi access point, your micro will change channel to match the channel used by the phone's wifi hot spot. That is what connecting to an AP means. Whatever channel you set beforehand is irrelevant.

I should also clarify that I was referring to devices which connect via the STA_IF interface to an Access Point. In that case, (until espnow came along) there was no need to set the channel on the STA_IF interface as the STA interface was constrained to always run on the channel used by the AP.

If you are using the AP_IF interface to run an access point, of course you need a mechanism to set the channel.

(In fact, there is still no support for setting the channel on the STA_IF for the ESP8266 - you have to use the AP_IF interface to set the channel even if you use the STA_IF interface for sending and receiving espnow messages - you can see this in my wifi.py utility code).

[glenn20]

@massimosala - FYI - I have updated PR #11558 so that you can set the wifi channel on the ESP8266 with the STA_IF interface, not just the AP_IF interface. This helps make code more portable between the esp32 and esp8266.

[massimosala]

Great!

[massimosala]

Thanks.
Ehm ... I don't understand the use cases, if the client has to adapt (auto-magically) to the router/hotspot channel.

[massimosala]

I verified.

In AP_IF, after setting the channel (step 2 of the above sequence), calling
nic.config("channel")
reports channel 1.

[peterhinch]

Another observation that others might want to check. I have three Feather S3 boards. These have 8MB of PSRAM. I have found them to suffer bouts of failed ESPNow messages. It is hard to replicate this behaviour as they can work for hours. Another time after power up I get repeated errors before it settles down. Initially I thought I had a duff board, but it isn't as simple as that.

The ESP32 reference board works very well - I've never seen this behaviour.

The question in my mind is whether it is the S3 or a latency effect from SPIRAM.

[glenn20]

Mmm - interesting. I wonder if this could be gc related. I have found a busy app can trigger very long gc cycles on an 8MB PSRAM device. I don't know if your use case alloc a lot of memory, but is the problem mitigated by pre-emptive garbage collection? I'm not aware of how this would cause send()s to fail, but thought it might be a simple test. I also have a few featherS3 boards and will try running some tests this week.

[glenn20]

PS. Another user once commented that they were told by a manufacturer that the S3 can have wifi signal issues, however, I have never heard that myself. Note that if such issues exist, they may not be detected in many wifi use cases due to tcp retransmissions.

[massimosala]

Without knowing the manufacturer and its reliability...
Who knows if he is discrediting S3 because he wants to sell other products?

[massimosala]

About garbage collection.

Try this in your boot.py and let me know:

import gc
gc.enable()
gc.threshold(64)

A frequent gc, which works on little, is better than a delayed gc (from a high threshold) and more work to do.

[peterhinch]

I wonder if this could be gc related

@glenn20 I wondered this too. Two thoughts. GC on machines with SPIRAM takes 100ms in my measurements but others have measured 200ms. That is even if you manually trigger a GC immediately after a prior one. However the typical behaviour I've seen occurs immediately after boot, when there can't be enough garbage in 8MiB of RAM to trigger a GC. This occurs even with trivial scripts like echo servers. It's very sporadic: just when you're confident you've isolated it to one unit it rears its head again.

I'll be very interested to see how you fare.

[glenn20]

OK - GC was a long shot.

Can I just check? You see this behaviour when the sending module is a Feathers3 - is that correct? I have a spare one each of Seon's FeatherS3 and TinyS3 I can test.

[peterhinch]

Yes there are send fails when using synchronous or async interfaces.