Question about receiving UART serial communications and framing

[ejgeiger]

Hi, I am using a ESP32 board to receive data over UART. The sending device sends 23 Bytes every 100 ms including a 4 byte start header (F4,F3,F2,F1) and 4 byte end of message (F8,F7,F6,F5). After much trial and and error I discovered I needed to initialize my UART with the timeout keyword uart1 = UART(1, baudrate=9600, tx=4, rx=3, bits=8, parity=None, stop=1, timeout=100) in order to read at all. Otherwise, UART.read(23) would immediately return with none. However, if I only read 23 bytes with the timeout, I am not guaranteed to read from the beginning of the frame. It will sometimes read from the middle of one frame until the middle of another frame. Furthermore, it will jump around like this:

b'6447005500f8f7f6f5f4f3f2f10d0002aa035000645800'
b'f5f4f3f2f10d0002aa036000642c00643a005500f8f7f6'

I realize this is a timing issue, so I was wondering if there was a way to ensure the read waits for the beginning of a full message since the TX line is mostly quiet. It is only sending data about 25% of the time. As a work around, I am now reading 46 bytes and parsing the first full message. This might be the best way to handle it, but I wanted to ask the community.

[karfas]

Great when this works for your use-case.
The drawbacks are obviously

• read() will (in absence of anything to read) block for 100ms
• read() might block forever when you get less than the requested 46 bytes (UART docs: "timeout specifies the time to wait for the first character (in ms)").
• you need (relative) complex code to decode the result of multiple read() calls into N frames.

In my experience reading single characters (with timeout=0, checking UART.any() or similar) and to accumulate the frame in a buffer (provided you have seen start-of-frame) until a full frame is complete seems to be a more flexible approach:

• your code has full control over timeouts etc.
• you can act immediate on a frame after receiving end-of-frame

[ejgeiger]

Is this more or less in line with what you are suggesting?

uart = UART(1, baudrate=9600, tx=4, rx=3, bits=8, parity=None, stop=1)

i = 0
r = bytearray(23)

while True:
    tmp = uart.read(1)
    i = i + 1
    if tmp==b'\xf4':
        r[0] = 0xf4
        for j in range(1,23):
            r[j]=uart.read(1)[0]
        print(hexlify(r))
        break
    if i == 5000: # It took some trial and error to find what value would consistently return a message  
        print(hexlify(r))
        break

I think one issue with this approach is if the baud rate was faster than the loop, but maybe that is overthinking it?

[robert-hh]

read() might block forever when you get less than the requested 46 bytes (UART docs: "timeout specifies the time to wait for the first character (in ms)").

There is as well timeout_char which defines the timeout after the first byte has been received. The default value is 0, meaning that it will not wait forever, but usually only for 1 character time.

[peterhinch]

The approach of @karfas is the standard way and will work.

However in this instance there may be an easier way. Sending 23 chars at 9600 baud takes about 23ms, so there is a gap of about 77ms between frames. If you set timeout=50 and timeout_char=50 you could synchronise by waiting for a timeout:

while uart.read(1) is not None:
    pass
# time is ~73ms from start of last frame. About 27ms later, next frame arrives.
while True:
    frame = uart.read(23)
    # process frame

This does assume that the sender's timings are consistent and accurate.

[ejgeiger]

I haven't exhaustively tested, but it does look like the messages are very consistent in this case based on limited probing with a network analyser. One comment on your approach, which is more inline with what I thought might be possible, is that the timeout should be closer to 70ms. With a timeout of 50ms I would still sometimes get a ``None'' result:

from machine import UART
from binascii import hexlify

uart = UART(1, baudrate=9600, tx=4, rx=3, bits=8, parity=None, stop=1, timeout=50, timeout_char=50)

for _ in range(100):
    while uart.read(1) is not None:
        pass
    while True:
        frame = uart.read(23)
        if frame is not None:
            print(hexlify(frame))
        else:
            print(frame)
        break

gives:

...
None
None
b'f4f3f2f10d0002aa023101003101647a005500f8f7f6f5'
b'f4f3f2f10d0002aa027d004b31016478005500f8f7f6f5'
...
b'f4f3f2f10d0002aa021e00001e006424005500f8f7f6f5'
b'f4f3f2f10d0002aa021e00001e006421005500f8f7f6f5'
None
None
...

[peterhinch]

There was a problem with my suggestion, in that a timeout was occurring in the main loop. Try this:

from machine import UART
from binascii import hexlify
from time import sleep_ms

uart = UART(1, baudrate=9600, tx=4, rx=3, bits=8, parity=None, stop=1, timeout=50, timeout_char=50)

for _ in range(100):
    while uart.read(1) is not None:
        pass
    while True:
        frame = uart.read(23)
        if frame is not None:
            print(hexlify(frame))
        else:
            print(frame)
        sleep_ms(50)  # Prevent timeout on next pass
        break

This should give better timing margins than setting the timeout to 70ms.

[GitHubsSilverBullet]

I have a feeling that all those timeout based approaches are fairly unstable.

Why not read byte-wise, put the byte data into a 23size ringbuffer and when receiving the ›end of frame‹ byte (the ›F5‹), check the remainder of the ringbuffer for validity.

Done.

[ejgeiger]

Do you know at what baud rate would you start to run into timing issues when reading byte-wise?

[GitHubsSilverBullet]

Don't have or use the ESP, but on a Pi Pico, north of 230400, actually far far north.
Especially if one makes good use of the hardware-FIFO.
Would have to try, but using viper code most certainly a least a Megabit per second.
And thumb, well, as fast as it'll come in.

[peterhinch]

@GitHubsSilverBullet The ringbuffer approach depends on the time in the sender's message sequence when reception starts. The first message received may be partial.

An industrial-standard solution would include provision to re-synchronise in the event of a sender outage.

[GitHubsSilverBullet]

@peterhinch I don't think so. I synchronizes automagically because it triggers the validation every time an »end-byte (F5)« is received. And if the buffer is only partially filled, then the validation (four start bytes, four end bytes) simply fails.

[shariltumin]

I suggest using the pin interrupt mechanism as an alternative solution. Here I am trying to simulate the UART communication you are trying to solve.

You will need to use an extra pin, i.e. 4 wires (Tx, Rx, Gnd, Trig).

ESP32-S2 is the producer and ESP32 is the consumer.

ESP32-S2 script:

# esp32s2 - producer
import random
from machine import UART, Pin
from time import sleep, sleep_ms

uart = UART(1, baudrate=9600) # buf=256 tx=10, rx=9
trig = Pin(12, Pin.OUT, value=0)

hdr = b'\xf4\xf3\xf2\xf1'
tal = b'\xf8\xf7\xf6\xf5'
dat = b''

while True:
    #wait = random.randint(1,10); sleep(wait)
    trig(0)
    sleep_ms(100)
    d = b''.join([random.randint(0,255).to_bytes(1, 'big') for i in range(15)])
    dat = hdr + d + tal
    w = uart.write(dat)
    print('msg:', dat)
    trig(1) # trigger data ready

ESP32 script:

# esp32 - consumer
from machine import UART, Pin, disable_irq, enable_irq
from time import sleep

uart = UART(1, baudrate=9600, tx=13, rx=14) #  buf=256 
trig = Pin(12, Pin.IN, Pin.PULL_DOWN)

hdr = b'\xf4\xf3\xf2\xf1'
tal = b'\xf8\xf7\xf6\xf5'

def get(pin):
    state = disable_irq()
    m=uart.read()
    if m:
       # print(len(m), m[4:-4])
       print(len(m), m)
    enable_irq(state)

irq = trig.irq(trigger=Pin.IRQ_RISING, handler=get)

Here we use pin12 as the data ready pin. When a packet has been written, the ESP32-S2 signals the ESP32 by pulling pin12 high and then low, and communication is ready for the next packet.

I've had good results:

ESP32-S2 test output:

....
msg: b'\xf4\xf3\xf2\xf1\x86M\x13\x11[\x1cC\xba\\,\xbes\xa9\xf3\xe3\xf8\xf7\xf6\xf5'
msg: b'\xf4\xf3\xf2\xf1\xe0\xc2[z\x82=\xcb\xcd\x9c_2R\x10\xbfI\xf8\xf7\xf6\xf5'
msg: b'\xf4\xf3\xf2\xf1$\xaf+,\x9aM\x0f\xad\x0f\x1f\xf1\xbb\x19\xfc\xd8\xf8\xf7\xf6\xf5'
msg: b'\xf4\xf3\xf2\xf1\xe7[\xd8\xd2QC\x9d\xfe\x926\x16\xfe\xd6\xe5\xcb\xf8\xf7\xf6\xf5'
Traceback (most recent call last):
  File "<stdin>", line 15, in <module>
KeyboardInterrupt: 

ESP32 test output:

....
23 b'\xf4\xf3\xf2\xf1\x86M\x13\x11[\x1cC\xba\\,\xbes\xa9\xf3\xe3\xf8\xf7\xf6\xf5'
23 b'\xf4\xf3\xf2\xf1\xe0\xc2[z\x82=\xcb\xcd\x9c_2R\x10\xbfI\xf8\xf7\xf6\xf5'
23 b'\xf4\xf3\xf2\xf1$\xaf+,\x9aM\x0f\xad\x0f\x1f\xf1\xbb\x19\xfc\xd8\xf8\xf7\xf6\xf5'

Hope this helps.

[ejgeiger]

I don't have control of the sender in this case.

[shariltumin]

It seems that we do not need an extra trigger line. We need to add an interrupt to RX on the receiver:

from machine import UART, Pin, disable_irq, enable_irq

uart = UART(1, baudrate=9600, tx=13, rx=14) #  buf=256
trig = Pin(14, Pin.IN)

def get(pin):
    state = disable_irq()
    m=uart.read()
    enable_irq(state)
    if m:
       print(len(m), m)

irq = trig.irq(trigger=Pin.IRQ_FALLING, handler=get)

The handler code between disable_irq() and enable_irq() must be within the IWDT time period, otherwise you will get a WDT reset.

I don't know how good or stable this solution is for your application, but you can give it a try.

[peterhinch]

The comments on my solution are valid. It's a "quick and dirty" approach in the spirit of the code originally posted. I think the latest iteration will work, but only if communications, once established, remain working consistently.

A production grade solution would be able to re-synchronise if communications were disrupted. I would use asyncio to write such an application and would not use interrupts or UART timeouts.

[ejgeiger]

Thanks everyone for the great discussion here. There isn't much regarding UART on the forums so I hope this helps someone in the future!

[Fre19]

I just want to report a buffer issue I had using ESP32 flashed with micropython 1.23. The UART was not able to transmit more than 2 characters, whatever I tried to increase tx and rx buffers, it did'nt work. Even more the bufferparameter was not accepted. When I reflashed mp v1.22 all went well regarding tx and receive buffers. This was the setup I used for uart 1 (and I use now with mp v1.22):
uart1 = UART(1, rx=32, tx=33, baudrate=115200, bits=8, parity=None, stop=1, timeout=5)