OSError: [Errno 110] ETIMEDOUT, STM32, asyncio

[darrenweinhold]

I'm getting Errno 110 on an STM32 (Nucleo H723ZG) using asyncio and could use some thoughts on what might be causing this.

I have two slightly different code bases running on two different systems. The one runs fine. Never has any issues. The other is running into this error every 12 to 24 hours. I don't know what I should be looking for to troubleshoot this. There is nothing that I am actually waiting on in my code.

My code is setup with a couple of tasks that run forever. I am reading some data from devices connected to UARTs. I poll them every 100 to 500 ms depending on what is connected. Interestingly enough, the one setup that is running fine is polling data every 10 ms.

The tasks just look like the below. This is just pseudocode but its nothing complicated.

while True:
   await self.read_lock.acquire()
   await self.request_data()
   await asyncio.sleep_ms(100)
   data = await self.read_response()
   self.read_lock.release()
   await asyncio.sleep_ms(100)

I have been running low on ram when importing modules, but once everything is running I have around 16k consistently free.

Edit: I should note that this error is occurring in different tasks but always on the call to write data to a UART. Specifically it is occurring in the uasyncio/stream line 1 in write.

[peterhinch]

How is the UART configured? I recommend timeout=0 in asyncio applications. That said, a timeout on write is unusual. Is there any flow control in place?

[darrenweinhold]

Thanks for taking the time to respond!

I do not have the timeout set on the UART. I will make that change. I take it that timeout=0 will disable the timeout?
I have a class for each device that handles everything. Nothing fancy. No flow control is in place.

class UART_INST:
   def __init__(self, uart_number,baudrate):
        self.uart = UART(uart_number)
        self.uart.init(baudrate, bits=8, parity=None, stop=1)
       
        self.buffer = array.array('f',bytearray(80))
        self.rx_stream = asyncio.StreamReader(self.uart)
        self.tx_stream = asyncio.StreamWriter(self.uart, {})
        self.read_task = asyncio.create_task(self._stream_read())


   async def _stream_read(self):
        while True:
            await self.rx_stream.readinto(self.buffer)
            await asyncio.sleep_ms(50)

    async def _read(self, n=None):
        data = self.buffer[0:n].decode("utf-8")
        return data

    async def _write(self, data):
        self.tx_stream.write(data)
        await self.tx_stream.drain()

Interestingly enough, I replaced tx_stream.awrite with a separate write/drain and haven't seen an issue yet in around 36 hours of testing. I don't see why this should change anything having looked through the asyncio code.

I have six UART devices attached and have seen this error on multiple devices. They are from different manufacturers as well so I don't think there is anything going on there.

[darrenweinhold]

Quick update, the device did fail again. After about 36 hours. I have set the timeout to 0 and restarted it.

[peterhinch]

This is definitely the approved approach.

    async def _write(self, data):
        self.tx_stream.write(data)
        await self.tx_stream.drain()

The following is unorthodox as it's declared as async, but the body is synchronous:

    async def _read(self, n=None):
        data = self.buffer[0:n].decode("utf-8")
        return data

I also can't help feeling there should be some kind of synchronisation between _read() and _stream_read().

While I can't see anything definitively wrong, in seeking long term reliability I would start with known-good solutions such as zero timeout, .drain and synchronisation.

[darrenweinhold]

Not sure why I was declaring _read as async. May have been a holdover. I have updated that now. Hopefully that fixes it.
I had the program error out again after setting the timeout to 0. So it wasn't that.

I have two questions though for next steps:

1. What would you recommend for synchronization between the stream_read and read? I (perhaps naively) thought this wouldn't be necessary as in asyncio there shouldn't be any pre-emption while reading into the buffer and pulling from the buffer. Would you put a lock in place? I already have a lock for the uart so adding another specifically for managing the buffer would be pretty trivial.
2. I am thinking about recovering from this error if I can't prevent it. My question is: how to cancel and then restart tasks? The most important thing is to not lose the state variables. Currently my critical variables are stored in a settings.py, which has a class called Data. All of the class attributes are my different state variables. My main.py looks like this:

main = Main() # initializes states in one main class
asyncio.run(main.create_tasks()) # creates tasks in the form asyncio.create_task(self.instrument_x.instrument_handler())

Note that each instrument_handler is an infinite loop that looks like what I had in a previous post.

while True:
   await self.read_lock.acquire()
   await self.request_data()
   await asyncio.sleep_ms(100)
   data = await self.read_response()
   self.read_lock.release()
   await asyncio.sleep_ms(100)

I attempted to change my main.py to the following:

main = Main()
while True:
   try:
         asyncio.run(main.create_tasks())
    except KeyboardInterrupt:
         asyncio.close()
    except:
         print('resetting')
         loop = asyncio.get_event_loop()
         loop.stop()
         loop.close()
         main.close_tasks() # function that closes each task 
         asyncio.new_event_loop()

The idea being to cancel all of the tasks and restart them without losing the states though. Is .cancel() the correct call to close out the tasks? I assume I would want to both close out the overall task and the close out the stream_reader task for each as well? Or is that not necessary with the loop.stop and loop.close functions? The tasks aren't actually restarting with this, but it isn't clear to me why they are not restarting.

[peterhinch]

Re synchronisation the need for this depends on the code that calls your synchronous _read() function. If it is called from asynchronous code, the call might occur while the buffer was being filled. This is because the following is asynchronous:

await self.rx_stream.readinto(self.buffer)

It can therefore be preempted by another task. The general solution is to use a Lock, however in your actual code I think there is a better way. Considering your actual code

   await self.request_data()
   await asyncio.sleep_ms(100)
   data = await self.read_response()

If all the data has been sent after the 100ms delay, preemption won't occur. But in that case you need to ensure that the UART read buffer is big enough to hold the worst-case message. I don't think this is ideal. I would use the following approach. The class would have a read_complete Event. This would synchronise acquisition and reading without the need for a Lock.

class UART_INST:
   def __init__(self, uart_number,baudrate):
        self.uart = UART(uart_number)
        self.uart.init(baudrate, bits=8, parity=None, stop=1)
        self.read_complete = asyncio.Event()
        self.buffer = array.array('f',bytearray(80))
        self.rx_stream = asyncio.StreamReader(self.uart)
        self.tx_stream = asyncio.StreamWriter(self.uart, {})
        self.read_task = asyncio.create_task(self._stream_read())

   async def _stream_read(self):
        while True:
            await self.rx_stream.readinto(self.buffer)
            self.read_complete.set()
            await asyncio.sleep_ms(50)  # Do you actually need this?

    async def _read(self, n=None):  # Now it does need to be async
        self.read_complete.clear()
        await  self.read_complete.wait()
        data = self.buffer[0:n].decode("utf-8")
        return data

    async def _write(self, data):
        self.tx_stream.write(data)
        await self.tx_stream.drain()

Then your calling code would do:

   await self.request_data()
   data = await self.read_response()  # Won't return until a complete message has been received

You immediately await data after requesting it, reducing the demands on the UART buffer.

Restarting asyncio is utterly drastic. Once you have issued asyncio.run(main_task()), main_task should run "forever", i.e. until the application quits. In typical firmware applications, this never occurs.

Within a running asyncio system it is entirely valid to cancel tasks or subject them to timeouts - see the tutorial. However my focus would be on trying to understand why your application is failing rather than crafting an elaborate recovery mechanism. I have asyncio applications which run indefinitely, and the STM32 micropython port is highly reliable. Occasional failures have a cause.

Incidentally your use of async def _read was not a cause of your problem - asyncio does handle this even though it is unorthodox.