Is there a way to recover from errors and restart my application?

[neliel123]

Hi,
I am using asyncio on an ESP32 microcontroller and am doing several tasks like my code below.

I am running one coroutine for the web server and other coroutines for monitoring.

async def main():
    await asyncio.gather(run_web_server(), run_monitor_app(), clear_data(), listen_for_messages())

try:
    asyncio.run(main())
except KeyboardInterrupt:
    print('Interrupted')
finally:
    asyncio.new_event_loop()

I just noticed that my web server stopped receiving requests since I cannot access it anymore randomly.
Some of the errors that I am encountering are the following.

Is there a way in my code to catch these errors and restart my coroutines or hard reset my device when these error occurs? Thanks

Traceback (most recent call last):
  File "main.py", line 139, in <module>
  File "asyncio/core.py", line 1, in run
  File "asyncio/core.py", line 1, in run_until_complete
  File "asyncio/core.py", line 1, in run_until_complete
  File "main.py", line 136, in main
  File "asyncio/funcs.py", line 1, in gather
  File "asyncio/core.py", line 1, in run_until_complete
  File "main.py", line 84, in wait_for_message
MemoryError: memory allocation failed, allocating 108 bytes

Traceback (most recent call last):
  File "main.py", line 139, in <module>
  File "asyncio/core.py", line 1, in run
  File "asyncio/core.py", line 1, in run_until_complete
  File "asyncio/core.py", line 1, in wait_io_event
OSError: [Errno 5] EIO

[karfas]

Just pack your entire app in a try/except block and reboot when an exception occurs.

You might be able to minimize the memory errors by calling gc.collect() frequently - this should prevent memory fragmentation.

[neliel123]

Just pack your entire app in a try/except block and reboot when an exception occurs.

Sorry, what do you mean by this? This is my main.py and I have wrapped this in a try except. Should I put this in a loop?

This is my first app where I have used asyncio so I am still learning the ropes.

async def main():
    await asyncio.gather(run_web_server(), run_monitor_app(), clear_data(), listen_for_messages())

try:
    asyncio.run(main())
except KeyboardInterrupt:
    print('Interrupted')
finally:
    asyncio.new_event_loop()

You might be able to minimize the memory errors by calling gc.collect() frequently - this should prevent memory fragmentation.

About this one, should I put this in a coroutine and run this every minute or something?

Thanks for your inputs.

[karfas]

Please read about python exceptions - e.g. https://www.w3schools.com/python/python_try_except.asp .

In your case:

try: 
  asyncio.run(main()) 
except KeyboardInterrupt:
  print('Interrupted')
except:
  machine.reset()
finally: 
  asyncio.new_event_loop()

should do it.

Regarding gc.collect(): Wherever it fits in your application. I would do it when some larger(?) memory areas aren't needed anymore - maybe after serving a http request.

[peterhinch]

This doc is worth reading for techniques to control RAM usage. Rebooting on running out of RAM as above, will work, but it is a hack.

If, after optimisation, the app is still too big to fit there is the option to use an ESP32 with SPIRAM.

[DavesCodeMusings]

Just curious, when you are serving HTTP requests are you opening files (like index.html, style.css, etc.) to send as a reply? This can eat up a lot of memory. I've found that reading in small chunks can help.

Something like this:

def read_file_chunk(file):
"""
Given a file handle, read the file in small chunks to avoid large buffer requirements.

Args:
    file (object): the file handle returned by open()

Returns:
    bytes: a chunk of the file until the file ends, then nothing
"""
while True:
    chunk = file.read(64)  # small chunks to avoid out of memory errors
    if chunk:
        yield chunk
    else:  # empty chunk means end of the file
        return


with open(file_path, 'rb') as file:
    for chunk in read_file_chunk(file):
        writer.write(chunk)
        await writer.drain()

[neliel123]

Yes, I am using MicroDot framework to serve my index.html file using the asyncio version.
My application works perfectly for some time like 1 hour or two until the coroutine for Microdot Web Server stops responding.
The other coroutines are running fine though.

[DavesCodeMusings]

Ah, okay. I thought maybe you had created your own web server.

[peterhinch]

Another thought. Are you sure all sockets are closed under all circumstances? Failure to close sockets is a common cause of gradual RAM use. A debug option is to run a task which monitors RAM use:

async def ram_use():
    import gc
    while True:
        await asyncio.sleep(30)
        print(gc.mem_free()

[donskytech]

async def ram_use():
import gc
while True:
await asyncio.sleep(30)
print(gc.mem_free()

Yes, this is interesting. I would take a look at this. What is the best way in MicroPython to log for application logs? This is for debugging purposes also, Not sure if writing logs to to text file is okay as this might consume memory or am I overthinking it?

[davefes]

Not sure if writing logs to to text file is okay as this might consume memory or am I overthinking it?

I gather hourly data (~1K) using a RAM file and once an hour save it to a file on flash. That file gets send to Gmail once a day and then deleted. Yes, you have to keep an eye on how much RAM and FLASH you are filling up.