Capturing and storing UART data to SD card (pyboard)

[skysurf76]

I'm working on developing a upython script for the pyboard that captures all data coming in over a UART and writing it in binary form to a file. Being able to capture all raw data from a serial stream for later analysis would be a huge tool for me. I have made a few attempts with varying levels of success, and before going further I figured it would be a good idea to get some input from the pros.

-Firstly, I would like this to work on continuous serial streams up to 115200 bps. Is this possible on the pyboard, or would this project require more performance than the pyboard has. Moving to a teensy 4.1 is an option if this is the case.

-Any general design advice for the script would be greatly appreciated. I am very new and still in the process of learning, and any advice that would stop me from going down a dead end path would be greatly appreciated.

-From the research I have done so far it seems like reading one character at a time is the best way handle serial data, and I have learned how to do that. I'm not sure how to handle the data once I have it, and how best to get it on the SD card.

Thanks in advance!

[rkompass]

With DMA on the RPI Pico you should be able to read from UART into two buffers in continuous alteration, completely in the background. Then you could poll the DMA state and transfer the buffer that is full to SD card in the foreground. So overall speed/transfer rate then should depend on what speed is achievable with the SD card. One (you?) could do a search on that or test it, if Pico is an option. Note that SD card on the Pico is attached via SPI.
Also different SD cards have different speeds, of course.
Perhaps I modify my DMA class to work on the pyboard (assuming Pyboard 1.1). That was planned anyway. I suppose SD card writing will be faster there, it might be done with DMA there already, but different DMA transfers should be possible in parallel ?!

[peterhinch]

The Pyboard UART uses interrupts and is buffered. The key is probably to set a sufficiently large receive buffer size (.rxbuf) so that bytes are not lost while the SD card is being updated. The advice to handle data one character at a time is really targeted at text data where the data must be parsed. With binary data there is no real merit. If you have a 1024 byte buffer, at 115200 baud, it will take 88ms to fill. Assuming that the SD card write blocks for less than this, it will work.

I would start out with a 1024 byte .rxbuf buffer, reading 128 bytes at a time into a pre-allocated 128 byte bytearray (uart.readinto). If bytes are lost, increase the size of the .rxbuf.

[skysurf76]

Increasing the rx buffer worked perfectly. Thanks to both of you for the advice!