Nonblocking methods for machine.SPI

[adeuring]

While tinkering with micropython-micro-gui I noticed that transferring data to the display takes quite some time. Peter Hinch, the author of micropython-micro-gui, mentions this too.

I'm using an ESP32 board and an ILI9341 LCD display with 360x240 pixels, driven by the ILI9486 driver from the micropython-micro-gui library.

Looking a bit closer at the SPI bus (using a DSO) and at the source code of the driver (with my eyes ;) it turned out that the driver uses the SPI bus only for a bit more than 50% of the entire runtime of a display update. The remaining time is mostly needed to map framebuffer data, having 4 bits per pixel, to a representation with 16 bit per pixel, as needed by the ILI9341.

The ESP32 uses DMA for its hardware SPI interfaces, hence methods like machine.SPI.write() spend most of the time waiting until a chunk of data has been transferred.

This time spent waiting for data to be transferred to the display could be used by the display driver to do the 4->16bit conversion for the next SPI transfer. Out of curiosity I turned the file ports/esp32/machine_hw_spi.c into an experimental C module that provides nonblocking access to the SPI bus. (As written in the README file there, I do not consider the module in any way as "ready for general usage" – it is just intended to be a proof of concept.)

If the ILI9486 module uses an instance of machine.SPI to control the display, a full image update takes 101ms .. 117ms, with my experimental _ASPI class it takes 58ms .. 60ms.

IMHO this is enough of a performance increase to ask if it would make sense to add support for nonblocking data transfer to class machine.SPI. I know that implementing this is a bigger task than what I wrote in mp-aspi since most Micropython ports are affected. I am not even sure if such methods would be useful for each microcontroller supported by Micropython - do all of them support DMA?

Anyway – what about this: struct mp_machine_spi_p_t (defined in extmod/machine_spi.h) could get additional fields start_transfer, finish_transfer, transfer_finished, queue_full. These would be pointers to functions for nonblocking data transfers.

Ports that do not support nonblocking SPI data transfers simply set these pointers to NULL. (Such NULL pointers already exist in some ports for the field deinit.)

Ports that support the nonblocking SPI data transfers would implement an internal queue that allows to queue at least two SPI transfers.

int start_transfer(mp_obj_base_t *obj, size_t len, const uint8_t *src, uint8_t *dest) queues an SPI data transfer and returns an ID for the queued transfer. The function blocks when the internal queue is full until another transfer is finished. (It might make sense to add a timeout parameter.)

void finish_transfer(int id) blocks until the given transfer is finished. Again, an additional timeout parameter might make sense.

int transfer_finished(int id) returns 0 if the given transfer is still under way, otherwise it returns 1. (Hrm... What about invalid IDs? Return -1 or perhaps raise a ValueError?)

Changes on the "Python side" would mostly mean to add a parameter blocking=True to the methods read(), read_into(), write(), write_readinto(). Depending on the value of blocking, these methods would either call the existing C function transfer() or the new function start_transfer(). The methods would have to return a transfer ID.

If a port does not support nonblocking transfers, the methods would raise a NotImplementedError if they are called with blocking=False.

Three additional methods would be necessary, finish_transfer(id), transfer_finished(id), queue_full(). They would also raise NotImplementedError if a port does not support nonblocking transfer.

So... Did I miss something important? Does the entire idea make sense? I'd like to offer to work on extmod/machine_spi.c and on the ESP32 port; perhaps also on the ESP8266 port. Can't promise much for other ports. Currently, I have only modules with these processors aside from two Raspberry Pico modules that I bought a few weeks ago – but I did not even open their anti-static bags yet...

[stephanelsmith]

This is a bit beyond me as I haven't explored this at all beyond the top level functions. But SPI being so central/universal for embedded systems, anything with a 50% boost is pretty important in my opinion!

[rkompass]

This is impressing. If you do not get the full attention/feedback of developers here it might be advantageous to submit your developments in form of a pull request or an issue with enhancement label.

As for the Raspberry Pico, after reading your descriptions and findings, I got the idea that all the transfer from the framebuffer memory to the SPI, including color lookup, might be achieved by a chain like this:
The 4-bit-colors are transferred byte-wise via DMA to a PIO channel; The PIO program pushes two look-up-table-addresses from each such byte with two color values. A second DMA-channel does 1-word transfer from the PIO output to the data source address of a third DMA channel thereby triggering its 1-word transfer. The third DMA channel is programmed to transfer the contents of this address - the looked-up color to the SPI. The second channel should be timed according to the SPI transfer rate. The PIO and the first DMA then synchronize themselfes accordingly if the PIO pulls and pushes are in blocking mode.
If you like have a look at https://github.com/orgs/micropython/discussions/11259

[peterhinch]

There is a lot to unpack here.

First the general point: in the nine years I've been involved with MP there have been periodic discussions of nonblocking device drivers. I can't speak for the maintainers but I think there would be interest in a solution subject to agreement on an API. A possible approach would be to raise an RFC issue.

On the specific of micro-gui I'd make the following comments:

• The ILI9341 is a large display requiring a large framebuf, even using 4-bit color. Testing was done on RP2 because I didn't believe that the ESP32 had sufficient RAM to enable it to work. If you have succeeded, I'm impressed :)
• The 4-16 bit conversion is a bottleneck which I've optimised as far as I can in a cross-platform manner using Viper. It is a tradeoff whereby a large RAM saving is achieved at a cost of longer refresh times.
• The latency of long refresh times can cause button-press events to be missed. A latency on the order of 50ms is my target for avoiding this.
• Latency is reduced by splitting the screen into segments. After each segment is updated, the task yields to the scheduler - see code. (The blocking show method is not used). On RP2 response to button presses is good.
• The ESP32 with SPIRAM offers the theoretical possibility of a 16-bit driver, removing the need for the Viper routine. However GC times on such hardware are 100ms (my measurements) and 200ms (other people's). It's possible that button presses will be missed. Further, SPIRAM is much slower than built-in SRAM. I have done no testing with such devices: if you are using one I'd be interested to hear your experience.

the driver uses the SPI bus only for a bit more than 50% of the entire runtime of a display update

Some of that downtime is caused by the mechanism described above where uasyncio periodically yields to run other tasks (e.g. checking pushbuttons). Before spending a lot of time on nonblocking SPI I would check just how much time the Viper code is taking and how much is being used in necessary uasyncio concurrency. Also consider exactly what is the problem you aim to solve. Total refresh time or uasyncio latency?

[peterhinch]

I experimented with delegating refresh to core 1 of the RP2. It worked, but I didn't spend nearly enough time on it to be sure it was reliable. An RP2-only solution fails to meet the project goal of portability.

The "problem" is that, in the work I've done with micro-gui, display refresh time isn't really an issue. There are two criteria I use:

• Responsiveness to button presses - in particular not missing presses.
• Visual refresh artefacts.

I have aimed to ensure adequate performance for practical machine control applications; putting refresh on core 1 made no obvious visual difference.

I appreciate that there are applications requiring lightning-fast responses, particularly games. In my view a framebuffer-based interface is on a complete loser there. The big gain of the framebuffer approach is portability between hosts and between displays.

This isn't to pour cold water on nonblocking SPI: it is something I've wanted for years and I'd back it to the hilt. I'm interested to know whether it makes a visually significant difference to the behaviour of these GUIs.

This time increases of course, if other tasks really run.

Crucially other tasks do run, but quickly. They poll and debounce the buttons. Of course user tasks may also run.

[megazhuk]

Thanks for the suggestion of writing an RFC, And thanks as well to rkompass for the suggestion to write a pull request. I'll think about them. For now, I just want to see if the proposal of nonblocking access to the SPI bus looks overall interesting enough. After all, performance is, albeit important, just one aspect. Another is that a nonblocking API in inherently more complicated to use. Then there would be a slight increase of the size of a firmware image. And there would be more code to maintain...

Non-blocking SPI is very important!
Some time ago, i was think about spent some time and write my own implementation as MP C-module, but I don't have enough knowledge in ESP-IDF, but i keep dreaming about Non-blocking SPI with asyncio. 🥲

On ESP32 i do many of calculations and send them permanently with SPI. Something about 50..100 kilobytes per second.
If I can do calculations and send data at the same time, I can process 1.5-2 times more data. I will be extremely happy!

When I thought about implementing this, I thought that I should do something like this:
spi.write_non_blocked(data :bytes|memoryview, callback_on_finish:awaitable=None)
then check SPI.is_transfer_done() ->bool
or handle flags in callback_on_finish

But your queue-like method is good too. 👍

[peterhinch]

You guys might like to look at the machine.I2S implementation which does offer nonblocking DMA based interface. See this doc from the author Mike Teachman. A lot of discussion went into the API.

This demo uses a StreamWriter to asynchronously output music data from a wav file on disk.

[adeuring]

You guys might like to look at the machine.I2S implementation which does offer nonblocking DMA based interface. See this doc from the author Mike Teachman. A lot of discussion went into the API.

This demo uses a StreamWriter to asynchronously output music data from a wav file on disk.

The nonblocking I2S API is intriguingly simple with its single callback. But I am not sure if this approach works that well with SPI. I2S interfaces are, for obvious reasons, designed to prevent "pauses" in the data stream. The ESP32, for example, has a 32*64 bit FIFO. (Page 305 in the ESP32 technical reference). For 16bit stereo data with 48kHz sampling rate, this means a "buffered time" of 64/48 -> 1.33ms. That's roughly the time the microcontroller software has to start the next DMA transfer. For SPI, the total time for one DMA transfer can be much shorter, like 144µs in my example setup of micro-gui. If you want a "nearly continuous" usage of an SPI interface, using a queue for DMA transfers seems like a more reasonable approach. This DSO screeshot from my experiment with nonblocking SPI transfers shows "gaps" in the SPI clock of about 15 .. 30µs. These "gaps" are probably the time needed by an "ISR mechanism" of ESP-IDF/FreeRTOS to put the "bureaucratic data" of the first transfer into an FreeRTOS queue for finished transfers, to get the next transfer from another FreeRTOS queue and to start that transfer. I doubt that it is possible to initiate a new SPI transfer in this quite short time from a Python callback that runs when the previous SPI transfer finishes.

A callback that runs for a finished SPI transfer, be it defined in a new method SPI.irq() similar to the I2S interface, or be it defined as a parameter of an SPI.read() call, as proposed by megazhuk, still makes sense: After all, one needs a "mechanism" to start processing read data.

But completely avoiding the usage of queues is IMHO not a good idea. Problem is that a queue makes the API a bit more complicated: One would need to know if the next read() or write() call will or will not block.

[peterhinch]

SPI transfers vary in length from one byte to tens of KB. If you assume an SPI clock rate of 10MHz (the low end of the scale), a 10 byte transfer will take 8μs. In a MicroPython context that is "quick": comparable to the overhead of calling the function to send the data. At the short end of the length scale there is little to be gained by attempting concurrent transfer and a simple synchronous function will suffice. Typically with short transfers there are pins to waggle before and after the transfer and concurrency might add complexity to the code.

Where background transfers and uasyncio compliance come into their own is with large block transfers.

I think real-world applications will mix synchronous and asynchronous methods.

[ItsShadowCone]

I'm currently building an embedded platform that needs a lot of bulk transfer. From a software / firmware development view I'd very much appreciate an async API with the semantics that it does the minimum necessary work to get data rolling before yielding, and returns as soon as the operation is finished.

On platform that support e.g. DMA, this could mean the operation yields after writing to memory (and the wait is asynchronous), on platforms that only support blocking operations, this might mean a fallback to the blocking api, a purely software impl impl might or might not yield depending on selected clock speeds or buffer sizes.

Ideally, this behavior could be mirrored across different hardware APIs, and the documentation would state which specific behavior is chosen on which platform.