Working on a more complete binding for LVGL

[kdschlosser]

Take a look...

https://github.com/kdschlosser/lvgl_micropython

There are instructions for how to compile. At the moment ESP32 compiled properly. I am not knowledgeable with the other MCU's that MicroPython supports so I have not done anything to get those working. I am currently working on the unix port.

If there is anyone that is familiar with MicroPython C code and wants to lend a hand that would be fantastic. If there is anyone proficient with the RP2 or STM32 MCUs and is willing to lend a hand that would be great!

[AndreoBotelho]

Good work, managed to build to stm32 with little errors but for some reason lvgl is hanging changing tab on tabview and I cant debug so changed the lvgl source with the on from lv_bindings and worked fine.

[kdschlosser]

OK so the problem is in LVGL then?

Maybe open an issue on the LVGL repo about it.

I am using the code from the main branch where as the code use in the official binding doesn't . It is attached to the9.0 release of LVGL. so something is different in the code in the master branch that has broken something.

If you don't mind could you paste the code you are using for the event callback so I can see what part is potentially broken.

If you had to correct any errors in the build system for the STM3 if you could make a PR with what you fixed that would be helpful.

[kdschlosser]

I would love to be able to get DMA memory working for SPI and also get a proper I8080 driver with DMA working as well.

I have been dinking about with it using another library as a guide. The issue is the other library is written to be compile time and not runtime so all the pins and things like that are done using macros.

[kdschlosser]

ok so specifically look at this...

This is from the link you posted.

if (dest == NULL) {
        // send only
        if (len == 1 || query_irq() == IRQ_STATE_DISABLED) {
            status = HAL_SPI_Transmit(self->spi, (uint8_t *)src, len, timeout);
        } else {
            DMA_HandleTypeDef tx_dma;
            dma_init(&tx_dma, self->tx_dma_descr, DMA_MEMORY_TO_PERIPH, self->spi);
            self->spi->hdmatx = &tx_dma;
            self->spi->hdmarx = NULL;
            MP_HAL_CLEAN_DCACHE(src, len);
            uint32_t t_start = HAL_GetTick();
            do {
                uint32_t l = MIN(len, 65535);
                status = HAL_SPI_Transmit_DMA(self->spi, (uint8_t *)src, l);
                if (status != HAL_OK) {
                    break;
                }
                status = spi_wait_dma_finished(self, t_start, timeout);    <------------------  THIS RIGHT HERE 
                if (status != HAL_OK) {
                    break;
                }
                len -= l;
                src += l;
            } while (len);
            dma_deinit(self->tx_dma_descr);
        }

Where I marked completely negates any benefits gotten from DMA memory. It stalls the program until the transfer has completed. The largest purpose to DMA is that it is non blocking to send the data from the memory. That single function call makes it block.

So for all intents and purposes DMA memory is really not being used.

[kdschlosser]

There is also another issue with the code.

It should be written like this. The timeout being cumulative is not ideal because on a large transfer should a problem happen when the first chunk is being sent then guess what? It has to set there for a really long time before it errors. where as setting the smaller amount is the better way to go.

            DMA_HandleTypeDef tx_dma;
            dma_init(&tx_dma, self->tx_dma_descr, DMA_MEMORY_TO_PERIPH, self->spi);
            self->spi->hdmatx = &tx_dma;
            self->spi->hdmarx = NULL;
            MP_HAL_CLEAN_DCACHE(src, len);
            uint32_t t_start;
            do {
                t_start = HAL_GetTick();
               uint32_t l = MIN(len, 65535);
                status = HAL_SPI_Transmit_DMA(self->spi, (uint8_t *)src, l);
                if (status != HAL_OK) {
                    break;
                }
                status = spi_wait_dma_finished(self, t_start, timeout); 
                if (status != HAL_OK) {
                    break;
                }
                len -= l;
                src += l;
            } while (len);
            dma_deinit(self->tx_dma_descr);

[kdschlosser]

I can fix these issues and write a new SPI driver specifically for STM32. But I need to know if there is a way to register a callback that can get called at the end of the last chunk that gets sent. If there is no way to do that then it makes it rather hard to make things work properly.

[andrewleech]

The existing DMA usage was done I believe to be simply slightly faster than just looping in C and/or iterating through a buffer in an interrupt, but yes it's true that DMA could be used for background communication but certainly is not currently.

This is the sort of stuff I wanted to discuss in https://github.com/orgs/micropython/discussions/13081#discussioncomment-7925594 to look at what API changes should be made to support breaking all the blocking code points like this one.

The consensus I got from maintainers is that keeping existing read/write functions is preferred, but add a .setblocking() function like on socket to enable / disable the non-blocking functionality. On top of that in C an .ioctl function responding to MP_STREAM_POLL would provide the standard usage like a stream and from asyncio (not your goal I know) and with the select module as a way to query if finished. Maybe for integration into lvgl usage also having a .irq() that let you register a "finished interrupt callback" would be suitable.

Also, as pointed out in https://github.com/orgs/micropython/discussions/13081#discussioncomment-7926319 any background usage of SPI / I2C should have some consideration for multple devices, ie in SPI case usage of CS line. Certainly when one piece of code is using the SPI bus in a background fashion, some protections will be needed to stop a different part of code also trying to use it at the same time on a different (cs) device.

[kdschlosser]

The CS line has been left up to the user to deal with, The only way to make that work using DMA is the addition of setting start and stop callback functions that get called so the user can set the CS line however they want to. The real trick is how do you go about making the DMA transfers non blocking when multiple devices are involved? I know that the ESP32 handles the CS line internally and you can queue the transfers without any issue. I don't know if that ability exists with STM32.

[AndreoBotelho]

Yes, STM32 has hardware SS pin, I've said it was using DMA, not that it was using in background, someone answered that the non block code must be done on mpy side but DMA is non blocking by nature Em ter., 23 de abr. de 2024 03:10, Kevin Schlosser ***@***.***> escreveu:

…

The CS line has been left up to the user to deal with, The only way to make that work using DMA is the addition of setting start and stop callback functions that get called so the user can set the CS line however they want to. The real trick is how do you go about making the DMA transfers non blocking when multiple devices are involved? I know that the ESP32 handles the CS line internally and you can queue the transfers without any issue. I don't know if that ability exists with STM32. — Reply to this email directly, view it on GitHub <#14317 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AK4TXJ4LL6GWFQ46ANTQB4TY6X3LXAVCNFSM6AAAAABGLFBLMCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCOJWG4YTO> . You are receiving this because you commented.Message ID: ***@***.*** com>

[kdschlosser]

the non blocking code should be written on the C c side of things.

The thing is you need to be able to tell the user when the transfer has finished. This is the key to making the entire thing work. They need to be able to know when they are able to send the next bunch of data. That needs to be in place no matter what. That is what I am currently trying to figure out how to do for the STM32. That and also making use of the slave select pin.
It is looking like the user is not able to select whatever pins they want. It looks like the user has to use pins that are specifically set for the purpose of SPI.

[kdschlosser]

I guess it can be done in C code in order to hold up the next transfer and to handle the callback to release the lock. The real trick here is not stalling until the transfer is done, It is how to hold things up in a manner that doesn't create a spinning wheel and allow other threads to get processed and also scheduled tasks to run.

With the ESP32 threading is available but due to the GIL only one thing is able to run at a time. and in order to get something to run in the main thread from another thread is going to require scheduling the task to run. The time.sleep function is not coded correctly in the unix port.

https://github.com/micropython/micropython/blob/e60e8079a7e1f6edd308e2e8981096e42ec6a78c/ports/unix/modtime.c#L88C17-L88C30

Here you can see the GIL gets released but there is no loop with running of the scheduled tasks between each loop like what you see in say the ESP32 port. I don't know if other ports have a similar problem. The time that is being passed to sleep for should be cut into 1 millisecond chunks and select should be called and stall only for a single millisecond until the total duration is used up. This would allow for running scheduled tasks while a sleep is occurring.

There are inconsistencies in behavior between different ports that makes it very hard to put together something like that I am trying to do. I makes for a very large amount of additional code to be able to handle these differences for the various MCU's. an example is DMA memory and SPI. This is available for ESP32 and for STM32 but what about RP2?

Also, in order to create a DMA buffer for the ESP32 very specific flags need to be used with heap_caps_malloc which is apart of the ESP32 SDK. The best way to deal with transfers like this is to let the user create the buffer to be used and they would be responsible for making the buffer in DMA memory. Then how does one go about checking if the buffer was made in DMA memory? There is no way of finding that information out unless there is something added to the micropython code that would indicate this has been done.

If we look at the code for the ESP32 for SPI we will see that it is wrong. It will never use DMA for the transfer.

int offset = 0;
        int bits_remaining = bits_to_send;
        int optimum_word_size = 8 * self->bits / gcd(8, self->bits);
        int max_transaction_bits = MP_HW_SPI_MAX_XFER_BITS / optimum_word_size * optimum_word_size;
        spi_transaction_t *transaction, *result, transactions[2];
        int i = 0;

        spi_device_acquire_bus(self->spi, portMAX_DELAY);

        while (bits_remaining) {
            transaction = transactions + i++ % 2;
            memset(transaction, 0, sizeof(spi_transaction_t));

            transaction->length =
                bits_remaining > max_transaction_bits ? max_transaction_bits : bits_remaining;

            if (src != NULL) {
                transaction->tx_buffer = src + offset;
            }
            if (dest != NULL) {
                transaction->rx_buffer = dest + offset;
            }

            spi_device_queue_trans(self->spi, transaction, portMAX_DELAY);
            bits_remaining -= transaction->length;

            if (offset > 0) {
                // wait for previously queued transaction
                MP_THREAD_GIL_EXIT();
                spi_device_get_trans_result(self->spi, &result, portMAX_DELAY);
                MP_THREAD_GIL_ENTER();
            }

            // doesn't need ceil(); loop ends when bits_remaining is 0
            offset += transaction->length / 8;
        }

        // wait for last transaction
        MP_THREAD_GIL_EXIT();
        spi_device_get_trans_result(self->spi, &result, portMAX_DELAY);
        MP_THREAD_GIL_ENTER();
        spi_device_release_bus(self->spi);
    }

we can see that here

                transaction->tx_buffer = src + offset;

Unless scr has been allocated using the DMA flag it is not going to use DMA memory.

This is also wrong.

        int max_transaction_bits = MP_HW_SPI_MAX_XFER_BITS / optimum_word_size * optimum_word_size;

DMA transfers are limited to 65595 bytes not 4092 bytes which is what MP_HW_SPI_MAX_XFER_BITS is equal to.

#define MP_HW_SPI_MAX_XFER_BYTES (4092)
#define MP_HW_SPI_MAX_XFER_BITS (MP_HW_SPI_MAX_XFER_BYTES * 8) // Has to be an even multiple of 8

this is from the ESP-IDF

int max_transfer_sz;  ///< Maximum transfer size, in bytes. Defaults to 4092 if 0 when DMA enabled, or to `SOC_SPI_MAXIMUM_BUFFER_SIZE` if DMA is disabled.

The 4092 is NOT the actual maximum. It is simply what it defaults to. That limit only applies when the ESP32 is acting as a slave not a master...
What needs to be set into that field is the actual size of the buffer and not a limited size.

The underlying problem is making the buffer in DMA memory and being able to identify that the buffer is in fact made in DMA memory. that is the only way to know how to send the data.

While it might look like DMA is being used it really is not being used. It would also be a waste of memory and also processor time to create the buffer in DMA memory and then copy the contents from one buffer to the next in order to send it DMA. you loose all advantages of using DMA if that gets done.

[kdschlosser]

I did also want to mention this in the ESP-IDF

esp_err_t spi_bus_get_max_transaction_len(spi_host_device_t host_id, size_t *max_bytes)
{
    SPI_CHECK(is_valid_host(host_id), "invalid host", ESP_ERR_INVALID_ARG);
    if (bus_driver_ctx[host_id] == NULL || max_bytes == NULL) {
        return ESP_ERR_INVALID_ARG;
    }

    spi_host_t *host = bus_driver_ctx[host_id];
    if (host->bus_attr->dma_enabled) {
        *max_bytes = MIN(host->bus_attr->max_transfer_sz, (SPI_LL_DMA_MAX_BIT_LEN / 8));
    } else {
        *max_bytes = MIN(host->bus_attr->max_transfer_sz, (SPI_LL_CPU_MAX_BIT_LEN / 8));
    }

    return ESP_OK;
}

The macro used for the DMA size is dependent on the SOC. This is what it ends up being for the maximum transfer sizes for the different SOC's

The max size for a DMA transfers are

• ESP32-S2: 1,048,576 bytes
• ESP32: 2,097,152 bytes
• ESP32-S2, S3, C2, C3 and H2: 32,768 bytes

[AndreoBotelho]

the dma capable memory allocation on ESP32 is really a error from the port, it could at least consider a dma buffer to split the non dma buffer if it is too large, there's some TFT library I've used on esp32 that does this.
On STM32 most of memory is dma capable and the one allocated for mpy is normally the user memory that has access to all dma channel so it is not a problem, the max transfer size could be fixed to 32k that supports all esp32 variants and is a large size to lvgl fb( you could send the whole screen on ESP32 and S2 fb but will have to allocate 2 fbs and this is a lot of ram to spare), stm32 supports 64k dma transfer too.

[kdschlosser]

with respect to the ESP32 the 32K buffer size isn't really all that large. a 480 x 320 display at 16 bit color depth is going to have a buffer size of 30,720 bytes.bump that to a 480 x 480 you are looking at a buffer size of 46,080 bytes... Where the issue comes in is the spi_transaction_t need to be placed into SRAM, it cannot be put in PSRAM. and you don't want to keep on allocating these things over and over again because of fragmentation.

With how the SPI works is the queue_size needs to be known at the time the device gets created on the bus. That is also the same time the callback gets set for the post transaction cb. what gets passed to the callback is the transaction and only the transaction so __containerof needs to be used for the transaction in order to get the structure that the transaction resides in and in that structure is where the flag is set to indicate if that is the last transaction that is sent. In the callback if that flag is set then the call gets made to the user supplied callback indicating the message has been sent. With this kind of a design you would not need to stall at all if the queue size is large enough because all of the transactions are allocated when the device is added to the SPI.

[kdschlosser]

I am going to write an spi driver for LCD displays from scratch. The reason why is because of esp_lcd using interrupts for the transactions regardless of the type of memory being used.

When using DMA memory the transaction sizes can be pretty large, if the memory is not DMA the maximum transaction size is 64 bytes. when dealing with large amount of data that is being send there is a 20us hit for each transaction because that is how long the interrupt takes. so when sending 30720 bytes of data that turns into 480 transactions that need to occur which results in 480 interrupts that get fired.

This is what the numbers look like when sending 30730 bytes at 80Mhz

interrupt CPU = 0.013152 seconds (13.152 milliseconds) 18,686,131 bps
interrupt DMA = 0.003094 seconds (3.094 milliseconds) 79,431,157 bps

As you can see due to the large number or transactions that occur the speed is 25% of what the DMA speed is.
Now if I change from interrupt to polling transactions this is what you get.

polling CPU = 0.006432 seconds (6.432 milliseconds) 38,208,955 bps

The reason why I am able to change to polling is because the CPU is not going to be able to do any other work while the buffer is sending unless DMA memory is used. so using double buffering becomes pointless at that point so only a single buffer exists and the buffer is not able to be written to at the same time it is transferring so the transmit is able to be a blocking call which is what the polling is. It cuts the transfer time in 1/2 when using a single buffer. The esp_lcd spi bus driver doesn't take this into consideration because it is completely blind to how many buffers exist. With how I wrote the driver for MicroPython the bus driver is aware of the number of buffers being used and the memory allocation flags that are used when that buffer is created.

Since I will be writing a new bus driver from the ground up for SPI I may as well make it so I can use DMA memory from PSRAM which is not supported using the SPI drivers that are in the ESP-IDF. the I8080 and RGB bus drivers in the esp_lcd component do support DMA transfers from PSRAM. This might only for for the ESP32-S3. I have to check on that to be sure.

[AndreoBotelho]

Yes, STM32 does not have gpio matrix like ESP32, you need to use the fixed Peripheral pins Em ter., 23 de abr. de 2024 12:11, Kevin Schlosser ***@***.***> escreveu:

…

the non blocking code should be written on the C c side of things. The thing is you need to be able to tell the user when the transfer has finished. This is the key to making the entire thing work. They need to be able to know when they are able to send the next bunch of data. That needs to be in place no matter what. That is what I am currently trying to figure out how to do for the STM32. That and also making use of the slave select pin. It is looking like the user is not able to select whatever pins they want. It looks like the user has to use pins that are specifically set for the purpose of SPI. — Reply to this email directly, view it on GitHub <#14317 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AK4TXJZRZ4HFZET45HX7WL3Y6Z22HAVCNFSM6AAAAABGLFBLMCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TEMBSGI4TG> . You are receiving this because you commented.Message ID: ***@***.*** com>

[kdschlosser]

OK that is good to know.

[Carglglz]

@kdschlosser
you may be interested in this lvgl/lv_binding_micropython#242 (comment)

[kdschlosser]

??? why would that be of interest? All that needs to be done to update the MicroPython version for what I have is change the version number it is using. 1.23 is a preview which is an alpha test version. It has a while before it is going to be a full release. It is very common for releases of MicroPython to have bugs in it that need to be corrected. Waiting until a release has been out for a few months before pulling it in is a smart thing to do.

That post and linked fork only builds LVGL as a user c module. It doesn't bring anything else to it so it only supports SPI and has different API for the different MCU's. There are no additional display drivers and no additional touch drivers. Once you get into dealing with the busses it will be quickly realized that it is impossible to build without modifying parts of MicroPython. from things like changing the macro definitions to adding repeat include protection in some of the header files needs to be done. That is why I have it done using python, it's a single entry point that is able to make the adjustments that are needed. Important things like building a custom partition file when compiling for ESP32 so the build doesn't fail or compile with partition size that is too large if a user modifies things and disables some of the internal features of MicroPython.

[Carglglz]

??? why would that be of interest?

Sorry I should have provided a bit of context

All that needs to be done to update the MicroPython version for what I have is change the version number it is using. 1.23 is a preview which is an alpha test version.

There is this API change, so gen/lvgl_api_gen_mpy.py and ext_mod/lcd_bus need to be update. Also I think this change is backwards compatible so it should work with previous versions too...

That post and linked fork only builds LVGL as a user c module. It doesn't bring anything else to it so it only supports SPI and has different API for the different MCU's. There are no additional display drivers and no additional touch drivers.

Yes this is intended, AFAIK LVGL is independent of hardware drivers, also you've already done quite a work to provide that 👍🏼 , so I've no intention to recreate that, in fact this is what I'm testing:

lvgl-mpy-drivers$ tree
.
├── README.md
├── display -> ../../lvgl_micropython/api_drivers/common_api_drivers/display
├── display-lvgl -> ../../lvgl_micropython/api_drivers/lvgl_api_drivers/frozen/display
├── indev -> ../../lvgl_micropython/api_drivers/common_api_drivers/indev
├── indev-lvgl -> ../../lvgl_micropython/api_drivers/lvgl_api_drivers/frozen/indev
├── lcd_bus -> ../../lvgl_micropython/ext_mod/lcd_bus
├── manifest.py
├── manifest_display.py
└── manifest_indev.py

6 directories, 4 files

lvgl-mpy-drivers$ cat ../micropython_lvgl.cmake
include(${CMAKE_CURRENT_LIST_DIR}/lv_binding_micropython/micropython.cmake)
include(${CMAKE_CURRENT_LIST_DIR}/lvgl-mpy-drivers/lcd_bus/micropython.cmake)

Once you get into dealing with the busses it will be quickly realized that it is impossible to build without modifying parts of MicroPython. from things like changing the macro definitions to adding repeat include protection in some of the header files needs to be done.

I wasn't aware of this and it does not seems ideal, is this a MicroPython build process bug?

But what you may be interested is in this (from your comment on micropython.cmake)

# this MUST be an execute_process because of the order in which cmake does things
# if add_custom_command is used it errors becasue add_custom_command doesn't
# actually run before the lv_mp.c file gets added to the source list. That causes
# the compilation to error because the source file doesn't exist. It needs to
# exist before it gets added to the source list and this is the only way I have
# found to go about doing it.

I've a solution for this so you can use add_custom_command which is basically writing an empty file, e.g.

file(WRITE ${LV_MP} "")

So now the file exists, the configuring phase will find it then the generating phase will run the custom command and it will generate the proper file with api bindings 👍🏼

[kdschlosser]

I wasn't aware of this and it does not seems ideal, is this a MicroPython build process bug?

There are all kinds of bugs and weird glitches in MicroPython. Here is an example. Trying to import mphalport.h from a user c module doesn't play nice. It is because some, not all of the mphalport.h files do not have protection against getting included more than once. So those files need to be edited and the macros added to guard against declaring things more than once.

You also have things like not all of the ports having an mphalport.h file. It is sometimes named differently. The API's between the things that are in that file are also different, some things that you would think are a function are actually a macro and some things that are macros you would think is a function. There is no way to test to see if the things exist other then putting in specific handling for specific ports. It makes a complete pain in the ass to do.

The single biggest annoyance I had was the partition files in the ESP32 port. The ones that come with MicroPython have the partition sizes set to small to be able to fit LVGL. so you had to manually go in and change the file and then compile again. If you made a change that made the firmware smaller or larger then you have to go back and edit that file and compile again. There is no way to push a custom partition file into the MicroPython build system easily....

OH and the hair pulling trying to figure out how to add the esp_lcd component to the build from the user C module. It's not that the source files weren't getting compiled, it was getting the damned include paths added to the build so they could be included into the user C module. That was quite the hair pulling thing to get working.

There are no guards in place on some of the macros that are set in mpconfigport.h. Do I am not able to change the value of a macro using the -D compiler argument, Those values need to be manually changed in order for it to work.

This is why I used a python script to do all of this. It made it a lot easier to deal with. It handles downloading the ESP-IDF and setting it up properly, and collecting the RP2 SDK. Compiling SDL for use with LVGL on the Unix port instead of using a 2 year old version that is the most recent in the Ubuntu package manager.

It was the way to go to provide a user the ability to do everything needed by entering a single command. It will do everything from start to finish right down to flashing the MCU if that ability is supported.

Yes this is intended, AFAIK LVGL is independent of hardware drivers, also you've already done quite a work to provide that 👍🏼 , so I've no intention to recreate that

My intention was to separate the display IC from what is being used to transfer the data. I put the driver IC code into python so it becomes a runtime thing. The user is able to flash the firmware and change some lines of python code if they want to change displays. No need to compile and write the firmware. I have kept a pretty stable API across all of the displays and different busses. No matter what port is being used. I am able to develop on a desktop using the unix port and literally take the exact same python source file and drop it right into an MCU and it will run without needing to make any changes.

display drivers and indev drivers have frameworks that I wrote that do the majority of the heavy lifting. for displays all that needs to be done is the init commands need to get sent, for the most part everything else is handled by the framework that gets baked into the firmware.

I've a solution for this so you can use add_custom_command which is basically writing an empty file, e.g.

Mine is working, so I am not screwing with it at all. LOL...

There is this API #13763, so gen/lvgl_api_gen_mpy.py and ext_mod/lcd_bus need to be update. Also I think this change is backwards compatible so it should work with previous versions too...

This is a work ion progress. It is not functional at the moment. I had it compiling but it would segfault/crash when it was run. It still needs more work.

[kdschlosser]

While it looks like everything is intertwined in the binding I have been working on it's actually not. The bus drivers are a separate user c module from LVGL. there is nothing in those bus drivers that uses any part of LVGL at all. Those bus drivers can be used with any GUI framework that is able to run on MicroPython.

The drivers that are written in python code for the displays and the indevs, those are LVGL specific, but once again the only thing that makes them a requirement is the build system I wrote. In fact you can compile the binding using make and specifying the user c modules and it will compile without any of the additional drivers... Not all of those drivers get written when running my build system, You have to specify which ones to use. You can tell it to use as many different ones as you want.