How to pass a readable and writeable bytearray to a C module? Or how can I prevent the C module allocate memory section that is already used by micropython?

[nspsck]

Hi guys, I am currently working on a tft display driver. The tft panel is huge, 536*240. For 16bits-color, a whole frame will take roughly 250KB of ram. By doing this, I learned, that using a framebuffer in C modules can be very dangerous. Since the C modules have no idea about if either a section of RAM has been used or not, it could cause data corruption. In my case, the device just triggered a reset. Half of the amount RAM I needed is quite stable, but again, I have no idea if it is guaranteed, not to use the section that has meaningful data.

If I use the bytearray, I could not find a way to modify it in the C module, hence I have to use a Frambuffer in the framebuf module. This cost me 22% performance. (Simply filling the whole screen with one color dropped down to 54 fps from 66 fps) So... ye... I really do not want to do complicated drawing in python, since they will be done very slowly.

Since I am also new to C and python and tft drivers, please do assume that I haven't thought through some very simple and effective solutions and please do let me know about those! Thank you very much in advance!

ps. chunking is not preferred, unless there is a way to guarantee that the buffer for the chunks are safe to use.

[glenn20]

You can pass a readable and writable bytearray to a C module quite easily, in fact the following example function in a C module will accept a bytearray, memoryview or array (or any r/w python object which supports the "buffer protocol") (I'm not sure if that actually will help with your application):

STATIC mp_obj_t my_function(mp_obj_t buf_obj) {
    // Get a pointer to the data of the supplied buffer (eg. bytearray)
    mp_buffer_info_t buf;
    mp_get_buffer_raise(buf_obj, &buf, MP_BUFFER_RW);

    // buf.len is length of the supplied data buffer.
    // buf.buf is a void * ptr to the contiguous data underlying the bytearray or array
    // Process the bytearray underlying data....

    return mp_const_none;
}
STATIC MP_DEFINE_CONST_FUN_OBJ_1(mymod_my_function, my_function);

I assume you know how to add that function to your module.

[iBigDane]

https://github.com/micropython/micropython/blob/01c758e26a846eca42e9c4444639fa822adf005f/extmod/modframebuf.c#L226C1-L232C2
use the memset function,set one line pixel color.

[nspsck]

Actually, that's what I have been using (Or I think I am using it...). If you want to view the whole implementation you can check out this.

STATIC void fill_color_buffer(mp_lcd_rm67162_obj_t *self, uint16_t color, int len /*in pixel*/) {
    uint16_t *buffer = self->frame_buffer;
    size_t size = len;
    while (size--) {
        *buffer++ = color;
    }
    write_color(self, self->frame_buffer, len * 2);
}

This is probably, at least to my eyes, the same function as memset(). Meanwhile I found out the bug that causes mpy to reboot. That's because I set more memory in this function than I actually allocated. (Ye... This function, as same as memset(), does not check the end of the frame_bufferpointer, which should have been done.. Absolutely my bad.) If the way how frame_buffer is allocated has been changed the performance difference will occur. And I use this function to allocate memory for the buffer:

STATIC void frame_buffer_alloc(mp_lcd_rm67162_obj_t *self, int len) {
    // create a constant DMA-enabled frambuffer.
    self->frame_buffer_size = len;
    //self->frame_buffer = heap_caps_malloc(self->frame_buffer_size, MALLOC_CAP_DMA);
    self->frame_buffer = gc_alloc(self->frame_buffer_size, 0);
    
    if (self->frame_buffer == NULL) {
        mp_raise_msg(&mp_type_OSError, MP_ERROR_TEXT("Failed to allocate DMA'able framebuffer."));
    }
    memset(self->frame_buffer, 0, self->frame_buffer_size);
}

Using gc_alloc() results a lower performance than using heap_caps_malloc().

[iBigDane]

For example, the screen resolution is 240*240, and the number of loops in your code is 57600. If you use memset to set the pixels of one row at a time, the number of loops is 240.I think so.
The gc_alloc() function should allocate SPIRAM

[nspsck]

Hi, thank you for all your advices so far. You are right! I checked later and memset() is indeed faster than my original function. It looks like the compiler knows how to optimise for 8-bits memory manipulation but does not know how to optimise for 16-bits memory manipulation. The source code of memset() seems to be this:

void *memset(void *s, int c, size_t n) {
    if (c == 0 && ((uintptr_t)s & 3) == 0) {
        // aligned store of 0
        uint32_t *s32 = s;
        for (size_t i = n >> 2; i > 0; i--) {
            *s32++ = 0;
        }
        if (n & 2) {
            *((uint16_t*)s32) = 0;
            s32 = (uint32_t*)((uint16_t*)s32 + 1);
        }
        if (n & 1) {
            *((uint8_t*)s32) = 0;
        }
    } else {
        uint8_t *s2 = s;
        for (; n > 0; n--) {
            *s2++ = c;
        }
    }
    return s;
}

Sadly, If i just call that function as below

memset(self->frame_buffer, color, len * 2); 
// the color is 2 bytes, and since color != 0, it will be a byte-wise operation
// hence if I were calling len, only half of the screen area will be drawn.

The color on the display seems to be broken. I.e. (255, 0, 0) will look like pink/purple and (0, 255, 0) will be green and (0, 0, 255) will just stay black. So I tried to modify my own code in order to match the performance. So I did this:

STATIC void fill_color_buffer(mp_lcd_rm67162_obj_t *self, uint16_t color, int len /*in pixel*/) {
    uint32_t *buffer = (uint32_t *)self->frame_buffer;
    // this ensures that the framebuffer is overfilled rather than unfilled.
    size_t size = (len + 1) / 2; 
    while (size--) {
        // ye, well, this should not work, but it works................but why?
        *buffer++ = color;
    }
    write_color(self, self->frame_buffer, len * 2);
}

and I've noticed, the *buffer++ = color; part should not work since the upper 16 bits is not controled by me at all... but to my surprise.. it works flawlessly...... Could you please give some possible explanations? I am using a display controler named RM67162.
Either way, the performance are very similar right now.

Also, the bytearray and gc_alloc thing... Due to this issue #11853 , I can not make any accurate statements about what gc actually allocates.

[iBigDane]

This is my mistake, if you are using rgb565 color mode, the method of setting pixels cannot use memset().
https://github.com/micropython/micropython/blob/01c758e26a846eca42e9c4444639fa822adf005f/extmod/modframebuf.c#L130C1-L138C2
As for the performance problem, there is no idea at present.