declaring an "big buffer" on esp32-s3 (16MB Flash & 8Mb RAM) need "long" time to recover connection after crash

[chf-6309]

Port, board and/or hardware

esp32-s3 "generic" (from AliExpress) YD-ESP32-23 / 2022-v1.3

MicroPython version

MicroPython v1.25.0 on 2025-04-15; Generic ESP32S3 module with Octal-SPIRAM with ESP32S3

Reproduction

#LCD TFT size
width = 320
height = 480

# === Create a framebuffer RGB888 (3 bytes per pixel) ===
def create_framebuffer(width, height):
    return bytearray(width * height * 3)  # 460800 bytes - 461KB

called from main program:
buf = create_framebuffer(width, height)

Expected behaviour

with 8MB available RAM, expect that, even is the buffer isn't used, the access to the esp32 card will be possible.
The size of that buffer should be around 460KB, small compared to the 8MB available

When my program crash during tests, after the buffer allowance, it need minutes or more, to recover the USB link to the card, or before I include an:

del buf
gc.collect()

at the end of the program.

The fastest way is to do an hard RESET

Observed behaviour

the esp32 need minutes (more than five) before communication with PC and (portable) Thonny v4.17 is (again) possible, without the use of the hard reset.

First I didn't use the "Generic ESP32S3 module with Octal-SPIRAM with ESP32S3" firmware, so, the available RAM was little as ~300KB. Withe the Octal version I get;

Mémoire libre : **8312704 octets** [free memory]
stack: 736 out of 15360
GC: total: 64000, used: 8832, free: 55168, max new split: 8257536
 No. of 1-blocks: 176, 2-blocks: 22, max blk sz: 32, max free sz: 3410

MicroPython v1.25.0 on 2025-04-15; Generic ESP32S3 module with Octal-SPIRAM with ESP32S3

Type "help()" for more information.

>>> import gc
>>> gc.mem_alloc()
**10640**
>>> gc.mem_free()
**7785024**

Additional Information

No, I've provided everything above.

Code of Conduct

Yes, I agree

[Josverl]

I think this is too early in troubleshooting to declare this a MicroPython bug. Best to change this tool a Discussion

When using large buffers, the best thing is to allocate them first, before any other activities.

You did not share what other things your main.py does, so it's not clear at what stage you allocate.
You describe issues with network connectivity, but you have not shared how and when that is setup.

It would be helpful if you can share a minimal reproduction, that has the issue. I usually do this by commenting out chunks of code to find the root cause.
Adding prints / logging / turning on leds at certain stages also helps getting insight in what is happening on the board while you wait for reconnection.

[chf-6309]

Hello Josverl,

thank you for your answer.
some more precision, as you ask:

• the size of my"(frame)buffer" is dynamic, i I need full screen (3204803 [one byte for each color]), some time I reduce the size, if I have to display only a small band of pixel on the display.
• on my tests, no network is used, but I mean the USB connection to the board (an USB to serial), sorry for the confusion, so, when a large buffer (max size ~600KB), even if I stop the program with a "CTRL+C", it seams that the program stop, give the hand... but is freezed for long minutes... or back after reset and Thonne IDE re-connection.
• here the content of the main program:

# main start
# hardware configuration
rst = machine.Pin(5)           # reset
cs_ili9488 = machine.Pin(2)    # chip select, for ili9488 module 3.50"
dc = machine.Pin(7)            # data/command for display - common

hspi = machine.SPI(1, baudrate=20000000, bits=8, sck=machine.Pin(14), mosi=machine.Pin(13), miso=machine.Pin(12))
hspi.init()            # Ensure right baudrate
display = ILI9488(hspi, cs=cs_ili9488, dc=dc, rst=rst)

# ili9488 initialyzation
print('init display')
display.init_display()

print('clear display')
display.cls()                          #use direct command to the ili9488
time.sleep_ms(500)
print('green background')
display.cls(0, 255, 0)
time.sleep_ms(500)

gc.collect()

buf = create_framebuffer(width, height) # full screen buffer ~600KB
#buf = create_framebuffer(128, 128)

print('red background using framebuffer') # check gain of speed with framebuffer
color = bytes([255, 0, 0])                # fsetup red color
buf[:] = color * (len(buf) // 3)          # fille the buffer with red color
display.draw_bitmap(0, 0, width, height, buf) # write buffer to ili9488
time.sleep_ms(50)                             # some time, and do the others colors
print('green background using framebuffer')
color = bytes([0, 255, 0])                # green
buf[:] = color * (len(buf) // 3)
display.draw_bitmap(0, 0, width, height, buf)
time.sleep_ms(50)
print('blue background using framebuffer')
color = bytes([0, 0, 255])                # blue
buf[:] = color * (len(buf) // 3)
display.draw_bitmap(0, 0, width, height, buf)
time.sleep_ms(50)


print('display image using framebuffer')
bufimg = bytearray(width * height * 3)        # setup to read a picture
load_bin_to_buffer('homerRGB888.bin', bufimg, 460804) # load picture
display.draw_bitmap(0, 0, width, height, bufimg)      # xfert picture buffer to screen
del bufimg                                    # delet one time image buffer
gc.collect()
time.sleep_ms(2000)

fill_buffer(buf)                              # black background by default
display.cls()                                 # clear screen

gc.collect()

# draw messages at different color/size on the buffer
draw_text_to_buffer(buf, width, 10, 30, "uPython ESP32-S3", 0, 255, 0, scale=2)
display.draw_bitmap(0, 0, width, height, buf)
draw_text_to_buffer(buf, width, 10, 54, "Size 3 test", 0, 0, 255, scale=3)
display.draw_bitmap(0, 0, width, height, buf)
draw_text_to_buffer(buf, width, 10, 86, "Size 4", 255, 0, 0, scale=4)
# then copy the buffer to the ili9488 (and LCD)
display.draw_bitmap(0, 0, width, height, buf)

del buf                                       # no more buffer use 
gc.collect()

for i in range (width):
    y=300+int(128*math.sin(2*math.radians(i)))
    #print(y)
    display.draw_pixel(i, y, 0, 255, 0)       # green sinusoid on display / no frame buffer

buf = create_framebuffer(width, 16)           # create a small framebuffer for 320x16 pixels
t = rtc.datetime()                            # read inernal RTC time
print ('Date: {:02d}/{:02d}/{:04d}_{:02d}:{:02d}:{:02d} * temperature: {:02d}°C'.format(t[2], t[1], t[0], t[4]+5, t[5], t[6], (esp32.mcu_temperature())))

while True:
    tx=0
    ty=12       # time position
    t = rtc.datetime()
    fill_buffer(buf)                         # fill buffer with "0,0,0" color, black
    draw_text_to_buffer(buf, width, 0, 0, 'Date: {:02d}/{:02d}/{:04d}_{:02d}:{:02d}:{:02d} * temp.: {:02d}°C'.format(t[2], t[1], t[0], t[4]+5, t[5], t[6], (esp32.mcu_temperature())), 0, 255, 0, scale=1)
    display.draw_bitmap(tx, ty, width, ty-height, buf) #xfert bramebuffer to LCD controler (320x16)
    time.sleep_ms(980)

del buf
gc.collect()

On the display class: (writes to the LCD controler)

def draw_bitmap(self, x, y, width, height, buf):
        self.write_cmd(0x2A)  # Column Address Set "x"
        self.write_data(bytearray([ x >> 8, x & 0xFF, (x + width - 1) >> 8, (x + width - 1) & 0xFF ]))
        self.write_cmd(0x2B)  # Page Address Set
        self.write_data(bytearray([ y >> 8, y & 0xFF, (y + height - 1) >> 8, (y + height - 1) & 0xFF ]))
        self.write_cmd(0x2C)  # Memory Write
        self.write_data(buf)

The things are working, but if I let that "big" buffer on memory, to have access to the esp32 throught the USB/serial connection, even after an CTRL-C stop,

I hope that those information will help

[GitHubsSilverBullet]

color = bytes([255, 0, 0])                # fsetup red color
buf[:] = color * (len(buf) // 3)          # fille the buffer with red color

The second line actually doubles the (buf) RAM usage temporarily.
Usually not a good idea!

[chf-6309]

Hello GutHubsSilverBullet,

thank your for your comment, but I don't see why the (buf) double in size?
OK, the (buf) is defined by
buf = bytearray(320 * 480 * 3)
so, the size is 460800 bytes
and the 'color' take 3 bytes
color = bytes([255, 0, 0])
so, when I want to fill the (buf) with red color on that example, I "multiply" the 3 color bytes by 1/3rd of the (buf) size:
buf[:] = color * (len(buf) **// 3**)

can you help me to find my error?, thanks

[GitHubsSilverBullet]

This one:
color * (len(buf) **// 3**)
creates a temporary list of the same size as buf[] has. Then copies it, then deletes it.

MicroPython v1.25.0 on 2025-04-15; Raspberry Pi Pico with RP2040
Type "help()" for more information.
>>> import micropython
>>> micropython.mem_info()
stack: 548 out of 7936
GC: total: 245952, used: 4496, free: 241456
 No. of 1-blocks: 55, 2-blocks: 13, max blk sz: 64, max free sz: 15078
>>> ba = bytearray(150_000)
>>> ba[:] = bytes((1,2,3)) * (len(ba)// 3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
MemoryError: memory allocation failed, allocating 150001 bytes

[chf-6309]

OK, thanks, I gat some sense of that on (just finishing) read the "Memoryviews¶"
...
But I thought that I was safe with my "8Mb" on board...

[chf-6309]

I tried your code, to check the differences between the two processors/boards:

MPY: soft reboot
MicroPython v1.25.0 on 2025-04-15; Generic ESP32S3 module with Octal-SPIRAM with ESP32S3

Type "help()" for more information.

>>> import micropython
>>> micropython.mem_info()
stack: 736 out of 15360
GC: total: 64000, used: 11984, free: 52016, max new split: 7733248
 No. of 1-blocks: 256, 2-blocks: 36, max blk sz: 32, max free sz: 3239
>>> ba=bytearray(320*480)
>>> micropython.mem_info()
stack: 736 out of 15360
GC: total: 1306752, used: 772496, free: 534256, max new split: 6684672
 No. of 1-blocks: 81, 2-blocks: 13, max blk sz: 38401, max free sz: 27224
>>> ba[:] = bytes((1,2,3)) * (len(ba)// 3)
>>> micropython.mem_info()
stack: 736 out of 15360
GC: total: 1306752, used: 772496, free: 534256, max new split: 6684672
 No. of 1-blocks: 81, 2-blocks: 13, max blk sz: 38401, max free sz: 27224
>>> 

strange, that after declaring the 320480 buffer, the GC tol jump from 64000 to 1306752
I notice that, when asking "ba[:] = bytes((1,2,3)) * (len(ba)// 3)", I've access immediately to the prompt... but just after, I launch the the last "micropython.mem_info()", my esp32 need around 17 seconds to give me the prompt back!

With 264KB total RAM on RP2040, you "could" make a buffer of 3 * (150_000 // 3), but I understand that the old buffer is replaced(/destroyed may be, GC) after the new buffer is done... got it!

[peterhinch]

I have converted this to a discussion for the reason stated above.

The code sample is incomplete: the symbol ILI9488 is undefined. What driver are you using? It's possible that the driver declares its own FrameBuffer instance (as per this ILI9488 driver).

In any event, as @Josverl states, a large buffer should be declared early. Further, if you want to vary the size, it's best to use a memoryview to perform allocation-free slicing of the buffer.

As a general point to declare a MicroPython bug you should produce a minimal code sample that reproduces the behaviour.

[Josverl]

I think you are trying to do the right thing ( minimizing the data shuttled back and forth ) but by using slices you are creating ( many ) unneeded copies. Its like buying a new car each time you go shopping, its fun, but rather expensive and the parkingspace gets fragmented.

From : https://docs.micropython.org/en/latest/reference/speed_python.html#memoryviews

ba = bytearray(10000)  # big array
func(ba[30:2000])      # a copy is passed, ~2K new allocation
mv = memoryview(ba)    # small object is allocated
func(mv[30:2000])      # a pointer to memory is passed

[chf-6309]

@peterhinch

Hello, first thank you for the huge work done on the graphic LCDs world, and, also, to move my question to the discussions.

Some elements you ask for:

• he symbol ILI9488 is undefined.
  yes, defined, but not "published", here the start of the class:

class ILI9488:
   def __init__(self, spi, cs, dc, rst):
       self.spi = spi
       self.cs = Pin(cs, Pin.OUT, value=1)
       self.dc = Pin(dc, Pin.OUT, value=0)
       self.rst = Pin(rst, Pin.OUT, value=1)

       self.reset()
       self.init_display()

   def reset(self):
       self.rst.value(0)
       time.sleep_ms(20)
       self.rst.value(1)
       time.sleep_ms(120)  # initialization needed time 120 ms (datasheet)

   def write_cmd(self, command):
       self.dc.value(0)
       self.cs.value(0)
       time.sleep_us(1)  # bus stabilization time
       self.spi.write(bytearray([command]))
       self.cs.value(1)

   def write_data(self, data):
       self.cs.value(0)
       time.sleep_us(1)  # bus stabilization time
       self.dc.value(1)
       if isinstance(data, int):
           self.spi.write(bytearray([data]))
       else:
           self.spi.write(data)
       time.sleep_us(1) # write finalization time
       self.cs.value(1)

   def init_display(self):
       self.reset()
       
       # LCD TFT init parameters
       commands = [
           (0xF6, b'\x01\x00\x00'),  # SPI mode
           (0xE0, b'\x00\x03\x09\x08\x16\x0A\x3F\x78\x4C\x09\x0A\x08\x16\x1A\x0F'),  # Positive Gamma Control
           (0xE1, b'\x00\x16\x19\x03\x0F\x05\x32\x45\x46\x04\x0E\x0D\x35\x37\x0F'),  # Negative Gamma Control
           (0xC0, b'\x0d\x0d'),      # Power Control 1 (b'\x17\x15')
           (0xC1, b'\x43\x00'),      # Power Control 2 (0x41)
           (0xC2, b'\x22'),          # Power Control 3 (0x22) ***
           (0xC5, b'\x00\x53\x80'),  # VCOM Control 1 (b'\x00\x12\x80\x12') ***
           (0x3A, b'\x66'),          # Interface Pixel Format, 16 bits par pixel
           (0x36, b'\x48'),          # MX, BGR 48, 28, 08, E8 - BGR, portrait mode, no flip
           (0xB0, b'\x80'),          # Interface Mode Control (0x00)
           (0xB1, b'\xA0\x11'),      # Frame Rate Control 60.76Hz
           (0xB4, b'\x02'),          # Display Inversion Control
           (0xB6, b'\x00'),          # Display Function Control
           (0xE9, b'\x02'),          # Set Image Function
           (0xF7, b'\xA9\x51\x2C\x02'), # Adjust Control b'\xA9\x51\x2C\x82'
           (0x11, None),                # Sleep Out
       ]

       for command, data in commands:
           self.write_cmd(command)
           if data:
               self.write_data(data)

       #self.write_cmd(0x11)   # Sleep OUT
       time.sleep_ms(150)     # wait some time
       self.write_cmd(0x29)   # Display ON : OFF 0x28
       # continue with some graphic definitions

def set_window(self, x0, y0, x1, y1):
def fill_rectangle(self, x, y, width, height, r, g, b):
def cls(self, r=0, g=0, b=0):
...

• What driver are you using?
  I use my "own" driver after reading the datasheet, and after trying some drivers picked from GitHub.
  The problem is that many are designed for a two bytes color (RGB565), but the ili9488 need 18bits, so, three bytes.
  After moving to a three bytes color definition, the LCD start to display things, and what I'm asking for.

• It's possible that the driver declares its own FrameBuffer instance
  no, I was on your drivers definition, and, the MicroPython internal 'framebuf' seams to be well done, but only for 1bit,
  2bits, 4bits, 8bits and RGB565 color schemes...
  My program start with:

import machine
import time
import math
import gc
import esp32
from machine import Pin, SPI, RTC
rtc=RTC()
rtc.datetime()

from fonts.font8x8 import basic #, lisa_console

#LCD TFT size
width = 320
height = 480
...

Because the drawing of pixel/text, directly on the LCD controler (ili9488) was verrrrryyyy long, I tryied that "external framebufer" that fit the ili9488 color request (18bits/3 bytes).
Working on that framebuffer make the things faster, but not instantaneous, even with an 20MHz SPI... OK for my use, an to try to learn MicroPython!

• As a general point to declare a MicroPython bug you should produce a minimal code sample that reproduces the behaviour.
  Yes, but after trying some things on the Thonny IDE, here is some short code, that, when run, freeze the USB communication, and I need to hard reset the esp32:
  MicroPython v1.25.0 on 2025-04-15; Generic ESP32S3 module with Octal-SPIRAM with ESP32S3
  Type "help()" for more information.

>>> buff = bytearray(320 * 480 * 3)
        print(len(buff))
        print(len(buff)//3)
        460800
        153600
>>> print(len(buff))
        print(len(buff)//3)

PROBLEM IN THONNY'S BACK-END: Exception while handling 'get_globals' (serial.serialutil.SerialTimeoutException: Write timeout).
See Thonny's backend.log for more info.
You may need to press "Stop/Restart" or hard-reset your MicroPython device and try again.

So, the issue maybe on the IDE code... :(

@Josverl
Thank you Josver for your complementary informations!
[you must be an very early bird, or verry late...programmer ;o) ]

So, for the cars, I could get an driving licence, but I'm not supposed to know what under the hood (even If I had knowledge and did many fix on my old 205XRD, even replacing the timing belt... ;o)
So, I did read, twice, the MemoryViews on the MicroPython library docs and I will try to apply that "slice" when needed, and declare my FrameBuffer, once, at the top of the program, as you suggest it.

Mybe it could be an enhancement of the internal "FrameBuf", to allow the 18bits colors... but I believe that, expect the buffer filling, would be a long work to fix all graphic primitives embodied on that library, for only one mode byte!

[peterhinch]

@chf-6309 You are right that the ILI9488 doesn't support RGB565 over SPI (despite a contrary indication in the datasheet). The situation is confusing because it does support RGB565 over a parallel interface, and some ILI9488 boards have an SPI to parallel convertor. This can lead developers to think they have written an ILI9488 driver when in fact they have not. The driver I mentioned converts 4-bit color to RGB888 on the fly using a LUT to enable the framebuffer to be small. However it is intended for use with the GUIs and obviously can't display full color images.

Re Thonny I use mpremote on the command line so I have no personal experience but it has emerged as the culprit in a number of support issues.