Issues with framebuffer i2c display locking up due to memory allocation failure after a few iterations

[matteius]

Howdy, thanks for micropython -- its great! I am the author of a deployable routine for automating indoor plant growing https://github.com/opensensor/growmax and I have run into an issue with the pico w and the Framebuffer which is trying to allocate memory when I call .text() and failing after some iterations of the routine. I don't have this issue on a similar sized 2-bit grayscale display, so part of it may be the extra contiguous memory required. I can run the same code fine on an ESP32S3 so I don't believe there is a memory leak or it would eventually run into the same issue -- also I invoke the garbage collector consistently throughout the routine and it is reporting back most of the memory is free before I invoke the framebuffer.text calls, which apparently each one re-allocates the buffer? I am confused on this point as well: I passed in an initially allocated buffer and never assumed it was getting reallocated, but I imagine its possible for reasons. Still I am confused how it could be failing to allocate memory.

Currently running v1.19.1-966-g05bb26010 on 2023-03-13 and also tried 04-11 with the same issue. I get a different issue with the I2C not being able to initialize properly at all on the latest stable 1.20 release and latest nightly builds unless I use SoftI2C which I don't want to do.

Local to Thonny I've tried many refactors without success (removing all print() statements, refactoring the SSD1327 display class down to be a subclass of framebuffer so it was even lighter weight, removing the Timer class I recently added for motion sensors and just had the display on the entire time, and within 1-4 iterations the display freezes, Thonny sees that memory allocation failed, and the rest of the routine continues to function albeit with a frozen display. Any ideas?

Free mem after garbage collection:  138560
memory allocation failed, allocating 8193 bytes
Exception:  memory allocation failed, allocating 8193 bytes
memory allocation failed, allocating 8193 bytes
Exception:  memory allocation failed, allocating 8193 bytes
memory allocation failed, allocating 8193 bytes
Exception:  memory allocation failed, allocating 8193 bytes
memory allocation failed, allocating 8193 bytes
Exception:  memory allocation failed, allocating 8193 bytes
memory allocation failed, allocating 8193 bytes
Exception:  memory allocation failed, allocating 8193 bytes
memory allocation failed, allocating 8193 bytes
Exception:  memory allocation failed, allocating 8193 bytes
memory allocation failed, allocating 8193 bytes
Exception:  memory allocation failed, allocating 8193 bytes
memory allocation failed, allocating 8193 bytes
Exception:  memory allocation failed, allocating 8193 bytes

[peterhinch]

It might be best to take this up with the author.

Evidently you have plenty of free RAM - 138560bytes - but something in the code is attempting to allocate 8193 bytes. You are probably suffering from memory fragmentation. You could prove this (rather than the memory leak hypothesis) by having your exception handler do

gc.collect()
print(gc.mem_free())  # I'll bet you still have plenty

You can't defragment RAM.

I would try to find out what is allocating such a large buffer and check whether it can be pre-allocated. It is wise to allocate any large buffers at the program start, before fragmentation can take hold. Then re-use the existing buffers as needed.

[matteius]

I am the author (sorry if that wasn't clear) so that is why I created this discussion about the Framebuffer.

I would try to find out what is allocating such a large buffer

The code for the Framebuffer is what is allocating 8193 bytes but it doesn't really make sense that it continues to re-allocate it. I added a print statement to the initialize and have shown its only ever initialized once. However, it seems that calling .text() on the micropython framebuffer causes it to reallocate the framebuffer (and presumably free the old one since there isn't actually a leak). The size being allocated is the sames as:

bytearray(self.width * self.height // 2) == 128*128/2 == 8192 which is one byte less than what is being allocated when calling .text() on the framebuffer.

Here is the framebuffer sub-class:

from micropython import const
import framebuf

# commands
SET_COL_ADDR          = const(0x15)
SET_SCROLL_DEACTIVATE = const(0x2E)
SET_ROW_ADDR          = const(0x75)
SET_CONTRAST          = const(0x81)
SET_SEG_REMAP         = const(0xA0)
SET_DISP_START_LINE   = const(0xA1)
SET_DISP_OFFSET       = const(0xA2)
SET_DISP_MODE         = const(0xA4) # 0xA4 normal, 0xA5 all on, 0xA6 all off, 0xA7 when inverted
SET_MUX_RATIO         = const(0xA8)
SET_FN_SELECT_A       = const(0xAB)
SET_DISP              = const(0xAE) # 0xAE power off, 0xAF power on
SET_PHASE_LEN         = const(0xB1)
SET_DISP_CLK_DIV      = const(0xB3)
SET_SECOND_PRECHARGE  = const(0xB6)
SET_GRAYSCALE_TABLE   = const(0xB8)
SET_GRAYSCALE_LINEAR  = const(0xB9)
SET_PRECHARGE         = const(0xBC)
SET_VCOM_DESEL        = const(0xBE)
SET_FN_SELECT_B       = const(0xD5)
SET_COMMAND_LOCK      = const(0xFD)

# registers
REG_CMD  = const(0x80)
REG_DATA = const(0x40)

class SSD1327(framebuf.FrameBuffer):
    def __init__(self, width=128, height=128):
        self.width = width
        self.height = height
        self.buffer = bytearray(self.width * self.height // 2)
        super().__init__(self.buffer, self.width, self.height, framebuf.GS4_HMSB)

        self.col_addr = ((128 - self.width) // 4, 63 - ((128 - self.width) // 4))
        self.row_addr = (0, self.height - 1)
        self.offset = 128 - self.height
        self.framebuf = self

        self.poweron()
        self.init_display()

    def init_display(self):
        for cmd in (
            SET_COMMAND_LOCK, 0x12, # Unlock
            SET_DISP, # Display off
            # Resolution and layout
            SET_DISP_START_LINE, 0x00,
            SET_DISP_OFFSET, self.offset, # Set vertical offset by COM from 0~127
            # Set re-map
            # Enable column address re-map
            # Disable nibble re-map
            # Horizontal address increment
            # Enable COM re-map
            # Enable COM split odd even
            SET_SEG_REMAP, 0x51,
            SET_MUX_RATIO, self.height - 1,
            # Timing and driving scheme
            SET_FN_SELECT_A, 0x01, # Enable internal VDD regulator
            SET_PHASE_LEN, 0x51, # Phase 1: 1 DCLK, Phase 2: 5 DCLKs
            SET_DISP_CLK_DIV, 0x01, # Divide ratio: 1, Oscillator Frequency: 0
            SET_PRECHARGE, 0x08, # Set pre-charge voltage level: VCOMH
            SET_VCOM_DESEL, 0x07, # Set VCOMH COM deselect voltage level: 0.86*Vcc
            SET_SECOND_PRECHARGE, 0x01, # Second Pre-charge period: 1 DCLK
            SET_FN_SELECT_B, 0x62, # Enable enternal VSL, Enable second precharge
            # Display
            SET_GRAYSCALE_LINEAR, # Use linear greyscale lookup table
            SET_CONTRAST, 0x7f, # Medium brightness
            SET_DISP_MODE, # Normal, not inverted
            SET_COL_ADDR, self.col_addr[0], self.col_addr[1],
            SET_ROW_ADDR, self.row_addr[0], self.row_addr[1],
            SET_SCROLL_DEACTIVATE,
            SET_DISP | 0x01): # Display on
            self.write_cmd(cmd)
        self.fill(0)
        self.write_data(self.buffer)

    def poweroff(self):
        self.write_cmd(SET_FN_SELECT_A)
        self.write_cmd(0x01) # Enable internal VDD regulator
        self.write_cmd(SET_DISP)

    def poweron(self):
        self.write_cmd(SET_FN_SELECT_A)
        self.write_cmd(0x01) # Enable internal VDD regulator
        self.write_cmd(SET_DISP | 0x01)

    def contrast(self, contrast):
        self.write_cmd(SET_CONTRAST)
        self.write_cmd(contrast) # 0-255

    def rotate(self, rotate):
        self.poweroff()
        self.write_cmd(SET_DISP_OFFSET)
        self.write_cmd(self.height if rotate else self.offset)
        self.write_cmd(SET_SEG_REMAP)
        self.write_cmd(0x42 if rotate else 0x51)
        self.poweron()

    def invert(self, invert):
        self.write_cmd(SET_DISP_MODE | (invert & 1) << 1 | (invert & 1)) # 0xA4=Normal, 0xA7=Inverted

    def show(self):
        self.write_cmd(SET_COL_ADDR)
        self.write_cmd(self.col_addr[0])
        self.write_cmd(self.col_addr[1])
        self.write_cmd(SET_ROW_ADDR)
        self.write_cmd(self.row_addr[0])
        self.write_cmd(self.row_addr[1])
        self.write_data(self.buffer)

    def write_cmd(self):
        raise NotImplementedError

    def write_data(self):
        raise NotImplementedError


class SSD1327_I2C(SSD1327):
    def __init__(self, width, height, i2c, addr=0x3c):
        self.i2c = i2c
        self.addr = addr
        self.cmd_arr = bytearray([REG_CMD, 0])
        self.data_list = [bytes((REG_DATA,)), None]
        super().__init__(width, height)

    def write_cmd(self, cmd):
        self.cmd_arr[1] = cmd
        self.i2c.writeto(self.addr, self.cmd_arr)

    def write_data(self, data_buf):
        self.data_list[1] = data_buf
        self.i2c.writevto(self.addr, self.data_list)

[matteius]

Ah the above class is my refactored one which has an infinite recursion error, none the less I checked gc.mem_free() and it was in fact fragmenting over time. I changed two things -- I moved all the utime.sleep() calls to time.sleep() and I changed in the moisture class:
self._history = self._history[:self._history_length] -> self._history = [] since it can reset the state after the reading is made. Not sure which change fixed it, but I don't see the fragmentation happening anymore.

[matteius]

On further analysis, it was the array slicing and not utime -- thanks for your help!