Mat.addref() sometimes freezes

[sfe-SparkFro]

While testing, I've run into occasional situations where the RP2350 freezes after running some OpenCV function. I eventually poked around with a debugger, and it turns out that calling Math.addref() can sometimes freeze for an unknown reason.

Math.addref() calls CV_XADD(&u->refcount, 1), which in turn calls the platform's atomic add function (CV_XADD macro magic here). I'm not sure what that actually is for the RP2350, nor why it would cause a freeze.

There are underlying OpenCV calls to both Math.addref() and CV_XADD(&u->refcount, 1), so can't necessarily do anything here. But might be able to define a special CV_XADD that just increments without an atomic operation. Should be safe because of the Python GIL, right?

Will follow up with replication code if I can find something that easily replicates the problem.

[sfe-SparkFro]

Replication code. This draws a moving square bouncing around the screen:

import cv2
from ulab import numpy as np
from st7789_spi import ST7789_SPI

display = ST7789_SPI(width=240,
                    height=320,
                    spi_id=0,
                    pin_cs=17,
                    pin_dc=16,
                    rotation=1)

img = np.zeros((240, 320, 3), dtype=np.uint8)

x = 0
y = 0
vx = 10
vy = 10

while True:
    # Update position
    x += vx
    y += vy
    if x <= 0 or x >= 320 - 50:
        vx = -vx  # Reverse direction on x-axis
    if y <= 0 or y >= 240 - 50:
        vy = -vy # Reverse direction on y-axis

    # Draw a square
    img[y:y+50, x:x+50] = (0, 0, 255)

    cv2.imshow(display, img)

    # Clear the square area for the next frame
    img[y:y+50, x:x+50] = (0, 0, 0)

This runs for 5-10 seconds, then freezes. With a debugger connected, it does not hit any kind of breakpoint, I have to manually halt the processor. The backtrace is below:

^C
Thread 1 "rp2350.dap.core0" received signal SIGINT, Interrupt.
0x101c15fa in cv::Mat::addref (this=this@entry=0x20081b40) at /home/dryw/micropython-opencv/src/opencv/modules/core/src/matrix.cpp:544
544             CV_XADD(&u->refcount, 1);
(gdb) bt
#0  0x101c15fa in cv::Mat::addref (this=this@entry=0x20081b40)
    at /home/dryw/micropython-opencv/src/opencv/modules/core/src/matrix.cpp:544
#1  0x10067b58 in ndarray_to_mat (ndarray=0x110674d0)
    at /home/dryw/micropython-opencv/src/convert.cpp:115
#2  0x10067bba in mp_obj_to_mat (obj=<optimized out>)
    at /home/dryw/micropython-opencv/src/convert.cpp:144
#3  0x100885ea in cv2_imgproc_cvtColor (n_args=<optimized out>, 
    pos_args=<optimized out>, kw_args=<optimized out>)
    at /home/dryw/micropython-opencv/src/imgproc.cpp:366
#4  0x100115f6 in fun_builtin_var_call (args=0x20081d04, n_kw=0, n_args=3, 
    self_in=0x102c94a0 <cv2_imgproc_cvtColor_obj>)
    at /home/dryw/micropython-opencv/micropython/py/objfun.c:118
#5  fun_builtin_var_call (self_in=0x102c94a0 <cv2_imgproc_cvtColor_obj>, n_args=3, 
    n_kw=0, args=0x20081d04)
    at /home/dryw/micropython-opencv/micropython/py/objfun.c:102
#6  0x1001bf16 in mp_execute_bytecode (code_state=code_state@entry=0x20081ce8, 
    inject_exc=inject_exc@entry=0x0)
    at /home/dryw/micropython-opencv/micropython/py/vm.c:1042
#7  0x10011716 in fun_bc_call (self_in=0x2003aad0, n_args=<optimized out>, n_kw=<optimized out>, args=0x110673c8) at /home/dryw/micropython-opencv/micropython/py/objfun.c:295
#8  0x1001bf16 in mp_execute_bytecode (code_state=code_state@entry=0x110673b0, inject_exc=inject_exc@entry=0x0) at /home/dryw/micropython-opencv/micropython/py/vm.c:1042
#9  0x10011716 in fun_bc_call (self_in=0x2003aae0, n_args=<optimized out>, n_kw=<optimized out>, args=0x20081e20) at /home/dryw/micropython-opencv/micropython/py/objfun.c:295
#10 0x1006e684 in cv2_highgui_imshow (n_args=<optimized out>, pos_args=<optimized out>, kw_args=<optimized out>) at /home/dryw/micropython-opencv/src/highgui.cpp:45
#11 0x100115f6 in fun_builtin_var_call (args=0x2003a4c0, n_kw=0, n_args=2, self_in=0x102c93c8 <cv2_highgui_imshow_obj>) at /home/dryw/micropython-opencv/micropython/py/objfun.c:118
#12 fun_builtin_var_call (self_in=0x102c93c8 <cv2_highgui_imshow_obj>, n_args=2, n_kw=0, args=0x2003a4c0) at /home/dryw/micropython-opencv/micropython/py/objfun.c:102
#13 0x1001bf16 in mp_execute_bytecode (code_state=code_state@entry=0x2003a4a0, inject_exc=inject_exc@entry=0x0) at /home/dryw/micropython-opencv/micropython/py/vm.c:1042
#14 0x10011716 in fun_bc_call (self_in=0x2003a440, n_args=<optimized out>, n_kw=<optimized out>, args=0x0) at /home/dryw/micropython-opencv/micropython/py/objfun.c:295
#15 0x100316e6 in parse_compile_execute (source=source@entry=0x20081f88, input_kind=<optimized out>, exec_flags=exec_flags@entry=22) at /home/dryw/micropython-opencv/micropython/shared/runtime/pyexec.c:119
#16 0x100319ae in pyexec_friendly_repl () at /home/dryw/micropython-opencv/micropython/shared/runtime/pyexec.c:671
#17 0x10034006 in main (argc=<optimized out>, argv=<optimized out>) at /home/dryw/micropython-opencv/micropython/ports/rp2/main.c:236

FWIW this is with the fix from a2bf9f2

[sfe-SparkFro]

Huh:

#0  0x101c15fa in cv::Mat::addref (this=this@entry=0x20081b40)
    at /home/dryw/micropython-opencv/src/opencv/modules/core/src/matrix.cpp:544
544             CV_XADD(&u->refcount, 1);
(gdb) p u->refcount
$4 = 784943609

Now why the heck would u->refcount = 784943609?

[sfe-SparkFro]

If I add gc.collect() to the end of the loop, it seems to run fine. Sounds like something is going wonky with memory allocation

[sfe-SparkFro]

Now why the heck would u->refcount = 784943609?

Because CV_XADD(&u->refcount, 1) is running in a continual loop for some reason. If I let it run for a moment then halt again, the refcount value has increased by a lot.

Now why the heck would CV_XADD(&u->refcount, 1) be running in a continual loop?

[sfe-SparkFro]

Did a clean build of OpenCV and MicroPython with this change, but still freezes.

Edit: I had 2 copies of this project on my machine, and I edited the wrong file. This actually fixes the problem!

[sfe-SparkFro]

Running this code #12 (comment) with the addition of print(gc.mem_free()) at the end of the loop, prints the following:

8077520
8076208
8074896
8073584
...
7969936
7968624
7967312
7966000

So the GC still has plenty of memory available. Or at least, so it claims.

[sfe-SparkFro]

Edited #12 (comment)

So, not using the atomic operation seems to fix it. Not sure why that would make a difference. Perhaps there's a bug in the Pico SDK? This comment states there was a bug in Pico SDK 2.0.0, so perhaps there's another issue? I can only speculate.

I think for now, we can just #define CV_XADD through a custom port header (see OPENCV_INCLUDE_PORT_FILE), as indicated by the #error message at the end of the CV_XADD macro block. Doesn't feel like the "proper" solution, but if it works...

[sfe-SparkFro]

I should have read a couple comments further. Sounds like atomic operations are not supported for data in PSRAM. Given that my last test showed the system froze around 8MB remaining, I suspect the reason the replication code failed after some time is because that's when it just happened to start using the PSRAM. That would also explain why adding gc.collect() fixed the problem.

[sfe-SparkFro]

Yep! Here's the iteration right before the replication code freezes:

(gdb) print &(u->refcount)
$19 = (int *) 0x2007ee74

And here's the iteration when it does freeze:

(gdb) print &(u->refcount)
$20 = (int *) 0x11067604

So the atomic operations don't seem to work with memory in PSRAM. Not sure what the current state of PSRAM support is in the Pico SDK, but that seems to be the root problem.

[sfe-SparkFro]

Created raspberrypi/pico-sdk#2505

[sfe-SparkFro]

Fixed in 1d27888