Finalization in Dynamic Native Modules (.mpy)

[dubiousjim]

I'm experimenting with extending Micropython via .mpy files, and want to provide finalizers to release resources (as a fallback if some extension objects aren't properly __exit__ed or closed). As a test of the basic finalizer functionality, I'm trying to modify examples/natmod/features4.

I'm using the Unix port, but I'd like to understand what's needed and how things work in a general way.

Here's how I started:

--- features4/features4.c	2024-11-29 07:53:11.000000000 -0500
+++ features5/features5.c	2025-04-27 06:21:35.908166739 -0400
@@ -33,7 +33,7 @@
 static mp_obj_t factorial_make_new(const mp_obj_type_t *type, size_t n_args, size_t n_kw, const mp_obj_t *args_in) {
     mp_arg_check_num(n_args, n_kw, 1, 1, false);
 
-    mp_obj_factorial_t *o = mp_obj_malloc(mp_obj_factorial_t, type);
+    mp_obj_factorial_t *o = mp_obj_malloc_with_finaliser(mp_obj_factorial_t, type);
     o->n = mp_obj_get_int(args_in[0]);
 
     if (o->n < 0) {
@@ -53,13 +53,20 @@
 // Implements Factorial.calculate()
 static mp_obj_t factorial_calculate(mp_obj_t self_in) {
     mp_obj_factorial_t *self = MP_OBJ_TO_PTR(self_in);
     return mp_obj_new_int(factorial_helper(self->n));
 }
 static MP_DEFINE_CONST_FUN_OBJ_1(factorial_calculate_obj, factorial_calculate);
 
+static mp_obj_t factorial_deinit(mp_obj_t self_in) {
+    mp_obj_factorial_t *self = MP_OBJ_TO_PTR(self_in);
+    mp_printf(&mp_plat_print, "finalizing calculator(" INT_FMT ")\n", self->n);
+    return mp_const_none;
+}
+static MP_DEFINE_CONST_FUN_OBJ_1(factorial_deinit_obj, factorial_deinit);
+
 // Locals dict for the Factorial type (will have a single method, calculate,
 // added in mpy_init).
-mp_map_elem_t factorial_locals_dict_table[1];
+mp_map_elem_t factorial_locals_dict_table[2];
 static MP_DEFINE_CONST_DICT(factorial_locals_dict, factorial_locals_dict_table);
 
 // This is the entry point and is called when the module is imported
@@ -73,6 +80,7 @@
     mp_type_factorial.name = MP_QSTR_Factorial;
     MP_OBJ_TYPE_SET_SLOT(&mp_type_factorial, make_new, factorial_make_new, 0);
     factorial_locals_dict_table[0] = (mp_map_elem_t){ MP_OBJ_NEW_QSTR(MP_QSTR_calculate), MP_OBJ_FROM_PTR(&factorial_calculate_obj) };
+    factorial_locals_dict_table[1] = (mp_map_elem_t){ MP_OBJ_NEW_QSTR(MP_QSTR___del__), MP_OBJ_FROM_PTR(&factorial_deinit_obj) };
     MP_OBJ_TYPE_SET_SLOT(&mp_type_factorial, locals_dict, (void*)&factorial_locals_dict, 1);
 
     // Make the Factorial type available on the module.

That builds and links fine, but the finalizer never triggers. (I import the .mpy file into micropython, created one of the objects bound to a function's local variable, exit the function, do a gc.collect() several times.)

Some testing revealed that on my configuration, MICROPY_ENABLE_FINALISER wasn't defined, at least when I was compiling the .mpy file. So I added:

CFLAGS += -DMICROPY_ENABLE_FINALISER

to the extension's Makefile. But now linking fails with undefined symbol: mp_obj_malloc_with_finaliser_helper.

I tried copying the definition of that function into my extension, but then linking just fails a further step down the road, with undefined symbol: m_malloc_with_finaliser. I didn't expect that was the right solution anyway.

I've been reading through the Micropython source tree, this forum, and elsewhere. But at the moment, I'm confused enough that I thought it'd be best to ask for help.

1. In addition to talk about "finalisers" (using the __del__ dunder method on the object, and making sure that the object is allocated with mp_obj_malloc_with_finaliser), I've seen some reference to "a deinit function," in a way that suggested this might be a special slot on the object, that worked independently of the __del__ mechanism. Is this right? If so how do I use it?
2. Concerning the __del__ method, do I need to make sure that my micropython runtime was compiled in a special way (specifically, with MICROPY_ENABLE_FINALISER defined) in order for the finalization infrastructure to be present?
3. Even if I do that, how do I get my .mpy toolchain to recognize that the finalization infrastructure is present and can be relied on? Just definining MICROPY_ENABLE_FINALISER seems to give me linking errors, before the .mpy file itself is created, and so before it gets the opportunity to see what is or isn't available in the runtime.

[jonnor]

Hi @dubiousjim . Cool that more are using dynamic native modules!

deinit() is a convention of the machine module. Used to set peripherals back to their initial state. It is symmetrical to init(). This is intended to be used explicitly. For example, before entering a sleep state. So it is not related to object lifecycle or finalisers.

What kind of resources are you looking too free? If it is just about allocated memory or similar, I believe you should use just del. This should get called when an object gets garbage collected. So usually a little bit after it goes out of scope. I am pretty sure that using mp_obj_malloc_with_finaliser is not needed to have this happen.

[dubiousjim]

Client = python code that calls the module supplied by the dynamic library. Am developing for my own use, so will be my own code, though I would share the library with others once I'm confident in its stability. Yes, want to expect the Python code to explicitly close or use a context manager --- but also want to allow for the possibility that especially during development of the client code, buggy code may fail to do so. Want the client code to be able to continue running without keeping the resources alive when they're no longer available. (For example, don't want a buggy client being iteratively developed in the interpreter to force me to restart the interpreter because some resource didn't get released.)

When the client process (which may be an interpreter session) using my library exits, yes the OS will release FDs. I think it won't release flocks, but not sure --- I always have to reload the details of these into my head when I'm working on them, and don't retain all the fine bits. (That's why I want to encapsulate flock handling in a library in the first place!) Child processes that weren't killed and waited for I think will run to completion (which may tie up memory/cpu unnecessarily) then become zombies.

[dubiousjim]

Checked. A flock will be released when its FD is closed, which will happen automatically when the process exits. (Unless the process forked and/or execed without O_CLOEXECing the FD, in which case the parent process' exit may leave the FD holding the flock still open.)

[jonnor]

If the interpreter is open interactively (or otherwise in a long-running process) - but the buggy code holds a reference to some object which has some resources (like flocks, FDs etc) - then finalisers will not be called on the object (as it is still referenced)? So I do not see how it would help...

[dubiousjim]

I was imagining: object from my library gets created, which involves the initiation of some shared resource (FD, flock, child process). But instead of calling inside an context manager, and/or a try...finally block, the calling code omits this, and the resource never gets released. But suppose the reference to the object holding the resource is lost (goes out of scope, gets overwritten, etc.) Then if finalizers were available, there'd be an opportunity to release the resource in a finalizer for the custom class. Of course if a reference to the object is retained, no finaliser would be (or should be) called.

[jonnor]

I think that __del__ will be called in that case, even without finalisers. But it would be wise to test it to make sure :)

[jonnor]

The build of a dynamic module is entirely separate from the MicroPython firmware build. So the symbols (and thus functions) that are available to dynamic native modules are separate. That is why you do not have mp_obj_malloc_with_finaliser_helper. Only symbols that are in the explicitly created symbol table are available.

[dubiousjim]

I wasn't sure whether the build of the interpreter/runtime also created a library that the dynamic modules linked against. So is the situation that, since the mp_obj_malloc_with_finaliser_helper implementation (and the implementation of functions it in turn calls) aren't available for a runtime module to link against, runtime modules aren't able to use the finaliser machinery? In that case, I'd have to either (1) forego the provision of a fallback mechanism to release these resources if the client somehow fails to do so (presumably during development when the client still has issues); or (2) forego the use of dynamic modules here, and instead have my module be part of the build of the interpreter/runtime.

These are in the first place for my own use, so the second option isn't terrible. But I'd still like to go with dynamic modules if possible. And others might find my modules useful (for Unix ports) if I'm able to do so.

[jonnor]

You need to check the symbol table, to see if the function you are interested is in there. I do not know this specific one is.

[dubiousjim]

Which file has the symbol table in question? Is there a runtime library that the dynamic module toolchain is linking against?

[jonnor]

py/nativeglue.h it should be. From the docs

Linker limitation: the native module is not linked against the symbol table of the full MicroPython firmware. Rather, it is linked against an explicit table of exported symbols found in mp_fun_table (in py/nativeglue.h), that is fixed at firmware build time.

https://docs.micropython.org/en/latest/develop/natmod.html

[dubiousjim]

Thanks! I missed that.

[dubiousjim]

I've now managed to get finalisers to work with Dynamic Native Modules. Thanks @jonnor for pointing me in the right direction. This did require making some changes to the interpreter/runtime, and recompiling. Based on the source for MicroPython 1.25.0, testing with the Unix port (specifically testing on Linux with musl libc), here are the changes I needed to make to the interpreter:

diff -ru src/py/dynruntime.h src1/py/dynruntime.h
--- src/py/dynruntime.h	2025-04-15 10:28:30.000000000 -0400
+++ src1/py/dynruntime.h	2025-04-30 11:14:40.000269389 -0400
@@ -66,6 +66,7 @@
 
 #define m_malloc_fail(num_bytes)        (m_malloc_fail_dyn((num_bytes)))
 #define m_malloc(n)                     (m_malloc_dyn((n)))
+#define m_malloc_with_finaliser(n)      (m_malloc_with_finaliser_dyn((n)))
 #define m_free(ptr)                     (m_free_dyn((ptr)))
 #define m_realloc(ptr, new_num_bytes)   (m_realloc_dyn((ptr), (new_num_bytes)))
 #define m_realloc_maybe(ptr, new_num_bytes, allow_move) (m_realloc_maybe_dyn((ptr), (new_num_bytes), (allow_move)))
@@ -92,6 +93,14 @@
     return m_realloc_checked_dyn(NULL, n, false);
 }
 
+static inline void *m_malloc_with_finaliser_dyn(size_t n) {
+    void *ptr = mp_fun_table.alloc_with_finaliser(n);
+    if (ptr == NULL && n != 0) {
+        m_malloc_fail(n);
+    }
+    return ptr;
+}
+
 static inline void m_free_dyn(void *ptr) {
     mp_fun_table.realloc_(ptr, 0, false);
 }
@@ -170,6 +179,7 @@
 #define mp_obj_dict_store(dict, key, val)   (mp_fun_table.dict_store((dict), (key), (val)))
 
 #define mp_obj_malloc_helper(n, t)          (mp_obj_malloc_helper_dyn(n, t))
+#define mp_obj_malloc_with_finaliser_helper(n, t)          (mp_obj_malloc_with_finaliser_helper_dyn(n, t))
 
 static inline mp_obj_t mp_obj_new_str_of_type_dyn(const mp_obj_type_t *type, const byte *data, size_t len) {
     if (type == &mp_type_str) {
@@ -213,6 +223,12 @@
     base->type = type;
     return base;
 }
+
+static inline void *mp_obj_malloc_with_finaliser_helper_dyn(size_t num_bytes, const mp_obj_type_t *type) {
+    mp_obj_base_t *base = (mp_obj_base_t *)m_malloc_with_finaliser(num_bytes);
+    base->type = type;
+    return base;
+}
 
 /******************************************************************************/
 // General runtime functions
diff -ru src/py/nativeglue.h src1/py/nativeglue.h
--- src/py/nativeglue.h	2025-04-15 10:28:30.000000000 -0400
+++ src1/py/nativeglue.h	2025-04-30 11:14:50.200247666 -0400
@@ -179,6 +179,7 @@
     const mp_obj_fun_builtin_var_t *stream_readinto_obj;
     const mp_obj_fun_builtin_var_t *stream_unbuffered_readline_obj;
     const mp_obj_fun_builtin_var_t *stream_write_obj;
+    void *(*alloc_with_finaliser)(size_t n_bytes);
 } mp_fun_table_t;
 
 #if (MICROPY_EMIT_NATIVE && !MICROPY_DYNAMIC_COMPILER) || MICROPY_ENABLE_DYNRUNTIME
diff -ru src/py/nativeglue.c src1/py/nativeglue.c
--- src/py/nativeglue.c	2024-11-29 07:53:11.000000000 -0500
+++ src1/py/nativeglue.c	2025-04-29 23:47:27.349491954 -0400
@@ -254,6 +254,14 @@
 
 #endif
 
+static void *gc_alloc_with_finaliser(size_t n_bytes) {
+#if MICROPY_ENABLE_FINALISER
+    return gc_alloc(n_bytes, GC_ALLOC_FLAG_HAS_FINALISER);
+#else
+    return gc_alloc(n_bytes, 0);
+#endif
+}
+
 // these must correspond to the respective enum in nativeglue.h
 const mp_fun_table_t mp_fun_table = {
     mp_const_none,
@@ -352,6 +360,7 @@
     &mp_stream_readinto_obj,
     &mp_stream_unbuffered_readline_obj,
     &mp_stream_write_obj,
+    gc_alloc_with_finaliser,
 };
 
 #elif MICROPY_EMIT_NATIVE && MICROPY_DYNAMIC_COMPILER
diff -ru src/tools/mpy_ld.py src1/tools/mpy_ld.py
--- src/tools/mpy_ld.py	2025-04-15 10:28:30.000000000 -0400
+++ src1/tools/mpy_ld.py	2025-04-30 11:15:11.970201328 -0400
@@ -1190,6 +1190,7 @@
                 "mp_stream_readinto_obj",
                 "mp_stream_unbuffered_readline_obj",
                 "mp_stream_write_obj",
+                # "gc_alloc_with_finaliser",
             ]
         )
     }

Note the commented out entry at the end of the fun_table dict comprehension in mpy_ld.py. I don't really understand why these entries are handled in this special way, and whether the newly added entry should be treated the same way they are, or should be treated the way entries 50-72 of the mp_fun_table struct in py/nativeglue.[ch] are handled. The former would correspond to uncommenting this line; the latter would correspond to leaving it commented (or deleting it). From my casual testing, it seems that both work. So I'm guessing this line could be omitted. But I wish I understood better why entries 73+ had to be special-cased here, whereas entries 50-72 didn't. (I do have some more understanding of why entries before 50 are handled yet a third way.)

If the interpreter/runtime is compiled like that, with the macro MICROPY_ENABLE_FINALISER defined, then Dynamic Native Modules can make use of finalisers like this. Here is a diff of examples/natmod/features4:

--- features4/Makefile	2025-03-20 02:52:25.869353343 -0400
+++ features5/Makefile	2025-04-30 16:08:00.762470993 -0400
@@ -2,13 +2,15 @@
 MPY_DIR = ../../..
 
 # Name of module
-MOD = features4
+MOD = features5
 
 # Source files (.c or .py)
-SRC = features4.c
+SRC = features5.c
 
 # Architecture to build for (x86, x64, armv7m, xtensa, xtensawin, rv32imc)
 ARCH = x64
 
+CFLAGS_EXTRA += -DMICROPY_ENABLE_FINALISER
+
 # Include to get the rules for compiling and linking the module
 include $(MPY_DIR)/py/dynruntime.mk
--- features4/features4.c	2024-11-29 07:53:11.000000000 -0500
+++ features5/features5.c	2025-04-30 16:07:42.852503056 -0400
@@ -33,7 +33,7 @@
 static mp_obj_t factorial_make_new(const mp_obj_type_t *type, size_t n_args, size_t n_kw, const mp_obj_t *args_in) {
     mp_arg_check_num(n_args, n_kw, 1, 1, false);
 
-    mp_obj_factorial_t *o = mp_obj_malloc(mp_obj_factorial_t, type);
+    mp_obj_factorial_t *o = mp_obj_malloc_with_finaliser(mp_obj_factorial_t, type);
     o->n = mp_obj_get_int(args_in[0]);
 
     if (o->n < 0) {
@@ -57,9 +57,16 @@
 }
 static MP_DEFINE_CONST_FUN_OBJ_1(factorial_calculate_obj, factorial_calculate);
 
+static mp_obj_t factorial_destructor(mp_obj_t self_in) {
+    mp_obj_factorial_t *self = MP_OBJ_TO_PTR(self_in);
+    mp_printf(MP_PYTHON_PRINTER, "finalizing calculator(" INT_FMT ")\n", self->n);
+    return mp_const_none;
+}
+static MP_DEFINE_CONST_FUN_OBJ_1(factorial_destructor_obj, factorial_destructor);
+
 // Locals dict for the Factorial type (will have a single method, calculate,
 // added in mpy_init).
-mp_map_elem_t factorial_locals_dict_table[1];
+mp_map_elem_t factorial_locals_dict_table[2];
 static MP_DEFINE_CONST_DICT(factorial_locals_dict, factorial_locals_dict_table);
 
 // This is the entry point and is called when the module is imported
@@ -73,6 +80,7 @@
     mp_type_factorial.name = MP_QSTR_Factorial;
     MP_OBJ_TYPE_SET_SLOT(&mp_type_factorial, make_new, factorial_make_new, 0);
     factorial_locals_dict_table[0] = (mp_map_elem_t){ MP_OBJ_NEW_QSTR(MP_QSTR_calculate), MP_OBJ_FROM_PTR(&factorial_calculate_obj) };
+    factorial_locals_dict_table[1] = (mp_map_elem_t){ MP_OBJ_NEW_QSTR(MP_QSTR___del__), MP_OBJ_FROM_PTR(&factorial_destructor_obj) };
     MP_OBJ_TYPE_SET_SLOT(&mp_type_factorial, locals_dict, (void*)&factorial_locals_dict, 1);
 
     // Make the Factorial type available on the module.

Note I had to add -DMICROPY_ENABLE_FINALISER to my Makefile, even though when I compiled the interpreter/runtime, this macro was already implicitly defined by the port I chose.

With these changes, then here is a session demonstrating the finaliser works:

MicroPython v1.25.0 on 2025-04-30; linux [GCC 14.2.0] version
Use Ctrl-D to exit, Ctrl-E for paste mode
>>> import gc, features5
>>> def f():
...     x = features5.Factorial(5)
...     print(x.calculate())
... 
>>> f()
120
>>> gc.collect()
finalizing calculator(5)
31

[andrewleech]

I'm working on this for exactly this kind of purpose: #16063