sigsegv on exit for static PyObject Py_XDECREF

[ovesh]

Bug report

Bug description:

Noticed that on py312 a C extension I maintain is segfaulting when python exits. Managed to create a minimal C extension that reproduces the issue:

#include <Python.h>
#include <memory>

struct PyObjectDecrementer {
	void operator()(PyObject* obj) const {
		Py_XDECREF(obj);
	};
};

static std::unique_ptr<PyObject, PyObjectDecrementer> stored_obj = nullptr;

static PyObject* set_stored_int(PyObject* self, PyObject* args) {
    const int* input_int;

    if (!PyArg_ParseTuple(args, "i", &input_int)) {
        printf("failed to parse input int\n");
        return NULL;
    }

    PyObject* new_int = PyLong_FromLong((long)input_int);
    if (!new_int) {
        printf("failed to create new int object\n");
        return NULL;
    }

    stored_obj.reset(new_int);

    Py_RETURN_NONE;
}

static PyMethodDef HelloMethods[] = {
    {"set_stored_int", set_stored_int, METH_VARARGS, "Store a Python int in a static variable."},
    {NULL, NULL, 0, NULL}
};

static struct PyModuleDef hellomodule = {
    PyModuleDef_HEAD_INIT, "hello", NULL, -1, HelloMethods
};

PyMODINIT_FUNC PyInit_hello(void) {
    return PyModule_Create(&hellomodule);
}

Minimal setup.py:

from setuptools import setup, Extension
hello_module = Extension('hello', sources=['library.cpp'])
setup(name='hello', version='1.0', description='xxxyyy', ext_modules=[hello_module])

% python
Python 3.12.7 | packaged by conda-forge | (main, Oct  4 2024, 15:55:29) [Clang 17.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import hello
>>> hello.set_stored_int(23)
>>> exit()
zsh: segmentation fault  python

Occurs at least on linux amd64 and MacOS. Narrowed it down with git bisect to this commit: df3173d

CPython versions tested on:

3.12

Operating systems tested on:

Linux, macOS

[s22chan]

(release) backtrace implies it wasn't able to get a non null OMState/interpreter (after skipping past the assert) at

cpython/Objects/obmalloc.c

Line 1378 in 2e95c5b

assert(interp->obmalloc != NULL); // otherwise not initialized or freed

if I read it right.

Process 15740 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = instruction step into
    frame #0: 0x00007ff814b4e592 libdyld.dylib`tlv_get_addr + 13
libdyld.dylib`tlv_get_addr:
->  0x7ff814b4e592 <+13>: testq  %rax, %rax
    0x7ff814b4e595 <+16>: je     0x7ff814b4e59c ; <+23>
    0x7ff814b4e597 <+18>: addq   0x10(%rdi), %rax
    0x7ff814b4e59b <+22>: retq
Target 0: (python) stopped.
(lldb)
Process 15740 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = instruction step into
    frame #0: 0x00007ff814b4e595 libdyld.dylib`tlv_get_addr + 16
libdyld.dylib`tlv_get_addr:
->  0x7ff814b4e595 <+16>: je     0x7ff814b4e59c ; <+23>
    0x7ff814b4e597 <+18>: addq   0x10(%rdi), %rax

that ends up crashing it here:

* thread #1, queue = 'com.apple.main-thread', stop reason = instruction step into
    frame #0: 0x00000001000f6ffc python`_PyObject_Free + 28
python`_PyObject_Free:
->  0x1000f6ffc <+28>: movq   0x10(%rax), %rbx

[asvetlov]

If I understand correctly, segfault is raised on calling stored_obj destructor which is C++ class instance.

At this moment Python interpreter is already shut down and destroyed.

Py_XDECREF cannot get the current interpreter which leads to segmentation fault.
Destructors for static C++ variables are not good place to call Python code.

While segfault is not desired behavior, I'm curious what should we do here.
No-op on object destruction if the interpreter is gone doesn't sound like a good thing, it hides the actual error.

Py_XDECREF cannot signal the error; in C we have no exceptions.

I can propose a workaround that fixes the reproducer:

struct PyObjectDecrementer {
	void operator()(PyObject* obj) const {
	        if(PyInterpreterState_Get() != NULL)  {
		    Py_XDECREF(obj);
		}
	};
};

It can leave resources leaked, e.g. a file is not closed. Probably unclosed resource is not a problem, it will be released on the process exit. Anyway, decref could call __del__ which is dangerous is the interpreter is destroyed already.

Another (and maybe better) option is registering m_clear callback of PyModuleDef to explicitly clean-up all static PyObject* instances. The callback will be called on the module reloading, when the interpreter is still alive.

[s22chan]

Another (and maybe better) option is registering m_clear callback of PyModuleDef to explicitly clean-up all static PyObject* instances. The callback will be called on the module reloading, when the interpreter is still alive.

For the multiple interpreter case, will that necessitate some state management on the extension side?

[asvetlov]

Yes probably. For multi-interpreter mode there is a recommendation to avoid static variables but put the shared data into the module state. See PyModuleDef.m_size and PyModule_GetState(...). Say, Modules/_abc.c could provide a relatively small usage example.

[ZeroIntensity]

Yeah, I don't think there's anything that's a bug here. I highly suggest looking into multi-phase initialization and module state for calling things upon finalization.

I can propose a workaround that fixes the reproducer:

Hmm, I'm not sure that will always work either. Acquiring the interpreter state is reliant on the thread state, which is probably cleared by the time the destructor gets called. You probably want PyThreadState_Get() != NULL instead.

[ovesh]

Future readers: calling PyThreadState_Get() or PyInterpreterState_Get() at that point results in an error, so it's not actually a usable workaround.

Fatal Python error: PyInterpreterState_Get: the function must be called with the GIL held, after Python initialization and before Python finalization, but the GIL is released (the current Python thread state is NULL)

[ZeroIntensity]

Right, I forgot about that. Use PyThreadState_GetUnchecked instead.

[ovesh]

PyThreadState_GetUnchecked only available from py313. Is calling _PyThreadState_UncheckedGet() for py<313 advised?