Refactor atexit.pyx

[cxzhong]

I have refactor the atexit.pyx since for python 3.14, the changes of atexit is so much for supporting the nogil version.

Python<=3.13 Implementation:

• The atexit module stores callbacks in a C array of structs (atexit_callback).
• The structure is defined in C as:

typedef struct {  PyObject *func;    PyObject *args;    PyObject *kwargs;} atexit_callback;

• Callbacks are stored in the interp->atexit.callbacks field as a C array
• The array can be directly accessed from Cython code using pointer arithmetic

Python 3.14 Implementation:

• The atexit module was refactored to use a Python PyList object instead of a C array.
• The structure is now:

state.callbacks = [(func, args, kwargs), ...]  # A Python list of tuples

• In the C implementation, callbacks are managed with:

PyObject *callbacks;  // This is now a PyList

• Callbacks are inserted at the beginning (LIFO order) using PyList_Insert(state->callbacks, 0, callback)

📝 Checklist

• The title is concise and informative.
• The description explains in detail what this PR is about.
• I have linked a relevant issue or discussion.
• I have created tests covering the changes.
• I have updated the documentation and checked the documentation preview.

⌛ Dependencies

python/cpython@3b76682 python/cpython#127935

[github-actions]

Documentation preview for this PR (built with commit 2ec6f1b; changes) is ready! 🎉
This preview will update shortly after each push to this PR.

[tobiasdiez]

I would prefer to simply do nothing (or throw a 'is deprecated/removed' exception) in atexit for Python >= 3.13 (since it's not needed in this case as explained in #40890).

[cxzhong]

I would prefer to simply do nothing (or throw a 'is deprecated/removed' exception) in atexit for Python >= 3.13 (since it's not needed in this case as explained in #40890).

I just tested the doctest in atexit.pyx is OK but the order changed. The output is reversed.

[cxzhong]

I would prefer to simply do nothing (or throw a 'is deprecated/removed' exception) in atexit for Python >= 3.13 (since it's not needed in this case as explained in #40890).

I also think it is useless in Python 3.13+. We can edit the build system to exclude this. If Python version bigger than 3.13. But we need to modify forker.py, maybe we can do this when we drop support of python 3.12

[Copilot]

Pull Request Overview

This PR refactors the atexit.pyx module to support both Python ≤3.13 and Python 3.14+, which changed how atexit callbacks are stored internally. The refactoring provides a uniform interface for accessing atexit callbacks across different Python versions.

• Adds version-specific implementations for accessing atexit callbacks
• Introduces C functions that handle both the legacy C array format (Python ≤3.13) and the new PyList format (Python 3.14+)
• Updates the _get_exithandlers() function to work with both storage formats while maintaining consistent behavior

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[dimpase]

I thought that everything atexit-related in 3.14 is already in 3.13, no?

[cxzhong]

I thought that everything atexit-related in 3.14 is already in 3.13, no?

It rewrite in python 3.14 for supporting no gil. you can refer to this commit
python/cpython@3b76682 and this PR python/cpython#127935
So there are no atexit_py_callback structure in python 3.14.
But it is internal change in python. So you can not find it in python document.

[cxzhong]

I thought that everything atexit-related in 3.14 is already in 3.13, no?

Yes, the open APIs is not changed. But it changes the internal implementation

[cxzhong]

Maybe we can refactor It to support 3.14 immediately. Then we deprecate the atexit module.

[cxzhong]

I would prefer to simply do nothing (or throw a 'is deprecated/removed' exception) in atexit for Python >= 3.13 (since it's not needed in this case as explained in #40890).

I suggest that we first do the refactor to support python3.14. Then we have one year to rewrite the forker.py to deprecate the atexit.pyx. @tobiasdiez

[nbruin]

Question:
We are testing the version at compile time or cythonization time (those are not necessarily the same and I'm not sure at which time this condition is tested) with:

#if PY_VERSION_HEX >= 0x030e0000

and also during runtime with

if sys.version_info >= (3, 14):

Are we sure those will always agree (I don't think so). What do we do if they don't agree? Do we get a decent error then? Or do we get undefined behaviour? I'm mainly worried because the conditional code defines a dummy function that's supposed to be never called in one situation.

[cxzhong]

Question: We are testing the version at compile time or cythonization time (those are not necessarily the same and I'm not sure at which time this condition is tested) with:

#if PY_VERSION_HEX >= 0x030e0000

and also during runtime with

if sys.version_info >= (3, 14):

Are we sure those will always agree (I don't think so). What do we do if they don't agree? Do we get a decent error then? Or do we get undefined behaviour? I'm mainly worried because the conditional code defines a dummy function that's supposed to be never called in one situation.

Maybe I need to have assert in the dummy function. Or throwing an error. I have to write as this. Because >=3.14, we need to receive Python List, for python <=3.14, we need to receive C array. So I defined two functions. Maybe I should throw an error: The compile version and the running version is not consistent. The function should not arrive there.

[cxzhong]

Question: We are testing the version at compile time or cythonization time (those are not necessarily the same and I'm not sure at which time this condition is tested) with:

#if PY_VERSION_HEX >= 0x030e0000

and also during runtime with

if sys.version_info >= (3, 14):

Are we sure those will always agree (I don't think so). What do we do if they don't agree? Do we get a decent error then? Or do we get undefined behaviour? I'm mainly worried because the conditional code defines a dummy function that's supposed to be never called in one situation.

Thank you very much for your reviewing. Do you think I should do throwing an error in runtime I should throw error in the dummy function.

[nbruin]

Sorry, I don't know what's the best way to deal with cythonize time/compile time/run time discrepancies like the one that can arise here. I was hoping someone else would. Perhaps the cython people have a suggestion?

Questions that may be relevant: does Python consider 3.13 and 3.14 to be binary compatible? They might because they may view the internal changes as not changing the publishes ABI. In that case we're on our own, because then I think you might end up being linked against a 3.14 runtime even if you compiled against a 3.13.

If 3.13 and 3.14 are NOT considered binary compatible then we may not have to worry about this: then there should be other safeguards in place that prevent objects compiled for 3.13 to be dynamically linked at runtime to 3.14 and vice versa.

Because there is also the subtle question that cythonization (making the c source from the cython) may not coincide with the actual compilation, it's also at least theoretically possible to have a mismatch there. The cython people may have good input about that. Question here: https://groups.google.com/g/cython-users/c/CjAn4sPDFK0/m/jeQLWu5QAgAJ

[cxzhong]

Sorry, I don't know what's the best way to deal with cythonize time/compile time/run time discrepancies like the one that can arise here. I was hoping someone else would. Perhaps the cython people have a suggestion?

Questions that may be relevant: does Python consider 3.13 and 3.14 to be binary compatible? They might because they may view the internal changes as not changing the publishes ABI. In that case we're on our own, because then I think you might end up being linked against a 3.14 runtime even if you compiled against a 3.13.

If 3.13 and 3.14 are NOT considered binary compatible then we may not have to worry about this: then there should be other safeguards in place that prevent objects compiled for 3.13 to be dynamically linked at runtime to 3.14 and vice versa.

Because there is also the subtle question that cythonization (making the c source from the cython) may not coincide with the actual compilation, it's also at least theoretically possible to have a mismatch there. The cython people may have good input about that. Question here: https://groups.google.com/g/cython-users/c/CjAn4sPDFK0/m/jeQLWu5QAgAJ

I think the changes of python 3.14 is huge, the python group did a lot for the no gil support. I do not think the wheels in python 3.13 is also compatible in 3.14. But the python group does not say they are compatible or not in an official document.

I suggest that we can merge it so we can support python 3.14 quickly. Then we have one year to discuss how to deal with the problem of atexit.pyx. Maybe when we drop support of python 3.12. we can remove this module.

[cxzhong]

Sorry, I don't know what's the best way to deal with cythonize time/compile time/run time discrepancies like the one that can arise here. I was hoping someone else would. Perhaps the cython people have a suggestion?

Questions that may be relevant: does Python consider 3.13 and 3.14 to be binary compatible? They might because they may view the internal changes as not changing the publishes ABI. In that case we're on our own, because then I think you might end up being linked against a 3.14 runtime even if you compiled against a 3.13.

If 3.13 and 3.14 are NOT considered binary compatible then we may not have to worry about this: then there should be other safeguards in place that prevent objects compiled for 3.13 to be dynamically linked at runtime to 3.14 and vice versa.

Because there is also the subtle question that cythonization (making the c source from the cython) may not coincide with the actual compilation, it's also at least theoretically possible to have a mismatch there. The cython people may have good input about that. Question here: https://groups.google.com/g/cython-users/c/CjAn4sPDFK0/m/jeQLWu5QAgAJ

Because the atexit module only has two open api, register and unregister. without internal api, we can not do the restore_atexit.

[cxzhong]

Sorry, I don't know what's the best way to deal with cythonize time/compile time/run time discrepancies like the one that can arise here. I was hoping someone else would. Perhaps the cython people have a suggestion?

Questions that may be relevant: does Python consider 3.13 and 3.14 to be binary compatible? They might because they may view the internal changes as not changing the publishes ABI. In that case we're on our own, because then I think you might end up being linked against a 3.14 runtime even if you compiled against a 3.13.

If 3.13 and 3.14 are NOT considered binary compatible then we may not have to worry about this: then there should be other safeguards in place that prevent objects compiled for 3.13 to be dynamically linked at runtime to 3.14 and vice versa.

Because there is also the subtle question that cythonization (making the c source from the cython) may not coincide with the actual compilation, it's also at least theoretically possible to have a mismatch there. The cython people may have good input about that. Question here: https://groups.google.com/g/cython-users/c/CjAn4sPDFK0/m/jeQLWu5QAgAJ

But for python 3.12, the doctest needs this module. we should support python 3.12 this year and next year.

[nbruin]

Yes, I think your support plan makes perfect sense. We just need to check if we're sufficiently defensive about version checks. Presently you define a "never called" get_atexit_callbacks_array if the compile-time version is >=3.14, but if the runtime version ends up being <=3.13 then it is called after all! If we know that won't happen then we're fine. Otherwise, I think things need to be guarded a bit, because you wouldn't want to dereference the NULL pointer you get back in that situation.

[cxzhong]

Yes, I think your support plan makes perfect sense. We just need to check if we're sufficiently defensive about version checks. Presently you define a "never called" get_atexit_callbacks_array if the compile-time version is >=3.14, but if the runtime version ends up being <=3.13 then it is called after all! If we know that won't happen then we're fine. Otherwise, I think things need to be guarded a bit, because you wouldn't want to dereference the NULL pointer you get back in that situation.

I think it will directly throw Python error with my last commit. But how to run Python 3.14 wheel in Python 3.13? the Python abi is not compatible

[cxzhong]

Yes, I think your support plan makes perfect sense. We just need to check if we're sufficiently defensive about version checks. Presently you define a "never called" get_atexit_callbacks_array if the compile-time version is >=3.14, but if the runtime version ends up being <=3.13 then it is called after all! If we know that won't happen then we're fine. Otherwise, I think things need to be guarded a bit, because you wouldn't want to dereference the NULL pointer you get back in that situation.

in fact, The origin atexit.pyx file also use if PY_VERSION_HEX to control the version

[nbruin]

I think it will directly throw Python error with my last commit. But how to run Python 3.14 wheel in Python 3.13? the Python abi is not compatible

If we can confirm that it does, then that would mean we don't have to worry about compile-time and runtime versions disagreeing.

Note that python does have a Stable ABI: https://docs.python.org/3/c-api/stable.html#stable . Their API already has stronger compatibility guarantees. So they're definitely considering scenarios where code compiled for one version gets run with a different one. That's why I'm not so sure that it's impossible to encounter in the wild.

[cxzhong]

I think it will directly throw Python error with my last commit. But how to run Python 3.14 wheel in Python 3.13? the Python abi is not compatible

If we can confirm that it does, then that would mean we don't have to worry about compile-time and runtime versions disagreeing.

Note that python does have a Stable ABI: https://docs.python.org/3/c-api/stable.html#stable . Their API already has stronger compatibility guarantees. So they're definitely considering scenarios where code compiled for one version gets run with a different one. That's why I'm not so sure that it's impossible to encounter in the wild.

Thank you very much. I will try to do this

cdef extern from "Python.h":
    cdef int PY_MAJOR_VERSION
    cdef int PY_MINOR_VERSION


cdef int _built_py_major = PY_MAJOR_VERSION
cdef int _built_py_minor = PY_MINOR_VERSION


__built_for_python__ = (_built_py_major, _built_py_minor)

If runtime is not equal that. It will return importerror immediately.
Do you think it is OK? But I have to sleep now. I will commit tomorrow.

[nbruin]

D Woods https://groups.google.com/g/cython-users/c/CjAn4sPDFK0/m/gl7_GcNVAgAJ actually thinks that compile-time/runtime version match is (softly) enforced through filenames, so normally we should not encounter a mismatch. So I'm now less worried about this issue. As a consequence I don't understand why Python is bothering with their "Stable ABI": if you're going to insist on version matches anyway, stability of the API is the only thing that matters (there are the micro versions, but there they already try to be ABI compatible). Sleep well.

[dimpase]

I'd guess Python ABI is very important for binary wheels.

[cxzhong]

@nbruin I just have a question like this

    // Dummy function for Python < 3.14 (never called) 
     static PyObject* get_atexit_callbacks_list(PyObject *self) { 
         PyErr_SetString(PyExc_RuntimeError, "Python < 3.14 has no atexit list"); 
         return NULL; 
     }

If I run this accidently, It will safely raise a Python error or It will get a segfault. Cython can safely handle this error？If this is Ok. I think maybe we do not need so complicated

[nbruin]

If I run this accidently, It will safely raise a Python error or It will get a segfault. Cython can safely handle this error？If this is Ok. I think maybe we do not need so complicated

I think the cdef needs an except NULL clause in order for cython to be able to check for the error condition and raise. Otherwise the error may just get set but not detected at the callsite.

And with that in place, I think the body could just be

raise RuntimeError("Python < 3.14 has no atexit list")

Cython should be able to figure out the right error setting and error-signalling return value then.

Since these errors then get raised in atexit code, it could be in an environment where exceptions are hard to be raised. There's not much you can do against that other than forcing a hard exit (via a segfault or some other unhandled interrupt).