diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 1c58bc21620..47b1f46246f 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -673,6 +673,7 @@ peps/pep-0791.rst @vstinner peps/pep-0792.rst @dstufft peps/pep-0793.rst @encukou peps/pep-0794.rst @brettcannon +peps/pep-0797.rst @ZeroIntensity peps/pep-0798.rst @JelleZijlstra peps/pep-0799.rst @pablogsal peps/pep-0800.rst @JelleZijlstra diff --git a/peps/pep-0797.rst b/peps/pep-0797.rst new file mode 100644 index 00000000000..c5257ace44f --- /dev/null +++ b/peps/pep-0797.rst @@ -0,0 +1,270 @@ +PEP: 797 +Title: Shared Object Proxies for Subinterpreters +Author: Peter Bierma +Discussions-To: Pending +Status: Draft +Type: Standards Track +Created: 08-Aug-2025 +Python-Version: 3.15 +Post-History: `01-Jul-2025 `__ + + +Abstract +======== + +This PEP introduces a new :func:`~concurrent.interpreters.share` function to +the :mod:`concurrent.interpreters` module, which allows *any* arbitrary object +to be shared for a period of time. + +For example:: + + from concurrent import interpreters + + with open("spanish_inquisition.txt") as unshareable: + interp = interpreters.create() + with interpreters.share(unshareable) as proxy: + interp.prepare_main(file=proxy) + interp.exec("file.write('I didn't expect the Spanish Inquisition')") + + +Motivation +========== + +Many Objects Cannot be Shared Between Subinterpreters +----------------------------------------------------- + +In Python 3.14, the new :mod:`concurrent.interpreters` module can be used to +create multiple interpreters in a single Python process. This works well for +stateless code (that is, code that doesn't need any external data) and objects +that can be serialized, but it is fairly common for applications to want to use +highly-complex data structures (that cannot be serialized) with their +concurrency. + +Currently, :mod:`!concurrent.interpreters` can only share +:ref:`a handful of types ` natively, and then falls back +to the :mod:`pickle` module for other types. This can be very limited, as many +types of objects cannot be pickled. + +Rationale +========= + +A Fallback for Object Sharing +----------------------------- + +A shared object proxy is designed to be a fallback for sharing an object +between interpreters, because it's generally slow and causes increased memory +usage (due to :term:`immortality `, which will be discussed more +later). As such, this PEP does not make other mechanisms for sharing objects +(namely, serialization) obsolete. A shared object proxy should only be used as +a last-resort silver bullet for highly complex objects that cannot be +serialized or shared in any other way. + +Specification +============= + +The ``SharedObjectProxy`` Type +------------------------------ + +.. class:: concurrent.interpreters.SharedObjectProxy + + A proxy type that allows thread-safe access to an object across multiple + interpreters. This cannot be constructed from Python; instead, use the + :func:`~concurrent.interpreters.share` function. + + When interacting with the wrapped object, the proxy will switch to the + interpreter in which the object was created. Arguments passed to anything + on the proxy are also wrapped in a new shared object proxy if the type + isn't natively shareable (so, for example, strings would not be wrapped + in an object proxy, but file objects would). The same goes for return + values. + + For thread-safety purposes, an instance of ``SharedObjectProxy`` is + always :term:`immortal`. This means that it won't be deallocated for the + lifetime of the interpreter. When an object proxy is done being used, it + clears its reference to the object that it wraps and allows itself to be + reused. This prevents extreme memory accumulation. + + In addition, all object proxies have an implicit context that manages them. + This context is determined by the most recent call to + :func:`~concurrent.interpreters.share` in the current thread. When the context + finishes, all object proxies created under that context are cleared, allowing + them to be reused in a new context. + +Thread State Switching +********************** + +At the C level, all objects in Python's C API are interacted with through their +type (a pointer to a :c:type:`PyTypeObject`). For example, to call an object, +the interpreter will access the :c:member:`~PyTypeObject.tp_call` field on the +object's type. This is where the magic of a shared object proxy can happen. + +The :c:type:`!PyTypeObject` for a shared object proxy must be such a type that +implements wrapping behavior for every single field on the type object +structure. So, going back to ``tp_call``, an object proxy must be able to +"intercept" the call in such a way where the wrapped object's ``tp_call`` +slot can be executed without thread-safety issues. This is done by switching +the :term:`attached thread state`. + +In the C API, a :term:`thread state` belongs to a certain interpreter, and by +holding an attached thread state, the thread may interact with any object +belonging to its interpreter. This is because holding an attached thread state +implies things like holding the :term:`GIL`, which make object access thread-safe. + +.. note:: + + On the :term:`free threaded ` build, it is still required + to hold an :term:`attached thread state` to interact with objects in the + C API. + +So, with that in mind, the only thing that the object proxy has to do to call +a type slot is hold an attached thread state for the object's interpreter. +This is the fundamental idea of how a shared object proxy works: allow access +from any interpreter, but switch to one the wrapped object needs when a type +slot is called. + +Sharing Arguments and Return Values +*********************************** + +Once the attached thread state has been switched to match a wrapped object's +interpreter, passed arguments and the return value of the slot need to be shared +back to the caller. This is done by first attempting to share them natively +(for example, with objects such as ``True`` or ``False``), and then falling +back to creating a new shared object proxy if all else fails. The new proxy +is given the same context as the current proxy, meaning the newly wrapped object +will be able to be freed once the :func:`~concurrent.interpreters.share` context +is closed. + +The Sharing APIs +---------------- + +.. function:: concurrent.interpreters.share(obj) + + Wrap *obj* in a :class:`~concurrent.interpreters.SharedObjectProxy`, + allowing it to be used in other interpreter APIs as if it were natively shareable. + + This returns a :term:`context manager`. The resulting object is the proxy + that can be shared (meaning that *obj* is left unchanged). After the context + is closed, the proxy will release its reference to *obj* and allow itself to + be reused for a future call to ``share``. + + If this function is used on an existing shared object proxy, it is assigned + a new context, preventing it from being cleared when the parent ``share`` + context finishes. + + For example: + + .. code-block:: python + + from concurrent import interpreters + + with open("spanish_inquisition.txt") as unshareable: + interp = interpreters.create() + with interpreters.share(unshareable) as proxy: + interp.prepare_main(file=proxy) + interp.exec("file.write('I didn't expect the Spanish Inquisition')") + + + .. note:: + + ``None`` cannot be used with this function, as ``None`` is a special + value reserved for dead object proxies. Since ``None`` is natively + shareable, there's no need to pass it to this function anyway. + +.. function:: concurrent.interpreters.share_forever(obj) + + Similar to :func:`~concurrent.interpreters.share`, but *does not* give the resulting + proxy a context, meaning it will live forever (unless a call to ``share`` + explicitly gives the proxy a new lifetime). As such, this function does not + return a :term:`context manager`. + + For example: + + .. code-block:: python + + from concurrent import interpreters + + with open("spanish_inquisition.txt") as unshareable: + interp = interpreters.create() + proxy = interpreters.share_forever(unshareable) + interp.prepare_main(file=proxy) + # Note: the bound method object for file.write() will also live + # forever in a proxy. + interp.exec("file.write('I didn't expect the Spanish Inquisition')") + + .. warning:: + + Proxies created as a result of the returned proxy (for example, bound + method objects) will also exist for the lifetime of the interpreter, + which can lead to high memory usage. + + +Multithreaded Scaling +--------------------- + +Since an object proxy mostly interacts with an object normally, there shouldn't +be much additional overhead on using the object once the thread state has been +switched. However, this means that when the :term:`GIL` is enabled, you may lose +some of the concurrency benefits from subinterpreters, because threads will be +stuck waiting on the GIL for a wrapped object. + +Backwards Compatibility +======================= + +In order to implement the immortality mechanism used by shared object proxies, +several assumptions had to be made about the object lifecycle in the C API. +So, some best practices in the C API (such as using the object allocator for +objects) are made harder requirements by the implementation of this PEP. + +The author of this PEP believes it is unlikely that this will cause breakage, +as he has not ever seen code in the wild that violates the assumptions made +about the object lifecycle as required by the reference implementation. + +Security Implications +===================== + +The largest issue with shared object proxies is that in order to have +thread-safe reference counting operations, they must be :term:`immortal`, +which prevents any concurrent modification to their reference count. +This can cause them to take up very large amounts of memory if mismanaged. + +The :func:`~concurrent.interpreters.share` context manager does its best +to avoid this issue by manually clearing references at the end of an object +proxy's usage (allowing mortal objects to be freed), as well as avoiding +the allocation of new object proxies by reusing dead ones (that is, object +proxies with a cleared reference). + +How to Teach This +================= + +New APIs and important information about how to use them will be added to the +:mod:`concurrent.interpreters` documentation. An informational PEP regarding +the new immortality mechanisms included in the reference implementation will +be written if this PEP is accepted. + +Reference Implementation +======================== + +The reference implementation of this PEP can be found +`here `_. + +Rejected Ideas +============== + +Why Not Atomic Reference Counting? +---------------------------------- + +Immortality seems to be the driver for a lot of complexity in this proposal; +why not use atomic reference counting instead? + +Atomic reference counting has been tried before in previous :term:`GIL` +removal attempts, but unfortunately added too much overhead to CPython to be +feasible, because atomic "add" operations are much slower than their non-atomic +counterparts. Immortality, while complex, has the benefit of being efficient +and thread-safe without needing to slow down single-threaded performance with +reference counting. + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive.