PEP 797: Shared Object Proxies for Subinterpreters

.github/CODEOWNERS

@@ -673,6 +673,7 @@ peps/pep-0791.rst  @vstinner
 peps/pep-0792.rst  @dstufft
 peps/pep-0793.rst  @encukou
 peps/pep-0794.rst  @brettcannon
+peps/pep-0797.rst  @ZeroIntensity
 peps/pep-0798.rst  @JelleZijlstra
 peps/pep-0799.rst  @pablogsal
 peps/pep-0800.rst  @JelleZijlstra

peps/pep-0797.rst

@@ -0,0 +1,271 @@
+PEP: 797
+Title: Shared Object Proxies for Subinterpreters
+Author: Peter Bierma <[email protected]>
+Discussions-To: Pending
+Status: Draft
+Type: Standards Track
+Created: 08-Aug-2025
+Python-Version: 3.15
+Post-History: `01-Jul-2025 <https://discuss.python.org/t/97306>`__
+
+
+Abstract
+========
+
+This PEP introduces a new :func:`~concurrent.interpreters.share` function to
+the :mod:`concurrent.interpreters` module, which allows *any* arbitrary object
+to be shared for a period of time.
+
+For example::
+
+    from concurrent import interpreters
+
+    with open("spanish_inquisition.txt") as unshareable:
+        interp = interpreters.create()
+        with interpreters.share(unshareable) as proxy:
+            interp.prepare_main(file=proxy)
+            interp.exec("file.write('I didn't expect the Spanish Inquisition')")
+
+
+Motivation
+==========
+
+Many Objects Cannot be Shared Between Subinterpreters
+-----------------------------------------------------
+
+In Python 3.14, the new :mod:`concurrent.interpreters` module can be used to
+create multiple interpreters in a single Python process. This works well for
+stateless code (that is, code that doesn't need any state from a caller) and
+objects that can be serialized, but it is fairly common for applications to
+want to use highly-complex data structures (that cannot be serialized) with
+their concurrency.
+
+Currently, :mod:`!concurrent.interpreters` can only share
+:ref:`a handful of types <interp-object-sharing>` natively, and then falls back
+to the :mod:`pickle` module for other types. This can be very limited, as many
+types of objects cannot be pickled. For example, file objects returned by
+:func:`open` cannot be serialized through ``pickle``.
+
+Rationale
+=========
+
+A Fallback for Object Sharing
+-----------------------------
+
+A shared object proxy is designed to be a fallback for sharing an object
+between interpreters, because it's generally slow and causes increased memory
+usage (due to :term:`immortality <immortal>`, which will be discussed more
+later). As such, this PEP does not make other mechanisms for sharing objects
+(namely, serialization) obsolete. A shared object proxy should only be used as
+a last-resort for highly complex objects that cannot be serialized or shared
+in any other way.
+
+Specification
+=============
+
+The ``SharedObjectProxy`` Type
+------------------------------
+
+.. class:: concurrent.interpreters.SharedObjectProxy
+
+    A proxy type that allows thread-safe access to an object across multiple
+    interpreters. This cannot be constructed from Python; instead, use the
+    :func:`~concurrent.interpreters.share` function.
+
+    When interacting with the wrapped object, the proxy will switch to the
+    interpreter in which the object was created. Arguments passed to anything
+    on the proxy are also wrapped in a new shared object proxy if the type
+    isn't natively shareable (so, for example, strings would not be wrapped
+    in an object proxy, but file objects would). The same goes for return
+    values.
+
+    For thread-safety purposes, an instance of ``SharedObjectProxy`` is
+    always :term:`immortal`. This means that it won't be deallocated for the
+    lifetime of the interpreter. When an object proxy is done being used, it
+    clears its reference to the object that it wraps and allows itself to be
+    reused. This prevents extreme memory accumulation.
+
+    In addition, all object proxies have an implicit context that manages them.
+    This context is determined by the most recent call to
+    :func:`~concurrent.interpreters.share` in the current thread. When the context
+    finishes, all object proxies created under that context are cleared, allowing
+    them to be reused in a new context.
+
+Thread State Switching
+**********************
+
+At the C level, all objects in Python's C API are interacted with through their
+type (a pointer to a :c:type:`PyTypeObject`). For example, to call an object,
+the interpreter will access the :c:member:`~PyTypeObject.tp_call` field on the
+object's type. This is where the magic of a shared object proxy can happen.
+
+The :c:type:`!PyTypeObject` for a shared object proxy must implement
+wrapping behavior for every single field on the type object structure.
+So, going back to ``tp_call``, an object proxy must be able to "intercept" the
+call in such a way where the wrapped object's ``tp_call`` slot can be executed
+without thread-safety issues. This is done by switching the
+:term:`attached thread state`.
+
+In the C API, a :term:`thread state` belongs to a certain interpreter, and by
+holding an attached thread state, the thread may interact with any object
+belonging to its interpreter. This is because holding an attached thread state
+implies things like holding the :term:`GIL`, which make object access thread-safe.
+
+.. note::
+
+   On the :term:`free threaded <free threading>` build, it is still required
+   to hold an :term:`attached thread state` to interact with objects in the
+   C API.
+
+So, with that in mind, the only thing that the object proxy has to do to call
+a type slot is hold an attached thread state for the object's interpreter.
+This is the fundamental idea of how a shared object proxy works: allow access
+from any interpreter, but switch to the wrapped object's interpreter when a type
+slot is called.
+
+Sharing Arguments and Return Values
+***********************************
+
+Once the attached thread state has been switched to match a wrapped object's
+interpreter, arguments and the return value (if it's a ``PyObject *``) of the
+slot need to be shared back to the caller. This is done by first attempting to
+share them natively (for example, with ``pickle``), and then falling back to
+creating a new shared object proxy if all else fails. The new proxy is given
+the same context as the current proxy, meaning the newly wrapped object will
+be able to be freed once the :func:`~concurrent.interpreters.share` context
+is closed.
+
+The Sharing APIs
+----------------
+
+.. function:: concurrent.interpreters.share(obj)
+
+    Wrap *obj* in a :class:`~concurrent.interpreters.SharedObjectProxy`,
+    allowing it to be used in other interpreter APIs as if it were natively shareable.
+
+    This returns a :term:`context manager`. The resulting object from the
+    context is the proxy that can be shared. After the context is closed, the
+    proxy will release its reference to *obj* and allow itself to be reused
+    for a future call to ``share``.
+
+    If this function is used on an existing shared object proxy, it is assigned
+    a new context, preventing it from being cleared when the parent ``share``
+    context finishes.
+
+    For example:
+
+    .. code-block:: python
+
+        from concurrent import interpreters
+
+        with open("spanish_inquisition.txt") as unshareable:
+            interp = interpreters.create()
+            with interpreters.share(unshareable) as proxy:
+                interp.prepare_main(file=proxy)
+                interp.exec("file.write('I didn't expect the Spanish Inquisition')")
+
+
+    .. note::
+
+        ``None`` cannot be used with this function, as ``None`` is a special
+        value reserved for dead object proxies. Since ``None`` is natively
+        shareable, there's no need to pass it to this function anyway.
+
+.. function:: concurrent.interpreters.share_forever(obj)
+
+    Similar to :func:`~concurrent.interpreters.share`, but *does not* give the resulting
+    proxy a context, meaning it will live forever (unless a call to ``share``
+    explicitly gives the proxy a new lifetime). As such, this function does not
+    return a :term:`context manager`.
+
+    For example:
+
+    .. code-block:: python
+
+        from concurrent import interpreters
+
+        with open("spanish_inquisition.txt") as unshareable:
+            interp = interpreters.create()
+            proxy = interpreters.share_forever(unshareable)
+            interp.prepare_main(file=proxy)
+            # Note: the bound method object for file.write() will also live
+            # forever in a proxy.
+            interp.exec("file.write('I didn't expect the Spanish Inquisition')")
+
+    .. warning::
+
+        Proxies created as a result of the returned proxy (for example, bound
+        method objects) will also exist for the lifetime of the interpreter,
+        which can lead to high memory usage.
+
+
+Multithreaded Scaling
+---------------------
+
+Since an object proxy mostly interacts with an object normally, there shouldn't
+be much additional overhead on using the object once the thread state has been
+switched. However, this means that when the :term:`GIL` is enabled, you may lose
+some of the concurrency benefits from subinterpreters, because threads will be
+stuck waiting on the GIL of a wrapped object's interpreter.
+
+Backwards Compatibility
+=======================
+
+In order to implement the immortality mechanism used by shared object proxies,
+several assumptions had to be made about the object lifecycle in the C API.
+So, some best practices in the C API (such as using the object allocator for
+objects) are made harder requirements by the implementation of this PEP.
+
+The author of this PEP believes it is unlikely that this will cause breakage,
+as he has not ever seen code in the wild that violates the assumptions made
+about the object lifecycle as required by the reference implementation.
+
+Security Implications
+=====================
+
+The largest issue with shared object proxies is that in order to have
+thread-safe reference counting operations, they must be :term:`immortal`,
+which prevents any concurrent modification to their reference count.
+This can cause them to take up very large amounts of memory if mismanaged.
+
+The :func:`~concurrent.interpreters.share` context manager does its best
+to avoid this issue by manually clearing references at the end of an object
+proxy's usage (allowing mortal objects to be freed), as well as avoiding
+the allocation of new object proxies by reusing dead ones (that is, object
+proxies with a cleared reference).
+
+How to Teach This
+=================
+
+New APIs and important information about how to use them will be added to the
+:mod:`concurrent.interpreters` documentation. An informational PEP regarding
+the new immortality mechanisms included in the reference implementation will
+be written if this PEP is accepted.
+
+Reference Implementation
+========================
+
+The reference implementation of this PEP can be found
+`here <https://github.com/python/cpython/compare/main...ZeroIntensity:cpython:shared-object-proxy>`_.
+
+Rejected Ideas
+==============
+
+Why Not Atomic Reference Counting?
+----------------------------------
+
+Immortality seems to be the driver for a lot of complexity in this proposal;
+why not use atomic reference counting instead?
+
+Atomic reference counting has been tried before in previous :term:`GIL`
+removal attempts, but unfortunately added too much overhead to CPython to be
+feasible, because atomic "add" operations are much slower than their non-atomic
+counterparts. Immortality, while complex, has the benefit of being efficient
+and thread-safe without needing to slow down single-threaded performance with
+reference counting.
+
+Copyright
+=========
+
+This document is placed in the public domain or under the
+CC0-1.0-Universal license, whichever is more permissive.