PEP 734: Simplify Explanations of "Shareable" Objects (#4374)

Author: ericsnowcurrently

peps/pep-0734.rst

@@ -356,9 +356,10 @@ Attributes and methods:
       Bind one or more objects in the interpreter's ``__main__`` module.
 
       The keyword argument names will be used as the attribute names.
-      The values will be bound as new objects, though exactly equivalent
-      to the original.  Only objects specifically supported for passing
-      between interpreters are allowed.  See `Shareable Objects`_.
+      For most objects a copy will be bound in the interpreter, with
+      pickle used in between.  For some objects, like ``memoryview``,
+      the underlying data will be shared between the interpreters.
+      See `Shareable Objects`_.
 
       ``prepare_main()`` is helpful for initializing the
       globals for an interpreter before running code in it.
@@ -449,12 +450,14 @@ As with other queues in Python, for each "put" the object is added to
 the back and each "get" pops the next one off the front.  Every added
 object will be popped off in the order it was pushed on.
 
-Only objects that are specifically supported for passing
-between interpreters may be sent through an ``interpreters.Queue``.
-Note that the actual objects aren't sent, but rather their
-underlying data.  However, the popped object will still be
-strictly equivalent to the original.
-See `Shareable Objects`_.
+Any object that can be pickled may be sent through an
+``interpreters.Queue``.
+
+Note that the actual objects aren't sent, but rather their underlying
+data is sent.  The resulting object is strictly equivalent to the
+original.  For most objects the underlying data is serialized (e.g.
+pickled).  In a few cases, like with ``memoryview``, the underlying data
+is sent (and shared) without serialization.  See `Shareable Objects`_.
 
 The module defines the following functions:
 
@@ -514,23 +517,17 @@ Attributes and methods:
       This is only a snapshot of the state at the time of the call.
       Other threads or interpreters may cause this to change.
 
-* ``put(obj, timeout=None, *, syncobj=None)``
+* ``put(obj, timeout=None)``
       Add the object to the queue.
 
       If ``maxsize > 0`` and the queue is full then this blocks until
       a free slot is available.  If *timeout* is a positive number
       then it only blocks at least that many seconds and then raises
       ``interpreters.QueueFull``.  Otherwise is blocks forever.
 
-      If "syncobj" is true then the object must be
-      `shareable <Shareable Objects_>`_, which means the object's data
-      is passed through rather than the object itself.
-      If "syncobj" is false then all objects are supported.  However,
-      there are some performance penalties and all objects are copies
-      (e.g. via pickle).  Thus mutable objects will never be
-      automatically synchronized between interpreters.
-      If "syncobj" is None (the default) then the queue's default
-      value is used.
+      Nearly all objects can be sent through the queue.  In a few cases,
+      like with ``memoryview``, the underlying data is actually shared,
+      rather than just copied.  See `Shareable Objects`_.
 
       If an object is still in the queue, and the interpreter which put
       it in the queue (i.e. to which it belongs) is destroyed, then the
@@ -539,7 +536,7 @@ Attributes and methods:
       sentinel or to raise an exception for the corresponding ``get()``
       call.)
 
-* ``put_nowait(obj, *, syncobj=None)``
+* ``put_nowait(obj``
       Like ``put()`` but effectively with an immediate timeout.
       Thus if the queue is full, it immediately raises
       ``interpreters.QueueFull``.
@@ -558,51 +555,41 @@ Attributes and methods:
 Shareable Objects
 -----------------
 
-``Interpreter.prepare_main()`` only works with "shareable" objects.
-The same goes for ``interpreters.Queue`` (optionally).
-
 A "shareable" object is one which may be passed from one interpreter
-to another.  The object is not necessarily actually directly shared
-by the interpreters.  However, even if it isn't, the shared object
-should be treated as though it *were* shared directly.  That's a
-strong equivalence guarantee for all shareable objects.
-(See below.)
-
-For some types (builtin singletons), the actual object is shared.
-For some, the object's underlying data is actually shared but each
-interpreter has a distinct object wrapping that data.  For all other
-shareable types, a strict copy or proxy is made such that the
-corresponding objects continue to match exactly.  In cases where
-the underlying data is complex but must be copied (e.g. ``tuple``),
-the data is serialized as efficiently as possible.
-
-Shareable objects must be specifically supported internally
-by the Python runtime.  However, there is no restriction against
-adding support for more types later.
-
-Here's the initial list of supported objects:
-
-* ``str``
-* ``bytes``
-* ``int``
-* ``float``
-* ``bool`` (``True``/``False``)
-* ``None``
-* ``tuple`` (only with shareable items)
-* ``interpreters.Queue``
-* ``memoryview`` (underlying buffer actually shared)
-
-Note that the last two on the list, queues and ``memoryview``, are
-technically mutable data types, whereas the rest are not.  When any
-interpreters share mutable data there is always a risk of data races.
-Cross-interpreter safety, including thread-safety, is a fundamental
-feature of queues.
-
-However, ``memoryview`` does not have any native accommodations.
-The user is responsible for managing thread-safety, whether passing
-a token back and forth through a queue to indicate safety
-(see `Synchronization`_), or by assigning sub-range exclusivity
-to individual interpreters.
+to another.  The object is not actually directly shared by the
+interpreters.  However, the shared object should be treated as though
+it *were* shared directly, with caveats for mutability.
+
+All objects that can be pickled are shareable.  Thus, nearly every
+object is shareable.  ``interpreters.Queue`` objects are also shareable.
+
+In nearly every case where an object is sent to an interpreter, whether
+with ``interp.prepare_main()`` or ``queue.put()``, the actual object
+is not sent.  Instead, the object's underlying data is sent.  For
+most objects the object is pickled and the receiving
+interpreter unpickles it.
+
+A notable exception is objects which implement the "buffer" protocol,
+like ``memoryview``.  Their underlying ``Py_buffer`` is actually shared
+between interpreters.  ``interp.prepare_main()`` and ``queue.get()``
+wrap the buffer in a new ``memoryview`` object.
+
+For most mutable objects, when one is sent to another interpreter, it is
+copied.  Thus any changes to the original or to the copy will never be
+synchronized to the other.  Mutable objects shared through pickling fall
+into this category.  However, ``interpreters.Queue`` and objects that
+implement the buffer protocol are notable cases where the underlying
+data *is* shared between interpreters, so objects stay synchronized.
+
+When interpreters genuinely share mutable data there is always a risk
+of data races.  Cross-interpreter safety, including thread-safety,
+is a fundamental feature of ``interpreters.Queue``.
+
+However, the buffer protocol (i.e. ``Py_buffer``) does not have any
+native accommodations against data races.  Instead, the user is
+responsible for managing thread-safety, whether passing a token back
+and forth through a queue to indicate safety (see `Synchronization`_),
+or by assigning sub-range exclusivity to individual interpreters.
 
 Most objects will be shared through queues (``interpreters.Queue``),
 as interpreters communicate information between each other.
@@ -612,23 +599,6 @@ to set up an interpreter prior to running code in it.  However,
 to provide another interpreter with a means
 of further communication.
 
-Finally, a reminder: for a few types the actual object is shared,
-whereas for the rest only the underlying data is shared, whether
-as a copy or through a proxy.  Regardless, it always preserves
-the strong equivalence guarantee of "shareable" objects.
-
-The guarantee is that a shared object in one interpreter is strictly
-equivalent to the corresponding object in the other interpreter.
-In other words, the two objects will be indistinguishable from each
-other.  The shared object should be treated as though the original
-had been shared directly, whether or not it actually was.
-That's a slightly different and stronger promise than just equality.
-
-The guarantee is especially important for mutable objects, like
-``Interpreters.Queue`` and ``memoryview``.  Mutating the object
-in one interpreter will always be reflected immediately in every
-other interpreter sharing the object.
-
 Synchronization
 ---------------
 
@@ -1007,24 +977,6 @@ in the calling interpreter.  This is because ``Interpreter.call()`` is
 a higher level method that uses pickle to support objects that can't
 normally be passed between interpreters.
 
-Limited Object Sharing
-----------------------
-
-As noted in `Interpreter Isolation`_, only a small number of builtin
-objects may be truly shared between interpreters.  In all other cases
-objects can only be shared indirectly, through copies or proxies.
-
-The set of objects that are shareable as copies through queues
-(and ``Interpreter.prepare_main()``) is limited for the sake of
-efficiency.
-
-Supporting sharing of *all* objects is possible (via pickle)
-but not part of this proposal.  For one thing, it's helpful to know
-in those cases that only an efficient implementation is being used.
-Furthermore, in those cases supporting mutable objects via pickling
-would violate the guarantee that "shared" objects be equivalent
-(and stay that way).
-
 Objects vs. ID Proxies
 ----------------------