[GR-60660] Multi context C extensions with copying

PullRequest: graalpython/3537

Author: timfel

CHANGELOG.md

@@ -19,6 +19,8 @@ language runtime. The main focus is on user-observable behavior of the engine.
 * Added `GRAALPY_VERSION` and `GRAALPY_VERSION_NUM` C macros.
 * Remove `ginstall` module. It hasn't been necessary for several releases. Please, use `pip install`.
 * Remove experimental `SetupLLVMLibraryPaths` option. It was used to pre-set library path for LLVM toolchain's libc++. The path can still be set manually.
+* Added `GRAALPY_VERSION` and `GRAALPY_VERSION_NUM` C macros
+* Added experimental `python.IsolateNativeModules` option to allow loading native extensions multiple times in different contexts. See [the documentation](https://github.com/oracle/graalpython/blob/master/docs/user/Native-Extensions.md) for more information.
 
 ## Version 24.1.0
 * GraalPy is now considered stable for pure Python workloads. While many workloads involving native extension modules work, we continue to consider them experimental. You can use the command-line option `--python.WarnExperimentalFeatures` to enable warnings for such modules at runtime. In Java embeddings the warnings are enabled by default and you can suppress them by setting the context option 'python.WarnExperimentalFeatures' to 'false'.

docs/contributor/IMPLEMENTATION_DETAILS.md

@@ -302,3 +302,16 @@ checking the same `MANAGED_REFCNT` limit for all objects, the subsequent `Py_Dec
 call for this object will not detect that the reference should be made weak again!
 However, this is OK, it only prolongs the collection: we will make it weak again in
 the next run of the cycle GC on the native side.
+
+## C extension copying
+
+On Linux, Python native extensions expect to lookup Python C API functions in the global namespace and specify no explicit dependency on any libpython.
+To isolate them, we copy them with a new name, change their `SONAME`, add a `DT_NEEDED` dependency on a copy of our libpython shared object, and finally load them with `RTLD_LOCAL`.
+
+On Windows there is no global namespace so native extensions already have a dependency on our libpython DLL.
+We copy them and just change the dependency to point to the context-local copy of libpython rather than the global one.
+
+On macOS, while two-level namespaces exist, Python extensions historically use `-undefined dynamic_lookup` where they (just like in Linux) expect to find C API functions in any loaded image.
+We have to apply a similar workaround as on Linux, copy to a new name, change the `LC_ID_DYLIB` to that name, and add a `LC_LOAD_DYLIB` section to make the linker load the symbols from our libpython.
+
+Note that any code signatures are invalidated by this process.

docs/user/Native-Extensions.md

@@ -18,9 +18,39 @@ Please do not update `pip` or use alternative tools such as `uv`.
 ## Embedding limitations
 
 Python native extensions run by default as native binaries, with full access to the underlying system.
-Native code is entirely unrestricted and can circumvent any security protections Truffle or the JVM may provide.
-Native data structures are not subject to the Java GC and the combination of them with Java data structures may lead to memory leaks.
-Native libraries generally cannot be loaded multiple times into the same process, and they may contain global state that cannot be safely reset.
-Thus, it is not possible to create multiple GraalPy contexts that access native modules within the same JVM.
-This includes the case when you create a context, close it, and then create another context.
-The second context will not be able to access native extensions.
+This has a few implications:
+
+1. Native code is entirely unrestricted and can circumvent any security protections Truffle or the JVM may provide.
+2. Native data structures are not subject to the Java GC and the combination of them with Java data structures may lead to increased memory pressure or memory leaks.
+3. Native libraries generally cannot be loaded multiple times into the same process, and they may contain global state that cannot be safely reset.
+
+### Full Native Access
+
+The Context API allows to set options such as `allowIO`, `allowHostAccess`, `allowThreads` and more on the created contexts.
+To use Python native extensions on GraalPy, the `allowNativeAccess` option must be set to true, but this opens the door to full native access.
+This means that while Python code may be denied access to the host file system, thread- or subprocess creation, and more, the native extension is under no such restriction.
+
+### Memory Management
+
+Python C extensions, like the CPython reference implementation, use reference counting for memory management.
+This is fundamentally incompatible with JVM GCs.
+
+Java objects may end up being referenced from native data structures which the JVM cannot trace, so to avoid crashing, GraalPy keeps such Java objects strongly referenced.
+To avoid memory leaks, GraalPy implements a cycle detector that regularly traces references between Java objects and native objects that have crossed between the two worlds and cleans up strong references that are no longer needed.
+
+On the other side, reference-counted native extension objects may end up being referenced from Java objects, and in this case GraalPy bumps their reference count to make them unreclaimable.
+Any such references to native extension objects are registered with a `java.lang.ref.WeakReference` and when the JVM GC has collected the owning Java object, the reference count of the native object is reduced again.
+
+Both of these mechanisms together mean there is additional delay between objects becoming unreachable and their memory being reclaimed when compared to the CPython implementation.
+This can manifest in increased memory usage when running C extensions.
+You can tweak the Context options `python.BackgroundGCTaskInterval`, `python.BackgroundGCTaskThreshold`, and `BackgroundGCTaskMinimum` to mitigate this.
+They control the minimum interval between cycle detections, how much RSS memory must have increased since the last time to trigger the cycle detector, and the absolute minimum RSS under which no cycle detection should be done.
+You can also manually trigger the detector with the Python `gc.collect()` call.
+
+### Multi-Context and Native Libraries
+
+To support creating multiple GraalPy contexts that access native modules within the same JVM or Native Image, we need to isolate them from each other.
+The current strategy for this is to copy the libraries and modify them such that the dynamic library loader of the operating system will isolate them for us.
+To do this, all GraalPy contexts in the same process (not just those in the same engine!) must set the `python.IsolateNativeModules` option to `true`.
+
+For more details on this, see [our implementation details](https://github.com/oracle/graalpython/blob/master/docs/contributor/IMPLEMENTATION_DETAILS.md#c-extension-copying).

graalpython/com.oracle.graal.python.cext/include/cpython/object.h

@@ -1,4 +1,4 @@
-/* Copyright (c) 2020, 2023, Oracle and/or its affiliates.
+/* Copyright (c) 2020, 2024, Oracle and/or its affiliates.
  * Copyright (C) 1996-2020 Python Software Foundation
  *
  * Licensed under the PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2
@@ -42,8 +42,8 @@ PyAPI_FUNC(Py_ssize_t) _Py_GetRefTotal(void);
 typedef struct _Py_Identifier {
     const char* string;
     /* XXX Truffle change: CPython migrated away from keeping the pointer directly
-     * in the struct to support subinterpreters. We don't have subinterpreters, so
-     * we keep the object pointer for now */
+     * in the struct to support subinterpreters. We do subinterpreters differently,
+     * so we keep the object pointer for now */
     PyObject *object;
 } _Py_Identifier;
 

graalpython/com.oracle.graal.python.cext/modules/clinic/_bz2module.c.h

@@ -1,4 +1,4 @@
-/* Copyright (c) 2019, 2023, Oracle and/or its affiliates.
+/* Copyright (c) 2019, 2024, Oracle and/or its affiliates.
  * Copyright (C) 1996-2020 Python Software Foundation
  *
  * Licensed under the PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2

graalpython/com.oracle.graal.python.cext/modules/clinic/_testmultiphase.c.h

@@ -1,4 +1,4 @@
-/* Copyright (c) 2022, 2023, Oracle and/or its affiliates.
+/* Copyright (c) 2022, 2024, Oracle and/or its affiliates.
  * Copyright (C) 1996-2022 Python Software Foundation
  *
  * Licensed under the PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2

graalpython/com.oracle.graal.python.cext/modules/clinic/pyexpat.c.h

@@ -1,4 +1,4 @@
-/* Copyright (c) 2021, 2023, Oracle and/or its affiliates.
+/* Copyright (c) 2021, 2024, Oracle and/or its affiliates.
  * Copyright (C) 1996-2021 Python Software Foundation
  *
  * Licensed under the PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2

graalpython/com.oracle.graal.python.cext/src/capi.c

@@ -280,6 +280,13 @@ PyObject* _Py_EllipsisObjectReference;
 PyObject* _Py_NoneStructReference;
 PyObject* _Py_NotImplementedStructReference;
 
+/*
+ * This holds the thread-local reference to the PythonThreadState on the
+ * managed side. If additional contexts used the C API with the same Python
+ * library, we have to either update when switching contexts, or before each
+ * downcall, or just leave this NULL at all times and incur an upcall in the
+ * getter.
+ */
 THREAD_LOCAL PyThreadState *tstate_current = NULL;
 
 static void initialize_globals() {
@@ -916,7 +923,34 @@ void initialize_hashes();
 // defined in 'floatobject.c'
 void _PyFloat_InitState(PyInterpreterState* state);
 
+/*
+ * This is used to allow Truffle to enter/leave the context on native threads
+ * that were not created by NFI/Truffle/Java and thus not previously attached
+ * to the context. See e.g. PyGILState_Ensure. This is used by some C
+ * extensions to allow calling Python APIs from natively created threads. This
+ * poses a problem if multiple contexts use the same library, since we cannot
+ * know which context should be entered. CPython has the same problem (see
+ * https://docs.python.org/3/c-api/init.html#bugs-and-caveats), in particular
+ * the following quote:
+ *
+ *   Furthermore, extensions (such as ctypes) using these APIs to allow calling
+ *   of Python code from non-Python created threads will probably be broken
+ *   when using sub-interpreters.
+ *
+ * If we try to use the same libpython for multiple contexts, we can only
+ * behave in a similar (likely broken) way as CPython: natively created threads
+ * that use the PyGIL_* APIs to allow calling into Python will attach to the
+ * first interpreter that initialized the C API (and thus set the
+ * TRUFFLE_CONTEXT pointer) only.
+ */
 Py_LOCAL_SYMBOL TruffleContext* TRUFFLE_CONTEXT;
+
+/*
+ * This is only set during VM shutdown, so on the native side can only be used
+ * to guard things that do not work during VM shutdown, not to guard things
+ * that do not work during context shutdown! (This means that it would be safe
+ * to share this global across multiple contexts.)
+ */
 Py_LOCAL_SYMBOL int32_t graalpy_finalizing;
 
 PyAPI_FUNC(void) initialize_graal_capi(TruffleEnv* env, void **builtin_closures, GCState *gc) {
@@ -928,6 +962,30 @@ PyAPI_FUNC(void) initialize_graal_capi(TruffleEnv* env, void **builtin_closures,
 
     _PyGC_InitState(gc);
 
+    /*
+     * Initializing all these global fields with pointers to different contexts
+     * could be ok even if all contexts share this library and its globals.
+     * Each native stub is allocated using the AllocateNode and filled by each
+     * context. The long value of the stub pointer is then given an index via
+     * nativeStubLookupReserve, and the index is stored both in the stub for
+     * fast access from native as well as in the PythonObjectReference which
+     * wraps the stub on the managed side. The table is per context. Given that
+     * C API initialisation is deterministic and we initialise only with
+     * quasi-immortal objects, those indices are never repurposed for other
+     * objects. So, while later contexts override the pointers for those global
+     * objects with pointers to their own stubs, the index stored in those
+     * stubs are the same across all contexts, and thus mapping to
+     * context-specific objects works as intended. It'd a bit dodgy that the
+     * ob_refcnt fields of those objects would show whacky behaviour. Maybe we
+     * could just assert that everything stored during initialisation of the
+     * GraalPy C API has IMMORTAL_REFCNT and that all stub indices actually
+     * match. The only real problem would be if the last context exists and
+     * actually frees the stub memory. We would have to VM-globally delay
+     * freeing the "latest" stubs, but that's no big deal, we would simply keep
+     * a reference to the "latest" handle stub table globally. When it changes
+     * and the associated context already exited, we can free it now, when
+     * context exits and its table is the "latest", we delay freeing it.
+     */
     initialize_builtins(builtin_closures);
     PyTruffle_Log(PY_TRUFFLE_LOG_FINE, "initialize_builtins: %fs", ((double) (clock() - t)) / CLOCKS_PER_SEC);
     Py_Truffle_Options = GraalPyTruffle_Native_Options();

graalpython/com.oracle.graal.python.shell/src/com/oracle/graal/python/shell/GraalPythonMain.java

@@ -177,6 +177,7 @@ protected List<String> preprocessArguments(List<String> givenArgs, Map<String, S
         boolean posixBackendSpecified = false;
         boolean sha3BackendSpecified = false;
         boolean installSignalHandlersSpecified = false;
+        boolean isolateNativeModulesSpecified = false;
         for (Iterator<String> argumentIterator = arguments.iterator(); argumentIterator.hasNext();) {
             String arg = argumentIterator.next();
             origArgs.add(arg);
@@ -276,6 +277,9 @@ protected List<String> preprocessArguments(List<String> givenArgs, Map<String, S
                             if (matchesPythonOption(arg, "InstallSignalHandlers")) {
                                 installSignalHandlersSpecified = true;
                             }
+                            if (matchesPythonOption(arg, "IsolateNativeModules")) {
+                                isolateNativeModulesSpecified = true;
+                            }
                             // possibly a polyglot argument
                             unrecognized.add(arg);
                             continue;
@@ -432,6 +436,9 @@ protected List<String> preprocessArguments(List<String> givenArgs, Map<String, S
         if (!installSignalHandlersSpecified) {
             polyglotOptions.put("python.InstallSignalHandlers", "true");
         }
+        if (!isolateNativeModulesSpecified) {
+            polyglotOptions.put("python.IsolateNativeModules", "false");
+        }
         // Never emit warnings that mess up the output
         unrecognized.add("--engine.WarnInterpreterOnly=false");
         return unrecognized;

graalpython/com.oracle.graal.python.test.integration/pom.xml

@@ -177,7 +177,7 @@ Additionally, one can change the polyglot artifacts version with
             <groupId>org.graalvm.python</groupId>
             <artifactId>python-embedding</artifactId>
             <version>${com.oracle.graal.python.test.polyglot.version}</version>
-          </dependency>
+        </dependency>
         <dependency>
             <groupId>junit</groupId>
             <artifactId>junit</artifactId>