Skip to content

Commit 83f27a3

Browse files
committed
A little more warning about multi-context C extensions
1 parent 1a2eaf2 commit 83f27a3

File tree

2 files changed

+8
-0
lines changed

2 files changed

+8
-0
lines changed

docs/contributor/IMPLEMENTATION_DETAILS.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -307,11 +307,14 @@ the next run of the cycle GC on the native side.
307307

308308
On Linux, Python native extensions expect to lookup Python C API functions in the global namespace and specify no explicit dependency on any libpython.
309309
To isolate them, we copy them with a new name, change their `SONAME`, add a `DT_NEEDED` dependency on a copy of our libpython shared object, and finally load them with `RTLD_LOCAL`.
310+
The ELF format is not really meant to allow such modifications after the fact.
311+
The most widely used tool to do so, [patchelf](https://github.com/NixOS/patchelf), still sees regular bugfixes and can corrupt ELF files.
310312

311313
On Windows there is no global namespace so native extensions already have a dependency on our libpython DLL.
312314
We copy them and just change the dependency to point to the context-local copy of libpython rather than the global one.
313315

314316
On macOS, while two-level namespaces exist, Python extensions historically use `-undefined dynamic_lookup` where they (just like in Linux) expect to find C API functions in any loaded image.
315317
We have to apply a similar workaround as on Linux, copy to a new name, change the `LC_ID_DYLIB` to that name, and add a `LC_LOAD_DYLIB` section to make the linker load the symbols from our libpython.
318+
This is currently not fully implemented.
316319

317320
Note that any code signatures are invalidated by this process.

docs/user/Native-Extensions.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,11 @@ You can also manually trigger the detector with the Python `gc.collect()` call.
4949

5050
### Multi-Context and Native Libraries
5151

52+
Using C extensions in multiple contexts is only possible on Linux for now, and many C extensions still have issues in this mode.
53+
You should test your applications thoroughly if you want to use this feature.
54+
There are many possiblities for native code to sidestep the library isolation through other process-wide global state, corrupting the state and leading to incorrect results or crashing.
55+
The implementation also relies on a venv to work, even if you are not using external packages.
56+
5257
To support creating multiple GraalPy contexts that access native modules within the same JVM or Native Image, we need to isolate them from each other.
5358
The current strategy for this is to copy the libraries and modify them such that the dynamic library loader of the operating system will isolate them for us.
5459
To do this, all GraalPy contexts in the same process (not just those in the same engine!) must set the `python.IsolateNativeModules` option to `true`.

0 commit comments

Comments
 (0)