Skip to content

Commit 3da5572

Browse files
timfeltimfel
committed
[GR-69322] Update docs to clarify platform support levels.
PullRequest: graalpython/3974
2 parents c6da3bd + 67e47e9 commit 3da5572

File tree

4 files changed

+245
-61
lines changed

4 files changed

+245
-61
lines changed

‎docs/contributor/IMPLEMENTATION_DETAILS.md‎

Lines changed: 174 additions & 51 deletions
Original file line numberDiff line numberDiff line change
@@ -170,7 +170,7 @@ safepoint action mechanism can thus be used to kill threads waiting on the GIL.
170170
### High-level
171171

172172
C extensions assume reference counting, but on the managed side we want to leverage
173-
Java tracing GC. This creates a mismatch. The approach is to do both, reference
173+
Java's tracing garbage collector (GC). This creates a mismatch. The approach is to do both, reference
174174
counting and tracing GC, at the same time.
175175

176176
On the native side we use reference counting. The native code is responsible for doing
@@ -180,65 +180,188 @@ code is created and when the last reference from the native code is destroyed.
180180

181181
On the managed side we rely on tracing GC, so managed references are not ref-counted.
182182
For the ref-counting scheme on the native side, we approximate all the managed references
183-
as a single reference, i.e., we increment the refcount when object is referenced from managed
184-
code, and using a `PhantomReference` and reference queue we decrement the refcount when
183+
as a single reference, i.e., we increment the `refcount` when object is referenced from managed
184+
code, and using a `PhantomReference` and reference queue we decrement the `refcount` when
185185
there are no longer any managed references (but we do not clean the object as long as
186186
`refcount > 0`, because that means that there are still native references to it).
187187

188188
### Details
189189

190190
There are two kinds of Python objects in GraalPy: managed and native.
191191

192+
Below is a rough draft of the types and memory layouts involved and how they connect:
193+
194+
- On the left is the managed (Java) heap
195+
- On the right are memory layouts allocated natively
196+
- At the top are the classes involved with making built-in and pure Python objects available to native code (with PInt as an example)
197+
- At the bottom are the classes involved in making our Java code work when natively allocated memory is passed into GraalPy.
198+
199+
```
200+
Managed Heap Native Heap
201+
Stub allocated to represent managed object
202+
+------------------------+ +-------------------------------+
203+
| PInt |<---+ | struct PyGC_Head { |
204+
+------------------------+ | | uintptr_t _gc_next |
205+
| BigInteger value | | | uintptr_t _gc_prev |
206+
+------------------------+ | | } |
207+
| +------------>| struct GraalPyObject { |
208+
+------------------------+ | | | Py_ssize_t ob_refcnt |
209+
| PythonNativeWrapper |<---+ | +------>| PyObject *ob_type |
210+
+------------------------+------------+ | | int32_t handle_table_index |---+
211+
| +<---+ | | } | |
212+
+------------------------+ | | +-------------------------------+ |
213+
| | |
214+
| | |
215+
+------------------------+ | | |
216+
+---| PythonObjectReference |<---+ | |
217+
| +------------------------+------------------+ |
218+
| | boolean gc | |
219+
| | boolean freeAtCollect | - |
220+
| +------------------------+ |
221+
| |
222+
| +------------------------+ |
223+
|_/\| ReferenceQueue |<---+ |
224+
| \/+------------------------+ | |
225+
| | |
226+
| +---------------------------------------------------+ |
227+
| | HandleContext | |
228+
| +---------------------------------------------------+ index into nativeStubLookup |
229+
| | PythonObjectreference[] nativeStubLookup |-----------------------------------+
230+
| | HashMap<Long, NativeObjectReference> nativeLookup |
231+
| +---------------------------------------------------+
232+
| |
233+
| +-----------------------+ | Object allocated by native extension
234+
+---| NativeObjectReference | | +--------------------------------+
235+
+-----------------------+ +-----| struct PyObject { |
236+
+->| |---+ | Py_ssize_t ob_refcnt |
237+
| +-----------------------+ | | PyObject *ob_type |
238+
| +---------------------->| ... |
239+
| +--------------------+ | // extension defined memory |
240+
+->| PythonNativeObject |----------------------------->| |
241+
+--------------------+ +--------------------------------+
242+
| |
243+
+--------------------+
244+
245+
```
246+
247+
Managed objects are associated with `PythonNativeWrapper` subinstances when
248+
they go to native, native objects are represented throughout the interpreters
249+
as `PythonAbstractNativeObject`. Both have associated weak references,
250+
`PythonObjectReference` and `NativeObjectReference`, respectively.
251+
192252
#### Managed Objects
193253

194-
Managed objects are allocated in the interpreter. If there is no native code involved,
195-
we do not do anything special and let the Java GC handle them. When a managed object
196-
is passed to a native extension code:
197-
198-
* We wrap it in `PythonObjectNativeWrapper`. This is mostly in order to provide different
199-
interop protocol: we do not want to expose `toNative` and `asPointer` on Python objects.
200-
201-
* When NFI calls `toNative`/`asPointer` we:
202-
* Allocate C memory that will represent the object on the native side (including the refcount field)
203-
* Add a mapping of that memory address to the `PythonObjectNativeWrapper` object to a hash map `CApiTransitions.nativeLookup`.
204-
* We initialize the refcount field to a constant `MANAGED_REFCNT` (larger number, because some
205-
extensions like to special case on some small refcount values)
206-
* Create `PythonObjectReference`: a weak reference to the `PythonObjectNativeWrapper`,
207-
when this reference is enqueued (i.e., no managed references exist), we decrement the refcount by
208-
`MANAGED_REFCNT` and if the recount falls back to `0`, we deallocate the native memory of the object,
209-
otherwise we need to wait for the native code to eventually call `Py_DecRef` and make it `0`.
210-
211-
* When extension code wants to create a new reference, it will call `Py_IncRef`.
212-
In the C implementation of `Py_IncRef` we check if a managed object with
213-
`refcount==MANAGED_REFCNT` wants to increment its refcount. In such case, the native code is
214-
creating a first reference to the managed object, we must make sure to keep the object alive
215-
as long as there are some native references. We set a field `PythonObjectReference.strongReference`,
216-
which will keep the `PythonObjectNativeWrapper` alive even when all other managed references die.
217-
218-
* When extension code is done with the object, it will call `Py_DecRef`.
219-
In the C implementation of `Py_DecRef` we check if a managed object with `refcount == MANAGED_REFCNT+1`
220-
wants to decrement its refcount to MANAGED_REFCNT, which means that there are no native references
221-
to that object anymore. In such case we clear the `PythonObjectReference.strongReference` field,
222-
and the memory management is then again left solely to the Java tracing GC.
254+
Managed objects are allocated in the interpreter. If there is no native code
255+
involved, we do not do anything special and let the Java GC handle them. If
256+
native code wants to create any kind of object that is implemented as a
257+
built-in (in GraalPy) or in pure Python, we do an upcall and create the managed
258+
object. This object is immediately passed back to native code directly,
259+
and goes through the same transformation as one that was created from Python or
260+
Java code and is, for example, passed as an argument.
261+
262+
When a managed object is passed to a native extension code:
263+
264+
* We create `PythonNativeWrapper`. We create `PythonNativeWrapper` to
265+
provide a different interop protocol, and not expose `toNative` and
266+
`asPointer` on Python objects. The wrapper is stored inside the
267+
`PythonAbstractObject`, because pointer identity is relied upon by some
268+
extensions we saw in the wild. (See PythonToNativeNode)
269+
* If the object was a "primitive" (TruffleString, int, long, boolean) we must
270+
box it first into a `PythonAbstractObject`, so we create a `PString` or
271+
`PInt` wrapper (or retrieve one for the singletons).
272+
* For container types (such as tuples, lists), when their elements are
273+
accessed any primitive elements are (like the previous step) boxed into a
274+
`PythonAbstractObject`.
275+
276+
* When NFI calls `toNative`/`asPointer`, we:
277+
* Allocate a native stub that will represent the object on the native side.
278+
We allocate room for the `refcount` and type pointer to avoid upcalls for
279+
reading those. For some types such as floats, we also store the
280+
double value into native memory to avoid upcalls (see
281+
`FirstToNativeNode`). We also store a custom 32-bit integer into the native
282+
memory with an index into the `nativeStubLookup` array.
283+
* Create `PythonObjectReference`: a weak reference to the
284+
`PythonObjectNativeWrapper`. It is stored in `nativeStubLookup` for
285+
lookup. When this reference is enqueued (meaning no managed references to
286+
the object exist anymore), we decrement the `refcount` by `MANAGED_REFCNT`
287+
and if the recount reached `0`, we deallocate the object's native memory.
288+
Otherwise, we need to wait for the native code to eventually
289+
call `Py_DecRef` and make it `0` (see `AllocateNativeObjectStubNode`). The
290+
field `gc` indicates if this object has a GC header prepended to the
291+
pointer. The field `freeAtCollect` indicates this pointer can be free'd
292+
immediately when the reference is enqueued.
293+
* Initialize the `refcount` field to a constant `MANAGED_REFCNT` (larger
294+
number, because some extensions like to special case on some small
295+
`refcount` values)
296+
* Set the high bit of the allocated pointer to quickly identify stubs
297+
in native code. This allows us to make small
298+
modifications to the existing macros for checking types and refcounts in
299+
the Python C API.
300+
301+
* When extension code creates a new reference, it calls `Py_IncRef`.
302+
In the C implementation of `Py_IncRef`, we check if a managed
303+
object with `refcount==MANAGED_REFCNT` wants to increment its `refcount`.
304+
To ensure the object remains alive while native references exist,
305+
we set the `PythonObjectReference.strongReference` field which keeps
306+
the `PythonObjectNativeWrapper` alive even after all managed references are gone.
307+
308+
* When extension code is done with the object, it calls `Py_DecRef`. In the
309+
C implementation of `Py_DecRef`, we check if a managed object with
310+
`refcount == MANAGED_REFCNT+1` wants to decrement its `refcount` to `MANAGED_REFCNT`,
311+
which means that there are no native references to that object anymore.
312+
We then clear the `PythonObjectReference.strongReference` field, and the
313+
memory management is then again left solely to the Java tracing GC.
223314

224315
#### Native Objects
225316

226-
Native objects allocated using `PyObject_GC_New` in the native code are backed by native memory
227-
and may never be passed to managed code (as a return value of extension function or as an argument
228-
to some C API call). If a native object is not made available to managed code, it is just reference
229-
counted as usual, where `Py_DecRef` call that reaches `0` will deallocate the object. If a native
230-
object is passed to managed code:
317+
Native objects allocated using `PyObject_GC_New` in the native code are backed
318+
by native memory and may never be passed to managed code (as a return value of
319+
extension function or as an argument to some C API call). If a native object is
320+
not made available to managed code, it is just reference counted as usual,
321+
and when `Py_DecRef` reduces its count to `0`, it deallocates the object.
322+
231323

232-
* We increment the refcount of the native object by `MANAGED_REFCNT`
324+
If a native object is passed to managed code (see
325+
CApiTransitions.createAbstractNativeObject):
326+
* We increment the `refcount` of the native object by `MANAGED_REFCNT`
233327
* We create:
234328
* `PythonAbstractNativeObject` Java object to mirror it on the managed side
235-
* `NativeObjectReference`, a weak reference to the `PythonAbstractNativeObject`.
236-
* Add mapping: native object address => `NativeObjectReference` into hash map `CApiTransitions.nativeLookup`
329+
* `NativeObjectReference`, a weak reference to the
330+
`PythonAbstractNativeObject` which will be enqueued when it is no longer
331+
referenced from managed objects.
332+
* Add mapping: native object address => `NativeObjectReference` into hash map
333+
`CApiTransitions.nativeLookup`
237334
* Next time we just fetch the existing wrapper and don't do any of this
238-
* When `NativeObjectReference` is enqueued, we decrement the refcount by `MANAGED_REFCNT`
239-
* If the refcount falls to `0`, it means that there are no references to the object even from
240-
native code, and we can destroy it. If it does not fall to `0`, we just wait for the native
241-
code to eventually call `Py_DecRef` that makes it fall to `0`.
335+
* When `NativeObjectReference` is enqueued, we decrement the `refcount` by
336+
`MANAGED_REFCNT`
337+
* If the `refcount` falls to `0`, it means that there are no references to the
338+
object even from native code, and we can destroy it. If it does not fall to
339+
`0`, we just wait for the native code to eventually call `Py_DecRef` that
340+
makes it fall to `0`.
341+
342+
##### Memory Pressure
343+
344+
Since native allocations are not visible to the JVM, we can run into trouble if
345+
an application allocates only small managed objects that have large amounts of
346+
off-heap memory attached. In such cases, the GC does not see any memory
347+
pressure and may not collect the objects with associated native memory.
348+
To address this, we track off-heap allocations and count the allocated bytes.
349+
In order to correctly account for free'd memory, we prepend a header that
350+
stores the allocated size in the Python APIs memory management functions.
351+
352+
When we exceed a configurable threshold (`MaxNativeMemory`), we force a full Java GC,
353+
which also triggers a native Python cycle collection (see below). Unfortunately
354+
not all extensions use that API, so we *also* run a background thread
355+
that watches process RSS (see the `BackgroundGCTask*` context options). If
356+
RSS increases too quickly and we are allocating weak references for native objects,
357+
we force GCs more frequently.
358+
359+
For things that allocate GPU memory this is still not enough, since there are no
360+
APIs we can use to get GPU allocation rate across platforms. For PyTorch (for example)
361+
we apply a patch that forces a GC whenever a CUDA allocation fails and retries.
362+
Additionally, some extensions cause such a high rate of stub allocations that polling
363+
weak references on a lower priority background thread was too slow. To address this,
364+
we also poll during transitions to prevent our handle table from growing too rapidly.
242365

243366
#### Weak References
244367

@@ -263,7 +386,7 @@ The high level solution is that when we see a "dead" cycle going through an obje
263386
which may be, however, referenced from managed),
264387
we fully replicate the object graphs (and the cycle) on the managed side (refcounts of native objects
265388
in the cycle, which were not referenced from managed yet, will get new `NativeObjectReference`
266-
created and refcount incremented by `MANAGED_REFCNT`). Managed objects already refer
389+
created and `refcount` incremented by `MANAGED_REFCNT`). Managed objects already refer
267390
to the `PythonAbstractNativeObject` wrappers of the native objects (e.g., some Python container
268391
with managed storage), but we also make the native wrappers refer to whatever their referents
269392
are on the Java side (we use `tp_traverse` to find their referents).
@@ -279,17 +402,17 @@ count them into this limit. Let us call this limit *weak to strong limit*.
279402
After this, if the objects on the managed side (the managed objects or `PythonAbstractNativeObject`
280403
mirrors of native objects) are garbage, eventually Java GC will collect them.
281404
This will push their references to the reference queue. When polled from the queue (`CApiTransitions#pollReferenceQueue`),
282-
we decrement the refcount by `MANAGED_REFCNT` (no managed references anymore) and
283-
if their refcount falls to `0`, they are freed - as part of that, we call the
405+
we decrement the `refcount` by `MANAGED_REFCNT` (no managed references anymore) and
406+
if their `refcount` falls to `0`, they are freed - as part of that, we call the
284407
`tp_clear` slot for native objects, which should call `Py_CLEAR` for their references,
285-
which does `Py_DecRef` - eventually all objects in the cycle should fall to refcount `0`.
408+
which does `Py_DecRef` - eventually all objects in the cycle should fall to `refcount` `0`.
286409

287-
*Example: managed object `o1` has refcount `MANAGED_REFCNT+1`: `MANAGED_REFCNT` representing all managed
410+
*Example: managed object `o1` has `refcount` `MANAGED_REFCNT+1`: `MANAGED_REFCNT` representing all managed
288411
references, and `+1` for some native object `o2` referencing it. Native object `o2` has
289412
refcount `MANAGED_REFCNT`, because it is referenced only from managed (from `o1`).
290413
Both `o1` and `o2` form a cycle that was already transformed to managed during cycle GC.
291-
The reference queue processing will subtract `MANAGED_REFCNT` from `o1`'s refcount making it `1`.
292-
Then the reference queue processing will subtract `MANAGED_REFCNT` from `o2`'s refcount making it fall
414+
The reference queue processing will subtract `MANAGED_REFCNT` from `o1`'s `refcount` making it `1`.
415+
Then the reference queue processing will subtract `MANAGED_REFCNT` from `o2`'s `refcount` making it fall
293416
to `0` - this triggers the `tp_clear` of `o2`, which should subtract the final `1` from `o1`'s refcount.*
294417

295418
If some of the managed objects are not garbage, and they passed back to native code,

‎docs/site/01-docs.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,5 +22,6 @@ Below are the options you can set on contexts for GraalPy.
2222
{% gfm_docs ../user/Embedding-Permissions.md %}
2323
{% gfm_docs ../user/Tooling.md %}
2424
{% gfm_docs ../user/Troubleshooting.md %}
25+
{% gfm_docs ../user/Test-Tiers.md %}
2526

2627
{% copy_assets ../user/assets %}

0 commit comments

Comments
 (0)