Skip to content

V8 GC does not know if PyObjects grow after initialization #74

@mmomtchev

Description

@mmomtchev

The V8 GC has a complex algorithm that decides when it is going to make a collection run and this requires tracking of the allocated memory. Currently pymport reports only the size of objects it has referenced and does only once.

In particular these two cases can be problematic:

  • Allocating small Python objects, then growing them - for example a dict or a list that gradually becomes huge - V8 won't be aware of the size of this object and won't be in a hurry to free it when it is no longer referenced
  • Python subroutines allocate a large number of Python objects that are never seen by V8 - V8 may continue allocating memory beyond its administrative limits

These problem do not cause leaks - as the memory is still tracked - it is simply not correctly included in the GC statistics.

Alas, solving these problems would require instrumenting Python's allocator (which is costly) and there is no possible solution compatible with worker_threads - as all V8 threads share the same Python instance - it won't be possible to know whose account is to be charged when allocating memory.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions