[Python-C] _PyThreadState_UncheckedGet() changed behavior in python 3.12 and code from 3.11 no longer works

[AnnaAraslanova]

Bug report

Bug description:

Hello

I have code that works in Python 3.11 and no longer works in Python 3.12. I'd like to get an explanation of what changed as I see that the full GIL Python-C API was rewritten in Python 3.12 (./Python/pystate.c).

Basically I check that GIL lock is held by checking it this way:

bool IsGilHeld() {
  return _PyThreadState_UncheckedGet() != nullptr;
}

And I test it with doing an intencive operation:

# Simulate some heavy ML workload.
for _ in range(10000000):
    _ = random.random() * random.random()

In Python 3.11 _PyThreadState_UncheckedGet() returns a pointer at some point and in 3.12 the function always returns a nullptr.

@ericsnowcurrently

CPython versions tested on:

3.12

Operating systems tested on:

Linux

[picnixz]

The docs say (about the public PyThreadState_GetUnchecked()):

Added in version 3.13: In Python 3.5 to 3.12, the function was private and known as _PyThreadState_UncheckedGet().

I'm not sure that private API has the same guarantees as the public one so I don't know if we need to consider it as a bug. However, some private APIs were removed recently and then re-added because it caused downstream issues. So maybe, this specific function may also be considered similarly.

[encukou]

_PyThreadState_UncheckedGet returns NULL if the GIL is not held. I don't think that changed.

It looks like the change is somewhere in the surrounding code. Do you have some code to guarantee that IsGilHeld is eventually called with the GIL held?

[vstinner]

Your reproducer is incomplete, so it's not possible for us to help you. You should describe more what you are doing. As @encukou said, please explain how IsGilHeld() is called. Do you have a full reproducer?

[jgrave3]

I don't have full repro code handy at the moment but can summarize it.

Imagine there is a c++ (non-Python) background thread that polls regularly
_PyThreadState_UncheckedGet() != nullptr

Before 3.12, UncheckedGet would return the PyThreadState of the thread holding the GIL when called by a c++ thread that had released the GIL. In 3.12, it only returns the PyThreadState if called while holding the GIL.

We have a workaround: PyGILState_GetQueueLength > 0 returns whether anything is using or awaiting the GIL or not.

[ZeroIntensity]

Yeah, that makes sense. Thread states were made thread-local in 3.12.