Error when using Lock from thread in asyncio

[davidbrochart]

Things to check first

• I have searched the existing issues and didn't find my bug already reported there

• I have checked that my bug is still present in the latest release

AnyIO version

4.4.0

Python version

3.12.5

What happened?

With the asyncio backend, this code fails with TypeError: cannot create weak reference to 'NoneType' object. It works with the trio backend.

Traceback

Traceback (most recent call last):
  File "/home/david/git/pycrdt/foo.py", line 11, in 
    run(main)
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/_core/_eventloop.py", line 74, in run
    return async_backend.run(func, args, {}, backend_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2034, in run
    return runner.run(wrapper())
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2022, in wrapper
    return await func(*args)
           ^^^^^^^^^^^^^^^^^
  File "/home/david/git/pycrdt/foo.py", line 9, in main
    await to_thread.run_sync(worker, lock)
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2177, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 859, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/git/pycrdt/foo.py", line 5, in worker
    from_thread.run_sync(lock.release)
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/from_thread.py", line 80, in run_sync
    return async_backend.run_sync_from_thread(func, args, token=token)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2239, in run_sync_from_thread
    return f.result()
           ^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2230, in wrapper
    f.set_result(func(*args))
                 ^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/_core/_synchronization.py", line 202, in release
    if self._owner_task != get_current_task():
                           ^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/_core/_testing.py", line 63, in get_current_task
    return get_async_backend().get_current_task()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2484, in get_current_task
    return AsyncIOTaskInfo(current_task())  # type: ignore[arg-type]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 1839, in __init__
    task_state = _task_states.get(task)
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/weakref.py", line 452, in get
    return self.data.get(ref(key),default)
                         ^^^^^^^^
TypeError: cannot create weak reference to 'NoneType' object

How can we reproduce the bug?

from anyio import Lock, from_thread, to_thread, run

def worker(lock):
    from_thread.run(lock.acquire)
    from_thread.run_sync(lock.release)

async def main():
    lock = Lock()
    await to_thread.run_sync(worker, lock)

run(main)

[agronholm]

The immediate cause is that it's trying to create a weak reference to None. It's trying to do this because current_task() returns None, but I'm still trying to figure out why it would do that.

[davidbrochart]

BTW with #761 the error is different: RuntimeError: The current task is not holding this lock.

Traceback

Traceback (most recent call last):
  File "/home/david/git/anyio/foo.py", line 11, in 
    run(main)
  File "/home/david/git/anyio/src/anyio/_core/_eventloop.py", line 74, in run
    return async_backend.run(func, args, {}, backend_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/git/anyio/src/anyio/_backends/_asyncio.py", line 2173, in run
    return runner.run(wrapper())
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/anyio/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/anyio/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/david/git/anyio/src/anyio/_backends/_asyncio.py", line 2161, in wrapper
    return await func(*args)
           ^^^^^^^^^^^^^^^^^
  File "/home/david/git/anyio/foo.py", line 9, in main
    await to_thread.run_sync(worker, lock)
  File "/home/david/git/anyio/src/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/git/anyio/src/anyio/_backends/_asyncio.py", line 2326, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/home/david/git/anyio/src/anyio/_backends/_asyncio.py", line 872, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/git/anyio/foo.py", line 5, in worker
    from_thread.run_sync(lock.release)
  File "/home/david/git/anyio/src/anyio/from_thread.py", line 80, in run_sync
    return async_backend.run_sync_from_thread(func, args, token=token)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/git/anyio/src/anyio/_backends/_asyncio.py", line 2388, in run_sync_from_thread
    return f.result()
           ^^^^^^^^^^
  File "/home/david/micromamba/envs/anyio/lib/python3.12/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/anyio/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/david/git/anyio/src/anyio/_backends/_asyncio.py", line 2379, in wrapper
    f.set_result(func(*args))
                 ^^^^^^^^^^^
  File "/home/david/git/anyio/src/anyio/_backends/_asyncio.py", line 1728, in release
    raise RuntimeError("The current task is not holding this lock")
RuntimeError: The current task is not holding this lock

[smurfix]

RuntimeError: The current task is not holding this lock.

That's to be expected.

The workaround is to use a nice 'async with lock:` block, and call back to the thread from within that.

[davidbrochart]

Oh I see, thanks. I guess I should close this issue then, and not give Alex more work than needed.

[agronholm]

I agree with @smurfix 's suggestion, but I would still like to understand why there was an error, and why it only happens with asyncio, as the effects of the problem may not be limited to locking.

[agronholm]

For posterity, the reason for this was that I had opted to use call_soon_threadsafe() as the backing implementation for run_sync_from_thread(). As such, the code runs in an environment where there is no backing task, hence returning None from current_task(). I'm reopening this as this may cause trouble further down the line, but it's not a priority.

[samskiter]

I see this is reopened - I think we've just run onto something similar on 4.6 (previously using 4.4). I haven't dug into it much but googling the error brought me straight to this ticket

[samskiter]

Ok have confirmed the same issue on 4.5 and 4.6 of Anyio. Sample trace here:

time=2024-09-26T08:24:38 level=ERROR message="Child process (HeatPumpSetter) failed with exception (cannot create weak reference to 'NoneType' object) 1727335478.2475069" file=supervisor.py line=105
  + Exception Group Traceback (most recent call last):
  |   File "blah.py", line 93, in _child_monitor
  |   File "/app/hub/.venv/lib/python3.11/site-packages/trio/_core/_run.py", line 959, in __aexit__
  |     raise combined_error_from_nursery
  | ExceptionGroup: Exceptions from Trio nursery (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "blah.py", line 54, in run
    |   File "/usr/local/lib/python3.11/contextlib.py", line 210, in __aenter__
    |     return await anext(self.gen)
    |            ^^^^^^^^^^^^^^^^^^^^^
    |   File "blah.py", line 48, in process_context
    |   File "/usr/local/lib/python3.11/contextlib.py", line 210, in __aenter__
    |     return await anext(self.gen)
    |            ^^^^^^^^^^^^^^^^^^^^^
    |   File "blah.py", line 104, in controllable_pump
    |   File "/app/hub/.venv/lib/python3.11/site-packages/anyio/_core/_synchronization.py", line 147, in __aenter__
    |     await self.acquire()
    |   File "/app/hub/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 1742, in acquire
    |     await AsyncIOBackend.cancel_shielded_checkpoint()
    |   File "/app/hub/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2287, in cancel_shielded_checkpoint
    |     with CancelScope(shield=True):
    |   File "/app/hub/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 400, in __enter__
    |     task_state = _task_states[host_task]
    |                  ~~~~~~~~~~~~^^^^^^^^^^^
    |   File "/usr/local/lib/python3.11/weakref.py", line 415, in __getitem__
    |     return self.data[ref(key)]
    |                      ^^^^^^^^
    | TypeError: cannot create weak reference to 'NoneType' object
    +------------------------------------

[graingert]

@samskiter oh that's odd, it's using the AsyncIOBackend on trio, can you post your blah.py source code?

[agronholm]

@samskiter ping – would you share your use case? It's hard to determine exactly what went wrong without seeing what code caused it.

[Automaat]

Hitting this same issue with OpenAI SDK in Python 3.14.2 with Textual TUI worker threads.

Stack:

• Python 3.14.2
• anyio 4.12.1
• httpcore 1.0.9
• httpx 0.28.1
• openai 2.14.0 SDK
• Textual worker thread (@work(thread=True))

Error:

TypeError: cannot create weak reference to 'NoneType' object
  at anyio/_backends/_asyncio.py:428
  in task_state = _task_states[host_task]

Trigger:
Running async OpenAI API calls via asyncio.run() in a Textual worker thread. The OpenAI SDK uses httpx → httpcore → anyio, which fails when trying to track task state in thread context where current_task() returns None.

Workarounds attempted (all failed):

• nest_asyncio.apply()
• anyio.from_thread.start_blocking_portal()
• Manual event loop creation with loop.run_until_complete()
• Disabling aiohttp transport in litellm

Impact:
Blocks using OpenAI SDK in any thread-based async execution context in Python 3.14. Standalone asyncio.run() works fine, but fails when called from threads.

Would appreciate any guidance on a workaround or timeline for a fix. This is becoming a blocker for Python 3.14 adoption in projects using httpx-based SDKs with threaded async execution.

[agronholm]

Can you point me to the code that is triggering this? I don't think I'm properly understanding what you're doing there.