run_coroutine_threadsafe document can not be reproduced

[PaleNeutron]

Documentation

https://docs.python.org/3/library/asyncio-task.html#asyncio.run_coroutine_threadsafe

My test code:

import asyncio


async def main():
    """main is async and started as normal with asyncio.run"""
    print("BEGIN main")

    loop = asyncio.get_running_loop()
    timeout = 3
    # Create a coroutine
    coro = asyncio.sleep(1, result=3)

    # Submit the coroutine to a given loop
    future = asyncio.run_coroutine_threadsafe(coro, loop)

    # Wait for the result with an optional timeout argument
    assert future.result(timeout) == 3


if __name__ == "__main__":
    asyncio.run(main())

Tested with Python 3.11.6 and Python 3.10.14, Both get TimeoutError

[sobolevn]

yes, the same on main. cc @kumaraditya303

[picnixz]

In the docs it says "This function is meant to be called from a different OS thread than the one where the event loop is running." But in your example I feel that it's in the same thread (or maybe I misunderstood this sentence)?

[PaleNeutron]

@picnixz the tricky thing is if we do not use .result, the coroutine works well, see this code:

copied from https://stackoverflow.com/a/76319360/2955827

import asyncio
from typing import Callable

Callback = Callable[[int], None]


class SomeSystem:
    """Some library you don't control that is mostly async, but provides a callback that
    is _not_ async."""

    def __init__(self, callback: Callback):
        self._callback = callback

    async def do_something(self):
        """do some work and then call the non-async callback"""
        await asyncio.sleep(1.0)
        self._callback(1)
        await asyncio.sleep(1.0)
        self._callback(2)


async def some_async_method(value: int):
    """some long-running operation normally called by async code"""
    await asyncio.sleep(0.1)
    print(f"long-running: {value}")


async def main():
    """main is async and started as normal with asyncio.run"""
    print("BEGIN main")

    loop = asyncio.get_running_loop()

    def cb(value: int) -> None:
        """This method _cannot_ be async, due to the underlying implementation of SomeSystem."""
        # some_async_method(value)  # RuntimeWarning: coroutine 'some_async_method' was never awaited
        asyncio.run_coroutine_threadsafe(some_async_method(value), loop)  # okay

    system = SomeSystem(cb)
    await system.do_something()

    # maybe ensure the last call to async method is awaited? Without this call, the final callback
    # won't be handled, since it's never being awaited. If anyone knows how to properly wait
    # for this, let me know in the comments!
    await asyncio.sleep(1.0)

    print("END main")


if __name__ == "__main__":
    asyncio.run(main())

---

If we change the code to use run_coroutine_threadsafe in main thread, it still doesn't work

import asyncio


async def foo(value):
    await asyncio.sleep(0.1)
    return value

def main():
    """main is async and started as normal with asyncio.run"""
    print("BEGIN main")

    timeout = 3
    # Create a coroutine
    loop = asyncio.new_event_loop()
    asyncio.run(foo(1))

    # Submit the coroutine to a given loop
    future = asyncio.run_coroutine_threadsafe(foo(2), loop)

    # Wait for the result with an optional timeout argument
    assert future.result(timeout) == 2


if __name__ == "__main__":
    main()

RuntimeWarning: coroutine 'foo' was never awaited

[picnixz]

No I mean, in your example, you are using the same thread, namely loop is created in the same thread as the one which calls asyncio.run_coroutine_threadsafe which is why it fails.

Maybe the name is misleading but run_coroutin_threadsafe is meant to be used as follows:

import asyncio
from threading import Thread

def start_event_loop(loop):
    asyncio.set_event_loop(loop)
    loop.run_forever()

if __name__ == '__main__':
    loop = asyncio.new_event_loop()
    loop_thread = Thread(target=start_event_loop, args=(loop,), daemon=True)

    try:
        loop_thread.start()
        coro = asyncio.sleep(1, result=3)
        future = asyncio.run_coroutine_threadsafe(coro, loop)
        assert future.result(3) == 3
    finally:
        loop.call_soon_threadsafe(loop.stop)
        loop_thread.join()

Rememeber that an event loop runs in a thread (here in my loop_thread) and executes all callbacks and Tasks in its thread. In your example, both the loop and asyncio.run_coroutine_threadsafe live in the same thread. The correct example is given in https://docs.python.org/3/library/asyncio-dev.html#asyncio-multithreading:

async def coro_func():
     return await asyncio.sleep(1, 42)

# Later in >> another << OS thread:

future = asyncio.run_coroutine_threadsafe(coro_func(), loop)
# Wait for the result:
result = future.result()

[picnixz]

Concerning the documentation, I agree that the example in run_coroutine_threadsafe is indeed misleading. Because it does not mention that it should be in another OS thread (the paragraph does but not the code example).

[kumaraditya303]

The example is updated now https://docs.python.org/3.14/library/asyncio-task.html#asyncio.run_coroutine_threadsafe

[picnixz]

@kumaraditya303 just for the record, can you add a ref for a commit/PR that updated that example please?

[kumaraditya303]

Fixed by #127576