[Feature Request] Async Support

[logan-markewich]

Hey! We are adding BM25 search to LlamaIndex here run-llama/llama_index#7342

It would be super cool if the library supported async retrieval at some point 🙏🏻 If that's not possible though, that's ok too, but it would improve the usability of the library quite a bit

[andersfylling]

How does coroutine support matter here?

[logan-markewich]

Being able to call the retriever asynchronously helps a lot if you are running on a server for example

For example, in LlamaIndex, we can call embeddings or LLMs asynchronously so that they don't block the entire server thread

[andersfylling]

coroutine scheduling happens within a single thread. Given this is CPU bound, you won't be able to execute any other coroutines at the time. Meaning, it will actually harm your performance instead of improving it.

For CPU bound tasks I suggest using anyio and send off the work to a background process:

result = await anyio.to_thread.run_sync(some_task, "arg")