Skip to content

Commit 27ad604

Browse files
author
Andrew J Westlake
committed
Added PyO3 guide to README
1 parent 0ffcea7 commit 27ad604

File tree

1 file changed

+289
-1
lines changed

1 file changed

+289
-1
lines changed

README.md

Lines changed: 289 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -189,7 +189,295 @@ Type "help", "copyright", "credits" or "license" for more information.
189189
>
190190
> This restriction may be lifted in a future release.
191191
192+
## PyO3 Asyncio Primer
193+
194+
If you are working with a Python library that makes use of async functions or wish to provide
195+
Python bindings for an async Rust library, [`pyo3-asyncio`](https://github.com/awestlake87/pyo3-asyncio)
196+
likely has the tools you need. It provides conversions between async functions in both Python and
197+
Rust and was designed with first-class support for popular Rust runtimes such as
198+
[`tokio`](https://tokio.rs/) and [`async-std`](https://async.rs/). In addition, all async Python
199+
code runs on the default `asyncio` event loop, so `pyo3-asyncio` should work just fine with existing
200+
Python libraries.
201+
202+
In the following sections, we'll give a general overview of `pyo3-asyncio` explaining how to call
203+
async Python functions with PyO3, how to call async Rust functions from Python, and how to configure
204+
your codebase to manage the runtimes of both.
205+
206+
## Awaiting an Async Python Function in Rust
207+
208+
Let's take a look at a dead simple async Python function:
209+
210+
```python
211+
# Sleep for 1 second
212+
async def py_sleep():
213+
await asyncio.sleep(1)
214+
```
215+
216+
**Async functions in Python are simply functions that return a `coroutine` object**. For our purposes,
217+
we really don't need to know much about these `coroutine` objects. The key factor here is that calling
218+
an `async` function is _just like calling a regular function_, the only difference is that we have
219+
to do something special with the object that it returns.
220+
221+
Normally in Python, that something special is the `await` keyword, but in order to await this
222+
coroutine in Rust, we first need to convert it into Rust's version of a `coroutine`: a `Future`.
223+
That's where `pyo3-asyncio` comes in.
224+
[`pyo3_asyncio::into_future`](https://docs.rs/pyo3-asyncio/latest/pyo3_asyncio/fn.into_future.html)
225+
performs this conversion for us:
226+
227+
228+
```rust no_run
229+
use pyo3::prelude::*;
230+
231+
#[pyo3_asyncio::tokio::main]
232+
async fn main() -> PyResult<()> {
233+
let future = Python::with_gil(|py| -> PyResult<_> {
234+
// import the module containing the py_sleep function
235+
let example = py.import("example")?;
236+
237+
// calling the py_sleep method like a normal function
238+
// returns a coroutine
239+
let coroutine = example.call_method0("py_sleep")?;
240+
241+
// convert the coroutine into a Rust future using the
242+
// tokio runtime
243+
pyo3_asyncio::tokio::into_future(coroutine)
244+
})?;
245+
246+
// await the future
247+
future.await?;
248+
249+
Ok(())
250+
}
251+
```
252+
253+
> If you're interested in learning more about `coroutines` and `awaitables` in general, check out the
254+
> [Python 3 `asyncio` docs](https://docs.python.org/3/library/asyncio-task.html) for more information.
255+
256+
## Awaiting a Rust Future in Python
257+
258+
Here we have the same async function as before written in Rust using the
259+
[`async-std`](https://async.rs/) runtime:
260+
261+
```rust
262+
/// Sleep for 1 second
263+
async fn rust_sleep() {
264+
async_std::task::sleep(std::time::Duration::from_secs(1)).await;
265+
}
266+
```
267+
268+
Similar to Python, Rust's async functions also return a special object called a
269+
`Future`:
270+
271+
```rust compile_fail
272+
let future = rust_sleep();
273+
```
274+
275+
We can convert this `Future` object into Python to make it `awaitable`. This tells Python that you
276+
can use the `await` keyword with it. In order to do this, we'll call
277+
[`pyo3_asyncio::async_std::future_into_py`](https://docs.rs/pyo3-asyncio/latest/pyo3_asyncio/async_std/fn.future_into_py.html):
278+
279+
```rust
280+
use pyo3::prelude::*;
281+
282+
async fn rust_sleep() {
283+
async_std::task::sleep(std::time::Duration::from_secs(1)).await;
284+
}
285+
286+
#[pyfunction]
287+
fn call_rust_sleep(py: Python) -> PyResult<&PyAny> {
288+
pyo3_asyncio::async_std::future_into_py(py, async move {
289+
rust_sleep().await;
290+
Ok(Python::with_gil(|py| py.None()))
291+
})
292+
}
293+
```
294+
295+
In Python, we can call this pyo3 function just like any other async function:
296+
297+
```python
298+
from example import call_rust_sleep
299+
300+
async def rust_sleep():
301+
await call_rust_sleep()
302+
```
303+
304+
## Managing Event Loops
305+
306+
Python's event loop requires some special treatment, especially regarding the main thread. Some of
307+
Python's `asyncio` features, like proper signal handling, require control over the main thread, which
308+
doesn't always play well with Rust.
309+
310+
Luckily, Rust's event loops are pretty flexible and don't _need_ control over the main thread, so in
311+
`pyo3-asyncio`, we decided the best way to handle Rust/Python interop was to just surrender the main
312+
thread to Python and run Rust's event loops in the background. Unfortunately, since most event loop
313+
implementations _prefer_ control over the main thread, this can still make some things awkward.
314+
315+
### PyO3 Asyncio Initialization
316+
317+
Because Python needs to control the main thread, we can't use the convenient proc macros from Rust
318+
runtimes to handle the `main` function or `#[test]` functions. Instead, the initialization for PyO3 has to be done from the `main` function and the main
319+
thread must block on [`pyo3_asyncio::run_forever`](https://docs.rs/pyo3-asyncio/latest/pyo3_asyncio/fn.run_forever.html) or [`pyo3_asyncio::async_std::run_until_complete`](https://docs.rs/pyo3-asyncio/latest/pyo3_asyncio/async_std/fn.run_until_complete.html).
320+
321+
Because we have to block on one of those functions, we can't use [`#[async_std::main]`](https://docs.rs/async-std/latest/async_std/attr.main.html) or [`#[tokio::main]`](https://docs.rs/tokio/1.1.0/tokio/attr.main.html)
322+
since it's not a good idea to make long blocking calls during an async function.
323+
324+
> Internally, these `#[main]` proc macros are expanded to something like this:
325+
> ```rust compile_fail
326+
> fn main() {
327+
> // your async main fn
328+
> async fn _main_impl() { /* ... */ }
329+
> Runtime::new().block_on(_main_impl());
330+
> }
331+
> ```
332+
> Making a long blocking call inside the `Future` that's being driven by `block_on` prevents that
333+
> thread from doing anything else and can spell trouble for some runtimes (also this will actually
334+
> deadlock a single-threaded runtime!). Many runtimes have some sort of `spawn_blocking` mechanism
335+
> that can avoid this problem, but again that's not something we can use here since we need it to
336+
> block on the _main_ thread.
337+
338+
For this reason, `pyo3-asyncio` provides its own set of proc macros to provide you with this
339+
initialization. These macros are intended to mirror the initialization of `async-std` and `tokio`
340+
while also satisfying the Python runtime's needs.
341+
342+
Here's a full example of PyO3 initialization with the `async-std` runtime:
343+
```rust no_run
344+
use pyo3::prelude::*;
345+
346+
#[pyo3_asyncio::async_std::main]
347+
async fn main() -> PyResult<()> {
348+
// PyO3 is initialized - Ready to go
349+
350+
let fut = Python::with_gil(|py| -> PyResult<_> {
351+
let asyncio = py.import("asyncio")?;
352+
353+
// convert asyncio.sleep into a Rust Future
354+
pyo3_asyncio::async_std::into_future(
355+
asyncio.call_method1("sleep", (1.into_py(py),))?
356+
)
357+
})?;
358+
359+
fut.await?;
360+
361+
Ok(())
362+
}
363+
```
364+
365+
## PyO3 Asyncio in Cargo Tests
366+
367+
The default Cargo Test harness does not currently allow test crates to provide their own `main`
368+
function, so there doesn't seem to be a good way to allow Python to gain control over the main
369+
thread.
370+
371+
We can, however, override the default test harness and provide our own. `pyo3-asyncio` provides some
372+
utilities to help us do just that! In the following sections, we will provide an overview for
373+
constructing a Cargo integration test with `pyo3-asyncio` and adding your tests to it.
374+
375+
### Main Test File
376+
First, we need to create the test's main file. Although these tests are considered integration
377+
tests, we cannot put them in the `tests` directory since that is a special directory owned by
378+
Cargo. Instead, we put our tests in a `pytests` directory.
379+
380+
> The name `pytests` is just a convention. You can name this folder anything you want in your own
381+
> projects.
382+
383+
We'll also want to provide the test's main function. Most of the functionality that the test harness needs is packed in the [`pyo3_asyncio::testing::main`](https://docs.rs/pyo3-asyncio/latest/pyo3_asyncio/testing/fn.main.html) function. This function will parse the test's CLI arguments, collect and pass the functions marked with [`#[pyo3_asyncio::async_std::test]`](https://docs.rs/pyo3-asyncio/latest/pyo3_asyncio/async_std/attr.test.html) or [`#[pyo3_asyncio::tokio::test]`](https://docs.rs/pyo3-asyncio/latest/pyo3_asyncio/tokio/attr.test.html) and pass them into the test harness for running and filtering.
384+
385+
`pytests/test_example.rs` for the `tokio` runtime:
386+
```rust
387+
#[pyo3_asyncio::tokio::main]
388+
async fn main() -> pyo3::PyResult<()> {
389+
pyo3_asyncio::testing::main().await
390+
}
391+
```
392+
393+
`pytests/test_example.rs` for the `async-std` runtime:
394+
```rust
395+
#[pyo3_asyncio::async_std::main]
396+
async fn main() -> pyo3::PyResult<()> {
397+
pyo3_asyncio::testing::main().await
398+
}
399+
```
400+
401+
### Cargo Configuration
402+
Next, we need to add our test file to the Cargo manifest by adding the following section to the
403+
`Cargo.toml`
404+
405+
```toml
406+
[[test]]
407+
name = "test_example"
408+
path = "pytests/test_example.rs"
409+
harness = false
410+
```
411+
412+
Also add the `testing` and `attributes` features to the `pyo3-asyncio` dependency and select your preferred runtime:
413+
414+
```toml
415+
pyo3-asyncio = { version = "0.13", features = ["testing", "attributes", "async-std-runtime"] }
416+
```
417+
418+
At this point, you should be able to run the test via `cargo test`
419+
420+
### Adding Tests to the PyO3 Asyncio Test Harness
421+
422+
We can add tests anywhere in the test crate with the runtime's corresponding `#[test]` attribute:
423+
424+
For `async-std` use the [`pyo3_asyncio::async_std::test`](https://docs.rs/pyo3-asyncio/latest/pyo3_asyncio/async_std/attr.test.html) attribute:
425+
```rust
426+
mod tests {
427+
use std::{time::Duration, thread};
428+
429+
use pyo3::prelude::*;
430+
431+
// tests can be async
432+
#[pyo3_asyncio::async_std::test]
433+
async fn test_async_sleep() -> PyResult<()> {
434+
async_std::task::sleep(Duration::from_secs(1)).await;
435+
Ok(())
436+
}
437+
438+
// they can also be synchronous
439+
#[pyo3_asyncio::async_std::test]
440+
fn test_blocking_sleep() -> PyResult<()> {
441+
thread::sleep(Duration::from_secs(1));
442+
Ok(())
443+
}
444+
}
445+
446+
#[pyo3_asyncio::async_std::main]
447+
async fn main() -> pyo3::PyResult<()> {
448+
pyo3_asyncio::testing::main().await
449+
}
450+
```
451+
452+
For `tokio` use the [`pyo3_asyncio::tokio::test`](https://docs.rs/pyo3-asyncio/latest/pyo3_asyncio/tokio/attr.test.html) attribute:
453+
```rust
454+
mod tests {
455+
use std::{time::Duration, thread};
456+
457+
use pyo3::prelude::*;
458+
459+
// tests can be async
460+
#[pyo3_asyncio::tokio::test]
461+
async fn test_async_sleep() -> PyResult<()> {
462+
tokio::time::sleep(Duration::from_secs(1)).await;
463+
Ok(())
464+
}
465+
466+
// they can also be synchronous
467+
#[pyo3_asyncio::tokio::test]
468+
fn test_blocking_sleep() -> PyResult<()> {
469+
thread::sleep(Duration::from_secs(1));
470+
Ok(())
471+
}
472+
}
473+
474+
#[pyo3_asyncio::tokio::main]
475+
async fn main() -> pyo3::PyResult<()> {
476+
pyo3_asyncio::testing::main().await
477+
}
478+
```
479+
192480
## MSRV
193481
Currently the MSRV for this library is 1.46.0, _but_ if you don't need to use the `async-std-runtime`
194482
feature, you can use rust 1.45.0.
195-
> `async-std` depends on `socket2` which fails to compile under 1.45.0.
483+
> `async-std` depends on `socket2` which fails to compile under 1.45.0.

0 commit comments

Comments
 (0)