Skip to content

Commit 49bfc33

Browse files
author
Andrew J Westlake
committed
Added tested examples for stream conversions, ran test feature-powerset
2 parents 58afb66 + b93e1ed commit 49bfc33

File tree

5 files changed

+810
-38
lines changed

5 files changed

+810
-38
lines changed

‎README.md

Lines changed: 41 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -52,8 +52,8 @@ Here we initialize the runtime, import Python's `asyncio` library and run the gi
5252
```toml
5353
# Cargo.toml dependencies
5454
[dependencies]
55-
pyo3 = { version = "0.14" }
56-
pyo3-asyncio = { version = "0.14", features = ["attributes", "async-std-runtime"] }
55+
pyo3 = { version = "0.16" }
56+
pyo3-asyncio = { version = "0.16", features = ["attributes", "async-std-runtime"] }
5757
async-std = "1.9"
5858
```
5959

@@ -82,8 +82,8 @@ attribute.
8282
```toml
8383
# Cargo.toml dependencies
8484
[dependencies]
85-
pyo3 = { version = "0.14" }
86-
pyo3-asyncio = { version = "0.14", features = ["attributes", "tokio-runtime"] }
85+
pyo3 = { version = "0.16" }
86+
pyo3-asyncio = { version = "0.16", features = ["attributes", "tokio-runtime"] }
8787
tokio = "1.9"
8888
```
8989

@@ -127,17 +127,17 @@ For `async-std`:
127127

128128
```toml
129129
[dependencies]
130-
pyo3 = { version = "0.14", features = ["extension-module"] }
131-
pyo3-asyncio = { version = "0.14", features = ["async-std-runtime"] }
130+
pyo3 = { version = "0.16", features = ["extension-module"] }
131+
pyo3-asyncio = { version = "0.16", features = ["async-std-runtime"] }
132132
async-std = "1.9"
133133
```
134134

135135
For `tokio`:
136136

137137
```toml
138138
[dependencies]
139-
pyo3 = { version = "0.14", features = ["extension-module"] }
140-
pyo3-asyncio = { version = "0.14", features = ["tokio-runtime"] }
139+
pyo3 = { version = "0.16", features = ["extension-module"] }
140+
pyo3-asyncio = { version = "0.16", features = ["tokio-runtime"] }
141141
tokio = "1.9"
142142
```
143143

@@ -431,8 +431,8 @@ name = "my_async_module"
431431
crate-type = ["cdylib"]
432432

433433
[dependencies]
434-
pyo3 = { version = "0.14", features = ["extension-module"] }
435-
pyo3-asyncio = { version = "0.14", features = ["tokio-runtime"] }
434+
pyo3 = { version = "0.16", features = ["extension-module"] }
435+
pyo3-asyncio = { version = "0.16", features = ["tokio-runtime"] }
436436
async-std = "1.9"
437437
tokio = "1.9"
438438
```
@@ -491,8 +491,8 @@ event loop before we can install the `uvloop` policy.
491491
```toml
492492
[dependencies]
493493
async-std = "1.9"
494-
pyo3 = "0.14"
495-
pyo3-asyncio = { version = "0.14", features = ["async-std-runtime"] }
494+
pyo3 = "0.16"
495+
pyo3-asyncio = { version = "0.16", features = ["async-std-runtime"] }
496496
```
497497

498498
```rust no_run
@@ -641,7 +641,7 @@ To make things a bit easier, I decided to keep most of the old API alongside the
641641

642642
> The `v0.13` API has been removed in version `v0.15`
643643
644-
### Migrating from 0.14 to 0.15
644+
### Migrating from 0.14 to 0.15+
645645

646646
There have been a few changes to the API in order to support proper cancellation from Python and the `contextvars` module.
647647

@@ -653,22 +653,20 @@ There have been a few changes to the API in order to support proper cancellation
653653
use pyo3::prelude::*;
654654

655655
Python::with_gil(|py| -> PyResult<()> {
656-
let event_loop = pyo3_asyncio::get_running_loop(py)?;
657656

658657
// *_with_loop conversions in 0.14
659658
//
659+
// let event_loop = pyo3_asyncio::get_running_loop(py)?;
660+
//
660661
// let fut = pyo3_asyncio::tokio::future_into_py_with_loop(
661662
// event_loop,
662663
// async move { Ok(Python::with_gil(|py| py.None())) }
663664
// )?;
664665
//
665-
// should be replaced with *_with_locals in 0.15
666-
//
667-
// contextvars can be copied with `copy_context` or supplied
668-
// manually via `with_context`
666+
// should be replaced with *_with_locals in 0.15+
669667
let fut = pyo3_asyncio::tokio::future_into_py_with_locals(
670668
py,
671-
pyo3_asyncio::TaskLocals::new(event_loop).copy_context(py)?,
669+
pyo3_asyncio::tokio::get_current_locals(py)?,
672670
async move { Ok(()) }
673671
)?;
674672

@@ -679,3 +677,27 @@ There have been a few changes to the API in order to support proper cancellation
679677
- `scope` and `scope_local` variants now accept `TaskLocals` instead of `event_loop`. You can usually just replace the `event_loop` with `pyo3_asyncio::TaskLocals::new(event_loop).copy_context(py)?`.
680678
- Return types for `future_into_py`, `future_into_py_with_locals` `local_future_into_py`, and `local_future_into_py_with_locals` are now constrained by the bound `IntoPy<PyObject>` instead of requiring the return type to be `PyObject`. This can make the return types for futures more flexible, but inference can also fail when the concrete type is ambiguous (for example when using `into()`). Sometimes the `into()` can just be removed,
681679
- `run`, and `run_until_complete` can now return any `Send + 'static` value.
680+
681+
### Migrating from 0.15 to 0.16
682+
683+
Actually, not much has changed in the API. I'm happy to say that the PyO3 Asyncio is reaching a
684+
pretty stable point in 0.16. For the most part, 0.16 has been about cleanup and removing deprecated
685+
functions from the API.
686+
687+
PyO3 0.16 comes with a few API changes of its own, but one of the changes that most impacted PyO3
688+
Asyncio was it's decision to drop support for Python 3.6. PyO3 Asyncio has been using a few
689+
workarounds / hacks to support the pre-3.7 version of Python's asyncio library that are no longer
690+
necessary. PyO3 Asyncio's underlying implementation is now a bit cleaner because of this.
691+
692+
PyO3 Asyncio 0.15 included some important fixes to the API in order to add support for proper task
693+
cancellation and allow for the preservation / use of contextvars in Python coroutines. This led to
694+
the deprecation of some 0.14 functions that were used for edge cases in favor of some more correct
695+
versions, and those deprecated functions are now removed from the API in 0.16.
696+
697+
In addition, with PyO3 Asyncio 0.16, the library now has experimental support for conversions from
698+
Python's async generators into a Rust `Stream`. There are currently two versions `v1` and `v2` with
699+
slightly different performance and type signatures, so I'm hoping to get some feedback on which one
700+
works best for downstream users. Just enable the `unstable-streams` feature and you're good to go!
701+
702+
> The inverse conversion, Rust `Stream` to Python async generator, may come in a later release if
703+
> requested!

‎src/async_std.rs

Lines changed: 187 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -489,7 +489,51 @@ pub fn into_future(awaitable: &PyAny) -> PyResult<impl Future<Output = PyResult<
489489
generic::into_future::<AsyncStdRuntime>(awaitable)
490490
}
491491

492-
/// Convert async generator into a stream
492+
/// <span class="module-item stab portability" style="display: inline; border-radius: 3px; padding: 2px; font-size: 80%; line-height: 1.2;"><code>unstable-streams</code></span> Convert an async generator into a stream
493+
///
494+
/// # Arguments
495+
/// * `gen` - The Python async generator to be converted
496+
///
497+
/// # Examples
498+
/// ```
499+
/// use pyo3::prelude::*;
500+
/// use futures::{StreamExt, TryStreamExt};
501+
///
502+
/// const TEST_MOD: &str = r#"
503+
/// import asyncio
504+
///
505+
/// async def gen():
506+
/// for i in range(10):
507+
/// await asyncio.sleep(0.1)
508+
/// yield i
509+
/// "#;
510+
///
511+
/// # #[cfg(all(feature = "unstable-streams", feature = "attributes"))]
512+
/// # #[pyo3_asyncio::async_std::main]
513+
/// # async fn main() -> PyResult<()> {
514+
/// let stream = Python::with_gil(|py| {
515+
/// let test_mod = PyModule::from_code(
516+
/// py,
517+
/// TEST_MOD,
518+
/// "test_rust_coroutine/test_mod.py",
519+
/// "test_mod",
520+
/// )?;
521+
///
522+
/// pyo3_asyncio::async_std::into_stream_v1(test_mod.call_method0("gen")?)
523+
/// })?;
524+
///
525+
/// let vals = stream
526+
/// .map(|item| Python::with_gil(|py| -> PyResult<i32> { Ok(item?.as_ref(py).extract()?) }))
527+
/// .try_collect::<Vec<i32>>()
528+
/// .await?;
529+
///
530+
/// assert_eq!((0..10).collect::<Vec<i32>>(), vals);
531+
///
532+
/// Ok(())
533+
/// # }
534+
/// # #[cfg(not(all(feature = "unstable-streams", feature = "attributes")))]
535+
/// # fn main() {}
536+
/// ```
493537
///
494538
/// # Availability
495539
///
@@ -503,7 +547,55 @@ pub fn into_stream_v1<'p>(
503547
generic::into_stream_v1::<AsyncStdRuntime>(gen)
504548
}
505549

506-
/// Convert async generator into a stream
550+
/// <span class="module-item stab portability" style="display: inline; border-radius: 3px; padding: 2px; font-size: 80%; line-height: 1.2;"><code>unstable-streams</code></span> Convert an async generator into a stream
551+
///
552+
/// # Arguments
553+
/// * `locals` - The current task locals
554+
/// * `gen` - The Python async generator to be converted
555+
///
556+
/// # Examples
557+
/// ```
558+
/// use pyo3::prelude::*;
559+
/// use futures::{StreamExt, TryStreamExt};
560+
///
561+
/// const TEST_MOD: &str = r#"
562+
/// import asyncio
563+
///
564+
/// async def gen():
565+
/// for i in range(10):
566+
/// await asyncio.sleep(0.1)
567+
/// yield i
568+
/// "#;
569+
///
570+
/// # #[cfg(all(feature = "unstable-streams", feature = "attributes"))]
571+
/// # #[pyo3_asyncio::async_std::main]
572+
/// # async fn main() -> PyResult<()> {
573+
/// let stream = Python::with_gil(|py| {
574+
/// let test_mod = PyModule::from_code(
575+
/// py,
576+
/// TEST_MOD,
577+
/// "test_rust_coroutine/test_mod.py",
578+
/// "test_mod",
579+
/// )?;
580+
///
581+
/// pyo3_asyncio::async_std::into_stream_with_locals_v1(
582+
/// pyo3_asyncio::async_std::get_current_locals(py)?,
583+
/// test_mod.call_method0("gen")?
584+
/// )
585+
/// })?;
586+
///
587+
/// let vals = stream
588+
/// .map(|item| Python::with_gil(|py| -> PyResult<i32> { Ok(item?.as_ref(py).extract()?) }))
589+
/// .try_collect::<Vec<i32>>()
590+
/// .await?;
591+
///
592+
/// assert_eq!((0..10).collect::<Vec<i32>>(), vals);
593+
///
594+
/// Ok(())
595+
/// # }
596+
/// # #[cfg(not(all(feature = "unstable-streams", feature = "attributes")))]
597+
/// # fn main() {}
598+
/// ```
507599
///
508600
/// # Availability
509601
///
@@ -518,7 +610,55 @@ pub fn into_stream_with_locals_v1<'p>(
518610
generic::into_stream_with_locals_v1::<AsyncStdRuntime>(locals, gen)
519611
}
520612

521-
/// Convert async generator into a stream
613+
/// <span class="module-item stab portability" style="display: inline; border-radius: 3px; padding: 2px; font-size: 80%; line-height: 1.2;"><code>unstable-streams</code></span> Convert an async generator into a stream
614+
///
615+
/// # Arguments
616+
/// * `locals` - The current task locals
617+
/// * `gen` - The Python async generator to be converted
618+
///
619+
/// # Examples
620+
/// ```
621+
/// use pyo3::prelude::*;
622+
/// use futures::{StreamExt, TryStreamExt};
623+
///
624+
/// const TEST_MOD: &str = r#"
625+
/// import asyncio
626+
///
627+
/// async def gen():
628+
/// for i in range(10):
629+
/// await asyncio.sleep(0.1)
630+
/// yield i
631+
/// "#;
632+
///
633+
/// # #[cfg(all(feature = "unstable-streams", feature = "attributes"))]
634+
/// # #[pyo3_asyncio::async_std::main]
635+
/// # async fn main() -> PyResult<()> {
636+
/// let stream = Python::with_gil(|py| {
637+
/// let test_mod = PyModule::from_code(
638+
/// py,
639+
/// TEST_MOD,
640+
/// "test_rust_coroutine/test_mod.py",
641+
/// "test_mod",
642+
/// )?;
643+
///
644+
/// pyo3_asyncio::async_std::into_stream_with_locals_v2(
645+
/// pyo3_asyncio::async_std::get_current_locals(py)?,
646+
/// test_mod.call_method0("gen")?
647+
/// )
648+
/// })?;
649+
///
650+
/// let vals = stream
651+
/// .map(|item| Python::with_gil(|py| -> PyResult<i32> { Ok(item.as_ref(py).extract()?) }))
652+
/// .try_collect::<Vec<i32>>()
653+
/// .await?;
654+
///
655+
/// assert_eq!((0..10).collect::<Vec<i32>>(), vals);
656+
///
657+
/// Ok(())
658+
/// # }
659+
/// # #[cfg(not(all(feature = "unstable-streams", feature = "attributes")))]
660+
/// # fn main() {}
661+
/// ```
522662
///
523663
/// # Availability
524664
///
@@ -533,9 +673,51 @@ pub fn into_stream_with_locals_v2<'p>(
533673
generic::into_stream_with_locals_v2::<AsyncStdRuntime>(locals, gen)
534674
}
535675

536-
/// Convert async generator into a stream
676+
/// <span class="module-item stab portability" style="display: inline; border-radius: 3px; padding: 2px; font-size: 80%; line-height: 1.2;"><code>unstable-streams</code></span> Convert an async generator into a stream
537677
///
538-
/// # Availability
678+
/// # Arguments
679+
/// * `gen` - The Python async generator to be converted
680+
///
681+
/// # Examples
682+
/// ```
683+
/// use pyo3::prelude::*;
684+
/// use futures::{StreamExt, TryStreamExt};
685+
///
686+
/// const TEST_MOD: &str = r#"
687+
/// import asyncio
688+
///
689+
/// async def gen():
690+
/// for i in range(10):
691+
/// await asyncio.sleep(0.1)
692+
/// yield i
693+
/// "#;
694+
///
695+
/// # #[cfg(all(feature = "unstable-streams", feature = "attributes"))]
696+
/// # #[pyo3_asyncio::async_std::main]
697+
/// # async fn main() -> PyResult<()> {
698+
/// let stream = Python::with_gil(|py| {
699+
/// let test_mod = PyModule::from_code(
700+
/// py,
701+
/// TEST_MOD,
702+
/// "test_rust_coroutine/test_mod.py",
703+
/// "test_mod",
704+
/// )?;
705+
///
706+
/// pyo3_asyncio::async_std::into_stream_v2(test_mod.call_method0("gen")?)
707+
/// })?;
708+
///
709+
/// let vals = stream
710+
/// .map(|item| Python::with_gil(|py| -> PyResult<i32> { Ok(item.as_ref(py).extract()?) }))
711+
/// .try_collect::<Vec<i32>>()
712+
/// .await?;
713+
///
714+
/// assert_eq!((0..10).collect::<Vec<i32>>(), vals);
715+
///
716+
/// Ok(())
717+
/// # }
718+
/// # #[cfg(not(all(feature = "unstable-streams", feature = "attributes")))]
719+
/// # fn main() {}
720+
/// ```
539721
///
540722
/// **This API is marked as unstable** and is only available when the
541723
/// `unstable-streams` crate feature is enabled. This comes with no

0 commit comments

Comments
 (0)