Need Advice on How to Handle Async Reader in Pyo3 Python bindings

[abstractqqq]

Hello, I am trying to build a reader that automatically decompresses data and chunks data in a certain way and output the bytes to Python. This works perfectly for local files and I can easily get a 5x speed up. But I am facing a lot of technical difficulty when it comes to cloud files, which I am reading through object_store/aws-sdk-s3. The problem I am facing is that inevitably I need something like the following:

pub struct PyGzReader<R: AsyncBufRead> {
    ... ...
    reader: GzipDecoder<R>
}

because I need to keep the state of the reader in order for read_one_chunk() to work. The whole point of my project is to stream the data because it can be extremely large. All Rust cloud data store crates provide async APIs, and they often return an "impl AsyncBufRead" instead of a concrete type for data streams.

I have done enough reading and googling and I know Pyo3 won't be able to make the struct above into a Pyclass. I have also thought about creating the reader on Python side and passing it to Rust but it looks like passing file handle may slow down the process a lot because the bytes need to be copied...

Any advice on how should I proceed? Thank you in advance.

[davidhewitt]

I think you probably want to have something like #[pyclass] MyReader(Box<dyn AsyncBufRead>), and build methods on that which can be async? Note that our async support is still evolving...

[abstractqqq]

thank you. I tried something like that but then it complains it is not thread safe.. I then wrapped it in a Mutex and it is compiling and working.. I am still not 100% on why that isn't thread safe...

[davidhewitt]

You might want Box<dyn AsyncBufRead + Send + Sync>

[abstractqqq]

So my access pattern in Rust is that the reader only has a single mutable reference when it is being called. So + Send + Sync makes sense because they are only used single-threaded.

If I make sure I don't access the object in Python from multiple threads, then it should be safe to mark it as Send + Sync? Is my understanding correct? Thank you

[davidhewitt]

Adding Send + Sync would add a requirement for the objects to be internally thread-safe; if that's not the case then leaving the Mutex seems simpler.

[abstractqqq]

thank you!