How do I return rust iterator from a python module function using pyo3

[shivamka1]

So I have had a brief discussion on rust forum regarding a sensible way to return rust iterators in python.
https://users.rust-lang.org/t/how-do-i-return-rust-iterator-from-a-python-module-function-using-pyo3/89306

What remains unanswered is what's a proper way to return non-trivial rust iterators in python using pyo3! We can't always consume all of iterator before returning it to python. That's not practical for any non-trivial use case!

Please see the discussion on the rust forum. Let me know if I need to elaborate here again.

[adamreichold]

You cannot pass types containing generics or lifetimes to Python as these concepts are alien to the language, hence an iterator exposed to Python needs (shared) ownership of the data it iterates. (This is basically the same as if the iterator was implemented in Python itself which would also keep references to the underlying data structure around.)

[shivamka1]

I understand that but my question is how do we hack around this? The library I am working on builds this iterator many layers below python module. Neither can I consume whole iterator before passing it to python. It could potentially hold gigabytes of records.

As you can see in the discussion I have already made couple of attempts at time but still unclear of a feasible approach. Please suggest.

[adamreichold]

If the underlying iterator contains a lifetime, then you either need unsafely ignore that lifetime (using pointer casts etc.) or consume and stash the items before passing them to Python.

[birkenfeld]

See also #2707 (comment)