Implementing complex type conversion between Python and Rust.

[jaywonchung]

Hi!

I'm looking into PyO3 to speed up my Python project by rewriting its time-consuming part in Rust.

My project deals with graphs (nodes and edges).
Its Python-side graph representation is networkx.DiGraph and the Rust-side graph representation will be rs_graph::Digraph, but converting between these will involve some logic (contrary to having a head-to-head corresponding type in each language). For instance, the nodes of networkx.DiGraph has to be accessed with something like DiGraph.nodes(), and I'll have to construct a new object that implements rs_graph::Digraph.
Similarly, when returning to Python, I'll have to convert rs_graph::Digraph to networkx.DiGraph.

What would be a general recommendation for building something with some complexity in conversion?
I'm fine with getting hit with some conversion overhead when calling the Rust-side subroutine from Python.

Thank a lot.

[adamreichold]

This is probably not overly helpful, but it usually depends. Let's your rather expensive conversion takes seconds. If the algorithm you want to implement takes seconds itself that overhead might be prohibitive, but if it takes hours then those extra seconds will most likely not matter at all. So this is hard to answer without concrete numbers and experimentation.

Alternative strategies would include wrapping your Rust-side types like rs_graph::Digraph in a custom #[pyclass] so that there is no conversion. But this means that you will have to implement a Python API so that the rest of your Python code can use your custom #[pyclass] instead of networkx.DiGraph which might be a lot of effort since you only want to rewrite the parts which actually consume a lot of time. Hence, this approach often works best when one aims to rewrite all of the code in Rust eventually.

Another option is to use a general purpose graph library like rustworkx which started out as the Rust reimplementation of networkx but could be used as it is to accelerate graph-heavy Python programs. But w.r.t. flexibility and possible speed-ups, this is not really different from existing Python modules implementing their time-consuming algorithms in C/C++/Cython.

My personal experience would suggest to give the most straight-forward conversion operation a try and measure the results. With involved computations even heavy pre and post processing often works out in the end. If it does not, you can often try to gradually move a larger part of the algorithm into Rust to avoid more conversion overheads until you achieve the desired speed up. But hopefully well before you had to rewrite your whole application...

[jaywonchung]

Thank you for the detailed answer and it makes a lot of sense. Maybe I could start by creating adaptors both in the Python side and the Rust side that distills the graph into a simple class that just holds nodes and edges, and have Python and Rust interface through that class definition. Thanks again for sharing your experience!