Fast exchange data between single rust process and multi python process

[sun-rs]

I have built a python extension by pyo3 for test some data, but it is a 1vs1 scenario. In a financial realtime scenario, I have a data source api wrote by rust and could get stream data. Also I have 20+ python consumer to use the same data to run some test work.

How to exchange data between rust and multiple python process in a low latency way? Maybe use pyo3 to call multiple python rather than run serveral pyo3 python extension?

[adamreichold]

I guess this is basically a question on "fast data exchange between multiple processes", because the receiver side could always be an extension using PyO3 that does what whatever works best to speak with the producer process.

As for the communication mechanism, I guess you could use either Unix domain sockets or shared memory. Shared memory has the potential to be much faster, but you will probably need to write more code to coordinate the data exchange. Sockets are most likely much simpler to get started with but will eventually be limited by the system call and data copying overhead.

A ready-to-use solution might be Servo's ipc-channel which uses bincode for serialization.

Alternatively, you might want to use a serialization format with limited overhead, e.g. rkyv or abomonation.

For shared memory, I guess the nix or shared_memory crates would be a place to start, but as written above the main work here would be a coordination scheme.

Finally, Eclipse's iceoryx seems to make using shared memory comparatively straight forward and appears to have gained Rust bindings.

[sun-rs]

Thank you! The reply do helps a lot!