How should I think about memory efficient concurrent programing when using pyo3?

[l-monninger]

I'm new to pyo3. I'd like to use it to interpret Python programs that handle their own multiprocessing. I came across this cool discussion today. And, I realized that I should probably have a much better mental model of how memory is used for processes spawned, forked, etc. by Python logic bubble up to memory usage by C and then Rust.

Where should I begin?

Keeping track of potential relevant content I've come across thusfar:

• How can we make Python's C-API more friendly for Rust? #2346
• How to parallel python function #2391

[davidhewitt]

I usually say that the main opportunity PyO3 offers for concurrent programming is that you can have Rust threads running free of the Python GIL, which avoids the need to go for the complexity of Python's multiprocessing module.

When multiprocessing comes into the picture it doesn't really change this model; the forked processes created by multiprocessing can all benefit from their own Rust threads independently. Note that #2346 is a separate problem; that's discussing a possible future Python where multiple interpreters run in the same process instead of forks like multiprocessing.

Is there a particular problem you're trying to solve, or topics that you are finding confusing? This is potentially a huge area so it's hard to know what to write for you without some further prompting.

[l-monninger]

Thanks, David!

As far as #2346 is concerned, more than anything that thread made me realize I didn't have the best understanding of what exists in memory when multiple interpreters are running under the pyo3 umbrella.

When multiprocessing comes into the picture it doesn't really change this model; the forked processes created by multiprocessing can all benefit from their own Rust threads independently.

I believe this speaks to my practical concerns. I assume this means when I'm calling a py.allow_threads method from my Python subprocesses, I can get useful multithreading. I also assume that everything is 1:1 threaded regardless of whatever flavor of threading a given Python library supposes. Awesome that we can get around the GIL though! I wonder how Sam Gross feels about this haha.

But, I still feel like I would like a more fundamental understanding multiprocessing. So, let's say I have an object of the following shape:

@dataclass
class Foo():
    name: str
    value : float

If I share this object across say 4 processes using Python abstractions (not that I necessarily will use multiprocessing now), what are the structures that are allocated? Is it roughly the same as when I multiprocess in Python normally?

Sorry if this all a bit nooby. I found this crate when I was simply looking for a way to call Python from Rust. I've been learning a bit about both sides of that coin since then.

[davidhewitt]

Sorry again for the slow reply. All questions are reasonable, just busy so might take me time to answer.

Yes, 4 processes or 4 subprocesses with multiprocessing, you'll end up with 4 separate allocations of this class in memory. IIRC the way the object would be "shared" between the processes using multiprocessing is by pickling & unpickling.