Allow Manual Control Over resource_tracker in shared_memory

[koshak01]

Feature or enhancement

Proposal:

Proposal: Allow Manual Control Over resource_tracker in shared_memory

Context and Problem Statement

I am working on a system where a main module is responsible for creating and deleting shared memory segments, while sub-modules only connect to these segments and write data. However, when a sub-module stops, resource_tracker incorrectly deletes the shared memory segment, even though it should persist.

This is a critical issue because resource_tracker assumes it has full control over shared memory cleanup, while in complex architectures like mine, only the main module should be responsible for managing these resources.

Proposed Solution

I propose adding a mechanism to control whether resource_tracker tracks a shared memory segment. There are several possible approaches:

1. Add a flag (e.g., track=False) when creating a SharedMemory object:

   shm = SharedMemory(name="my_shared_segment", create=True, track=False)

   This would allow developers to specify that they do not want resource_tracker to track this object.

2. Allow replacing resource_tracker with a custom implementation, so developers can define their own resource management logic.

3. Provide an API to disable resource_tracker entirely for advanced users who manage shared memory explicitly:

   from multiprocessing import resource_tracker
   resource_tracker.disable()

Why This is Needed

• In systems with complex shared memory architectures, resource_tracker does not have enough context to decide which segments should be deleted.
• Allowing manual control would prevent unintended resource deletion, reducing potential data loss or race conditions in multi-process applications.
• Manually patching resource_tracker or modifying the Python standard library is not an ideal solution.

Current Workaround

For now, I have resorted to commenting out resource_tracker code in the standard library, which I acknowledge is not a proper solution.

Would the Python maintainers consider adding such an option in multiprocessing.shared_memory?

Thank you!

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

No response

[YvesDup]

Provide an API to disable resource_tracker entirely for advanced users who manage shared memory explicitly:

Unfortunatly, resource_tracker is used with multiprocessing.[R]Lock, multiprocessing.[Bounded]Semaphore and multiprocessing shared datas. Proposition 3 is not possible.

[koshak01]

Yes, I understand that resource_tracker is used for managing locks, semaphores, and shared memory objects. However, for cases where a developer wants full control over shared memory management, wouldn't it be possible to introduce an optional parameter?

For example, in line 120 of multiprocessing/shared_memory.py, we have:

resource_tracker.register(self._name, "shared_memory")

Could we make this optional with a parameter, such as:

if track:
    resource_tracker.register(self._name, "shared_memory")

This way, users who need automatic cleanup can still use it, while those managing memory manually can disable tracking when creating a SharedMemory object:

shm = SharedMemory(name="my_shared_segment", create=True, track=False)

Would this approach be feasible? It would provide flexibility while keeping the default behavior unchanged.

[YvesDup]

Maybe the last part of this link will be useful as workaround

[koshak01]

I have already worked around this issue, and interestingly, after disabling the memory manager, memory leaks stopped occurring as well.

However, my goal is not just to find a workaround but to suggest a logical and non-intrusive improvement to the standard library. Adding an optional parameter to control whether resource_tracker tracks shared memory segments would not introduce breaking changes and would simply give developers more flexibility in managing shared resources.

This small addition would allow users with complex architectures to handle memory management explicitly, while keeping the default behavior unchanged for those who rely on automatic cleanup.

[YvesDup]

I understand your point of view. Your proposal should be submit to the core dev owner of this code: @gpshead.

[koshak01]

Should I create a new issue for this proposal, or is there another preferred way to proceed? Thank you.

[ericvsmith]

I think you should open a discussion on https://discuss.python.org/c/ideas/6

[koshak01]

I’ve posted the idea on the forum as suggested! Should we close this issue now and send it off to the land of unresolved-but-discussed problems? 😄

To be honest, I can’t shake the feeling that we’ve just moved the furniture around without actually fixing the leaky pipe. But hey, at least now it’s a well-documented leaky pipe! 🚰

Thanks for the guidance!

[ericvsmith]

I'll close this. Since this is a feature request, discussing it on d.p.o. is the correct procedure. It will have many more eyes on it than this issue ever will. If there's a consensus on adding some version of this feature, we can either re-open this issue or create a new one.

Thanks for your work! I've never used shared memory from Python, but this seems like a reasonable request to me.

[koshak01]

https://discuss.python.org/t/allow-manual-control-over-resource-tracker-in-shared-memory/81288