Skip to content

Conversation

@YvesDup
Copy link
Contributor

@YvesDup YvesDup commented Mar 6, 2025

Following #125829 discussions and after a long deeping in the _multiprocessing.semaphore.c C file, I suggest a workaround based on a shared memory that contains an array to store all the internal semaphore counters.

At the moment, all the tests are passing, but I'm having 2 side problems, the second of which is really embarrassing:

  1. How to calculate the size of the array stored in the shared memory, knowing that:
    1. I sized the array from the value of sysconf(SC_SEM_NSEMS_MAX) which is set to 87381 on a mac OSX 15. This is very high.
    2. But I can't create more than about 4900 semaphores regardless of the number of python instances. I get an OSError(28, 'No space left on device'). What is the right value ?
  2. When and how to call the system function shm_unlink to properly close the shared memory (First I called shm_open)?

Regarding this last point, I wonder if only the process that opened the shared memory should call this function? Or should all processes that called shm_open call shm_unlink? Unfortunately, the Posix documentation isn't precise.

I worked on the count of opened/closed processes, with the aim of calling shm_unlink when the last process has closed, without success. I also had a look to the posixshmem module, without success too.
Could the resource_tracker and/or the use of the Finalize class of the multiprocessing.util module help me to fix this problem ? Is it possible to use these 2 Python libs from C API ?

This memory management difficulty make me think that this choice may not be appropriate. I am a bit stuck.
Thank you for your feedback and ideas.

@YvesDup YvesDup marked this pull request as draft April 15, 2025 12:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant