Multiprocessing shared memory warnings about existing exported pointers during cleanup

[elidwa]

When running the test_multiprocess.py without collecting results via deep copy (line 131) and instead collecting results from shared memory (line 125), warnings about shared memory objects not being cleaned up are present. However, after the test completes, running ipcs shows that all shared memory has been properly released. This could be a cleanup bug in the code or a reference counting issue related to multiprocessing.

ipcs -m
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status

pytest -s test_multiprocess.py
======================================================================================= test session starts ========================================================================================
platform linux -- Python 3.13.1, pytest-8.3.4, pluggy-1.5.0
rootdir: /home/elidwa/H5Coro/h5coro
configfile: pytest.ini
plugins: anyio-4.8.0
collected 2 items

test_multiprocess.py MD5 checksum valid for /tmp/ATL03_20200401184200_00950707_005_01.h5

Reading with h5py
Reading with filedriver, multiProcess=False
Reading with s3driver, multiProcess=False
Reading with webdriver, multiProcess=False
Check results:
filedriver vs h5py
s3driver vs h5py
webdriver vs h5py
.MD5 checksum valid for /tmp/ATL03_20200401184200_00950707_005_01.h5

Reading with h5py
Reading with filedriver, multiProcess=True
Reading with s3driver, multiProcess=True
Reading with webdriver, multiProcess=True
Check results:
filedriver vs h5py
s3driver vs h5py
webdriver vs h5py
Exception ignored in: <function SharedMemory.del at 0xffff64b0fa60>
Traceback (most recent call last):
File "/home/elidwa/miniconda3/envs/h5coro-env/lib/python3.13/multiprocessing/shared_memory.py", line 189, in del
self.close()
File "/home/elidwa/miniconda3/envs/h5coro-env/lib/python3.13/multiprocessing/shared_memory.py", line 232, in close
self._mmap.close()
BufferError: cannot close exported pointers exist
.

========================================================================================= warnings summary =========================================================================================
tests/test_multiprocess.py::TestHDF::test_dataset_read[True]
/home/elidwa/miniconda3/envs/h5coro-env/lib/python3.13/site-packages/_pytest/unraisableexception.py:85: PytestUnraisableExceptionWarning: Exception ignored in: <function SharedMemory.del at 0xffff64b0fa60>

Traceback (most recent call last):
File "/home/elidwa/miniconda3/envs/h5coro-env/lib/python3.13/multiprocessing/shared_memory.py", line 189, in del
self.close()
~~~~~~~~~~^^
File "/home/elidwa/miniconda3/envs/h5coro-env/lib/python3.13/multiprocessing/shared_memory.py", line 232, in close
self._mmap.close()
~~~~~~~~~~~~~~~~^^
BufferError: cannot close exported pointers exist

warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))

[elidwa]

This is not a bug.

The shared memory warnings occur because of how references to shared memory objects are handled during cleanup. In my tests, using deep copy appeared to "work" because the h5obj was created inside a function, and the results from the promise were collected and returned. However, when the function completed, the h5obj was ready to be cleaned up while the shared memory references were still in use. This triggered the warnings when deep copy was not used.

I created a small test to demonstrate the issue. If I do not explicitly assign result = None after processing, the warnings reappear. Although objects are destroyed, the result still holds references to shared memory. Garbage collection behaves correctly, but the shared memory reference counting mechanism does not handle this scenario perfectly, leading to the warnings.

Example code:

h5obj = h5coro.H5Coro(self.local_file, filedriver.FileDriver, errorChecking=True, multiProcess=True)
promise = h5obj.readDatasets(self.datasets, block=True)
result = {dataset: promise[dataset] for dataset in promise}
result = None # Must be set to None to avoid shared memory leak warnings

Setting result = None ensures that shared memory references are properly released and prevents the warnings from appearing. Since this behavior is expected and tied to how shared memory references are counted, I am closing this issue.