[doc] creating large SharedMemory crashes on MacOs

[vinay0410]

BPO	39584
Nosy	@ronaldoussoren, @pitrou, @tiran, @ned-deily, @applio, @vinay0410
PRs	bpo-39584: raise ValueError when creating shared memory of size greater than 1TB #21877

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-02-08.14:18:24.896>
labels = ['OS-mac', '3.8', '3.9', 'expert-C-API', 'type-crash', 'docs']
title = 'multiprocessing.shared_memory: MacOS crashes by running attached Python code'
updated_at = <Date 2020-11-17.10:55:29.496>
user = 'https://github.com/vinay0410'

bugs.python.org fields:

activity = <Date 2020-11-17.10:55:29.496>
actor = 'ronaldoussoren'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation', 'macOS', 'C API']
creation = <Date 2020-02-08.14:18:24.896>
creator = 'vinay0410'
dependencies = []
files = []
hgrepos = []
issue_num = 39584
keywords = ['patch']
message_count = 9.0
messages = ['361629', '361924', '373806', '375229', '375245', '375249', '381221', '381222', '381223']
nosy_count = 7.0
nosy_names = ['ronaldoussoren', 'pitrou', 'christian.heimes', 'ned.deily', 'docs@python', 'davin', 'vinay0410']
pr_nums = ['21877']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue39584'
versions = ['Python 3.8', 'Python 3.9']

Linked PRs

• gh-83765: indicate the maximum amount of memory on macOS #138552

[vinay0410]

Consider the following python Code.

from multiprocessing.shared_memory import SharedMemory
shm = SharedMemory(name='test-crash', create=True, size=1000000000000000000)

This causes macOS Catalina, Mojave to freeze and then crash. Although, this works fine on ubuntu.

After, debugging I realised that this is due to the ftruncate call. I could replicate the same by calling os.ftruncate and also using ftruncate in C code.

Following C++ code also crashes, which confirms that ftruncate in macOS is broken:

#include <fcntl.h>
#include <sys/shm.h>
#include <sys/stat.h>
#include <iostream>
#include <sys/mman.h>
#include <unistd.h>
#include <sys/types.h>
#include <errno.h>
#include <time.h>

int main() {

    int shm_fd = shm_open("/test-shm2", O_CREAT | O_RDWR, 0666);

    if (shm_fd == -1) {
        throw "Shared Memory Object couldn't be created or opened";
    }

    int rv = ftruncate(shm_fd, (long long)1000000000000000000);

}

Should python, in any way handle this, so as to prevent any crashes using python code.

[applio]

My sense is that it would be nice if we can catch this before ftruncate does something nasty.

Where else is ftruncate used in CPython that this could similarly trigger a problem? How is it handled there (or not)?

[vinay0410]

Hi, I tried replicating this by truncating normal files but that doesn't crash. The above mentioned call of ftruncate only crashes for when the file descriptor passed points to a shared memory segment.
And only, multiprocessing.shared_memory is currently creating shared_memory using _posixshmem.shm_open.

So, it can be fixed in ftruncate implementation or this can also be handled by multiprocessing.shared_memory.

Please let me know your thoughts about the same.