Skip to content

gh-83765: indicate the maximum amount of memory on macOS #138552

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
Closed

gh-83765: indicate the maximum amount of memory on macOS #138552

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3002434
fix: indicate the maximum amount of memory on macOS (83765)
Sep 5, 2025
c431990
chore: numbers format
Sep 6, 2025
3c34ccc
fix: maximum size for a 64-bit process on mac 64
Sep 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Doc/library/multiprocessing.shared_memory.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ copying of data.
memory block may be larger or equal to the size requested.
When attaching to an existing shared memory block,
the *size* parameter is ignored.
On macOS, the maximum size of a shared memory block is 140256418463744 bytes.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At the very least please add some commas, but it might be more helpful to use a saner and more human readable unit:

Suggested change
On macOS, the maximum size of a shared memory block is 140256418463744 bytes.
On macOS, the maximum size of a shared memory block is 127 TiB (140,256,418,463,744 bytes).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, it would be useful to add a reference to where this number comes from. I can't see anything on the linked issue mentioning it.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi! Thank you for your comment! This number has been computed this way: https://gist.github.com/nilleb/e60e7adbfe2afaf9d727231c59b01100

I have tested it on macOS Sequoia 15.6.1.
That is the best I can afford for the moment, I don't have access to older operating systems.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, if this is an experimentally found number we should remove it, there are no guarantees.

Instead, say something like 'the maximum size is around 120 TiB'.

However, I'm now less sure about including this warning in the documentation, given the limit doesn't come from a documented or well-known property of macOS, and could change without warning. cc @python/macos-team for thoughts.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Searching further, I have found this SO post https://stackoverflow.com/a/77078609

Where they mention 0x00007ffffe000000 that corresponds to 140737454800896 (so a bit more that the size I get).
This would be compatible with the fact that the memory for a python process is limited to 128TiB on a 64bits macOS OS (and hence the maximum allowed value depends on the memory already allocated by python itself).

I would propose to add a link to https://github.com/apple-oss-distributions/xnu/blob/xnu-8796.141.3/osfmk/mach/arm/vm_param.h#L137

And a paraphrase of the explanation above.

WDYT?

Copy link
Member

@picnixz picnixz Sep 6, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe I'm wrong here but AFAICT, 127 TiB is not 140,256,418,463,744 bytes. Indeed 1 TiB is $2^{40}$ bytes so $2^{40} * 127 = 139,637,976,727,552$. Note that $140,737,454,800,896$ is a bit below $2^{40} * 128$ but above $127$ TiB.

$2^{40}$ is still a reasonable amount of memory, at least in my research area where we try to limit memory to that number (up to some constant factor, but most of the time, we want to limit the overall complexity of an attack to $O(2^{40})$ bits).

Instead, I would rather say that the maximum amount of memory on a 64-bit architecture is upperbounded by $128$ TiB and indicate that it is $128 * 2^{40}$ bytes). If we have mathjax enabled, we can use the math role, otherwise, you can write it down in full numbers.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, here is a further commit for your review! (the math role seems to be available, thank you!)


:param bool track:
When ``True``, register the shared memory block with a resource
Expand Down
Loading