Creating a bunch of `mmap.mmap` objects fails with "OSError: [Errno 12] Cannot allocate memory" (Python 3.13, Linux)

[Azureblade3808]

Bug description:

With the help of newly added trackfd=False option in mmap.mmap constructor, we can create far more mmap.mmap objects beyond the open file limit. But creating a lot of mmap.mmap instances seems to cause memory errors (“OSError: [Errno 12] Cannot allocate memory”).

Example:

import mmap, os

fn = "/tmp/some_file"
fd = os.open(fn, os.O_CREAT | os.O_RDWR | os.O_TRUNC, 0o600)
os.ftruncate(fd, 1)

mmap_objects = []
try:
    for i in range(1_000_000):
        mmap_object = mmap.mmap(fd, 1, trackfd=False)
        mmap_objects.append(mmap_object)
except:
    print(i)
    raise

I ran the snippet above on a CentOS8 server with 48GB RAM, and it stopped at the 65479th iterations with the mentioned error.

By contrast, creation of 1M builtins.object instances can be easily done and a C program that creates 1M mmap pointers can also finish with no error.

This issue were also reproduced on RHEL7.2.

CPython versions tested on:

3.13

Operating systems tested on:

Linux (CentOS 8, RHEL 7)

EDIT

I made a horrible mistake in my C program that it does not check if the pointer returned by mmap is valid. In fact, the max number of valid mmap pointers it can create is 65515. Sorry for the false and misleading statement.

[encukou]

On my Fedora with 64GB RAM, it gets to 1048437.
If you have the C code available, where does it stop if you try going up to 10 million?

[Azureblade3808]

If you have the C code available, where does it stop if you try going up to 10 million?

Sorry for having given some false and misleading info, but I made a mistake in my C program and it really didn't count how many mmap pointers it can create. The real number should be 65515 on my CentOS server.

I will check if there is a configurable limit for mmap in CentOS system.

[Azureblade3808]

As a reference, the incorrect C program is -

#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>

int main(int args, char *argv[])
{
    int fd = open("sample", O_CREAT | O_RDWR | O_TRUNC, 0600);
    ftruncate(fd, 1);

    for (int i = 0; i < 1000000; i++)
    {
        // Expected the following line would crash the program, but it really won't.
        mmap(NULL, 1, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
    }

    return 0;
}

, and the fixed version is -

#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>

int main(int args, char *argv[])
{
    int fd = open("sample", O_CREAT | O_RDWR | O_TRUNC, 0600);
    if (fd == -1)
    {
        printf("failed to open file\n");
        return -1;
    }

    if (ftruncate(fd, 1) == -1)
    {
        printf("failed to resize file\n");
        return -2;
    }

    for (int i = 0; i < 1000000; i++)
    {
        void *mmap_pointer = mmap(NULL, 1, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
        if (mmap_pointer == (void *)-1)
        {
            printf("failed to mmap in iteration #%d\n", i);
            return -3;
        }
    }

    return 0;
}

[Azureblade3808]

It's vm.max_map_count that is causing the problem.

It seems to have a default value of 65530 on my server. After I changed it to a larger number, the Python snippet in OP can run successfully.

Sorry again for wasting your time.