[Bug]: MongoDB crashes because ulimit is too low

[CLOVIS-AI]

Module

MongoDB

Testcontainers version

1.19.8

Using the latest Testcontainers version?

Yes

Host OS

Linux

Host Arch

x86

Docker version

Docker version 28.0.1, build 068a01ea94

What happened?

MongoDB crashes while the tests are running.

Relevant log output

{"t":{"$date":"2025-03-28T10:47:39.000+00:00"},"s":"E",  "c":"ASSERT",   "id":23077,   "ctx":"ftdc","msg":"Assertion","attr":{"error":"Location13538: couldn't open [/proc/1/stat] Too many open files","file":"src/mongo/util/processinfo_linux.cpp","line":83}}
{"t":{"$date":"2025-03-28T10:47:39.282+00:00"},"s":"E",  "c":"WT",       "id":22435,   "ctx":"conn24","msg":"WiredTiger error message","attr":{"error":24,"message":{"ts_sec":1743158859,"ts_usec":282251,"thread":"1:0x7142a7603640","session_name":"WT_SESSION.create","category":"WT_VERB_DEFAULT","category_id":9,"verbose_level":"ERROR","verbose_level_id":-3,"msg":"__posix_directory_sync:135:/data/db/: directory-sync: open","error_str":"Too many open files","error_code":24}}}
{"t":{"$date":"2025-03-28T10:47:39.282+00:00"},"s":"E",  "c":"WT",       "id":22435,   "ctx":"conn24","msg":"WiredTiger error message","attr":{"error":24,"message":{"ts_sec":1743158859,"ts_usec":282319,"thread":"1:0x7142a7603640","session_name":"WT_SESSION.create","category":"WT_VERB_DEFAULT","category_id":9,"verbose_level":"ERROR","verbose_level_id":-3,"msg":"__posix_directory_sync:151:/data/db/collection-945-3757964622610459637.wt: directory-sync","error_str":"Too many open files","error_code":24}}}
{"t":{"$date":"2025-03-28T10:47:39.282+00:00"},"s":"E",  "c":"WT",       "id":22435,   "ctx":"conn24","msg":"WiredTiger error message","attr":{"error":-31804,"message":{"ts_sec":1743158859,"ts_usec":282337,"thread":"1:0x7142a7603640","session_name":"WT_SESSION.create","category":"WT_VERB_DEFAULT","category_id":9,"verbose_level":"ERROR","verbose_level_id":-3,"msg":"__posix_directory_sync:151:the process must exit and restart","error_str":"WT_PANIC: WiredTiger library panic","error_code":-31804}}}
{"t":{"$date":"2025-03-28T10:47:39.282+00:00"},"s":"F",  "c":"ASSERT",   "id":23089,   "ctx":"conn24","msg":"Fatal assertion","attr":{"msgid":50853,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp","line":741}}
{"t":{"$date":"2025-03-28T10:47:39.282+00:00"},"s":"F",  "c":"ASSERT",   "id":23090,   "ctx":"conn24","msg":"\n\n***aborting after fassert() failure\n\n"}
{"t":{"$date":"2025-03-28T10:47:39.282+00:00"},"s":"F",  "c":"CONTROL",  "id":6384300, "ctx":"conn24","msg":"Writing fatal message","attr":{"message":"\n"}}
{"t":{"$date":"2025-03-28T10:47:39.282+00:00"},"s":"F",  "c":"CONTROL",  "id":6384300, "ctx":"conn24","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}

Additional Information

MongoDB recommends setting ulimit -n 64000, which is much more than Docker's default of 1024.

In a docker-compose file, I can use:

        ulimits:
            nofile: 64000

to increase the file ulimit. However, I can't find a way to do so with testcontainers-java.

[eddumelendez]

Use withCreateContainerCmdModifier

withCreateContainerCmdModifier(cmd -> cmd.getHostConfig().withUlimits())

[CLOVIS-AI]

Thanks for the workaround! Now I guess all that's left to do is make this the default for MongoDBContainer. I could create a PR, if you want.

[eddumelendez]

We are not having this issue, tests run with no issues on GHA Linux runners, so, not sure about having it as a default.

[CLOVIS-AI]

How much data are you creating? It takes quite a few before the limits gets reached. You can see the number of 64000 recommended in the official documentation.

[eddumelendez]

It takes quite a few before the limits gets reached

Can you share your numbers? How can be reproduced? TBH, I am not sure this should be enabled for testing purposes. I think is good to be aware of this configuration when writing tests, so, it can be verified once the application is going to be deployed.

[CLOVIS-AI]

How can be reproduced?

Create a ton of documents. I don't know exactly how many, but it must be quite a few. On my side, it starts failing ~10 minutes into the test suite.

I think is good to be aware of this configuration when writing tests, so, it can be verified once the application is going to be deployed.

I understand that viewpoint. In that case, maybe provide an easy accessor, like MongoDBContainer.withRecommendedULimits() and a link to the production documentation?

I couldn't figure out how to configure the ulimits at all, and there weren't any useful pages on the web either. If there had been an obvious function, I wouldn't have had created this issue.

[taole33]

@CLOVIS-AI
Hi there!

I believe you can configure the ulimits by using withCreateContainerCmdModifier to adjust the host configuration directly.

Here is a code snippet:

import com.github.dockerjava.api.model.Ulimit;

// ...

MongoDBContainer mongoDBContainer = new MongoDBContainer("mongo:6.0")
    .withCreateContainerCmdModifier(cmd -> {
        cmd.getHostConfig().withUlimits(new Ulimit[] {
            new Ulimit("nofile", 64000L, 64000L)
        });
    });

Could you please try this and see if it resolves the crash?

[CLOVIS-AI]

Hi! Yes, it does.

As mentioned in previous comments, either:

• This should be made the default when using MongoDBContainer (since it's the default recommended by MongoDB)
• Or, there should be a function or documentation on the MongoDBContainer class

[taole33]

Glad to hear it worked!

Regarding the implementation, here is the official MongoDB documentation recommending the ulimit setting:
https://www.mongodb.com/docs/manual/reference/ulimit/#recommended-ulimit-settings
(It explicitly states: "MongoDB recommends setting the file descriptor limit ( -n) to 64000")

@eddumelendez Given the official recommendation, aligning the default behavior seems reasonable. However, if you are concerned about side effects as mentioned, adding an optional helper method like .withUlimits() would be a safe middle ground. It would improve the DX significantly by documenting this requirement in the code itself.

@CLOVIS-AI If the maintainers agree with this approach, would you still be interested in proceeding with the PR as you offered earlier?