Elasticsearch 8.16.x Large Increase in MMAP Counts

[Evesy]

Elasticsearch Version

8.16.1

Installed Plugins

No response

Java Version

bundled && Java 17

OS Version

Linux elasticsearch-data-hot-1 6.1.112+ #1 SMP PREEMPT_DYNAMIC Sat Oct 19 17:09:54 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Problem Description

Elasticsearch 8.16.x onwards is requiring significantly more memory regions than prior versions.

https://discuss.elastic.co/t/heap-allocation-failures-on-8-17/372211/8
https://discuss.elastic.co/t/oom-since-8-16-1-with-openjdk23

In our experience (first link), we started observing semi-frequent heap allocation failures across all our hot nodes after upgrading from 8.15.x to 8.17.x. All our hot nodes would restart due to these errors within a couple of hours of each other, and then the same would happen again between 12 - 24 hours later.

After some digging we discovered that the max mmap count we had configured, based on the recommendations was being reached, resulting in these heap allocation failures.

We doubled the value to then observe if/where Elasticsearch would eventually top out at, which in our case was in the early 400k mark, and have yet to observe any failures since. The number of memory regions is not something we were previously collecting, however at the most conservative estimate if it was previously right below the limit prior to upgrading, the new numbers we were seeing after upgrading would be a roughly 60% increase in the amount of mmap regions being used, which does not feel like intended behaviour (or should at least be documented if so)

The second link provided above is another user with the same issue, after upgrading to 8.16.x (which indicates the change likes somewhere in the 8.16 series)

In our case we went from 8.15.1 to 8.17.0, without any JVM changes (using our own provided Java 21). In the other example it was upgrading from 8.15.1 to 8.16.1 including a change to the JVM version (preumably bundled JVM)

Steps to Reproduce

I've not been able to find a specific behaviour that may cause the increase between versions, and is difficult to reproduce in small clusters due to the relatively low activity of both indexing and search

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-perf (Team:Performance)

[Evesy]

#120021 has been pointed to as a potential cause for this, however it's worth noting our clusters do not make any use of enrichment processors, so I'm not clear whether this cache would be at all used/populated in our case

[Fumlop]

Can confirm this behaviour under 8.17.1, after around 48h ours the nodes are throwing lucene errors on different shards, its mostly caused after the daily rotation of some indexes started. Stats from cat /proc//maps where always over 150k, ulimit -n etc already set as in the recommendations. Some optimisations in linux at least helped that not the whole stack went down. But errors occured always after around 48h on a node.

[Evesy]

Confirmed this is still happening even after upgrading to a release with #120021 in it, which indicates a different cause

[radanliska]

I'm experiencing the same issue with version 8.17.3. I manage over 15 Elasticsearch 8.17.3 clusters, but the problem only appears on those under heavier load. The error occurs approximately every 30–60 minutes with vm.max_map_count set to 262144. When I increase it to 1048576, the issue only happens about once per day — but it still needs to be addressed.

Before upgrading to 8.17.3, I was using versions 8.13.2 and 8.15.1, and I didn’t encounter this problem at all.

Thank you for solving this issue.

[Evesy]

@radanliska I was meaning to close this off as I had success in setting -Dorg.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits=1 JVM arg on our nodes as per the comment here #118623 (comment)

Let me know if that works for you too!

[Thijsvandepoll]

We have been experiencing the same issue with version 8.17.6. We have a two large indices (both 21TB including 1 replica) spread out over 12 nodes. Both indices have a int8_hnsw dense vector field mapping, and therefore heavily rely on page cache.

We are indexing new documents every night. Recently we have been seeing node failures with You are running out of system memory on agent node when Java calls fork() and # Native memory allocation (malloc) failed to allocate 2097152 bytes. Error detail: AllocateHeap during indexing. In an attempt to solve it, we have been moving from hnsw to int8_hnsw, which was on our TODO list anyways. This did not solve it.

Only yesterday, we stumbled upon this thread, and found indeed that increasing the max_map_count to 500k+ did solve the issue for now.

Does it still make sense to set -Dorg.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits=1 as well?

[Fumlop]

@Thijsvandepoll

I implemented -Dorg.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits=1 on our loadtest and int currently and map count decreased dramatically, to around 5.5k instead.

Would recommend after testing to set this parameter.

[Thijsvandepoll]

@Fumlop is there any adversarial effect of doing this in your benchmarks?

[Fumlop]

@Thijsvandepoll currently no observed negative impact, but maybe my usecase is a lil smaller. Just 3TB data... split on 3 indices.
Tomorrow i rollout this jvm option to production