temporary files created in /dev/shm by JAX

[planetprivate]

Could some one using JAX please explain me how to resolve this?

May be my parlance is wrong as I am a sysadmin (not the person running the scientific code). We have a

2x 64-core AMD EPYC 7713
Kernel: 6.8.0-60-generic x86_64 Up: 62d 7h 6m Mem: 23.18/1007.65 GiB (2.3%)
Supermicro model: H12DSi-NT6

One of the user is reporting:

jax some times creates temporary files that will not get deleted if a process dies.
The user wants to delete old temporary files in the directory /dev/shm which apparently were left over files from crashed runs. We deleted those 500Gb but still there are a lot of yellow bars in htop.

When the process crashes I see in the dmesg the following.

[Wed Jul 23 15:22:06 2025] __vm_enough_memory: pid: 1826977, comm: python, not enough memory for the allocation
[Wed Jul 23 15:22:06 2025] __vm_enough_memory: pid: 1826977, comm: python, not enough memory for the allocation
[Wed Jul 23 15:22:06 2025] __vm_enough_memory: pid: 1826977, comm: python, not enough memory for the allocation
[Wed Jul 23 15:22:06 2025] __vm_enough_memory: pid: 1826977, comm: python, not enough memory for the allocation
[Wed Jul 23 15:22:06 2025] __vm_enough_memory: pid: 1826977, comm: python, not enough memory for the allocation
[Wed Jul 23 15:22:06 2025] __vm_enough_memory: pid: 1827322, comm: python, not enough memory for the allocation
[Wed Jul 23 15:22:06 2025] __vm_enough_memory: pid: 1827322, comm: python, not enough memory for the allocation
[Wed Jul 23 15:22:06 2025] __vm_enough_memory: pid: 1827322, comm: python, not enough memory for the allocation
[Wed Jul 23 15:22:06 2025] __vm_enough_memory: pid: 1827322, comm: python, not enough memory for the allocation
[Wed Jul 23 15:22:06 2025] __vm_enough_memory: pid: 1827322, comm: python, not enough memory for the allocation
[Wed Jul 23 15:22:06 2025] python[1827322]: segfault at c ip 00007184ecf75c6b sp 00007fff70f41a20 error 6 in kdtree.cpython-312-x86_64-linux-gnu.so[7184ecf70000+1c000] likely on CPU 125 (core 61, socket 1)
[Wed Jul 23 15:22:06 2025] Code: 0f 6e c7 66 0f 6e ce 48 83 ec 18 66 0f 62 c1 66 0f d6 44 24 08 85 d2 74 1e bf 28 00 00 00 ff 15 fb e0 01 00 f3 0f 7e 44 24 08 <66> 0f d6 40 0c 48 83 c4 18 c3 0f 1f 00 bf 38 00 00 00 ff 15 dd e0
[Wed Jul 23 15:28:49 2025] workqueue: xlog_ioend_work [xfs] hogged CPU for >13333us 128 times, consider switching to WQ_UNBOUND
[Wed Jul 23 17:43:00 2025] __vm_enough_memory: pid: 1852169, comm: python, not enough memory for the allocation

As you all know /dev/shm is RAM. How does one resolve this?

• I usually assumed that when a new job is being run then the RAM should be freed automatically? Does jax somehow prevent this?
• Is it related to this https://access.redhat.com/solutions/2062273

Thank you!