From e9bcf6e9a66f3067bea657b3d652d62f5f6990b0 Mon Sep 17 00:00:00 2001 From: Jonathan Coles Date: Tue, 12 Aug 2025 12:04:16 +0200 Subject: [PATCH] Document a known GPU out-of-memory bug caused by file caches residing in GPU memory --- docs/running/known-issues.md | 12 ++++++++++++ mkdocs.yml | 1 + 2 files changed, 13 insertions(+) create mode 100644 docs/running/known-issues.md diff --git a/docs/running/known-issues.md b/docs/running/known-issues.md new file mode 100644 index 00000000..edd39b6b --- /dev/null +++ b/docs/running/known-issues.md @@ -0,0 +1,12 @@ +[](){#ref-known-issues} +# Known Issues + +## Out of Memory on GH200 nodes +There is a known issue with Nvidia GPU driver version R550.54.15, currently installed on all GH200 nodes, that can reduce the amount of available GPU memory. + +Under normal conditions, Linux in-memory file caches may migrate from CPU to GPU memory. Allocating GPU memory should trigger the eviction of these caches, freeing up GPU memory. However, due to this bug, eviction does not occur, leading to out-of-memory errors. + +Applications with heavy I/O workloads are especially affected, as increased I/O generates more cached data. While we currently ensure that at least 90% of GPU memory is available when a node is allocated to a job, available memory may still decrease during the job’s execution. + +This issue is fixed in driver version R570, which will be deployed during a future system maintenance. + diff --git a/mkdocs.yml b/mkdocs.yml index b23efa9e..22d816e9 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -112,6 +112,7 @@ nav: - running/index.md - 'Slurm': running/slurm.md - 'Job report': running/jobreport.md + - 'Known issues': running/known-issues.md - 'Data Management and Storage': - storage/index.md - 'File Systems': storage/filesystems.md