What happened
CUDA_DEVICE_MEMORY_LIMIT=4Gi is parsed as 4 bytes, not 4 GiB. So is 4GB, 4GiB, and "4G " with a trailing space. CUDA_DEVICE_MEMORY_LIMIT=-1 is parsed as SIZE_MAX, which disables the limit entirely.
get_limit_from_env (src/multiprocess/multiprocess_memory_limit.c:117-163) computes a pointer to the unit suffix:
char* digit_end = env_limit + len;
if (env_limit[len - 1] == 'G' || env_limit[len - 1] == 'g') {
digit_end -= 1;
scalar = 1024 * 1024 * 1024;
}
...
size_t res = strtoul(env_limit, &digit_end, 0);
and then passes that same pointer to strtoul, which overwrites it. The suffix-stripping is dead code. Nothing ever checks that the text after the number is the suffix that matched — strtoul just stops at the first non-digit and the rest is dropped silently.
I compiled the function's arithmetic verbatim and ran it:
4G -> 4294967296 (4.000 GiB) ok
4096M -> 4294967296 (4.000 GiB) ok
4Gi -> 4 <-- 4 bytes
4GB -> 4 <-- 4 bytes
4GiB -> 4 <-- 4 bytes
4gb -> 4 <-- 4 bytes
" 4G " -> 4 <-- 4 bytes (trailing space)
-1 -> 18446744073709551615 <-- limit disabled
08G -> 0 <-- base 0 rejects "08" -> "no limit"
0x10G -> 17179869184 <-- base 0 hex, 16 GiB
Why it matters
4Gi is how a quantity is spelled everywhere else in Kubernetes. Someone setting the limit the way they set every other resource in their manifest gets a 4-byte GPU memory limit, under which the first CUDA allocation fails. The only signal is a LOG_WARN saying "invalid device memory limit", which is easy to miss and does not say the value was used anyway.
-1 is worse in the other direction: a typo silently removes the isolation the library exists to enforce.
The 0 case is subtle: base 0 means strtoul reads a leading 0 as octal, so 08G and 09G parse as 0 while 07G parses as 7 GiB.
What you expected to happen
Either the familiar spellings parse correctly, or they are refused loudly. Never silently reinterpreted as a different number.
How to reproduce it
CUDA_DEVICE_MEMORY_LIMIT=4Gi LD_PRELOAD=/path/libvgpu.so ./any_cuda_program
# first cudaMalloc fails; log shows a 4 byte limit
Anything else we need to know?
Proposed fix: parse explicitly — a non-negative decimal integer, optional surrounding whitespace, optional K/M/G suffix that may also be spelled Ki/KB/KiB. Keep every suffix binary, which is what a bare G has always meant here, so no currently-working value changes meaning. Refuse anything else and log at error level instead of inventing a limit.
The function is pure and needs no GPU, so it can carry a regression test alongside the existing GPU-free ones (test_postinit_owner_death, test_fork_child_exit_cleanup, test_pid_discovery). PR follows.
Environment
- HAMi-core version:
main (f01e9f2)
What happened
CUDA_DEVICE_MEMORY_LIMIT=4Giis parsed as 4 bytes, not 4 GiB. So is4GB,4GiB, and"4G "with a trailing space.CUDA_DEVICE_MEMORY_LIMIT=-1is parsed asSIZE_MAX, which disables the limit entirely.get_limit_from_env(src/multiprocess/multiprocess_memory_limit.c:117-163) computes a pointer to the unit suffix:and then passes that same pointer to
strtoul, which overwrites it. The suffix-stripping is dead code. Nothing ever checks that the text after the number is the suffix that matched —strtouljust stops at the first non-digit and the rest is dropped silently.I compiled the function's arithmetic verbatim and ran it:
Why it matters
4Giis how a quantity is spelled everywhere else in Kubernetes. Someone setting the limit the way they set every other resource in their manifest gets a 4-byte GPU memory limit, under which the first CUDA allocation fails. The only signal is aLOG_WARNsaying "invalid device memory limit", which is easy to miss and does not say the value was used anyway.-1is worse in the other direction: a typo silently removes the isolation the library exists to enforce.The
0case is subtle: base0meansstrtoulreads a leading0as octal, so08Gand09Gparse as 0 while07Gparses as 7 GiB.What you expected to happen
Either the familiar spellings parse correctly, or they are refused loudly. Never silently reinterpreted as a different number.
How to reproduce it
Anything else we need to know?
Proposed fix: parse explicitly — a non-negative decimal integer, optional surrounding whitespace, optional
K/M/Gsuffix that may also be spelledKi/KB/KiB. Keep every suffix binary, which is what a bareGhas always meant here, so no currently-working value changes meaning. Refuse anything else and log at error level instead of inventing a limit.The function is pure and needs no GPU, so it can carry a regression test alongside the existing GPU-free ones (
test_postinit_owner_death,test_fork_child_exit_cleanup,test_pid_discovery). PR follows.Environment
main(f01e9f2)