Skip to content

CUDA_DEVICE_MEMORY_LIMIT=4Gi is parsed as 4 bytes; -1 disables the limit #323

Description

@rapsealk

What happened

CUDA_DEVICE_MEMORY_LIMIT=4Gi is parsed as 4 bytes, not 4 GiB. So is 4GB, 4GiB, and "4G " with a trailing space. CUDA_DEVICE_MEMORY_LIMIT=-1 is parsed as SIZE_MAX, which disables the limit entirely.

get_limit_from_env (src/multiprocess/multiprocess_memory_limit.c:117-163) computes a pointer to the unit suffix:

char* digit_end = env_limit + len;
if (env_limit[len - 1] == 'G' || env_limit[len - 1] == 'g') {
    digit_end -= 1;
    scalar = 1024 * 1024 * 1024;
}
...
size_t res = strtoul(env_limit, &digit_end, 0);

and then passes that same pointer to strtoul, which overwrites it. The suffix-stripping is dead code. Nothing ever checks that the text after the number is the suffix that matched — strtoul just stops at the first non-digit and the rest is dropped silently.

I compiled the function's arithmetic verbatim and ran it:

4G      ->           4294967296   (4.000 GiB)   ok
4096M   ->           4294967296   (4.000 GiB)   ok
4Gi     ->                    4   <-- 4 bytes
4GB     ->                    4   <-- 4 bytes
4GiB    ->                    4   <-- 4 bytes
4gb     ->                    4   <-- 4 bytes
"  4G " ->                    4   <-- 4 bytes (trailing space)
-1      -> 18446744073709551615   <-- limit disabled
08G     ->                    0   <-- base 0 rejects "08" -> "no limit"
0x10G   ->          17179869184   <-- base 0 hex, 16 GiB

Why it matters

4Gi is how a quantity is spelled everywhere else in Kubernetes. Someone setting the limit the way they set every other resource in their manifest gets a 4-byte GPU memory limit, under which the first CUDA allocation fails. The only signal is a LOG_WARN saying "invalid device memory limit", which is easy to miss and does not say the value was used anyway.

-1 is worse in the other direction: a typo silently removes the isolation the library exists to enforce.

The 0 case is subtle: base 0 means strtoul reads a leading 0 as octal, so 08G and 09G parse as 0 while 07G parses as 7 GiB.

What you expected to happen

Either the familiar spellings parse correctly, or they are refused loudly. Never silently reinterpreted as a different number.

How to reproduce it

CUDA_DEVICE_MEMORY_LIMIT=4Gi LD_PRELOAD=/path/libvgpu.so ./any_cuda_program
# first cudaMalloc fails; log shows a 4 byte limit

Anything else we need to know?

Proposed fix: parse explicitly — a non-negative decimal integer, optional surrounding whitespace, optional K/M/G suffix that may also be spelled Ki/KB/KiB. Keep every suffix binary, which is what a bare G has always meant here, so no currently-working value changes meaning. Refuse anything else and log at error level instead of inventing a limit.

The function is pure and needs no GPU, so it can carry a regression test alongside the existing GPU-free ones (test_postinit_owner_death, test_fork_child_exit_cleanup, test_pid_discovery). PR follows.

Environment

  • HAMi-core version: main (f01e9f2)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions