Frequently got `resource exhausted error` when start up the server based on GPT-OSS vLLM inference recipe

[lepan-google]

Hey team,

I followed the setup provided by the TPU7X GPT-OSS vLLM inference recipe but frequently (63.64% error rate) encountered a RESOURCE_EXHAUSTED error when starting the inference server:

RESOURCE_EXHAUSTED: Error loading program 'jit__multi_slice': Attempting to reserve 6.14G at the bottom of memory. That was not possible.

Run Details:

• Manifests: [Link]
• Error Log: [Link]
• Full Logs: [Link]

I found that reducing the gpu-memory-utilization value from 0.93 to 0.83 allows the server to start as expected.

I would like to better understand:

• How gpu-memory-utilization impacts specific memory allocation and why this error occurs intermittently.
• The performance impacts of lowering gpu-memory-utilization.
• Whether there are better mitigation strategies for this issue.

Thanks for your help!

[karan]

It seems like the pod spec is different from the one provided in the recipe.

In any case, I found https://github.com/openxla/xla/blob/main/docs/errors/error_0101.md, specifically, I think this is relevant:

HBM Fragmentation: While the total free HBM on the device might be sufficient in aggregate, it is not available in a single, contiguous block large enough to fit the entire program.

I'm guessing this is happening because of the way the Job's bash is running - it's forking the vllm python process. This is probably dirtying the mmap.

There's another hint at this in the error message:

Attempting to reserve 6.14G at the bottom of memory

HBM doesn't have a "bottom" or "top" addresses - it's uniform. So I'm thinking that this error really is about the mmap (that xla/libtpu is using to manage HBM from host CPU).

Lowering gpu-memory-utilization then allows for tolerating this fragmentation. This also likely explains why this is intermittent (the kernel is managing the heap/stack so different memory offsets over multiple runs are expected).

So if you must deviate from the the recipe, I think the trade-off of lowering gpu utilization seems expected.

---

How gpu-memory-utilization impacts specific memory allocation and why this error occurs intermittently.

See above.

The performance impacts of lowering gpu-memory-utilization.

It's effectively lowering the kv cache size so - lower rps, lower tps, lower context window, higher latency.

Whether there are better mitigation strategies for this issue.

Remove the bash wrapper and the python forking (python3 .... "$tmp_fifo" &) is the best mitigation I can think of. You can maybe replace the bash script with a python script that does the same thing without forking - so subprocess.Popen instead of os.fork. I think this can help but I can't say if it's 100% guaranteed to fix the issue).

[lepan-google]

Thank you, @karan! This makes a lot of sense. I will update the benchmark code to run the monitoring task as a background process instead to avoid interfering with the server mmap.

However, I am still getting the same error when directly running the initial CMCS recipe: link. You might notice the server starts up successfully eventually, but it restarts once due to the same memory allocation failure.

Could you please take a look at the manifests and logs? I’ve left the deployment uninstalled to see if that helps.

[karan]

Let me take a look this week.

[lepan-google]

Sorry, I forgot to update this ticket. Based on our experiments, setting the GPU memory utilization to 0.93 does not guarantee a stable server startup, even with HuggingFace: https://docs.google.com/document/d/1vkEhP8qhOna1dRaLpzDKpayUEJ6FkytgNL2PQmcREsI/edit?resourcekey=0-vIpAVQKhoUTV3a_5dOjJ8A&tab=t.9f4j2i68ts0k#heading=h.unjds6xkncmy.

For GCS runs, the maximum value for a stable startup is 0.86: https://docs.google.com/document/d/1vkEhP8qhOna1dRaLpzDKpayUEJ6FkytgNL2PQmcREsI/edit?resourcekey=0-vIpAVQKhoUTV3a_5dOjJ8A&tab=t.0#heading=h.vbt9rxic4rj8.

Also verified this change doesn't have significant impacts on inference performance.