Scaling AI Inference Performance with vLLM on AMD Instinct MI355X GPUs — ROCm Blogs

[giscus[bot]]

Scaling AI Inference Performance with vLLM on AMD Instinct MI355X GPUs — ROCm Blogs

Explore how MI355X performs against B200 in vLLM benchmarks across DeepSeek-R1, GPT-OSS-120B, Qwen3-235B and Llama-3.3-70B.

https://rocm.blogs.amd.com/artificial-intelligence/scaling-ai-inference/README.html

[verhistoriasig]

Insights on scaling LLM inference with vLLM on AMD Instinct MI355X GPUs are impressive and highly practical for AI deployments. The detailed benchmarks comparing MI355X with NVIDIA B200 across models like DeepSeek-R1, GPT-OSS-120B, Qwen3-235B, and Llama-3.3-70B clearly highlight AMD’s advantage at mid-to-high concurrency levels. The explanation of optimizations such as AITER kernels, MoE/MLA fusion, and QuickReduce effectively demonstrates how hardware and software synergy boosts throughput. For detailed technical guides and performance tips, https://verhistoriasig.com.mx/ This post is a valuable resource for teams seeking scalable, high-performance, and cost-efficient LLM inference in production environments.

[itikhono]

hi!

Could you clarify which model is actually used for testing
Llama-3.3-70B or Llama-3.1-70B?

Everywhere in the article Llama-3.3-70B is mentioned.
But the command line contains
MODEL=amd/Llama-3.1-70B-Instruct-FP8-KV

[jhartika-amd]

Thanks for pointing this out! Indeed it's Llama-3.3-70B, and there was a typo in the command line example. Submitted a fix for it.

[Kavyaar3]

Hi Team,
Noticed a small typo under the GPT-OSS-120B (MI355X) section.
The documentation mentions: --disable-LOG-requests
However, the correct vLLM flag is: --disable-log-requests
(LOG should be lowercase)
Just flagging this to avoid confusion for users running the command.
Thanks