How to achieve 253 tok/sec with DeepSeek-R1-FP4 on 8xB200

[jeffye-dev]

I want to reproduce the DeepSeek-R1-FP4 on B200 deployment solution to align with the blog : https://developer.nvidia.com/blog/nvidia-blackwell-delivers-world-record-deepseek-r1-inference-performance

However, I just get 40 output tokens per per user, comparing with the 253 mentiond in this blog. It is a huge gap.
Here is my method to deploy on B200:

1. The latest official image (nvcr.io/nvidia/tritonserver:25.02-trtllm-python-py3) does not support DeepSeek V3 model, I have to download the newest source code of main branch from this project https://github.com/triton-inference-server/tensorrtllm_backend.git, and build it from scratch by the command:

DOCKER_BUILDKIT=1 docker build -t tritonserver_trtllm -f dockerfile/Dockerfile.triton.trt_llm_backend .

2. Setup the docker container and launch trtllm-serve inside container:

echo -e "enable_attention_dp: true\npytorch_backend_config:\n enable_overlap_scheduler: true\n print_iter_log: true\n use_cuda_graph: true\n cuda_graph_padding_enabled: true\n cuda_graph_batch_sizes: [1, 512]" > extra-llm-api-config.yml
trtllm-serve nvidia/DeepSeek-R1-FP4 --backend pytorch --max_batch_size 512 --max_num_tokens 1560 --tp_size 8 --pp_size 1 --ep_size 8 --kv_cache_free_gpu_memory_fraction 0.90 --extra_llm_api_options ./extra-llm-api-config.yml

With the engine is up, I constructed hundreds of requests with input length 1000 and output length 1000 and sent to engine in different batch size, the avg output speed is about 40 tokens per request.
3. I also run the trtllm-bench by following the official doc: https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/deepseek_v3/README.md#running-the-benchmark . Then I only got 7 output tokens per user.

Is my method wrong? And what's the correct method & configuration to run engine to serve DeepSeek-R1-FP4 on 8xB200. Appreciate for any help!

[juney-nvidia]

@jeffye-dev

Hi,

The 253 perf number is generated based on trtllm-bench command. Also, to achieve the 253 perf number, some needed MRs are being prepared to get merged to the main branch.

When it becomes ready, there will also be some concrete steps shared.

cc @Kefeng-Duan for vis on this ask from the community.

Thanks
June

[Kefeng-Duan]

Hi, @jeffye-dev
As Juney said, some MRs are still being prepared.
And besides that, our 253 TPS is for min-latency case (batch=1)

1. please set --max_batch_size=512
2. please use real data, better with ISL/OSL = 1K/2K
3. please disable enable_attention_dp
4. please don't print iteration log
5. please set MTP's nextn = 3
6. please use --ep_size 4

[jeffye-dev]

Thanks for explanations.
So I have to wait until the MRs are merged and use correct configuation?
BTW, enable_attention_dp=false might cause GPU hangs in my case.

[juney-nvidia]

So I have to wait until the MRs are merged and use correct configuation

Yes.

BTW, enable_attention_dp=false might cause GPU hangs in my case
Pls create another github issue to report this issue explicitly with providing the concrete reproducing steps and we will then follow up.

Thanks
June

[jeffye-dev]

When will these MRs be merged? I'd like to have a try in time.
It's better to have document about reproduce the performance.
@juney-nvidia @Kefeng-Duan

[Kefeng-Duan]

Hi, @jeffye-dev As Juney said, some MRs are still being prepared. And besides that, our 253 TPS is for min-latency case (batch=1)

1. please set --max_batch_size=512
2. please use real data, better with ISL/OSL = 1K/2K
3. please disable enable_attention_dp
4. please don't print iteration log
5. please set MTP's nextn = 3
6. please use --ep_size 4

@jeffye-dev 7. please do not capture cuda_graph with so large batch size 512, and for our case, 1 is OK
8. please set the env: sudo nvidia-smi -pm 0; sudo nvidia-smi -pm 1; sudo nvidia-smi boost-slider --vboost 4
9. please enable PDL by: export TRTLLM_ENABLE_PDL=1

We have tried our best to push all MRs + reproduce Doc to gitHub recently.
And I think you can start to try it now, even without the best perf (253), we will see some good perf based on the latest code.

Thanks

[jeffye-dev]

Thank @Kefeng-Duan for assistance.
I did the changes accordingly and ran some tests using trtllm-bench. Now I get very closed result: 207 tok/sec/user when setting batch=1. If set batch=10, the throughput is 127 tok/sec/user.

[juney-nvidia]

Thank @Kefeng-Duan for assistance. I did the changes accordingly and ran some tests using trtllm-bench. Now I get very closed result: 207 tok/sec/user when setting batch=1. If set batch=10, the throughput is 127 tok/sec/user.

Great to know this, @jeffye-dev . I believe you will see better perf with TensorRT-LLM in the upcoming weeks :)

Thanks
June

[juney-nvidia]

Why 253 tok/sec needs to enable MTP? If MTP is allowed, A100 can easily out-perform B200 ..

Hi @ghostplant

Do you mean that on A100, with MTP, it can also achieves up to 253 tok/sec?

If this is what you mean, can you share more details of the A100 benchmark workflow? We are happy to do the performance cross-check to learn more.

Thanks
June

[juney-nvidia]

@juney-nvidia Oh, we can just enlarge MTP and assume 100% success ratio, then it would be very close to running a large batch inference, which has been above 500 tok/sec at the moment using A100x8.

Thanks, so what is your benchmarked number without MTP on A100x8? And what is the precision?

June

[ghostplant]

The bsz=1 and MTP=0 would be far below, that's why 100% success MTP helps a lot.
But I have no idea how this question is related to this topic, we are seeking a faster solution like "253 tok/sec with DeepSeek-R1-FP4 on 8xB200" because we own solution cannot outperform it without large MTP and batch size.

[jeffye-dev]

So the point is MTP=3? It's nice to have MPT feature in production unless MTP decrease accuracy.
I cannot see the accepted rate at runtime, it's hard to judge how MTP speedup the performance. Probably we need to have A/B test.
Metrics are importance to ananlyze engine real behavior so I am also looking forward metrics related features.