[Usage]: [Question] Differences between trtllm-build and trtllm-bench --build

[kurosakiharachan]

System Info

TensorRT-LLM Version: 1.2.0rc2

How would you like to use TensorRT-LLM

Hello TensorRT-LLM team,

I have a question regarding the recommended workflow for building TensorRT engines that are intended to be used with the trtllm-serve tool and its --backend tensorrt option.

I've noticed that there are (at least) two different commands that can be used to build a TensorRT engine from a model:

The dedicated build command: trtllm-build

The build phase of the benchmarking tool: trtllm-bench --build ...

My Core Questions:

What are the functional and performance differences between an engine generated by trtllm-build and one generated by trtllm-bench --build for the same base model and architecture?

For the specific use case of serving a model using trtllm-serve --backend tensorrt, which build tool is the recommended or officially supported method to generate the engine?

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

[FrankD412]

Hi @kurosakiharachan, there is one primary difference between the standard trtllm-build and trtllm-bench build. The trtllm-bench variant has some calculations for trying to tune the max batch size and max tokens in it based on the ISL/OSL of the dataset being provided. This feature only takes effect when you provide a dataset. Additionally, it may not create engines with the same default settings as TRT-LLM itself.

Please keep in mind, while trtllm-bench build does do some tuning its primary purpose is for reproduction of benchmarks so it isn't the primary way to build engines. The official tool for building engines is trtllm-build in this case.

[github-actions]

Issue has not received an update in over 14 days. Adding stale label.

[github-actions]

This issue was closed because it has been 14 days without activity since it has been marked as stale.