Update on the development branch

[kaiyux]

Hi,

The TensorRT-LLM team is pleased to announce that we have pushed an update to the development branch (and the Triton backend) this Nov 5, 2024.

This update includes:

• Model Support
  • Added support for InternVL2, see examples/multimodal/README.md.
  • Added support for Qwen2-0.5B model. (Qwen2-1.5B-Instruct convert_checkpoint.py failed #2388)
• Features
  • Added support for per-token per-channel FP8 (namely row-wise FP8) on Ada.
  • The maximum supported beam_width is extended to 256.
  • Added FP8 and INT8 SmoothQuant quantization support for the InternVL2-4B variant (LLM model only), see examples/multimodal/README.md.
• API
  • [BREAKING CHANGE] Removed Python bindings of GptManager.
  • Exposed --trust_remote_code argument to the OpenAI API server. (openai_server error #2357)
• Bug fixes
  • Functional and performance fixes for Llama 3.2 11B and 90B.
  • Fixed an issue that appears when building BERT. (Bug in build bert #2373)
  • Fixed an issue that model is not loaded when building BERT. (2379)
  • Fixed the broken executor examples. (Succeeded in Python runtime, but failed in C++ runtime #2294)
• Infra
  • The dependent ModelOpt version is updated to 0.19.0.

Thanks,
The TensorRT-LLM Engineering Team