Update on the development branch

[kaiyux]

Hi,

The TensorRT-LLM team is pleased to announce that we have pushed an update to the development branch (and the Triton backend) this Sep 3rd, 2024.

This update includes:

• Features
  • Supported FP8 for MoE LoRA.
  • Supported KV cache reuse for P-Tuning and LoRA.
  • Supported in-flight batching for CogVLM models.
• API
  • [BREAKING CHANGE] Enabled strongly_typed by default in builder API.
  • [BREAKING CHANGE] Renamed maxNewTokens, randomSeed and minLength to maxTokens, seed and minTokens following OpenAI style.

Thanks,
The TensorRT-LLM Engineering Team