Update on the development branch

[kaiyux]

Hi,

The TensorRT-LLM team is pleased to announce that we are pushing an update to the development branch (and the Triton backend) this May 21, 2024.

This update includes:

• Features
  • Support INT8 quantization with embedding layer excluded
• Bug fixes
  • Fix brokens outputs for the cases when batch size is larger than 1 use_fp8_context_fmha broken outputs #1539
  • Fix top_k type in executor.py, thanks to the contribution from @vonjackustc in Fix top_k type (float => int32) executor.py #1329

Thanks,
The TensorRT-LLM Engineering Team