Update on the development branch

[kaiyux]

Hi,

Happy New Year!

The TensorRT-LLM team is pleased to announce that we are pushing an update to the development branch (and the Triton backend) this January 2nd, 2024.

This update includes:

• Model Support
  • Phi-1.5/2.0
• Features
  • StreamingLLM support for LLaMA
  • Baichuan OOTB functionality support
  • InternLM OOTB functionality support
  • LLaMA
    • Support enabling INT4-AWQ along with FP8 KV Cache
    • Support BF16 for weight-only plugin
  • Decoder Iteration-level Profiling Improvements
  • Add masked_select and cumsum function for modeling
• Bug fixes
  • Fix crash in gptManagerBenchmark gptManagerBenchmark launch failed #649
  • Fix Blip2 build error blip-2 build error #695
• Documentation
  • Typo fix INT4 support on Volta? #739
  • Refine TensorRT-LLM backend readme structure Proposal: restructure the README.md triton-inference-server/tensorrtllm_backend#133 thanks for @wangkuiyi 's nice suggestion.

Thanks,
The TensorRT-LLM Engineering Team