Update on the development branch

[kaiyux]

Hi,

The TensorRT-LLM team is pleased to announce that we are pushing an update to the development branch (also includes the Triton backend) this December 8th, 2023.

This update includes:

• Model Support
  • OpenAI Whisper support
• Features
  • Enable split-k for weight-only cutlass kernels
  • FP8 KV cache support for XQA kernel
  • New Python builder API and trtllm-build command(already applied to blip2 and OPT )
  • Add a Python class ModelRunnerCpp that wraps C++ gptSession
  • Support StoppingCriteria and LogitsProcessor in Python generate API (thanks to the contribution from @zhang-ge-hao)
• Bug fixes
  • LLaMA GPTQ failure [0.6.1] llama 13b gptq the value update is not the same shape as the original. updated: (2560, 3840), original (5120, 3840) #580
  • Python binding for InferenceRequest issue Issue in python binding for InferenceRequest for GptMananger #528
  • CodeLlama SQ accuracy issue Why are the human eval scores of smoothquant and int8_weight_only very very low? #453
• Performance
  • LoRA optimization: multi-stream gemm and cutlass grouped gemm
• Documentation
  • Add documentation for new builder workflow

Thanks,
The TensorRT-LLM Engineering Team