Update on the development branch

[kaiyux]

Hi,

The TensorRT-LLM team is pleased to announce that we are pushing an update to the development branch (also includes the Triton backend) this December 15th, 2023.

This update includes:

• Model Support
  • BART and mBART support in encoder-decoder models
  • Support FairSeq Neural Machine Translation (NMT) family
  • Mixtral-8x7B model support
    • Support weight loading for HuggingFace Mixtral model
• Features
  • MPT - Int4 AWQ / SmoothQuant support
  • Support speculative decoding with prefilled KV cache
  • Support AWQ and GPTQ support for QWEN
  • Support ReduceScatter plugin
• Bug fixes
  • Fix tokenizer usage in quantize.py Llama 2 FP8 quantization OOM #288, thanks to the contribution from @0xymoro
  • Fix LLaMa with LoRA error [Llama Model] Run LLaMa with LoRA Error #637
  • Fix the first token being abnormal issue when '--gather_all_token_logits' is enabled When '--gather_all_token_logits' is enabled, the first token appears to be abnormal." #639
  • Fix Expert and Tensor Parallelism for Mixture of Expert layers
  • Fix an accuracy bug in MMHA multi_block_mode when certain conditions are met (large TP & 32K sequence length)
  • Fix BERT FMHA flag
• Performance
  • Optimize Hopper warp specialized kernels
  • Optimize AllReduce for parallel attention on Falcon and GPT-J
  • Enable split-k for weight-only cutlass kernel when SM>=75

Thanks,
The TensorRT-LLM Engineering Team

[MrD005]

how to use speculative decoding?
is there any document for understanding it better?

[kaiyux]

As commented in #671, we are working on the documentation, thanks.

[MrD005]

Thanks for the information. it would be helpful if you can provide any expected ETA