Does tensorRT-LLM support serving 4bit quantised unsloth Llama model

[jayakommuru]

We want to deploy https://huggingface.co/unsloth/Llama-3.2-1B-Instruct-bnb-4bit which is 4-bit quantized version of llama-3.2-1B model. It is quantized using bitsandbytes. Can we deploy this using tensor RT-LLM backend ? If so, is there any documentation to refer?

[Tracin]

Sorry, can not support that for now.

[jayakommuru]

@Tracin is it because of the nf4 quantization of bitsandbytes used in the model ?

[karljang]

Closing this issue due to prolonged inactivity.
By the way, for quantization, the recommended tool is TensorRT Model Optimizer, available here:https://github.com/NVIDIA/TensorRT-Model-Optimizer