Mistral Small 3.2 architecture

[cyril23]

Please support https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506

Would love to run a FP8 or NVFP4 quant in TensortRT-LLM.

[karljang]

@cyril23 ,
I came across PR #6714, which adds support for Mistral Small 3.1, the predecessor to Mistral Small 3.2.
However, NVFP4 quantization might be an another story~

[cyril23]

@cyril23 , I came across PR #6714, which adds support for Mistral Small 3.1, the predecessor to Mistral Small 3.2. However, NVFP4 quantization might be an another story~

@karljang Mistral Small 3.1 seems to work now. Mistral Small 3.2 does not seem recognized although https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506 says:

Mistral-Small-3.2-24B-Instruct-2506 is a minor update of Mistral-Small-3.1-24B-Instruct-2503.

Small-3.2 improves in the following categories:

• Instruction following: Small-3.2 is better at following precise instructions
• Repetition errors: Small-3.2 produces less infinite generations or repetitive answers
• Function calling: Small-3.2's function calling template is more robust (see here and examples)

In all other categories Small-3.2 should match or slightly improve compared to Mistral-Small-3.1-24B-Instruct-2503.

Key Features

• same as Mistral-Small-3.1-24B-Instruct-2503

# /workspace/mymodel contains my models in HF format
sudo docker run --rm -it --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 --gpus=all \
  -v /home/freeuser/mymodel:/workspace/mymodel \
  nvcr.io/nvidia/tensorrt-llm/release:1.2.0rc1

from tensorrt_llm import LLM, SamplingParams

def main():
  llm = LLM(model="/workspace/mymodel/mistralai_Mistral-Small-3.2-24B-Instruct-2506") # error see below
  # llm = LLM(model="/workspace/mymodel/mistralai_Mistral-Small-3.1-24B-Instruct-2503") # runs OK

  # Sample prompts
  prompts = [
          "Hello, my name is",
          "The capital of France is",
          "The future of AI is",
  ]

  # Create sampling params
  sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

  for output in llm.generate(prompts, sampling_params):
          print(
                  f"Prompt: {output.prompt!r}, Generated text: {output.outputs[0].text!r}"
          )

if __name__ == '__main__':
  main()

run above test.py:

root@e0e0fdec2f19:/app/tensorrt_llm# python3 test.py
/usr/local/lib/python3.12/dist-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
/usr/local/lib/python3.12/dist-packages/modelopt/torch/utils/import_utils.py:32: UserWarning: Failed to import huggingface plugin due to: AttributeError("module 'transformers.modeling_utils' has no attribute 'Conv1D'"). You may ignore this warning if you do not need this plugin.
  warnings.warn(
/usr/local/lib/python3.12/dist-packages/modelopt/torch/__init__.py:36: UserWarning: transformers version 4.56.0 is incompatible with nvidia-modelopt and may cause issues. Please install recommended version with `pip install nvidia-modelopt[hf]` if working with HF models.
  _warnings.warn(
[TensorRT-LLM] TensorRT LLM version: 1.2.0rc1
[11/04/2025-23:14:45] [TRT-LLM] [I] Using LLM with PyTorch backend
[11/04/2025-23:14:45] [TRT-LLM] [W] Using default gpus_per_node: 1
[11/04/2025-23:14:45] [TRT-LLM] [I] Set PluginConfig.nccl_plugin to None.
[11/04/2025-23:14:45] [TRT-LLM] [I] neither checkpoint_format nor checkpoint_loader were provided, checkpoint_format will be set to HF.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
You are using a model of type mistral3 to instantiate a model of type . This is not supported for all configurations of models and can yield errors.
The following generation flags are not valid and may be ignored: ['temperature']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
Traceback (most recent call last):
  File "/app/tensorrt_llm/test.py", line 32, in <module>
    main()
  File "/app/tensorrt_llm/test.py", line 10, in main
    llm = LLM(model="/workspace/mymodel/mistralai_Mistral-Small-3.2-24B-Instruct-2506")
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm.py", line 1098, in __init__
    super().__init__(model, tokenizer, tokenizer_mode, skip_tokenizer_init,
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm.py", line 986, in __init__
    super().__init__(model,
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm.py", line 228, in __init__
    self._build_model()
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm.py", line 1039, in _build_model
    self.input_processor = create_input_processor(self._hf_model_dir,
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/inputs/registry.py", line 475, in create_input_processor
    return input_processor_cls(model_path_or_dir,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/models/modeling_mistral.py", line 227, in __init__
    tokenizer = AutoTokenizer.from_pretrained(model_path,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/tokenization_auto.py", line 1141, in from_pretrained
    tokenizer_class_py, tokenizer_class_fast = TOKENIZER_MAPPING[type(config)]
                                               ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/auto_factory.py", line 815, in __getitem__
    raise KeyError(key)
KeyError: <class 'transformers.models.mistral3.configuration_mistral3.Mistral3Config'>
root@e0e0fdec2f19:/app/tensorrt_llm#

Or did I do anything wrong in my test.py above?

[karljang]

Hi @dbari , Could you please share some guidance on how to run Mistral Small 3.1?

[dbari]

My implementation in #6714 is only for the Triton backend and uses the older TensorRT workflow, where you need to build the engine beforehand. For the pytorch implementation, please see #5529. I've used this one, but I'm not so familiar with the internals, maybe @2ez4bz can help?

The error points to the tokenizer configuration and I see that 3.2 is missing a few files in HuggingFace compared to 3.1. Have you tried loading the model in transformers? It could be that you need a newer version of transformers for this.

[karljang]

@cyril23 ,
As @dbari pointed out, it seems that the Mistral Small 3.2 uses tokenizer in a different way and currently, TRT-LLM only supports the 3.1 model via the Triton backend and the older TensorRT workflow, which is no longer the preferred approach for using TRT-LLM.

We don't have immediate plans to support the model, but in the meantime, contributions are always welcome, and guidelines for adding new model support can be found here

Still, if you'd like to try the model, here’s what you can do:

• Start by setting up your baseline using the 3.1 models, closely following the steps outlined in feat(models): Mistral3.1 VLM pytorch backend support #5529
• Then, update your setup with the 3.2 model.
• As noted, the 3.2 model differs from 3.1 in that it doesn’t include tokenizer files, which may lead to some issues.
• To address those, new tooling might be needed specifically for handling the tokenizer.

[github-actions]

Issue has not received an update in over 14 days. Adding stale label.

[github-actions]

This issue was closed because it has been 14 days without activity since it has been marked as stale.