Skip to content

Add traceable mistral and mistral3 classes#1343

Closed
anmarques wants to merge 17 commits intomainfrom
traceable_mistral3
Closed

Add traceable mistral and mistral3 classes#1343
anmarques wants to merge 17 commits intomainfrom
traceable_mistral3

Conversation

@anmarques
Copy link
Copy Markdown
Collaborator

@anmarques anmarques commented Apr 9, 2025

SUMMARY:
This PR adds traceable versions of Mistral and Mistral3.

NOTE:
The code fails quality and style tests, but I think we should ignore them. The failures occur at the model definitions ported from transformers and keeping the changes to a minimal would help to maintain these. I added ignore commands to the headers to automatically skip linting.

TEST PLAN:

llmcompressor.trace \
    --model_id "mistralai/Mistral-Small-3.1-24B-Instruct-2503" \
    --model_class "TraceableMistral3ForConditionalGeneration" \
    --sequential_targets "MistralDecoderLayer" \
    --ignore "language_model.lm_head" "re:vision_tower.*" "re:multi_modal_projector.*" \
    --modality vision
llmcompressor.trace \
    --model_id "mistralai/Mistral-Small-3.1-24B-Instruct-2503" \
    --model_class "TraceableMistral3ForConditionalGeneration" \
    --sequential_targets "MistralDecoderLayer" \
    --ignore "language_model.lm_head" "re:vision_tower.*" "re:multi_modal_projector.*" \
    --modality text

@anmarques anmarques requested a review from kylesayrs April 9, 2025 20:36
@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 9, 2025

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed.

@anmarques anmarques requested a review from dsikka April 9, 2025 20:36
@anmarques anmarques added the ready When a PR is ready for review label Apr 9, 2025
Copy link
Copy Markdown
Collaborator

@dsikka dsikka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think if we want to ignore linting/style, we should add lines to skip linting:
e.g:

@anmarques
Copy link
Copy Markdown
Collaborator Author

I think if we want to ignore linting/style, we should add lines to skip linting: e.g:

I wasn't aware this is an option. Done. Thanks

@anmarques anmarques requested a review from dsikka April 10, 2025 21:37
@anmarques anmarques marked this pull request as draft April 11, 2025 21:27
@anmarques
Copy link
Copy Markdown
Collaborator Author

Converted to draft because things are still in flux until we upgrade transformers support

@anmarques anmarques removed the ready When a PR is ready for review label Apr 11, 2025
@kylesayrs
Copy link
Copy Markdown
Collaborator

A traceable definition for this model is no longer required as of #1411 having been implemented. The accompanying example and test for this model is implemented here: #1490

@kylesayrs kylesayrs closed this May 29, 2025
kylesayrs added a commit that referenced this pull request Jun 3, 2025
## Purpose ##
* Add support for mistral3
* Related: #1343

## Prerequisites ##
* #1479

## Changes ##
* Added mistral3 example
* This model does not automatically change the dtype of pixel_values to
match the dtype of the model, so I had to do so manually in the collator
and sample generation
* This model has a [very verbose chat template by
default](https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/blob/main/chat_template.json),
which may be less conducive to calibration, so I added a custom
shortened version

## Testing ##
* Ran example to completion:
[nm-testing/Mistral-Small-3.1-24B-Instruct-2503-W4A16-G128](https://huggingface.co/nm-testing/Mistral-Small-3.1-24B-Instruct-2503-W4A16-G128)

---------

Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
aireilly pushed a commit to aireilly/llm-compressor that referenced this pull request Jul 30, 2025
## Purpose ##
* Add support for mistral3
* Related: vllm-project#1343

## Prerequisites ##
* vllm-project#1479

## Changes ##
* Added mistral3 example
* This model does not automatically change the dtype of pixel_values to
match the dtype of the model, so I had to do so manually in the collator
and sample generation
* This model has a [very verbose chat template by
default](https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/blob/main/chat_template.json),
which may be less conducive to calibration, so I added a custom
shortened version

## Testing ##
* Ran example to completion:
[nm-testing/Mistral-Small-3.1-24B-Instruct-2503-W4A16-G128](https://huggingface.co/nm-testing/Mistral-Small-3.1-24B-Instruct-2503-W4A16-G128)

---------

Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants