Mistral-7B-Instruct-v0.3 keeps repeating itself repeating itself repeating itself is repeating itself

[cshamis]

Hello all:

I wanted to upgrade my Mistral-7B-Instruct-v0.2 to v.03. So I:

Upgraded to llama-cpp-b3263, (I'm using an A10G GPU in AWS, RHEL9, 550.54.15, CUDA 12.4)
Built with CUDA support and no errors.
Untarred the Mistral-7B-Instruct-v0.3 model
Converted it to F16 GGUF
Quantized it down to Q6_K
No errors or warnings. So far so good.

Ran it with:
% llama-server -n 2000 -ngl 33 -m Mistral-7B-Instruct-v0.3_Q6_K.gguf

And when I connect my client using the OpenAI API, I get lots of repetition. It repeats the system prompt and it's own response several times with subtle variations. Sometimes it seems to get into a loop and never breaks out. --repeat-penalty n seems to have no observable effect.

b3263 runs the older Mistral-7B-Instruct-v.02_Q6_K.gguf seemingly fine. So it appears to be something funny with the new model, but I'm at a loss to narrow it down. Maybe this is the new tokenizer. Maybe the new v0.3 Instruct doesn't like the OpenAI chat template in llama-server. Maybe the new hf-to-gguf or quantizer scrambled the model.

Suggestions, Questions, Commiseration?

[steampunque]

Could be related to #7969
Instruct v0.3 moved to a special token for [INST], just like Mixtral 8x22B uses. Many models using special tokens were wiped out by the issue #7969. I temporarily use a revert patch to run the models.