Bug: imatrix quantization failing for nvidia Nemotron 49B v1.5

[erazortt]

What happened?

Trying to quantize the bf16 gguf and also using the imatrix file from here here: https://huggingface.co/bartowski/nvidia_Llama-3_3-Nemotron-Super-49B-v1_5-GGUF/tree/main

quantizing with the following command:
./llama-quantize --imatrix ../models/nvidia_Llama-3_3-Nemotron-Super-49B-v1_5-imatrix.gguf --allow-requantize --output-tensor-type q8_0 --token-embedding-type q8_0 ../models/nvidia_Llama-3_3-Nemotron-Super-49B-v1_5-bf16-00001-of-00003.gguf model-q4_k_l.gguf q4_k_m

fails with the error message:
load_imatrix: failed reading data for entry 1

quantization works when not using the imatrix flag

Name and Version

version: 1 (bb4c917)
built with MSVC 19.44.35213.0 for

What operating system are you seeing the problem on?

Windows

Relevant log output

load_imatrix: failed reading data for entry 1

[ubergarm]

@erazortt

So given bartowski's imatrix is in the new .gguf format only recently added in mainline I don't think that will work. You might be able to use mainlines feature to convert it from .gguf format to the .dat format used here and on mainline until a week ago.

I'm curious if you do convert it to .dat if you can get it to work. You will have to find the closed PR on mainline by compilade to read up on it, dont' have the reference handy atm.

Keep us posted!

[compilade]

@erazortt You should be able to use mainline llama-imatrix to convert nvidia_Llama-3_3-Nemotron-Super-49B-v1_5-imatrix.gguf to the .dat format:

$ ./llama-imatrix --in-file nvidia_Llama-3_3-Nemotron-Super-49B-v1_5-imatrix.gguf --output-format dat -o nvidia_Llama-3_3-Nemotron-Super-49B-v1_5-imatrix.dat

Note the .dat of the output file, and --output-format dat (since ggml-org/llama.cpp#14842).

See also ggml-org/llama.cpp#9400 (comment).

Then you should be able to use the resulting .dat imatrix file with llama-quantize from ik_llama.cpp (which is what you're using here).

[erazortt]

Yes the suggested conversion of the imatrix worked.

[ikawrakow]

GGUF imatrix confusion -> closing.