[Question] Can we convert FP8 or must we convert from BF16

[laterbreh]

Sorry if this is not the right place to ask this question, I have been quantizing bf16's using the convert.py with great success on the models that im wanting to use -- However im curious is it advisable/or a problem to use the convert.py on a FP8 version of a model? Will it yield a workable result or will this be a problem.

Again apologies I'm not sure where I could have asked this question.

[turboderp]

FP8 weights are converted on the fly to FP16 when loading an unquantized model. The same applies when quantizing, so there shouldn't be an issue converting an FP8 model to EXL3.

This hasn't been tested too extensively, though. For instance, I just had to make some tweaks on the dev branch to fix some FP8-related complications with the new Ministral instruct model. But it's supposed to work, so please report any issues you run into with FP8 models.

Of course, if you're converting a model that was originally in BF16 format, you'll lose a little bit of fidelity by first downcasting to FP8 before converting to EXL3. So you'd want to use BF16/FP16/FP32 weights whenever possible.

[laterbreh]

Yeah, that’s the exact model I’m trying to quantize. I just really don’t want to pull down the full 1.2+ TB BF16 version, so I figured I’d try grabbing the FP8 model (the big ~600B Mistral 3 one) instead—assuming there’s no major issue quantizing from FP8. Mostly just trying to confirm whether that’s a bad idea or a waste of time.

I’m going to start the download and run a quant/test overnight. I’ll report back with results. I’ll be testing on the latest commits from the dev branch.

[turboderp]

Please be aware that Mistral Large 3 isn't going to work yet. Ministral is supported, but the new Large model is a DeepSeek variant. It requires MLA, which isn't implemented yet.