Support quantizing native FP8 models

[mratsim]

New models are coming in native FP8 form, for example Minimax-M2.1

However trying to quantize them is a whack-in-a-mole of unsupported torch features in both compressed-tensors and llm-compressor.

In llm-compressor I'm hit by silent integer promotion:

  File "[...]/.venv/lib/python3.12/site-packages/compressed_tensors/quantization/lifecycle/forward.py", line 471, in _quantize
    scaled = x / scale
             ~~^~~~~~~
RuntimeError: Promotion for Float8 Types is not supported, attempted to promote Float8_e4m3fn and Float

compressed-tensors/src/compressed_tensors/quantization/lifecycle/forward.py

Lines 455 to 471 in 797d301

	def _quantize(
	x: torch.Tensor,
	scale: torch.Tensor,
	zero_point: torch.Tensor,
	q_min: torch.Tensor,
	q_max: torch.Tensor,
	args: QuantizationArgs,
	dtype: Optional[torch.dtype] = None,
	global_scale: Optional[torch.Tensor] = None,
	) -> torch.Tensor:
	# if a global scale is optionally provided, use it
	# to further scale the local `scale` parameter
	if global_scale is not None:
	scale = scale / global_scale
	scaled = x / scale

Unimplemented min/max/abs

  File "[...]/.venv/lib/python3.12/site-packages/compressed_tensors/quantization/utils/helpers.py", line 432, in generate_gparam
    min_vals = torch.min(updated_min_val, torch.zeros_like(updated_min_val))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
NotImplementedError: "min_elementwise_cuda" not implemented for 'Float8_e4m3fn'

  File "[...]/.venv/lib/python3.12/site-packages/compressed_tensors/quantization/utils/helpers.py", line 95, in calculate_qparams
    max_val_pos = torch.max(torch.abs(min_vals), torch.abs(max_vals))
                            ^^^^^^^^^^^^^^^^^^^
NotImplementedError: "abs_cuda" not implemented for 'Float8_e4m3fn'

Downstream issue: vllm-project/llm-compressor#2172 (comment)

[kylesayrs]

Hi @mratsim The issue is that LLM Compressor (and compressed tensors) only work with full precision models. This assumption is a key requirement for popular algorithms such as GPTQ, AutoRound, and AWQ.

Please use a full precision checkpoint, or skip the FP8 layers.

[mratsim]

The model is native FP8 though, wouldn't it be possible to add on the fly dequantization of each layer? With a warning. Materializing a BF16 model from FP8 costs an extra 420GB in this case.

Also I tried dequantizing just the layers I wanted to quantize, and skip the others but that's not possible.
And the native block fp8 format is ABI compatible with LLMCompressor but we need to change the metadata from weight_scale_inv to weight_scale

[brian-dellabetta]

Just FYI -- we are working on supporting checkpoints previously quantized in a different compression format through the convert entrypoint in compressed-tensors. On this PR, I have logic to dequantize from FP8_BLOCK to bfloat16, so that deepseek 3.2 can be quantized in a subsequent step:

• [convert] Support for DeepSeek-V3.2 and Dequantizing #641

We will have to tackle these on a case-by-case basis