RFC: Add Intel AutoRound Quantization Algorithm Support

[yiliu30]

Hi, here is the INC team from Intel. Thank you for developing this amazing project!

Our team has developed AutoRound, a novel tuning-based quantization algorithm that delivers state-of-the-art accuracy with low tuning cost and no extra inference overhead. The key features are listed below:

• Superior Accuracy: Achieves leading performance across various scenarios, including very low bits (2-3 bits).
• Multiple Data Types: Effective for a variety of data types (W4A16, MXFP8, MXFP4, FP8, NVFP4, etc.).
• Broad Model Support: Validated on both Large Language Models (LLMs) and Vision-Language Models (VLMs).
• Advanced Mixed-Precision: Supports layer-wise mixed-bit quantization and automatic bit-width search.
• Flexible Tuning: Offers a configurable tuning space to balance tuning cost and accuracy goals.

For more detailed information, please refer to our paper, and GitHub repository.

We propose integrating AutoRound into LLM Compressor to provide users with a simple, high-accuracy quantization option.

The key Idea of AutoRound

AutoRound quantizes a given tensor by introducing three trainable parameters (V, α and β) to adjust the rounding value and clipping range. For a given transformers model, AutoRound quantizes the decoder block one by one, using block-wise output reconstruction error as loss to train these parameters.

Integration Proposal

AutoRound is already integrated into popular LLM ecosystems such as Transformers, vLLM, SGLang and TorchAO. For LLM Compressor, we proposed two integration options as below:

Option 1. A New modifier for AutoRound (Recommend)

This approach integrates AutoRound as a new modifier within LLM Compressor. Specifically, we propose creating an AutoRoundModifier that wraps the AutoRound algorithm, delegating the actual tuning logic to the intel/auto-round library.
This method provides a lightweight, maintainable integration that directly leverages the AutoRound library while aligning perfectly with LLM Compressor's modifier architecture.

(Modified from deepwiki https://deepwiki.com/vllm-project/llm-compressor/)

The initial design and potential code changes are as follows:

# LLM Compressor/src/llmcompressor/modifiers/quantization/autoround/core.py

class AutoRoundModifier(Modifier, QuantizationMixin):

    def on_event(self, state: State, event: Event, **kwargs):
        if event.type_ == EventType.CALIBRATION_EPOCH_START:
            if not self.started_:
                self.on_start(state, None)

        if event.type_ == EventType.SEQUENTIAL_EPOCH_END:
            self.compress_modules()

        if event.type_ == EventType.CALIBRATION_EPOCH_END:
            self.compress_modules()

            if not self.ended_:
                self.on_end(state, None)

    def compress_modules(self):
        """
        Apply AutoRound tuning to a single decoding layer.
        This method delegates actual quantization to the AutoRound, 
        and updates the layer’s weights, scales, and zero points accordingly.
        """
        # TODO: Integrate with external tuning API.
        # Example (pseudo, TBD):
        #     tuned_layer = autoround.quant_block(decoding_layer, config=self.config)
        #     self._update_layer(decoding_layer, tuned_layer)
        pass

Option 2. Introduce a New Pipeline Wrapping Intel Neural Compressor (INC)

AutoRound is already implemented in Intel/neural-compressor. INC supports popular model compression techniques across mainstream deep learning frameworks, including PyTorch, TensorFlow, and Jax (planned for the near future).
In this option, we propose introducing a new pipeline to wrap INC. This approach would allow us to leverage INC's framework abstraction infrastructure, paving the way for supporting more frameworks in the future.

(Modified from deepwiki https://deepwiki.com/vllm-project/llm-compressor/)

# llm-compressor/src/llmcompressor/pipelines/inc/pipeline.py

@CalibrationPipeline.register("inc")
class INCPipeline(CalibrationPipeline):
    @staticmethod
    def __call__(
        model: torch.nn.Module,
        dataloader: DataLoader,
        dataset_args: Union["DatasetArguments", None],
    ):
        """
        Applies the INC calibration procedure.
        """
        dispatch_for_generation(model)  # basic dispatch is identical to generation
        model_device = get_execution_device(model)

        from neural_compressor.torch.quantization import prepare, convert
        inc_config = prepare_autoround_config(...)
        model = prepare(model, inc_config)
        # apply calibration
        for batch in tqdm.tqdm(dataloader, desc="Calibrating"):
            model(**batch)
        model = convert(model)

Comparison Between Two Options:

Item	Option 1: AutoRound Modifier	Option 2: INC Pipeline
Architectural Alignment	Native fit within LLM Compressor’s modifier paradigm.	A new pipeline
Dependencies	Only requires auto-roundlibrary.	Entire INC toolkit and its dependencies.
Implementation Effort	Utilizes existing APIs with localized modifications.	Requires designing a new pipeline and mapping configurations.
Model Scope	Focus on LLMs and VLMs	Broader, including LLMs, VLMs, and models from other domains like CV.
Framework Support	Focus PyTorch	Multi-framework: Inherits INC’s PyTorch/TF/JAX support.

Based on the comparison, we recommend going with Option 1 to add AutoRound support. Please feel free to comment on the flow mentioned above or suggest additional approaches:). Thank you in advance!

cc @hshen14 @thuang6 @wenhuach21

[robertgshaw2-redhat]

nice! very excited about this!

[dsikka]

Hi @yiliu30 thank you for your RFC!
Adding an AutoRound Modifier certainly makes sense and fits with the rest of the LLM Compressor PTQ integrations.

Just a few questions for you:

1. You mention 2-bit and 3-bit support. Do you have sample models at this precision? Are they supported in vLLM? It seems like the AutoRound integration is focused on using the GPTQ and AWQ integration: https://github.com/vllm-project/vllm/blob/e3d81866662e0ce398a2985b82e59075f327f51f/vllm/model_executor/layers/quantization/auto_round.py#L451
2. Seems like many of the results are focused on Int4 in comparison to GPTQ, RTN and AWQ. Do we have similar benchmarks for NVFp4?

[brian-dellabetta]

Thanks @yiliu30 for the nice summary! I agree the addition of an AutoRoundModifier makes the most sense, given that you can follow the template of GPTQModifier or possibly subclass it.

The INCPipeline idea would disallow use of our SequentialPipeline feature that allows for layer-wise compression without having to load the entire model up on a GPU. Has that been an issue for you at all when running it?

[hshen14]

CC @kylesayrs

[wenhuach21]

Hi @yiliu30 thank you for your RFC! Adding an AutoRound Modifier certainly makes sense and fits with the rest of the LLM Compressor PTQ integrations.

Just a few questions for you:

1. You mention 2-bit and 3-bit support. Do you have sample models at this precision? Are they supported in vLLM? It seems like the AutoRound integration is focused on using the GPTQ and AWQ integration: https://github.com/vllm-project/vllm/blob/e3d81866662e0ce398a2985b82e59075f327f51f/vllm/model_executor/layers/quantization/auto_round.py#L451
2. Seems like many of the results are focused on Int4 in comparison to GPTQ, RTN and AWQ. Do we have similar benchmarks for NVFp4?

1 https://huggingface.co/collections/OPEA/2-3-bits. And recently the accuracy has been improved, as shown in this doc. Since we currently lack efficient kernels for 2–3-bit quantization, we haven’t been generating these models recently and instead prefer using GGUF Q2_K_S.
2 This doc present some accuracy data for mxfp4/nvfp4. https://github.com/intel/auto-round/blob/main/docs/mxnv_acc.md.

[yiliu30]

Thanks @yiliu30 for the nice summary! I agree the addition of an AutoRoundModifier makes the most sense, given that you can follow the template of GPTQModifier or possibly subclass it.

The INCPipeline idea would disallow use of our SequentialPipeline feature that allows for layer-wise compression without having to load the entire model up on a GPU. Has that been an issue for you at all when running it?

Hi @brian-dellabetta, to clarify, the INCPipeline option would delegate the entire quantization process to INC. Similar to the SequentialPipeline, we also support a layer-wise compression mechanism to manage VRAM usage efficiently.

Thank you for your agreement on Option 1 — we also believe it’s the right starting point. The SequentialPipeline feature should be compatible with this approach, and we haven’t encountered any issues using it this way. And we’re aligned with your suggestion to follow the GPTQModifier implementation and will aim to share common components to keep the implementation clean.

[HDCharles]

agree, option 1 seems like a natural fit.

How is the speed of autoround with comparison to GPTQ? Are they roughly on par or is autoround slower to get better accuracy?

[wenhuach21]

agree, option 1 seems like a natural fit.

How is the speed of autoround with comparison to GPTQ? Are they roughly on par or is autoround slower to get better accuracy?

test on A100 with 80G VRAM.
We provide 3 recipes for users. The third column from the right is auto-round-light, while the second column form the right is auto-round.
please refer to this table for more infomation of AutoRound's recipes

[hshen14]

@robertgshaw2-redhat @dsikka @kylesayrs Please let us know if you get additional comments on this RFC. We'd like to start preparing the PR once the proposal is approved.

[kylesayrs]

Hi @yiliu30 @hshen14! I think this looks great, I'd be very excited to see this algorithm integrated into LLM Compressor and become more available for users interested in model compression.

You've covered most of the required integration with the outlined Modifier and Pipeline subclasses, and I deeply appreciate the time you've taken to understand the abstractions in LLM Compressor 🥳. The major decisions left are at what levels you would like to integrate with LLM Compressor.

save_pretrained Integration

Your convert function must be aware of with LC's modify_save_pretrained. This wrapped function is responsible for compressing (packing) model weights and saving the compressed tensors config in the config.json.

It seems like AutoRound already uses the same qparam names as compressed tensors (awesome!), so integration would just be a matter of attaching CT QuantizationSchemes to modules, disabling AutoRound's weight packing, and ensuring that the CT save_pretrained wrapper works with these converted modules.

If you don't want to integrate with the wrapped save_pretrained, you can unwrap the method (and assert that the only modifier in the recipe is AutoRoundModifier) and wrap with your own save_pretrained method which does your packing and config saving.

Moe Conversion Integration

Both LLM Compressor and AutoRound have logic for converting MoE models. If you choose, AutoRound can disable its module replacement code and use the MoE replacement repository provided by LLM Compressor (which supports more models atm).

You'll need to make sure that these overlapping functionalities don't conflict.

Data Pipeline Integration

LLM Compressor and AutoRound both use some form of sequential onloading in order to perform quantization. LLM Compressor does this by having modifiers hook into a CALIBRATION_EPOCH_END event, which is called by the SequentialPipeline. Choosing to integrate with LLM Compressor's SequentialPipeline would mean that AutoRound is only responsible (1) adding hooks to a module and (2) providing a function to optimize a layer using the cached data. This optimize function would be called on a CALIBRATION_EPOCH_END event. You can look at GPTQModifier for an example. Integration here comes with the benefit of gaining sequential onloading support for all the model architectures support by LLM Compressor.

Choosing not to integrate with LC's SequentialPipeline would mean creating your own pipeline (INCPipeline), and likely not calling dispatch_for_generation, instead letting autoround handle calibration and optimization within its own pipeline. When implementing AutoRoundModifier, you would not hook into SEQUENTIAL_EPOCH_END, instead implement everything within the on_initialize() method and make sure that the INCPipeline pipeline calls on_initialize()

Modifier Composing Integration

Quantization Parameter Standardization

As I called out previously, I believe that AutoRound uses the same quantization parameter names as CT (someone correct me if I'm wrong). There may be no work required here, but if AutoRoundModifier wants to be composed with different modifiers in the same recipe, it will have to be aware that qparams may change before/after the modifier is called.

Quantized Forward Integration

From LC's examples, you can see that models must be able to generate sample responses after being called with oneshot. It seems like this is already implemented by AutoRound's wrappers.

However, all quantization modifiers in LLM Compressor also wrap modules with a quantized forward method. If you want to integrate with other modifiers in the recipe, you'll need to make sure these don't conflict by converting autoround's module wrappers into CT wrappers.

If you don't want to integrate with the wrapped forward method, you can assert that the only modifier in the recipe is AutoRoundModifier and leave AutoRound's wrappers as is.

Compressed Tensors Config Integration

For the purposes of a providing easy-to-use recipes, I think the AutoRoundModifier should provide an interface for users to specify CT config/schemes, not just gguf.

Summary

I think before implementation starts, you should create a plan for deconflicting/ integrating with the following LLM Compressor features.

• Wrapped save_pretrained
• MoE model conversion
• Sequential data pipelines
• Multi-modifier recipes (standard params, quantized forward)
• CT modifier interface