feat: add importance-aware mixed-precision quantization

[wangwenmingaa]

This PR introduces and implements an importance-aware mixed-precision quantization method for large language models, with the following key features:

Importance-Guided Quantization Strategy:
We evaluate the contribution of each layer (or module) to the overall model performance and dynamically assign quantization precision accordingly. Empirical analysis shows that the first few and last few layers are typically more critical, so we keep them in full precision (e.g., FP16/FP32), while less sensitive middle layers are aggressively quantized.

Mixed-Precision Configuration:
High-importance layers: No quantization (kept in FP16/FP32)
Medium-importance layers: Quantized to FP8
Low-importance layers: Quantized to FP4
Strict Average Bitwidth Constraint (≤ 8-bit):
The overall average bitwidth across all layers is constrained to no more than 8 bits, ensuring efficient resource usage while maximizing performance.

Outperforms FP8 per-channel Baseline:
Accuracy: Achieves an average +1.04% accuracy gain over FP8 per-channel quantization on challenging benchmarks including AIME 2024–2025, LiveCodeBench, and MMLU-Pro, demonstrating superior preservation of model capabilities under aggressive quantization.
Inference Speed: Runs faster than the FP8 per-channel quantized baseline under identical hardware conditions, thanks to reduced memory bandwidth pressure from mixed-precision weight layouts.
Practical & Plug-and-Play: Requires no architectural modifications—only a principled, importance-aware assignment of quantization policies—making it immediately deployable for real-world large language model inference.

[gemini-code-assist]

Summary of Changes

Hello @wangwenmingaa, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a novel importance-aware mixed-precision quantization method designed to optimize large language models. By intelligently assigning different precision levels (FP4, FP8, FP16/FP32) to layers based on their impact on model performance, it aims to significantly reduce model size and improve inference speed while maintaining or even enhancing accuracy compared to standard FP8 quantization. The approach is presented as a practical, plug-and-play solution for LLM deployment.

Highlights

• Importance-Guided Quantization: Implements a strategy to dynamically assign quantization precision based on each layer's contribution to model performance, keeping critical layers in full precision and aggressively quantizing less sensitive ones.
• Mixed-Precision Configuration: Utilizes a tiered approach: high-importance layers remain FP16/FP32, medium-importance layers are quantized to FP8, and low-importance layers to FP4.
• Bitwidth Constraint: Ensures an overall average bitwidth across all layers does not exceed 8 bits for efficient resource usage.
• Performance Gains: Achieves an average +1.04% accuracy gain over FP8 per-channel quantization on benchmarks and runs faster due to reduced memory bandwidth pressure.
• Plug-and-Play Implementation: The method requires no architectural modifications, making it immediately deployable for real-world LLM inference.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[github-actions]

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed.

[gemini-code-assist]

Code Review

This pull request introduces a new example script for importance-aware mixed-precision quantization. The script is well-structured and serves as a good demonstration. I've provided a couple of suggestions to improve the readability and maintainability of the layer configurations within the script, mainly by clarifying comments and sorting layer indices.

[wangwenmingaa]

@dsikka @kylesayrs This PR is ready for review. Could someone please take a look? Thanks!