[Self-Review] Enhance Autoround to support multiple cards tuning

[yiliu30]

Given AutoRound uses block‑level reconstruction loss to fine‑tune quantization parameters, which requires running backward passes on each block. For large model, like Qwen35B, a single GPU often doesn’t have enough memory to hold an entire block during backward computation. To address this, we use the accelerator to dispatch the module across multiple devices.
In this PR, we enable this feature on LLMC side:

• Add device_map for tuning with multiple cards
• Map ignore to Autoround skipping layers
• Add Qwen235B as example for multiple cards

Test plan

pytest -svv ./llmcompressor/transformers/autoround/test_autoround_oneshot.py -k test_oneshot_with_device_map

Example results

# vllm (pretrained=INC4AI/Qwen3-235B-A22B-W4A16-G128-AutoRound-ITERS1-LLMC-TEST-ONLY,tensor_parallel_size=2,max_model_len=8192,max_num_batched_tokens=32768,max_num_seqs=128,add_bos_token=True,gpu_memory_utilization=0.8,dtype=bfloat16,max_gen_toks=2048,enable_prefix_caching=False), gen_kwargs: (None), limit: 1000.0, num_fewshot: None, batch_size: 128
# |Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
# |-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
# |gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.928|±  |0.0082|
# |     |       |strict-match    |     5|exact_match|↑  |0.930|±  |0.0081|
  

# vllm (pretrained=INC4AI/Qwen3-235B-A22B-W4A16-G128-AutoRound-ITERS200-LLMC-TEST-ONLY,tensor_parallel_size=2,max_model_len=8192,max_num_batched_tokens=32768,max_num_seqs=128,add_bos_token=True,gpu_memory_utilization=0.8,dtype=bfloat16,max_gen_toks=2048,enable_prefix_caching=False), gen_kwargs: (None), limit: 1000.0, num_fewshot: None, batch_size: 128
# |Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
# |-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
# |gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.934|±  |0.0079|
# |     |       |strict-match    |     5|exact_match|↑  |0.915|±  |0.0088|

[github-actions]

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed.

[gemini-code-assist]

Summary of Changes

Hello @yiliu30, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the AutoRoundModifier by improving its integration with models that leverage Hugging Face Accelerate for device management and offloading. The changes introduce robust handling of Accelerate hooks, provide explicit control over device mapping during quantization, and allow for the designation of layers to be kept in full precision. These updates aim to make the AutoRound quantization process more flexible and compatible with a wider range of large language models, as demonstrated by the new DeepSeek and Qwen examples.

Highlights

• New AutoRound Examples: Added two new example scripts, deepseek_example.py and qwen3_example.py, demonstrating how to apply AutoRound quantization to DeepSeek and Qwen models, respectively. These examples showcase model loading, dataset preparation, and the application of the AutoRoundModifier.
• Accelerate Hook Management: Introduced a suspend_accelerate_hooks context manager to temporarily detach and re-attach Hugging Face Accelerate hooks during the AutoRound quantization process. This ensures proper handling of models that use Accelerate for device management or offloading, preventing conflicts during quantization.
• Device Mapping for AutoRound: The AutoRoundModifier now includes a device_map parameter, allowing users to explicitly specify the devices (e.g., '0,1') on which the quantization should be performed. This parameter is passed directly to the underlying AutoRound library.
• Full Precision Layer Specification: Added a new method get_unquantized_layer_names to identify layers that should remain in full precision. These layers are then passed to the AutoRound algorithm via the fp_layers parameter, ensuring that specified modules are not quantized.
• Improved Auto-Offloading: Updated the auto_offload parameter in the AutoRound initialization from False to True, enabling AutoRound to manage model offloading more effectively during the quantization process.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[yiliu30]

/gemini review

[gemini-code-assist]

Code Review

This pull request adds support for multi-device execution in the AutoRound modifier and includes new example scripts. The core changes in the modifier logic seem solid, particularly the introduction of suspend_accelerate_hooks to manage device offloading. My feedback primarily focuses on improving the portability and clarity of the new example scripts by removing hardcoded local file paths. I've also included a couple of minor suggestions to improve code style and comment clarity in the modifier itself.

[gemini-code-assist]

The model_id is set to a hardcoded local path, which makes this example not portable. Please use a single, public model identifier from the Hugging Face Hub and remove the other commented or overwritten assignments for clarity.

Suggested change

	model_id = "meta-llama/Meta-Llama-3-8B-Instruct"
	model_id = "/storage/yiliu7/unsloth/DeepSeek-R1-BF16"
	# model_id = "/storage/yiliu7/deepseek-ai/DeepSeek-V2-Lite-Chat/"
	model_id = "deepseek-ai/DeepSeek-V2-Lite-Chat"

[gemini-code-assist]

The model_id is set to a hardcoded local path, which makes this example not portable. Please use a single, public model identifier from the Hugging Face Hub and remove the other commented or overwritten assignments for clarity. Using a smaller model might also be more suitable for an example script.

Suggested change

	model_id = "Qwen/Qwen3-30B-A3B"
	# model_id = "/storage/yiliu7/Qwen/Qwen3-30B-A3B"
	# model_id = "/storage/yiliu7/Qwen/Qwen2.5-0.5B/"
	model_id = "/storage/yiliu7/Qwen/Qwen3-235B-A22B/"
	model_id = "Qwen/Qwen1.5-0.5B-Chat"

[gemini-code-assist]

The SAVE_DIR is constructed with a hardcoded absolute path prefix, which will cause the script to fail on other machines. It should be a relative path so the model is saved in the current working directory.

SAVE_DIR = model_id.rstrip("/").split("/")[-1] + "-W4A16-G128-AutoRound"

[gemini-code-assist]

The comment # Leave offload for LLMC is now misleading since auto_offload is set to True. With the addition of suspend_accelerate_hooks, it seems the intention is now to use auto_round's internal offloading. The comment should be updated to reflect this change in behavior.

Suggested change

	# Leave offload for LLMC
	# Use auto_round's internal offloading

[gemini-code-assist]

Code Review

This pull request adds support for device mapping in the AutoRound modifier, including two new example scripts. My review focuses on improving the portability and security of the new examples and enhancing the robustness of the AutoRoundModifier. The main suggestions are to remove hardcoded local paths from the example scripts in favor of public models from the Hugging Face Hub, warn users about the security implications of trust_remote_code=True, and use a more flexible default for device_map.

[gemini-code-assist]

Using trust_remote_code=True can introduce a security vulnerability if the model repository contains malicious code. It is crucial to warn users about this risk, especially in an example script that they might copy and run. Please add a comment explaining that users should only enable this if they trust the source of the model.

[gemini-code-assist]

The model_id is hardcoded to a local path, which makes this example not portable or runnable for other users. It's better to default to a model identifier from the Hugging Face Hub and provide the local path as a commented-out alternative for users who wish to use a local model.

Suggested change

	model_id = "meta-llama/Meta-Llama-3-8B-Instruct"
	model_id = "/storage/yiliu7/unsloth/DeepSeek-R1-BF16"
	model_id = "meta-llama/Meta-Llama-3-8B-Instruct"
	# model_id = "/storage/yiliu7/unsloth/DeepSeek-R1-BF16"

[gemini-code-assist]

The model_id is hardcoded to a local path, which makes this example not runnable for other users. It's better to default to a model from the Hugging Face Hub and provide the local path as a commented-out alternative.

Suggested change

	model_id = "Qwen/Qwen3-30B-A3B"
	# model_id = "/storage/yiliu7/Qwen/Qwen3-30B-A3B"
	# model_id = "/storage/yiliu7/Qwen/Qwen2.5-0.5B/"
	model_id = "/storage/yiliu7/Qwen/Qwen3-235B-A22B/"
	model_id = "Qwen/Qwen3-30B-A3B"
	# model_id = "/storage/yiliu7/Qwen/Qwen3-30B-A3B"
	# model_id = "/storage/yiliu7/Qwen/Qwen2.5-0.5B/"
	# model_id = "/storage/yiliu7/Qwen/Qwen3-235B-A22B/"

[gemini-code-assist]

The SAVE_DIR is constructed using a hardcoded absolute path. This will cause the script to fail for any user who does not have the /storage/yiliu7/ directory. The output directory should be a relative path to make the example portable.

SAVE_DIR = model_id.rstrip("/").split("/")[-1] + "-W4A16-G128-AutoRound"

[gemini-code-assist]

The default value for device_map is hardcoded to '0'. This assumes a GPU is available at index 0 and will cause a crash on CPU-only systems or systems with a different GPU configuration. Consider changing the default to 'auto' to let accelerate handle device placement automatically, which is more robust and user-friendly.

Suggested change

	device_map: str = "0"
	device_map: str = "auto"

[yiliu30]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces support for multi-GPU tuning with device_map in the AutoRoundModifier. It also adds a new example for quantizing a large model (Qwen3-235B) using multiple GPUs. Additionally, it maps the ignore parameter to AutoRound's layer skipping mechanism and includes a bug fix related to accessing quantization scale attributes. The changes are well-implemented and include a new test case for the multi-GPU functionality. My main feedback is to clean up the new example file by removing development artifacts for better clarity.

[gemini-code-assist]

This section for selecting the model ID contains multiple overwritten assignments and a FIXME comment. This looks like leftover development code and can be confusing for users of this example. Please clean this up to have a single, clear model_id assignment. If you want to show other possible models, it's better to list them in comments.

Suggested change

	# FIXME: clean up model_id settings
	model_id = "Qwen/Qwen3-30B-A3B"
	model_id = "/storage/yiliu7/Qwen/Qwen3-30B-A3B"
	# model_id = "/storage/yiliu7/Qwen/Qwen2.5-0.5B/"
	model_id = "/storage/yiliu7/Qwen/Qwen3-235B-A22B/"
	model_id = "/models/Qwen3-30B-A3B"
	model_id = "Qwen/Qwen3-235B-A22B/"
	# You can try other models here, for example:
	# model_id = "Qwen/Qwen3-30B-A3B"
	model_id = "Qwen/Qwen3-235B-A22B/"

[yiliu30]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces support for multi-GPU tuning with device_map in the AutoRound modifier. It also adds a new example script for a large model (Qwen3-235B) to demonstrate this feature, along with a corresponding test case. The changes to the AutoRoundModifier correctly handle device mapping and suspend accelerate hooks during the process. My feedback focuses on improving the new example script for better clarity and robustness.

[gemini-code-assist]

The model_id is reassigned multiple times, with some lines commented out, and there's a FIXME comment. This seems like leftover code from development. It would be cleaner to remove the unused assignments and the FIXME comment, leaving only the desired model ID. For an example script, it's best to have a single, clear configuration.

Suggested change

	# FIXME: clean up model_id settings
	model_id = "Qwen/Qwen3-30B-A3B"
	model_id = "/storage/yiliu7/Qwen/Qwen3-30B-A3B"
	# model_id = "/storage/yiliu7/Qwen/Qwen2.5-0.5B/"
	model_id = "/storage/yiliu7/Qwen/Qwen3-235B-A22B/"
	model_id = "/models/Qwen3-30B-A3B"
	model_id = "Qwen/Qwen3-235B-A22B/"
	model_id = "Qwen/Qwen3-235B-A22B/"

[gemini-code-assist]

Using split('/')[-1] to get the model name from the path is not very robust as it assumes a Unix-like path separator. A more cross-platform and readable approach is to use os.path.basename. You will need to add import os at the top of the file.

Suggested change

	SAVE_DIR = model_id.rstrip("/").split("/")[-1] + "-W4A16-G128-AutoRound"
	SAVE_DIR = os.path.basename(model_id.rstrip("/")) + "-W4A16-G128-AutoRound"

[hshen14]

LGTM
you may prepare a readme to show the instruction and results for the example