NotImplementedError: Operator aten.amin.default does not have a sharding strategy registered.

[liuy-2019]

Describe the bug
use qwen_2_5_vl_example.py to quant a qwen2.5 vl 32b model
accelerate launch qwen_2_5_vl_example.py

but when I use python qwen_2_5_vl_example.py, there are another problem:

Environment
vllm 0.8.3
sentence-transformers 4.1.0
transformers 4.51.3
llmcompressor 0.5.1

[brian-dellabetta]

Hi @liuy-2019 , the error you are seeing with python qwen_2_5_vl_example.py is entirely in transformers/torch, you can see there is no reference to llm-compressor in your stack trace. You will have to investigate in transformers/torch docs. It provides a suggestion the bottom on how to resolve. It looks like your torch distributed configuration is causing the error

[liuy-2019]

Hi @brian-dellabetta , But I think the error is caused by the absence of distributed operators aten.amin
It only support GPTQ and SmoothQuant? My code is on the simple fp8 w8a8~

[brian-dellabetta]

Hi @liuy-2019 , the error seems to indicate you are sharding in an unexpected way. Are you using tensor parallelism? It should work if you shard with model parallelism. In future, this will be resolved by sequentially calibrating/quantizing the model one layer at a time:

• [Performance] Sequential onloading #1263

[liuy-2019]

I just want to accelerate the quantification process, I used the calibration set and it was very slow.

To enable accelerate features with llmcompressor, simple insert device_map in from_pretrained during model load.

from transformers import AutoModelForCausalLM
MODEL_ID = "meta-llama/Meta-Llama-3-70B-Instruct"

# device_map="auto" triggers usage of accelerate
# if > 1 GPU, the model will be sharded across the GPUs
# if not enough GPU memory to fit the model, parameters are offloaded to the CPU
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID, device_map="auto", torch_dtype="auto")
llmcompressor is designed to respect the device_map, so calls to oneshot will work properly out of the box for basic quantization with QuantizationModifier, even for CPU offloaded models.

To enable CPU offloading for second-order quantization methods such as GPTQ, we need to allocate additional memory upfront when computing the device map. Not doing so risks potentially going out-of-memory.

from llmcompressor.transformers.compression.helpers import calculate_offload_device_map
from transformers import AutoModelForCausalLM
MODEL_ID = "meta-llama/Meta-Llama-3-70B-Instruct"

# Load model, reserving memory in the device map for sequential GPTQ (adjust num_gpus as needed)
device_map = calculate_offload_device_map(MODEL_ID, reserve_for_hessians=True, num_gpus=1)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    device_map=device_map,
    torch_dtype="auto",
)

https://github.com/vllm-project/llm-compressor/blob/8b2c612f8669cd61ce085a0116f67d5325c2a8cb/examples/big_models_with_accelerate/README.md

[brian-dellabetta]

Hi @liuy-2019 , we very recently deprecated examples/big_models_with_accelerate in favor of examples/big_models_with_sequential_onloading. @kylesayrs please correct me if I'm wrong, but the accelerate examples showed how to shard a model across multiple GPUs, even though most algorithms (GPTQ, AWQ) have to run sequentially through each layer of an LLM. Sequential onloading removes that need, instead using CPU RAM, and only the layer being modified is onloaded to the GPU. This greatly reduces memory, but still largely runs sequentially.

There are aspects that can be parallelized within the layer, but it's a larger consideration that would need to take into account number of calibration samples, max sequence length, and user hardware capability.

[brian-dellabetta]

Be sure to track this draft for some of our work to improve parallel compression within a given layer

• [Performance] Parallelize modifier compression #1558

[liuy-2019]

Thank you for your answer！

[kylesayrs]

Tracked here #2180