W4A16 quantization option to handle/ignore layers with columns%groupsize != 0

[HDCharles]

When you apply W4A16 quantization to a model, all layers get quantized, if you then try to save the compressed model you'll run into errors saving layers where the groupsize doesn't evenly divide the columns.

https://github.com/vllm-project/compressed-tensors/blob/main/src/compressed_tensors/quantization/lifecycle/forward.py#L278

this has tripped up users before:

e.g. vllm-project/compressed-tensors#447

It feels like there should be an option to ignore or pad layers that don't have the necessary dimension so you don't have to manually ignore all the layers with that shape.

[dsikka]

Why wouldn't you just use an ignore list?
I think silently skipping the layer might not be behaviour we'd want?
As you may end up skipping the model in some cases?

[HDCharles]

UX is as of now:

1. user applies quantization, waits 15 minutes for GPTQ to run. all layers are quantized
2. go to save model -> ERROR
3. error message: one of your layers (and I can't say which one) isn't divisible by your groupsize. figure it out on your own and come back when you fix it, then wait 15 more minutes and HOPE you didn't miss one.

at which point the user has to figure out how to traverse the model and print the shape and identify divisibility for the right dimension....etc

it seems far from ideal and doing almost anything else would probably be better. We could

A) error at minute 0 instead of minute 15
B) notice that we're applying quantization to layers that will cause errors and proactively warn with fqn of such layers so they will be able to find what to add to ignore list
C) pad layers so we can quantize them
D) silently bypass said layers
E) non-silently bypass said layers
F) do nothing by default but add an option to bypass layers that aren't divisible by groupsize and point users to it.

[dsikka]

We should consider erroring out when we set-up up the group_scales or warn very loudly there.
And we would want to do this in the case of all gorup quantization schemes, not just W4A16

My fear is that silently failing isn't ideal as in many cases, all or most of the layers would be skipped due to shape issues which would just skip the whole model as often times the shapes are shared

cc @kylesayrs

[HDCharles]

I agree silently bypassing isn't the solution, but yeah, erroring early in a helpful way that can help users identify which layers could ignore would already be a big step forward.