RFC: AWQModifier user experience

[brian-dellabetta]

We want to provide users two new abilities with the AWQ algorithm:

1. Disable quantization, so the best scales are found and applied to the weights without round-to-nearest quantization. This is useful when someone wants to run AWQ and GPTQ, before ultimately quantizing weights -- [AWQ] Allow users to disable quantization during AWQ #1973
2. Run AWQ in a generalized way, without the current constraints where all quantization schemes have to match -- [WIP] Generalize AWQ quantization #1961

I believe these draft implementations are not compatible.

Currently in #1973

User explicitly sets AWQModifier(config_groups=None, targets=None, ...) (as opposed to its default value of targets=["Linear"]. This likely isn't compatible with other proposed changes in #1961 to generalize AWQ quantization, which use targets/config_groups to determine how observers and quantization config should be attached to each module, and would allow us to support more than just W4A16 group-wise.

Suggested Change

We add another field AWQModifier(disable_quantization=True, ...) that allows the resolved quantization config to be used to attach observers and find best scales, but will just not do any of the quantization. If we want to adopt #1961 , I think we need to do this instead. We would have to be careful about how we run apply_quantization_config here, there might be other side effects here.

[kylesayrs]

Can you clarify what you mean by "disable quantization"?

[brian-dellabetta]

Basically, use resolved targets and resolved config to compute best scales, as shown in #1961, but when it comes to actually running quantization here, we just don't do that if disable_quantization==True

[brian-dellabetta]

After discussing, disable_quantization field would be more suitable to allow for both PRs

[HDCharles]

Is that first use case actually effective? It seems like your almost undoing the work of the AWQ grid search by doing GPTQ after, since you've explicitly grid searches with respect to a particular quant function.

[brian-dellabetta]

Yeah, good point. This is more of a research request to experiment more than a proven path to open up to users. For that reason I think it should be clearly indicated by a boolean field like disable_quantization that 98% of the time wouldn't be modified.

[brian-dellabetta]

I can talk through this with you tomorrow. @kylesayrs said you might have suggestions to improve other aspects of our AWQ implementation.