Keep quantization enabled during calibration

[kylesayrs]

Purpose

• Revert the behavior regression introduced as a result of SQ and QM: Remove torch.cuda.empty_cache, use calibration_forward_context #1114
• When calibrating a model using the QuantizationModifier, quantization should be enabled when calibrating

Changes

• Remove "disabling quantization" from the calibration forward pass
• Add "disabling quantization" to the sequential pipelines in order to continue to disable quantization during calibration for GPTQ and SGPT
  • When calibration pipelines become shared between modifiers, the decision of whether to disabling quantization during calibration will have to be moved to the calibration pipelines themselves. Some work needs to be done to demonstrate that GPTQ and SGPT do not suffer accuracy regression from enabling activation quantization during calibration (in theory, the change should increase accuracy)

[github-actions]

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed.

[dsikka]

Thanks, we should get this in for the next release.

[markurtz]

If this fixes for release, that's great, but for the future, I think I'm missing some underlying assumptions here on why this is needed. I'm worried about general fragility for these and clarity of the code overall. Why do we need to disable completely and the modifiers aren't handling that logic properly?

[kylesayrs]

@markurtz Please see my comment in the description

the decision of whether to disabling quantization during calibration will have to be moved to the calibration pipelines themselves. Some work needs to be done to demonstrate that GPTQ and SGPT do not suffer accuracy regression from enabling activation quantization during calibration (in theory, the change should increase accuracy)

As currently implemented, activation quantization is enabled while calibrating with QuantizationModifier, but is not enabled when calibrating with GPTQModifier, and SGPT modifiers. In the future, we hope to demonstrate that enabling activation quantization while calibrating GPTQ and SGPT leads to accuracy improvements, not regressions

[markurtz]

As currently implemented, activation quantization is enabled while calibrating with QuantizationModifier, but is not enabled when calibrating with GPTQModifier, and SGPT modifiers. In the future, we hope to demonstrate that enabling activation quantization while calibrating GPTQ and SGPT leads to accuracy improvements, not regressions

Yep, that's all fine. My main concern is around the pipelines needing to know about disabling quantization within a context and that not being handled at the modifier level. We generally should never be setting up code where the pipeline is either specific to a compression pathway or needs to know about the contents of the recipes / modifiers. Leads to a ton of issues especially towards generality of recipes and implementations along with future support.