QLoRA + FSDP with compressed-tensors format quantized models

[kmehant]

Hi Team:

I am looking at if someone has looked at doing FSDP + QLoRA for models that quantised using llm-compressor and have compressed-tensors format. HF Transformers do support loading such models with CompressedLinear module for compressed linear layers, however, it does not seem to work out of the box for FSDP + QLoRA.

[BenjaminBossan]

Do you have a minimal reproducer available? Otherwise, we cannot possibly help to debug this. It should include:

• code that is needed to run (ideally stripped of anything that is not essential)
• versions of pertinent Python packages
• how you call the script
• if using accelerate, the accelerate config
• the full error message
• info about the hardware

[kmehant]

@BenjaminBossan thanks for the support, Its working for single gpu runs functionally, however, we have decided to try this format not at the moment but in the future so closing this down.