Evaluate AWQ + FP8 Example

[dsikka]

• Add an example applying AWQ with the FP8_DYNAMIC and FP8_BLOCK schemes
• Generate sample checkpoints using the examples and share the checkpoints with your PR

[Etelis]

I will take it

[dsikka]

@Etelis do you mind if I assign this to another community member as you work through block quantization support?

[Etelis]

@Etelis do you mind if I assign this to another community member as you work through block quantization support?

Go ahead

[gDINESH13]

Hey @dsikka Good day, I see examples are present already, So the ask is to execute them and share the quantised model config files in PR?

[dsikka]

@gDINESH13 Yup! And if you are able to, basic gsm8k evals as with vLLM: https://github.com/vllm-project/vllm/blob/main/tests/evals/gsm8k/gsm8k_eval.py that we can add to the examples.

I can assign this to you

[gDINESH13]

I would like to work on this but I need some more information.

you want the quantised model config files after running existing AWQ examples & run this gsm8k evals by serving those quantised model saved on disk.
Will that be all? I don't understand the changes or new examples to be added. Is this issue expected to create any new examples or just document and provide execution results?

[dsikka]

I would like to work on this but I need some more information.

you want the quantised model config files after running existing AWQ examples & run this gsm8k evals by serving those quantised model saved on disk. Will that be all? I don't understand the changes or new examples to be added. Is this issue expected to create any new examples or just document and provide execution results?

Hey sorry for the confusion.
Yes, no examples to add. Running the examples, sharing the configs and adding any evals to the existing examples is all we want to do for the scope of this task.

[rtj1]

Hi @dsikka, I'd like to work on this. I've been building quantization tools and have experience with AWQ + FP8 workflows. I can run the examples, generate configs, and add gsm8k evals. Happy to coordinate with others working on this.