Qualcomm AI Engine Direct - qat proto

[chunit-quic]

• Add qat proto
• Add Unit test test_qnn_backend_linear_qat
• Test command

python backends/qualcomm/tests/test_qnn_delegate.py -H $HOST -s $DEVICE -b $build-android/ -m "SM8650" -r $EXECUTORCH_ROOT -k TestQNNQuantizedOperator.test_qnn_backend_linear_qat

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/6222

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures

As of commit dd8ace6 with merge base ad0e5e8 ():

NEW FAILURES - The following jobs have failed:

• pull / test-eval_llama-mmlu-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 79b43d0cb5043918c582b7e464ad58c6ddae5c9748fe89d20875ea9f5b969ffe /exec failed with exit code 127
• pull / test-eval_llama-wikitext-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 17acd430af738d9587439a5a38c3d482a13a44b9e2825866660778b2cfd59d4a /exec failed with exit code 127
• pull / test-llama_runner_eager-linux / linux-job (gh)
  RuntimeError: Command docker exec -t c288be6a4c728e12946161713e45f485b7c1def8a43f1fba37b6feecb56b99d3 /exec failed with exit code 127

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[navsud]

Would it be possible to also add 16a4w qat config as well, in the same PR or as a follow-up?

[navsud]

How about using MovingAveragePerChannelMinMaxObserver instead of MovingAverageMinMaxObserver, since we are going to do per-channel quantization for the weights?

FusedMovingAvgObsFakeQuantize.with_args(observer=MovingAveragePerChannelMinMaxObserver)

[chunit-quic]

Thanks for the advice! Sorry for late reply because I was OoO.
If it's possible may we keep this for simplicity now? Currently we are working on different kind of quant confings for existing test cases. Once we finish we will update the quanziter with QAT.

[chunit-quic]

Would it be possible to also add 16a4w qat config as well, in the same PR or as a follow-up?
Hi @navsud,

Thank you very much for prompt reply!
Pardon that this PR is a draft to ensure that the QAT flow, like whether the unit case matches your expection. Because we have barely expreience about QAT before.

If it seems to be correct to you that will be really nice. Then we will move on to the coverage of qaunt configs.
One more question may you kindly give us some hints. Like is there any model sutible for testing the effect of QAT? Perpas one can be verify quickly?

Thank you!

Joey

[navsud]

The QAT unit test matches the intent.
Please update the observer to use per-channel, and after that we can go ahead with this PR. Overall, looks good to me.

For testing QAT on a real model, the easiest test is to apply QAT using the same dataset you use for PTQ calibration (e.g. Wiki) and the expectation is that loss should go down and the model metrics should be better than PTQ.

Update: You can also test the QAT flow on any open-source imagenet models (e.g. mobilenet-v2).

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[cccclai]

Hi @chunit-quic, should we merge this into main and iterate on this with real model? It will help internal running QAT as well.

[cccclai]

We just need to fix the lint error and it should be good.

[chunit-quic]

Hi @chunit-quic, should we merge this into main and iterate on this with real model? It will help internal running QAT as well.

Sorry for late reply. I was OoO previously .
Yes we can merge the PR, if it can help the internal iteration. I have fixed the lint errors.
We will provide a formal one recently, which includes more quantization settings

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.