Qualcomm AI Engine Direct - GA Albert, Bert, Distilbert, Eurobert

[winskuo-quic]

Summary

This PR consists of 4 Encoder-Only models.
Following stats are based on SM8750.

1. Albert (16a16w)

• Accuracy: ~22% (NOTE: nn.Module accuracy is around 24%, so the similarity between QNN and nn.Module is around 92%)
• Speed: 11ms/inf
• Script: python examples/qualcomm/oss_scripts/albert.py -b build-android -s $DEVICE -m SM8750 --dataset ../wikipedia-sentences/wikisent2.txt

2. Bert (16a8w)

• Accuracy: ~60%
• Speed: 9ms/inf
• Script: python examples/qualcomm/oss_scripts/bert.py -b build-android -s $DEVICE -m SM8750 --dataset ../wikipedia-sentences/wikisent2.txt

3. Distilbert (16a8w)

• Accuracy: ~59%
• Speed: 8ms/inf
• Script: python examples/qualcomm/oss_scripts/distilbert.py -b build-android -s $DEVICE -m SM8750 --dataset ../wikipedia-sentences/wikisent2.txt

4. Eurobert (16a16w)

• Accuracy: ~54%
• Speed: 40ms/inf
• Script: python examples/qualcomm/oss_scripts/eurobert.py -b build-android -s $DEVICE -m SM8750 --dataset ../wikipedia-sentences/wikisent2.txt

Test plan

• E2E Scripts under test_qnn_delegate.py
  • Example script: python backends/qualcomm/tests/test_qnn_delegate.py -k TestExampleOssScript.test_{BERT_MODEL} --model SM8750 -s $DEVICE --build_folder build-android/ -r ./ -a ./test --sentence_dataset ../wikipedia-sentences/wikisent2.txt
• Mainline CI

Author: @haowhsu-quic, @chunit-quic, @winskuo-quic

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/11546

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 4 New Failures

As of commit 4a42e26 with merge base 6af28c9 ():

NEW FAILURES - The following jobs have failed:

• Build Linux Wheels / pytorch/executorch / build-manywheel-py3_10-cpu (gh)
  ModuleNotFoundError: No module named 'torchvision'
• Build Linux Wheels / pytorch/executorch / upload / upload-manywheel-py3_10-cpu (gh)
  Unable to download artifact(s): Artifact not found for name: pytorch_executorch__3.10_cpu_x86_64
• Build macOS Wheels / pytorch/executorch / build-wheel-py3_10-cpu (gh)
  ModuleNotFoundError: No module named 'torchvision'
• Build macOS Wheels / pytorch/executorch / upload / upload-wheel-py3_10-cpu (gh)
  Unable to download artifact(s): Artifact not found for name: pytorch_executorch__3.10_cpu_

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[haowhsu-quic]

@pytorchbot label "release notes: qualcomm"

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[cccclai]

Does test_qnn_backend_conv2d_channel_last test pass on your side? I saw error like this

AssertionError: False is not true : ref_output:
tensor([[[[-0.2671, -0.2526, -0.3288],
          [-0.4589, -0.6176, -0.1747],
          [-0.0699, -0.0861, -0.2032]],

         [[ 0.0599, -0.0134, -0.0674],
          [ 0.0606,  0.1011,  0.1408],
          [ 0.1816,  0.0160, -0.1697]]]])

model_output:
tensor([[[[-0.2668, -0.3289, -0.6177],
          [-0.0699, -0.2032, -0.0134],
          [ 0.0605,  0.1409,  0.0161]],

         [[-0.2527, -0.4590, -0.1747],
          [-0.0861,  0.0599, -0.0674],
          [ 0.1013,  0.1816, -0.1697]]]])

[winskuo-quic]

Does test_qnn_backend_conv2d_channel_last test pass on your side? I saw error like this

AssertionError: False is not true : ref_output:
tensor([[[[-0.2671, -0.2526, -0.3288],
          [-0.4589, -0.6176, -0.1747],
          [-0.0699, -0.0861, -0.2032]],

         [[ 0.0599, -0.0134, -0.0674],
          [ 0.0606,  0.1011,  0.1408],
          [ 0.1816,  0.0160, -0.1697]]]])

model_output:
tensor([[[[-0.2668, -0.3289, -0.6177],
          [-0.0699, -0.2032, -0.0134],
          [ 0.0605,  0.1409,  0.0161]],

         [[-0.2527, -0.4590, -0.1747],
          [-0.0861,  0.0599, -0.0674],
          [ 0.1013,  0.1816, -0.1697]]]])

Thanks for catching this. Think I missed to update 1 file while rebasing. Please try again and it should be passing.

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[cccclai]

Looks like there is a conflict, can you rebase?

[winskuo-quic]

Looks like there is a conflict, can you rebase?

I have rebased but seems like the lintruner cannot be properly initialized in CI.

[cccclai]

Yeah it's fixed in main. Mind rebasing again?

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.