Qualcomm AI Engine Direct - gpu support part1

[haowhsu-quic]

Summary

• rename folders in backends/qualcomm/runtime/backends
• add gpu infra

Test plan

python backends/qualcomm/tests/test_qnn_delegate.py TestQNNFloatingPointOperator.test_qnn_backend_conv2d -b build-android/ -m SM8750 -s 5f396958 --online_prepare --backend gpu

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/12165

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures, 1 Cancelled Job

As of commit f21b2b8 with merge base 929ec94 ():

NEW FAILURES - The following jobs have failed:

• pull / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh)
  /pytorch/executorch/backends/qualcomm/runtime/backends/gpu/GpuBackendCustomConfig.h:35:7: error: private field 'gpu_backend_config_' is not used [-Werror,-Wunused-private-field]
• pull / test-llama-runner-qnn-linux (fp32, qnn_8a8w, qnn) / linux-job (gh)
  /pytorch/executorch/backends/qualcomm/runtime/backends/gpu/GpuBackendCustomConfig.h:35:7: error: private field 'gpu_backend_config_' is not used [-Werror,-Wunused-private-field]
• pull / test-static-llama-qnn-linux / linux-job (gh)
  /pytorch/executorch/backends/qualcomm/runtime/backends/gpu/GpuBackendCustomConfig.h:35:7: error: private field 'gpu_backend_config_' is not used [-Werror,-Wunused-private-field]

CANCELLED JOB - The following job was cancelled. Please retry:

• pull / test-models-linux (mobilebert, portable, linux.2xlarge) / linux-job (gh)
  ##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[haowhsu-quic]

@pytorchbot label "release notes: qualcomm"

[cccclai]

I'm slightly worried about the runtime size increase, that usually is a requirement for production. Do we know how much size increase with this PR? If I have a model runs on HTP only, can the runtime include HTP only?

[haowhsu-quic]

The libqnn_executorch_backend.so grows from 630984 to 652672 bytes. We'll deprecate few files in next PR, hopefully it could further reduce the number.

[cccclai]

What files will be deprecated in next PR?

[haowhsu-quic]

I think it will be aot/ir and runtime/backend/CustomProtocol*. We now switch to QNN IR backend (DLC) for online-prepare path, the qcir and the legacy code for multi-method compilation can be fully deprecated.
But it would break backward compatibility since we used to wrap preprocess result with custom protocol. Probably will let you to decide when will be the right time to apply the change.

[haowhsu-quic]

Hi, I was thinking wrong about the impact of deprecating files. We still need to keep the custom protocol implementation to make multi-graph path work.
The change is in #12583 now and will guarantee BC.

[cccclai]

Sorry I need to spend a bit more time on this, because we don't have CI to test the pllm model and I'm worried it will cause breakage

[haowhsu-quic]

Sorry I need to spend a bit more time on this, because we don't have CI to test the pllm model and I'm worried it will cause breakage

No worries, I think GA decoder models is way more important than this. This PR is mainly a proof of concept that we can extend the capability of QNN backend.

[cccclai]

Can we prioritize the stories.pte as part of CI to prevent BC breakage? Otherwise it's hard to catch failure

[github-actions]

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.

[cccclai]

We should be good to continue this PR, what do you think?