Qualcomm AI Engine Direct - Runtime Option #12297

winskuo-quic · 2025-07-09T04:32:43Z

Summary

Supporting following options that can be set during both AOT and runtime:

Log Level
Performance Mode
Profiling Level

Test plan

Log Level: Check debug message prefix exists.
Performance Mode: Ensure QNN SDK prints config log for performance, and ensure burst is faster than high power saver.
Profiling Level: Turn profiling off in compile spec and add profiling flag in runtime, ensure profiler gets expected number of events.

pytorch-bot · 2025-07-09T04:32:46Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/12297

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

ghstack-mergeability-check and Check labels failing with 'Resource not accessible by integration'

❌ 2 New Failures, 1 Unrelated Failure

As of commit 6e04866 with merge base 07b6059 ():

NEW FAILURES - The following jobs have failed:

Build documentation / build (buck2) / Build doc (gh)
At least one of the pre-conditions you specified did not hold
pull / test-moshi-linux / linux-job (gh)
RuntimeError: Command docker exec -t bf8037c911b960d30036d70419314f5cd2ab30eec41ac43b02bc4b470c7201d9 /exec failed with exit code 1

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / test-openvino-linux / linux-job (gh) (trunk failure)
AttributeError: '_OpNamespace' 'quantized_decomposed' object has no attribute 'convert_element_type'

This comment was automatically generated by Dr. CI and updates every 15 minutes.

github-actions · 2025-07-09T04:33:22Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

cccclai · 2025-07-09T06:21:32Z

examples/qualcomm/executor_runner/qnn_executor_runner.cpp

 * Currently we assume that the outputs are all fp32 tensors.
 */

+#include <executorch/backends/qualcomm/runtime/QnnBackendOptions.h>


Is it needed to have these headers?

#include <executorch/backends/qualcomm/runtime/QnnBackendOptions.h> #include <executorch/backends/qualcomm/runtime/QnnExecuTorchBackend.h>

And these two

#include <executorch/runtime/backend/backend_option_context.h> #include <executorch/runtime/backend/interface.h>

Thanks for reviewing the PR and catching this.
Some of these headers are not required and I have removed them.
However, QnnExecuTorchBackend.h would still be required since the backend name variable QNN_BACKEND is inside QnnExecuTorchBackend.h. If there are any concerns, I can move this variable to somewhere else.
Thanks

I see, I'm just trying to understand what headers are required to be added to the runtime. What is the alternative option for the QNN_BACKEND variable?

I have pushed a new commit that places all these macros under executorch/backends/qualcomm/runtime/QnnExecuTorch.h, so we don't need to include QnnExecuTorchBackend.h.
However, I will still need to include executorch/runtime/backend/interface.h since I need to call the set_options api.

I see, I might refactor to move the set_options API. But let's merge this PR for now

facebook-github-bot · 2025-07-17T22:51:41Z

@cccclai has imported this pull request. If you are a Meta employee, you can view this in D78522200.

cccclai · 2025-07-18T00:11:44Z

Hmm lots of internal failure, might be easier if I move the API to <options.h> now

winskuo-quic · 2025-07-18T13:57:33Z

Hmm lots of internal failure, might be easier if I move the API to <options.h> now

Sure! I will rebase to mainline and change the header once the API is moved to <options.h>

cccclai · 2025-08-04T02:01:00Z

Actually can you make this one line change

--- a/executorch/backends/qualcomm/runtime/targets.bzl
+++ b/executorch/backends/qualcomm/runtime/targets.bzl
@@ -75,11 +75,11 @@
                 "//executorch/backends/qualcomm:schema",
                 "//executorch/backends/qualcomm/aot/ir:qcir_utils",
                 "//executorch/backends/qualcomm/aot/wrappers:wrappers",
-                "//executorch/runtime/backend:interface",
                 "//executorch/runtime/core:core",
                 "//executorch/extension/tensor:tensor",
             ],
             exported_deps = [
+                "//executorch/runtime/backend:interface",
                 "//executorch/runtime/core/exec_aten/util:scalar_type_util",
                 "//executorch/runtime/core:event_tracer",
             ],

It seems fix the issue

winskuo-quic · 2025-08-05T04:01:38Z

Actually can you make this one line change

--- a/executorch/backends/qualcomm/runtime/targets.bzl
+++ b/executorch/backends/qualcomm/runtime/targets.bzl
@@ -75,11 +75,11 @@
                 "//executorch/backends/qualcomm:schema",
                 "//executorch/backends/qualcomm/aot/ir:qcir_utils",
                 "//executorch/backends/qualcomm/aot/wrappers:wrappers",
-                "//executorch/runtime/backend:interface",
                 "//executorch/runtime/core:core",
                 "//executorch/extension/tensor:tensor",
             ],
             exported_deps = [
+                "//executorch/runtime/backend:interface",
                 "//executorch/runtime/core/exec_aten/util:scalar_type_util",
                 "//executorch/runtime/core:event_tracer",
             ],

It seems fix the issue

I have the 1 line change fixed and also rebased to mainline. Thanks

facebook-github-bot · 2025-08-05T15:26:39Z

@cccclai has imported this pull request. If you are a Meta employee, you can view this in D78522200.

cccclai

Thanks for adding the feature!

### Summary Supporting following options that can be set during both AOT and runtime: - Log Level - Performance Mode - Profiling Level ### Test plan - Log Level: Check `debug` message prefix exists. - Performance Mode: Ensure QNN SDK prints config log for performance, and ensure burst is faster than high power saver. - Profiling Level: Turn profiling off in compile spec and add profiling flag in runtime, ensure profiler gets expected number of events.

winskuo-quic requested review from cccclai, kirklandsign and larryliu0820 as code owners July 9, 2025 04:32

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 9, 2025

cccclai reviewed Jul 9, 2025

View reviewed changes

winskuo-quic force-pushed the dev1/winskuo/runtime_option branch from 83b19ef to 141c7be Compare July 12, 2025 05:44

winskuo-quic added 3 commits August 5, 2025 09:37

Qualcomm AI Engine Direct - Runtime Option

f5bb567

Code Review

452d0d2

bzl file fix

6e04866

winskuo-quic force-pushed the dev1/winskuo/runtime_option branch from 141c7be to 6e04866 Compare August 5, 2025 01:51

cccclai reviewed Aug 5, 2025

View reviewed changes

cccclai approved these changes Aug 5, 2025

View reviewed changes

cccclai merged commit 047587e into pytorch:main Aug 5, 2025
101 of 104 checks passed

Qualcomm AI Engine Direct - Runtime Option #12297

Qualcomm AI Engine Direct - Runtime Option #12297

Uh oh!

Conversation

winskuo-quic commented Jul 9, 2025

Summary

Test plan

Uh oh!

pytorch-bot bot commented Jul 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/12297

❗ 1 Active SEVs

❌ 2 New Failures, 1 Unrelated Failure

Uh oh!

github-actions bot commented Jul 9, 2025

This PR needs a release notes: label

Uh oh!

cccclai Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

cccclai Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

winskuo-quic Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

cccclai Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

winskuo-quic Jul 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cccclai Jul 17, 2025

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Jul 17, 2025

Uh oh!

cccclai commented Jul 18, 2025

Uh oh!

winskuo-quic commented Jul 18, 2025

Uh oh!

cccclai commented Aug 4, 2025

Uh oh!

winskuo-quic commented Aug 5, 2025

Uh oh!

facebook-github-bot commented Aug 5, 2025

Uh oh!

cccclai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pytorch-bot bot commented Jul 9, 2025 •

edited

Loading

This PR needs a `release notes:` label

winskuo-quic Jul 12, 2025 •

edited

Loading