Qualcomm AI Engine Direct - Test QNN2.37 uplevel Fix

[winskuo-quic]

Summary

[PLEASE REMOVE] See CONTRIBUTING.md's Pull Requests for ExecuTorch PR guidelines.

[PLEASE REMOVE] If this PR closes an issue, please add a Fixes #<issue-id> line.

[PLEASE REMOVE] If this PR introduces a fix or feature that should be the upcoming release notes, please add a "Release notes: " label. For a list of available release notes labels, check out CONTRIBUTING.md's Pull Requests.

Test plan

[PLEASE REMOVE] How did you test this PR? Please write down any manual commands you used and note down tests that you have written if applicable.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/13927

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 6 New Failures, 5 Unrelated Failures

As of commit aad4cc7 with merge base bf22ab7 ():

NEW FAILURES - The following jobs have failed:

• android-perf / benchmark-on-device (Qwen/Qwen3-0.6B, et_xnnpack_custom_spda_kv_cache_8da4w, samsung_galaxy_s22, ... / mobile-job (android) (gh)
  An error occurred trying to start process '/usr/bin/bash' with working directory '/home/ec2-user/actions-runner/_work/executorch/executorch/test-infra/tools/device-farm-runner'. No such file or directory
• android-perf / benchmark-on-device (Qwen/Qwen3-0.6B, hf_xnnpack_custom_spda_kv_cache_8da4w, samsung_galaxy_s22, ... / mobile-job (android) (gh)
  An error occurred trying to start process '/usr/bin/bash' with working directory '/home/ec2-user/actions-runner/_work/executorch/executorch/test-infra/tools/device-farm-runner'. No such file or directory
• android-perf / export-models (Qwen/Qwen3-0.6B, et_xnnpack_custom_spda_kv_cache_8da4w, samsung_galaxy_s22, public... / linux-job (gh)
  RuntimeError: Command docker exec -t b0ea7b7f171a61d52b8b18883c6e0f01e81cab144c7d1cbce41a516524dedb25 /exec failed with exit code 2
• android-perf / export-models (Qwen/Qwen3-0.6B, hf_xnnpack_custom_spda_kv_cache_8da4w, samsung_galaxy_s22, public... / linux-job (gh)
  RuntimeError: Command docker exec -t 6ee6651337177eb8e1df326658978480e607069d575005b15985327b3ecc9cc8 /exec failed with exit code 2
• android-perf / upload-benchmark-results (gh)
  Could not assume role with user credentials: User: arn:aws:sts::308535385114:assumed-role/gh-ci-github-action-runners-runner-role/i-0a6215c272daef59f is not authorized to perform: sts:TagSession on resource: arn:aws:iam::308535385114:role/gha_workflow_upload-benchmark-results
• pull / test-vulkan-operators-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 09e0ff9bf99067dad15e3e52d43283a6665bcc91adb682b3e210d8e3f7b40b95 /exec failed with exit code 127

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• nightly / backend-test-linux (qnn_16a16w, operators) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)
• nightly / backend-test-linux (qnn_16a4w_block, operators) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)
• nightly / backend-test-linux (qnn_16a4w, operators) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)
• nightly / backend-test-linux (qnn_16a8w, operators) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)
• nightly / backend-test-linux (qnn_8a8w, operators) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[winskuo-quic]

Hi @cccclai,
This PR is trying to resolve the issue mentioned in: #13352
I can get the test build-benchmark-app resolved. However, there are still couple tests related to QWEN that is failing.
I also tried reverting #13352 in this PR: #13930, but these tests are still failing.

After looking at a couple of errors in CI job, it seems to be related to environment setup:

By any chance, do you know how my changes could affect these settings?

[cccclai]

I think some of the android perf jobs are failing in main branch already. For the screen shot, which CI job are you referring?

Oh nvm, find it

[cccclai]

Is the PR ready to merge? I think it should be good to fix some failure

[winskuo-quic]

Is the PR ready to merge? I think it should be good to fix some failure

Yes. This PR is ready. Thanks

[shoumikhin]

@winskuo-quic seems like nightly jobs still fail on this PR?

[winskuo-quic]

Hi @cccclai,
Could you trigger the nightly CI for me. I think it requires manual triggering.

[winskuo-quic]

Hi @cccclai,
I think the issue is now fixed.
I believe there are still a couple of errors for backend-test-linux that is operator related.
I checked the error log, and it seems like it is all using dynamic shape, which is probably not related to QNN version uplevel.
Could you take a look?
Thanks