Qualcomm AI Engine Direct - Llama Inference Refactor & Support SQNR Evaluation

[winskuo-quic]

Summary

This PR supports following:

1. Unify the inference flow to allow users to add their own inference method.
   Currently supports:

   • Output Prompt, Given a prompt, return the output.
   • Tasks Evaluation: Evaluate tasks such as perplexity.
   • SQNR Evaluation: Evaluate SQNR score based on prompt's logit.

2. Allow Mainline CI to check SQNR scores. Before, we eval PPL during nightly since it took too long. With SQNR evaluation, we can perform CI test for each commit pushed since it is a lot faster (Only runs a couple of tokens). We will still keep PPL eval in nightly and introduce a new test for SQNR evaluation.

Example Script:
python examples/qualcomm/oss_scripts/llama/llama.py -b build-android -s DEVICE -m SM8750 --temperature 0 --model_mode kv --max_seq_len 1024 --decoder_model smollm2_135m --prompt "I would like to learn python, could you teach me with a simple example?" --artifact ./smollm2_135m/ --eval_methods tasks_eval sqnr_eval --tasks wikitext --limit 1

Test plan

For x86 external CI usage:
python backends/qualcomm/tests/test_qnn_delegate.py -k TestExampleLLMScript.test_static_llm_model --model_name smollm2_135m --device DEVICE --model SM8750 --build_folder build-x86/ --executorch_root . --artifact_dir . --error_only --static_llm_eval_method sqnr --enable_x86_64

For Android Internal CI usage:
python backends/qualcomm/tests/test_qnn_delegate.py -k TestExampleLLMScript.test_static_llm_model --model_name smollm2_135m --device DEVICE --model SM8750 --build_folder build-android/ --executorch_root . --artifact_dir . --error_only --static_llm_eval_method sqnr

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/16506

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit 44aaad8 with merge base 8e8d97e ():

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

• pull / android / run-emulator (gh) (#16137)
  Timeout waiting for emulator to boot.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.