GA models benchmark script

[haowhsu-quic]

Test plan

# build xnnpack
cd executorch
export ANDROID_NDK_ROOT=path/to/ndk
./build_xnnpack.sh

# benchmark example for xnnpack
cd optimum
optimum-cli export executorch --model "microsoft/swin-tiny-patch4-window7-224" --task "image-classification" --recipe "xnnpack" --output_dir optimum/xnn_artifacts/swin
cd executorch
python benchmark.py -b xnn -p path/to/optimum/xnn_artifacts/swin/model.pte -s $DEVICE --iteration 150

# benchmark example for qnn
cd executorch
python benchmark.py -b qnn -p path/to/swin_qnn_q8.pte -s $DEVICE --iteration 400

Statistics will be something like:

load: 272.587000
1st: 4.648000
avg: 3.960253
peak_mem: 88766
avg_mem: 84006.000

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/11039

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• [ROCm][CI] Machines under the label linux.rocm.gpu.2 are undergoing maintenance.

❌ 9 New Failures, 2 Cancelled Jobs, 4 Unrelated Failures

As of commit 4f5a23c with merge base f32cdc3 ():

NEW FAILURES - The following jobs have failed:

• Lint / lintrunner / linux-job (gh)
  >>> Lint for examples/qualcomm/oss_scripts/t5/t5_model.py:
• Lint / lintrunner-mypy / linux-job (gh)
  >>> Lint for benchmark.py:
• pull / test-models-linux (mobilebert, portable, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t 5f4499df1f8446e015fa8470c1fb8d15a2fe4f9007bdefb0196e7f4de553930d /exec failed with exit code 134
• pull / test-models-linux (mobilebert, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t 76e34e9c782730f6764511f7e5e00da6af5c622055a6364aa0299be56550f8eb /exec failed with exit code 134
• pull / test-qnn-wheel-packages-linux (3.10) / linux-job (gh)
  RuntimeError: Command docker exec -t 16094b4f16250f45aea80df101297489f82a7ac1e3742e00ae4a1a4a8354eded /exec failed with exit code 1
• pull / test-qnn-wheel-packages-linux (3.11) / linux-job (gh)
  RuntimeError: Command docker exec -t 06c370e090451fcaedc25cc46693e2831b749302c3714a782a4b9d1cfce51c59 /exec failed with exit code 1
• pull / test-qnn-wheel-packages-linux (3.12) / linux-job (gh)
  RuntimeError: Command docker exec -t fd399e6b30085927c56ec25ea2a7f0a8905466dd06de4e79d77e372940fe301e /exec failed with exit code 1
• pull / test-samsung-models-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 7a0c35286c253dcb2d339d82af2e300cfb030ded13858b79e5ecf97b155093ce /exec failed with exit code 1
• Test CUDA Builds / export-voxtral-cuda-artifact / linux-job (gh)
  RuntimeError: Command docker exec -t 7c1d0be89d39f9de1e318f443ea238e576064765dd3b7a2113e863d2244f0932 /exec failed with exit code 2

CANCELLED JOBS - The following jobs were cancelled. Please retry:

• pull / test-models-linux (ic4, portable, linux.4xlarge.memory) / linux-job (gh)
  ##[error]The operation was canceled.
• pull / test-models-linux (phi_4_mini, portable, linux.4xlarge.memory) / linux-job (gh)
  ##[error]The operation was canceled.

FLAKY - The following job failed but was likely due to flakiness present on trunk:

• pull / unittest-editable / linux / linux-job (gh) (matched linux rule in flaky-rules.json)
  The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.

BROKEN TRUNK - The following jobs failed but was present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / unittest / macos / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• pull / unittest-buck / macos / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• pull / unittest-editable / macos / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:.

If not, please add the release notes: none label.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[cccclai]

This is helpful. Should we add this script to the devtools as a cli tool so others can also use it? And expand to other backend like vulkan as well.

[kimishpatel]

How are we generating pte file for QNN. It will be good to have that recipe and ideally if it is available on optimum-cli that would be great

[cccclai]

How are we generating pte file for QNN. It will be good to have that recipe and ideally if it is available on optimum-cli that would be great

Those are listed in https://github.com/pytorch/executorch/tree/main/examples/qualcomm/oss_scripts

[digantdesai]

This should generate extecutorch libraries, now you need to build a runner which uses this.

[github-actions]

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.