Distribute jobs to multiple private device pools

[huydhn]

Summary

We want to distribute jobs to multiple private device pools now that we have have more private devices on AWS mainly:

• 5 Samsung Galaxy S22 5G
• 4 Apple iPhone 15 Pro Max

A simple round-robin distribution algorithm will allocate different benchmark configs to devices of the same type from different pools. To achieve this, I refactor .ci/scripts/gather_benchmark_configs.py to introduce the concept of device variant in which the device name will be in the format DEVICE_NAME+VARIANT, for example samsung_galaxy_s22+private or apple_iphone_15+ios_18_public. Each can have more than one device pools.

I also re-enable the benchmark jobs on private iOS devices now that we have more of them to use.

Test plan

• Models are distributed round robin to different device pools:
  • samsung_galaxy_s22+private: https://github.com/pytorch/executorch/actions/runs/16509295068
    • https://github.com/pytorch/executorch/actions/runs/16509295068/job/46688761353#step:1:25
    • https://github.com/pytorch/executorch/actions/runs/16509295068/job/46688761345#step:1:25
  • apple_iphone_15+pro_private: https://github.com/pytorch/executorch/actions/runs/16513651804
• pytest .ci/script/test_generate_vllm_benchmark_matrix.py

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/12832

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 2 Unrelated Failures

As of commit 7384b78 with merge base 0479dcd ():

NEW FAILURE - The following job has failed:

• trunk / test-arm-backend (test_pytest_models_ethosu_fvp) / linux-job (gh)
  RuntimeError: Command docker exec -t 8426212ecad124caa0826d39babfd1d9d2b6b5167d4e8c5497a20f8bdea2c9ef /exec failed with exit code 1

FLAKY - The following job failed but was likely due to flakiness present on trunk:

• trunk / test-qnn-optimum-model (fp32, dit) / linux-job (gh) (matched linux rule in flaky-rules.json)
  The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.

BROKEN TRUNK - The following job failed but was present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• trunk / test-models-macos-coreml (emformer_transcribe) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[huydhn]

Let me take a look at the failures from https://github.com/pytorch/executorch/actions/runs/16513751841/job/46701693809 before landing this. I wasn't expect to see them there