Benchmark: Fix metrics high variability from run-to-run

[guangy10]

🐛 Describe the bug

Issue

The metrics are frequently exceeding the 10% threshold from run-to-run, making it hard to interpret the metrics and decided actions.

It's happening to pretty much ALL models and ALL configs:

link to the dashboard

What areas should we look into?

• iOS benchmark app
• Android benchmark app

Solution Space

I think we can start with parameterizing the number of iterations for each model and its benchmark config, find the "right" value so that the metrics (load, inference, tps) fluctuation from run-to-run is within a reasonable range (<10%).

Versions

trunk

cc @mergennachin @kimishpatel @iseeyuan @kirklandsign @cbilgin @huydhn @shoumikhin

[guangy10]

cc: @cbilgin @digantdesai @kimishpatel @GregoryComer

[digantdesai]

I would say, this can be "fine tuned" by the particular backend/benchmark expert, starting with high priority use cases we want to track, and improving stability for those should benefit the rest on the same platform/setup.

I.e. I/Max/Hangson can take a look at XNNPACK + Android + LLM for example, and dial that in, and so on.

[kirklandsign]

For Android, we have

executorch/extension/benchmark/android/benchmark/app/src/main/java/org/pytorch/minibench/BenchmarkActivity.java

Line 46 in 19cbafe

int numIter = intent.getIntExtra("num_iter", 50);

So probably add --ei <num_iterations>

executorch/extension/benchmark/android/benchmark/android-llm-device-farm-test-spec.yml.j2

Line 118 in 254eeca

--es "model_dir" "/data/local/tmp/minibench"

but how do we do that in GH setup? cc @huydhn

[guangy10]

For Android, we have

executorch/extension/benchmark/android/benchmark/app/src/main/java/org/pytorch/minibench/BenchmarkActivity.java

Line 46 in 19cbafe

int numIter = intent.getIntExtra("num_iter", 50);
So probably add --ei <num_iterations>

executorch/extension/benchmark/android/benchmark/android-llm-device-farm-test-spec.yml.j2

Line 118 in 254eeca

--es "model_dir" "/data/local/tmp/minibench"
but how do we do that in GH setup? cc @huydhn

I think you can extend the script to include iterations you want to run for each model+config. The script is here: https://github.com/pytorch/executorch/blob/main/.ci/scripts/gather_benchmark_configs.py#L327-L333.
Then read it in the workflow, i.e. android-perf.yml, either pass it down to test_spec during specialization, or pass it to the benchmark-on-device step which will instruct the app to run via ASW. About this step, @huydhn can probably suggest best option.

[guangy10]

@kirklandsign @shoumikhin I'm suggesting something we can try but don't want to be misleading. My suggestion assumes that the issue can be resolved by simply increasing the iterations, but this is not validated yet. I would suggest you should first try run the app locally, taking the llama1b w/ xnnpack for example, run as many iterations as you want, compare the run by switching to using binary maybe, check the mean of latency and RDS. We should go this solution only if we have verified the assumption.

[huydhn]

How's about increasing the value in

executorch/extension/benchmark/android/benchmark/app/src/main/java/org/pytorch/minibench/BenchmarkActivity.java

Line 46 in 19cbafe

int numIter = intent.getIntExtra("num_iter", 50);

to see if it helps? I guess we need to have it running for a few days to confirm anyway. IMO, this number like stays fixed, so either hard-coding it or having a fix configuration file are good. Making it configurable is doable via the test spec, but it seems overkill

[guangy10]

How's about increasing the value in

executorch/extension/benchmark/android/benchmark/app/src/main/java/org/pytorch/minibench/BenchmarkActivity.java

Line 46 in 19cbafe

int numIter = intent.getIntExtra("num_iter", 50);
to see if it helps? I guess we need to have it running for a few days to confirm anyway. IMO, this number like stays fixed, so either hard-coding it or having a fix configuration file are good. Making it configurable is doable via the test spec, but it seems overkill

Yeah, we can do both in parallel. It's trivial to bump it up and see how it goes whiling looking into the issue locally for a specific model+config. Note this 50 iters are for inference, load_time is also unstable as we only load once. Also it's unclear to users whether the 50 iterations is after warmup or not.

[huydhn]

I see, it might make sense to use a higher threshold for load time given that it's done only once