Skip to content

Conversation

@kenzieschmoll
Copy link
Member

@kenzieschmoll kenzieschmoll commented Nov 28, 2024

Fixes #8583

@kenzieschmoll kenzieschmoll requested a review from a team as a code owner November 28, 2024 01:03
@kenzieschmoll kenzieschmoll requested review from elliette and removed request for a team November 28, 2024 01:03
@nandnai-gurjar

This comment was marked as off-topic.

@kenzieschmoll
Copy link
Member Author

kenzieschmoll commented Dec 2, 2024

The WASM results are running much slower on the bots than locally. Locally, the benchmark tests pass and meet the expected thresholds, which are set to 16666 micros (16.6 ms is what is required for performant rendering on a 60 FPS device). On the CI the test is failing:

  [WASM Benchmarks] The following benchmark scores exceeded their expected thresholds:
  
  [devtools_offlineCpuProfilerScreen.wasm] flutter_frame.total_time.average was 35744.85030395137 μs, which exceeded the expected threshold, 16666.0 μs.
  [devtools_offlineCpuProfilerScreen.wasm] flutter_frame.total_time.p50 was 54035.0 μs, which exceeded the expected threshold, 16666.0 μs.
  [devtools_offlineCpuProfilerScreen.wasm] flutter_frame.total_time.p90 was 59095.0 μs, which exceeded the expected threshold, 16666.0 μs.
  [devtools_offlinePerformanceScreen.wasm] flutter_frame.total_time.p90 was 28900.0 μs, which exceeded the expected threshold, 16666.0 μs.

@eyebrowsoffire @yjbanov any idea what would cause WASM to be significantly slower on the CI? The JS benchmarks seem to be consistent, or at least they are not exceeding the 16666 micros threshold for any of the metrics (on the CI and locally).

@yjbanov
Copy link

yjbanov commented Dec 2, 2024

Does the CI use the same hardware+OS combination as the local device you use for benchmarking? A lot depends on the hardware the benchmark runs on. As for why the JS is consistent and Wasm is not, I'm not sure. Someone explained to me in the past that V8 runs in different modes for different situations. For example, when you have DevTools open, V8 switches into a mode that supports debugging, which changes the performance characteristics of the code.

@kenzieschmoll
Copy link
Member Author

@yjbanov looks like switching the action to run on macos-latest instead of ubuntu-latest did the trick. Thanks!

@kenzieschmoll kenzieschmoll merged commit aad5c0c into flutter:master Dec 5, 2024
24 checks passed
@kenzieschmoll kenzieschmoll deleted the benchmarks branch December 5, 2024 21:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Verify performance benchmark metrics against an expected threshold

4 participants