Skip to content

Commit d93663b

Browse files
author
Guang Yang
committed
Add more add points from benchmarking infra
1 parent 3befc8a commit d93663b

File tree

1 file changed

+6
-3
lines changed

1 file changed

+6
-3
lines changed

‎extension/benchmark/README.md‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -23,17 +23,20 @@ Below is a table summarizing some example data points obtained via the infra. Th
2323

2424
| Model | Backend | Model Load Time (ms) | Avg Inference Latency (ms) | Device |
2525
|-----------------------|--------------|----------------------|----------------------------|------------------------|
26-
| DeepLabV3 (dl3) | XNN Q8 | [90.616](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985758) | [666.219](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985758) | Samsung S22 |
27-
| DeepLabV3 (dl3) | QNN FP16 | [182.207](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985141) | [9.759](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985141) | Samsung S22 |
2826
| MobileNetV3 (mv3) | XNN Q8 | [34.024](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218988461) | [252.628](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218988461) | Samsung S22 |
2927
| MobileNetV3 (mv3) | QNN FP16 | [168.414](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987785) | [1.182](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987785) | Samsung S22 |
3028
| MobileNetV3 (mv3) | COREML FP16 | [61.583](https://github.com/pytorch/executorch/actions/runs/10836589365/job/30078681158) | [682.197](https://github.com/pytorch/executorch/actions/runs/10836589365/job/30078681158) | Apple iPhone 15 Pro |
3129
| MobileNetV2 (mv2) | XNN Q8 | [14.397](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987379) | [10.796](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987379) | Samsung S22 |
3230
| MobileNetV2 (mv2) | QNN FP16 | [136.862](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987097) | [0.673](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987097) | Samsung S22 |
31+
| MobileNetV2 (mv2) | COREML FP16 | [63.019](https://github.com/pytorch/executorch/actions/runs/10892026450/job/30269520397) | [551.031](https://github.com/pytorch/executorch/actions/runs/10892026450/job/30269520397) | Apple iPhone 15 Plus |
3332
| InceptionV4 (ic4) | XNN Q8 | [87.617](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986716) | [117.937](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986716) | Samsung S22 |
3433
| InceptionV4 (ic4) | QNN FP16 | [163.943](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986381) | [2.734](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986381) | Samsung S22 |
34+
| InceptionV4 (ic4) | COREML FP16 | [134.402](https://github.com/pytorch/executorch/actions/runs/10892026450/job/30269518983) | [1906.966](https://github.com/pytorch/executorch/actions/runs/10892026450/job/30269518983) | Apple iPhone 15 Pro |
3535
| InceptionV3 (ic3) | XNN Q8 | [60.708](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986023) | [98.390](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986023) | Samsung S22 |
3636
| InceptionV3 (ic3) | QNN FP16 | [134.732](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985425) | [1.351](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985425) | Samsung S22 |
37+
| InceptionV3 (ic3) | COREML FP16 | [91.808](https://github.com/pytorch/executorch/actions/runs/10892026450/job/30269518511) | [1500.712](https://github.com/pytorch/executorch/actions/runs/10892026450/job/30269518511) | Apple iPhone 15 Pro |
38+
| DeepLabV3 (dl3) | XNN Q8 | [90.616](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985758) | [666.219](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985758) | Samsung S22 |
39+
| DeepLabV3 (dl3) | QNN FP16 | [182.207](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985141) | [9.759](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985141) | Samsung S22 |
3740
| TorchVisionVit (vit) | QNN FP16 | [174.430](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218989581) | [199.279](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218989581) | Samsung S22 |
3841
| ResNet50 (resnet50) | COREML FP16 | [84.453](https://github.com/pytorch/executorch/actions/runs/10836589365/job/30078680731) | [1329.846](https://github.com/pytorch/executorch/actions/runs/10836589365/job/30078680731) | Apple iPhone 15 Pro |
3942
| Wave2Letter (w2l) | XNN Q8 | [33.913](https://github.com/pytorch/executorch/actions/runs/10857890364/job/30139445319) | [135.584](https://github.com/pytorch/executorch/actions/runs/10857890364/job/30139445319) | Apple iPhone 15 Pro |
@@ -65,7 +68,7 @@ Users can schedule a benchmarking workflow on a pull request through GitHub Acti
6568

6669
## Retrieving Benchmark Results
6770

68-
Currently, retrieving benchmark results involves manually extracting the `benchmark_results.json` from the `Customer_Artifacts.zip` stored on AWS S3 from the benchmarking job. This process is not yet streamlined. We are working on simplifying this process and linking the results directly to the dashboard, which will be available soon.
71+
Currently, retrieving benchmark results involves manually extracting the `benchmark_results.jsong` from the `Customer_Artifacts.zip` stored on AWS S3 from the benchmarking job. This process is not yet streamlined. We are working on simplifying this process and linking the results directly to the dashboard, which will be available soon.
6972

7073

7174
## Feedback and Issue Reporting

0 commit comments

Comments
 (0)