Skip to content

Commit b915b87

Browse files
author
Guang Yang
committed
Add more data points from benchmarking infra
1 parent aebc2e3 commit b915b87

File tree

1 file changed

+13
-9
lines changed

1 file changed

+13
-9
lines changed

‎extension/benchmark/README.md‎

Lines changed: 13 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -23,20 +23,24 @@ Below is a table summarizing some example data points obtained via the infra. Th
2323

2424
| Model | Backend | Model Load Time (ms) | Avg Inference Latency (ms) | Device |
2525
|-----------------------|--------------|----------------------|----------------------------|------------------------|
26-
| DeepLabV3 (dl3) | XNN Q8 | [90.616](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985758) | [666.219](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985758) | Samsung S22 |
27-
| DeepLabV3 (dl3) | QNN FP16 | [182.207](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985141) | [9.759](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985141) | Samsung S22 |
28-
| MobileNetV3 (mv3) | XNN Q8 | [34.024](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218988461) | [252.628](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218988461) | Samsung S22 |
26+
| MobileNetV3 (mv3) | XNNPACK Q8 | [34.024](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218988461) | [252.628](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218988461) | Samsung S22 |
2927
| MobileNetV3 (mv3) | QNN FP16 | [168.414](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987785) | [1.182](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987785) | Samsung S22 |
3028
| MobileNetV3 (mv3) | COREML FP16 | [61.583](https://github.com/pytorch/executorch/actions/runs/10836589365/job/30078681158) | [682.197](https://github.com/pytorch/executorch/actions/runs/10836589365/job/30078681158) | Apple iPhone 15 Pro |
31-
| MobileNetV2 (mv2) | XNN Q8 | [14.397](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987379) | [10.796](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987379) | Samsung S22 |
29+
| MobileNetV2 (mv2) | XNNPACK Q8 | [14.397](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987379) | [10.796](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987379) | Samsung S22 |
3230
| MobileNetV2 (mv2) | QNN FP16 | [136.862](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987097) | [0.673](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987097) | Samsung S22 |
33-
| InceptionV4 (ic4) | XNN Q8 | [87.617](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986716) | [117.937](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986716) | Samsung S22 |
31+
| MobileNetV2 (mv2) | COREML FP16 | [63.019](https://github.com/pytorch/executorch/actions/runs/10892026450/job/30269520397) | [551.031](https://github.com/pytorch/executorch/actions/runs/10892026450/job/30269520397) | Apple iPhone 15 Plus |
32+
| InceptionV4 (ic4) | XNNPACK Q8 | [87.617](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986716) | [117.937](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986716) | Samsung S22 |
3433
| InceptionV4 (ic4) | QNN FP16 | [163.943](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986381) | [2.734](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986381) | Samsung S22 |
35-
| InceptionV3 (ic3) | XNN Q8 | [60.708](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986023) | [98.390](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986023) | Samsung S22 |
34+
| InceptionV4 (ic4) | COREML FP16 | [134.402](https://github.com/pytorch/executorch/actions/runs/10892026450/job/30269518983) | [1906.966](https://github.com/pytorch/executorch/actions/runs/10892026450/job/30269518983) | Apple iPhone 15 Pro |
35+
| InceptionV3 (ic3) | XNNPACK Q8 | [60.708](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986023) | [98.390](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986023) | Samsung S22 |
3636
| InceptionV3 (ic3) | QNN FP16 | [134.732](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985425) | [1.351](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985425) | Samsung S22 |
37-
| TorchVisionVit (vit) | QNN FP16 | [174.430](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218989581) | [199.279](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218989581) | Samsung S22 |
37+
| InceptionV3 (ic3) | COREML FP16 | [91.808](https://github.com/pytorch/executorch/actions/runs/10892026450/job/30269518511) | [1500.712](https://github.com/pytorch/executorch/actions/runs/10892026450/job/30269518511) | Apple iPhone 15 Pro |
38+
| DeepLabV3 (dl3) | XNNPACK Q8 | [90.616](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985758) | [666.219](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985758) | Samsung S22 |
39+
| DeepLabV3 (dl3) | QNN FP16 | [182.207](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985141) | [9.759](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985141) | Samsung S22 |
40+
| ResNet50 (resnet50) | XNNPACK Q8 | [55.462](https://github.com/pytorch/executorch/actions/runs/10911725781/job/30285857102) | [37.021](https://github.com/pytorch/executorch/actions/runs/10911725781/job/30285857102) | Apple iPhone 15 Pro |
3841
| ResNet50 (resnet50) | COREML FP16 | [84.453](https://github.com/pytorch/executorch/actions/runs/10836589365/job/30078680731) | [1329.846](https://github.com/pytorch/executorch/actions/runs/10836589365/job/30078680731) | Apple iPhone 15 Pro |
39-
| Wave2Letter (w2l) | XNN Q8 | [33.913](https://github.com/pytorch/executorch/actions/runs/10857890364/job/30139445319) | [135.584](https://github.com/pytorch/executorch/actions/runs/10857890364/job/30139445319) | Apple iPhone 15 Pro |
42+
| TorchVisionVit (vit) | QNN FP16 | [174.430](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218989581) | [199.279](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218989581) | Samsung S22 |
43+
| Wave2Letter (w2l) | XNNPACK Q8 | [33.913](https://github.com/pytorch/executorch/actions/runs/10857890364/job/30139445319) | [135.584](https://github.com/pytorch/executorch/actions/runs/10857890364/job/30139445319) | Apple iPhone 15 Pro |
4044

4145

4246
## Supported Use Cases
@@ -65,7 +69,7 @@ Users can schedule a benchmarking workflow on a pull request through GitHub Acti
6569

6670
## Retrieving Benchmark Results
6771

68-
Currently, retrieving benchmark results involves manually extracting the `benchmark_results.json` from the `Customer_Artifacts.zip` stored on AWS S3 from the benchmarking job. This process is not yet streamlined. We are working on simplifying this process and linking the results directly to the dashboard, which will be available soon.
72+
Currently, retrieving benchmark results involves manually extracting the `benchmark_results.jsong` from the `Customer_Artifacts.zip` stored on AWS S3 from the benchmarking job. This process is not yet streamlined. We are working on simplifying this process and linking the results directly to the dashboard, which will be available soon.
6973

7074

7175
## Feedback and Issue Reporting

0 commit comments

Comments
 (0)