Skip to content

Commit faa2012

Browse files
yangw-devyangw-dev
committed
setup link
Signed-off-by: Yang Wang <[email protected]>
1 parent 4c0fdd2 commit faa2012

File tree

56 files changed

+1416
-6
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

56 files changed

+1416
-6
lines changed

‎backends/xnnpack/third-party/generate-cpuinfo-wrappers.py‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -78,10 +78,7 @@
7878
print(filepath)
7979
os.makedirs(os.path.dirname(filepath))
8080
with open(filepath, "w") as wrapper:
81-
print(
82-
"/* Auto-generated by generate-wrappers.py script. Do not modify */",
83-
file=wrapper,
84-
)
81+
print("/* Auto-generated by generate-wrappers.py script. Do not modify */", file=wrapper)
8582
print(file=wrapper)
8683
print("#ifdef __APPLE__", file=wrapper)
8784
print("\t#include <TargetConditionals.h>", file=wrapper)

‎extension/llm/custom_ops/spinquant/third-party/FFHT/example.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,6 @@
1515
t2 = timeit.default_timer()
1616

1717
if sys.version_info[0] == 2:
18-
print(t2 - t1 + 0.0) / (reps + 0.0)
18+
print (t2 - t1 + 0.0) / (reps + 0.0)
1919
if sys.version_info[0] == 3:
20-
print("{}".format((t2 - t1 + 0.0) / (reps + 0.0)))
20+
print('{}'.format((t2 - t1 + 0.0) / (reps + 0.0)))

‎private.xlsx‎

17.7 KB
Binary file not shown.

‎public.xlsx‎

13.2 KB
Binary file not shown.

‎stability_analysis_results/comprehensive_stability_summary.txt‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
Comprehensive Latency Stability Analysis Summary
2+
================================================================================
3+
4+
Primary (Private) Datasets Summary:
5+
+-----------+------------------+---------------------------------------------------+---------------------+----------+-------------------+--------------------+
6+
| Dataset | Model | Device | Mean Latency (ms) | CV (%) | Stability Score | Stability Rating |
7+
+===========+==================+===================================================+=====================+==========+===================+====================+
8+
| table1 | mv3(coreml_fp16) | Apple iPhone 15 (private)(iOS 18.0) | 1.00 | 0.00 | 100.00 | Excellent |
9+
+-----------+------------------+---------------------------------------------------+---------------------+----------+-------------------+--------------------+
10+
| table2 | mv3(coreml_fp16) | Apple iPhone 15 Plus (private)(iOS 17.4.1) | 1.00 | 0.00 | 100.00 | Excellent |
11+
+-----------+------------------+---------------------------------------------------+---------------------+----------+-------------------+--------------------+
12+
| table3 | mv3(coreml_fp16) | Apple iPhone 15 Pro (private)(iOS 18.4.1) | 1.00 | 0.00 | 100.00 | Excellent |
13+
+-----------+------------------+---------------------------------------------------+---------------------+----------+-------------------+--------------------+
14+
| table4 | mv3(mps) | Apple iPhone 15 (private)(iOS 18.0) | 4.00 | 0.00 | 100.00 | Excellent |
15+
+-----------+------------------+---------------------------------------------------+---------------------+----------+-------------------+--------------------+
16+
| table5 | mv3(mps) | Apple iPhone 15 Plus (private)(iOS 17.4.1) | 1.00 | 0.00 | 100.00 | Excellent |
17+
+-----------+------------------+---------------------------------------------------+---------------------+----------+-------------------+--------------------+
18+
| table8 | mv3(qnn_q8) | Samsung Galaxy S22 Ultra 5G (private)(Android 14) | 1.02 | 0.51 | 97.08 | Excellent |
19+
+-----------+------------------+---------------------------------------------------+---------------------+----------+-------------------+--------------------+
20+
| table9 | mv3(xnnpack_q8) | Apple iPhone 15 (private)(iOS 18.0) | 44.90 | 5.30 | 68.15 | Moderate |
21+
+-----------+------------------+---------------------------------------------------+---------------------+----------+-------------------+--------------------+
22+
| table7 | mv3(qnn_q8) | Samsung Galaxy S22 5G (private)(Android 13) | 1.03 | 6.03 | 62.53 | Moderate |
23+
+-----------+------------------+---------------------------------------------------+---------------------+----------+-------------------+--------------------+
24+
| table6 | mv3(mps) | Apple iPhone 15 Pro (private)(iOS 18.4.1) | 2.88 | 12.30 | 43.75 | Poor |
25+
+-----------+------------------+---------------------------------------------------+---------------------+----------+-------------------+--------------------+
26+
| table10 | mv3(xnnpack_q8) | Apple iPhone 15 Plus (private)(iOS 17.4.1) | 40.60 | 10.59 | 38.45 | Poor |
27+
+-----------+------------------+---------------------------------------------------+---------------------+----------+-------------------+--------------------+
28+
| table12 | mv3(xnnpack_q8) | Samsung Galaxy S22 5G (private)(Android 13) | 3.09 | 33.81 | 8.07 | Poor |
29+
+-----------+------------------+---------------------------------------------------+---------------------+----------+-------------------+--------------------+
30+
| table11 | mv3(xnnpack_q8) | Apple iPhone 15 Pro (private)(iOS 18.4.1) | 16.20 | 38.03 | 7.21 | Poor |
31+
+-----------+------------------+---------------------------------------------------+---------------------+----------+-------------------+--------------------+
32+
| table13 | mv3(xnnpack_q8) | Samsung Galaxy S22 Ultra 5G (private)(Android 14) | 2.98 | 39.37 | 4.91 | Poor |
33+
+-----------+------------------+---------------------------------------------------+---------------------+----------+-------------------+--------------------+
34+
35+
Reference (Public) Datasets Summary:
36+
+-----------+------------------+-----------------------------------+---------------------+----------+-------------------+--------------------+
37+
| Dataset | Model | Device | Mean Latency (ms) | CV (%) | Stability Score | Stability Rating |
38+
+===========+==================+===================================+=====================+==========+===================+====================+
39+
| table1 | mv3(coreml_fp16) | Apple iPhone 15(iOS 18.0) | 1.00 | 0.00 | 100.00 | Excellent |
40+
+-----------+------------------+-----------------------------------+---------------------+----------+-------------------+--------------------+
41+
| table2 | mv3(coreml_fp16) | Apple iPhone 15 Plus(iOS 17.4.1) | 1.00 | 0.00 | 100.00 | Excellent |
42+
+-----------+------------------+-----------------------------------+---------------------+----------+-------------------+--------------------+
43+
| table4 | mv3(mps) | Apple iPhone 15 Plus(iOS 17.4.1) | 1.00 | 0.00 | 100.00 | Excellent |
44+
+-----------+------------------+-----------------------------------+---------------------+----------+-------------------+--------------------+
45+
| table5 | mv3(qnn_q8) | Samsung Galaxy S22 5G(Android 13) | 1.02 | 1.63 | 89.31 | Good |
46+
+-----------+------------------+-----------------------------------+---------------------+----------+-------------------+--------------------+
47+
| table7 | mv3(xnnpack_q8) | Apple iPhone 15 Plus(iOS 17.4.1) | 40.00 | 10.10 | 41.69 | Poor |
48+
+-----------+------------------+-----------------------------------+---------------------+----------+-------------------+--------------------+
49+
| table3 | mv3(mps) | Apple iPhone 15(iOS 18.0) | 3.33 | 30.98 | 37.50 | Poor |
50+
+-----------+------------------+-----------------------------------+---------------------+----------+-------------------+--------------------+
51+
| table8 | mv3(xnnpack_q8) | Samsung Galaxy S22 5G(Android 13) | 2.77 | 25.21 | 19.91 | Poor |
52+
+-----------+------------------+-----------------------------------+---------------------+----------+-------------------+--------------------+
53+
| table6 | mv3(xnnpack_q8) | Apple iPhone 15(iOS 18.0) | 52.71 | 41.41 | 4.61 | Poor |
54+
+-----------+------------------+-----------------------------------+---------------------+----------+-------------------+--------------------+
55+
56+
Private vs Public Comparison:
57+
+----------------------------------------------------+----------------------------------------------+------------------------------------+-----------------+----------------+--------------+------------------+-----------------+---------------+
58+
| Dataset | Private Device | Public Device | Private Score | Public Score | Score Diff | Private CV (%) | Public CV (%) | CV Diff (%) |
59+
+====================================================+==============================================+====================================+=================+================+==============+==================+=================+===============+
60+
| mv3(xnnpack_q8) on Apple iPhone 15 (private) | Apple iPhone 15 (private) (iOS 18.0) | Apple iPhone 15 (iOS 18.0) | 68.15 | 4.61 | 63.54 | 5.30 | 41.41 | -36.12 |
61+
+----------------------------------------------------+----------------------------------------------+------------------------------------+-----------------+----------------+--------------+------------------+-----------------+---------------+
62+
| mv3(mps) on Apple iPhone 15 (private) | Apple iPhone 15 (private) (iOS 18.0) | Apple iPhone 15 (iOS 18.0) | 100.00 | 37.50 | 62.50 | 0.00 | 30.98 | -30.98 |
63+
+----------------------------------------------------+----------------------------------------------+------------------------------------+-----------------+----------------+--------------+------------------+-----------------+---------------+
64+
| mv3(coreml_fp16) on Apple iPhone 15 (private) | Apple iPhone 15 (private) (iOS 18.0) | Apple iPhone 15 (iOS 18.0) | 100.00 | 100.00 | 0.00 | 0.00 | 0.00 | 0.00 |
65+
+----------------------------------------------------+----------------------------------------------+------------------------------------+-----------------+----------------+--------------+------------------+-----------------+---------------+
66+
| mv3(coreml_fp16) on Apple iPhone 15 Plus (private) | Apple iPhone 15 Plus (private) (iOS 17.4.1) | Apple iPhone 15 Plus (iOS 17.4.1) | 100.00 | 100.00 | 0.00 | 0.00 | 0.00 | 0.00 |
67+
+----------------------------------------------------+----------------------------------------------+------------------------------------+-----------------+----------------+--------------+------------------+-----------------+---------------+
68+
| mv3(mps) on Apple iPhone 15 Plus (private) | Apple iPhone 15 Plus (private) (iOS 17.4.1) | Apple iPhone 15 Plus (iOS 17.4.1) | 100.00 | 100.00 | 0.00 | 0.00 | 0.00 | 0.00 |
69+
+----------------------------------------------------+----------------------------------------------+------------------------------------+-----------------+----------------+--------------+------------------+-----------------+---------------+
70+
| mv3(xnnpack_q8) on Apple iPhone 15 Plus (private) | Apple iPhone 15 Plus (private) (iOS 17.4.1) | Apple iPhone 15 Plus (iOS 17.4.1) | 38.45 | 41.69 | -3.24 | 10.59 | 10.10 | 0.49 |
71+
+----------------------------------------------------+----------------------------------------------+------------------------------------+-----------------+----------------+--------------+------------------+-----------------+---------------+
72+
| mv3(xnnpack_q8) on Samsung Galaxy S22 5G (private) | Samsung Galaxy S22 5G (private) (Android 13) | Samsung Galaxy S22 5G (Android 13) | 8.07 | 19.91 | -11.85 | 33.81 | 25.21 | 8.59 |
73+
+----------------------------------------------------+----------------------------------------------+------------------------------------+-----------------+----------------+--------------+------------------+-----------------+---------------+
74+
| mv3(qnn_q8) on Samsung Galaxy S22 5G (private) | Samsung Galaxy S22 5G (private) (Android 13) | Samsung Galaxy S22 5G (Android 13) | 62.53 | 89.31 | -26.78 | 6.03 | 1.63 | 4.40 |
75+
+----------------------------------------------------+----------------------------------------------+------------------------------------+-----------------+----------------+--------------+------------------+-----------------+---------------+
76+
77+
Private environment is more stable in 2 of 8 cases.
78+
Public environment is more stable in 3 of 8 cases.
79+
Both environments show equal stability in 3 of 8 cases.
80+
81+
Overall Insights and Recommendations:
82+
Stability Distribution in Private Datasets:
83+
- Excellent: 6 dataset(s)
84+
- Poor: 5 dataset(s)
85+
- Moderate: 2 dataset(s)
86+
87+
Best Configurations:
88+
- Most stable configuration: table1 (Score: 100.0/100)
89+
Model: mv3(coreml_fp16), Device: Apple iPhone 15 (private)(iOS 18.0)
90+
- Most stable OS version: iOS 18.0 (Avg. Score: 89.4/100)
91+
92+
General Recommendations:
93+
1. For datasets with 'Poor' or 'Moderate' stability, investigate potential causes
94+
such as thermal throttling, background processes, or power management settings.
95+
2. Consider increasing warm-up iterations for datasets with high CV values.
96+
3. For critical applications, prefer models and devices with 'Good' or 'Excellent' stability.
97+
4. Public environments show better stability in most cases. Consider investigating
98+
factors affecting stability in the private environment.

0 commit comments

Comments
 (0)