Skip to content

Commit 8780ebe

Browse files
veblushveblush
committed
Add new benchmark result
1 parent acd4073 commit 8780ebe

File tree

1 file changed

+112
-48
lines changed

1 file changed

+112
-48
lines changed

‎docs/Benchmark.md‎

Lines changed: 112 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -1,50 +1,114 @@
11
# Benchmark
22

3-
This benchmark is measured with a proprietary project in my company.
4-
5-
#### Project size
6-
7-
| Project | Count | Size |
8-
| :------------------------ | ----: | --------: |
9-
| Assembly-CSharp-firstpass | 10 | 158,858 |
10-
| Assembly-CSharp | 1,377 | 7,509,543 |
11-
| Assembly-CSharp-Editor | 223 | 1,967,076 |
12-
| Total | 1,610 | 9,635,477 |
13-
14-
### Benchmark Result
15-
16-
#### Unity Mono 3
17-
18-
| Project | Duration (sec) |
19-
| :------------------------ | -------------: |
20-
| Assembly-CSharp-firstpass | 1.72 |
21-
| Assembly-CSharp | 12.78 |
22-
| Assembly-CSharp-Editor | 4.34 |
23-
| Total | 18.84 |
24-
25-
#### Roslyn (Full)
26-
27-
| Project | Duration (sec) |
28-
| :------------------------ | -------------: |
29-
| Assembly-CSharp-firstpass | 3.96 + 0.39 |
30-
| Assembly-CSharp | 6.32 + 0.83 |
31-
| Assembly-CSharp-Editor | 2.16 + 0.46 |
32-
| Total | 14.12 (75%) |
33-
34-
#### Roslyn (Incremental)
35-
36-
| Project | Duration (sec) |
37-
| :------------------------ | -------------: |
38-
| Assembly-CSharp-firstpass | 1.16 + 0.48 |
39-
| Assembly-CSharp | 4.60 + 0.82 |
40-
| Assembly-CSharp-Editor | 1.67 + 0.47 |
41-
| Total | 9.2 (49%) |
42-
43-
#### Roslyn (Incremental with embedded MdbWriter)
44-
45-
| Project | Duration (sec) |
46-
| :------------------------ | -------------: |
47-
| Assembly-CSharp-firstpass | 1.31 |
48-
| Assembly-CSharp | 3.60 |
49-
| Assembly-CSharp-Editor | 1.00 |
50-
| Total | 5.91 (31%) |
3+
### How to measure
4+
5+
Benchmark is done with following steps.
6+
1. Run Unity3D.
7+
1. Open a target project.
8+
1. Make unity build scripts by editing one C# source for warming up.
9+
1. Do previous step again and measure compilation time.
10+
- Edit a script in Assembly-CSharp (not plugins nor editor)
11+
- Compilation time is recorded in UniversalCompiler log.
12+
13+
### Test environment
14+
15+
- CPU: Intel Core i5-4690 (3.50 GHz)
16+
- RAM: 8 GB
17+
- OS: Windows 10 Pro (64 bits)
18+
- SSD: Samsung SSD 840 EVO 1TB
19+
- Unity 5.3.1f1 (Win64)
20+
21+
### Target projects
22+
23+
1. Generated project whose script size is 1.4 MB (sample/Benchmark)
24+
1. Generated project whose script size is 6.7 MB (sample/Benchmark)
25+
1. In-house real project whose script size is 9.4 MB
26+
27+
### Tested compiler
28+
29+
1. Unity default compiler (Mono3)
30+
1. C# 6 roslyn compiler (CSC6)
31+
1. Incremental C# 6 Compiler with default configuration (ICS6)
32+
1. Incremental C# 6 Compiler with "WhenNoSourceChange" configuration (ICS6@)
33+
34+
### Results
35+
36+
#### Project 1 (Generated project: 1.4 MB)
37+
38+
This project is generated with `SourcePopulate.py 30 20 200` in ./samples/Benchmark.
39+
40+
##### Project size
41+
42+
| Project | Count | Size |
43+
| :------- | ----: | --------: |
44+
| Plugins | 20 | 122,980 |
45+
| Scripts | 200 | 1,175,260 |
46+
| Editor | 30 | 181,409 |
47+
| Total | 250 | 1,479,649 |
48+
49+
##### Compilation time (sec)
50+
51+
| | Mono3 | CSC6 | ICS6 | ICS6@ |
52+
| :-------- | ----: | ----: | ----: | ----: |
53+
| Scripts | 2.23 | 3.52 | 0.95 | 0.95 |
54+
| Editor | 0.60 | 2.88 | 0.23 | 0.17 |
55+
| Total | 2.83 | 6.40 | 1.18 | 1.12 |
56+
| Total (%) | 100% | 226% | 41% | 39% |
57+
58+
For small size project, it's hard to get significant benefit.
59+
2.83 seconds with Mono3 is not bad for work.
60+
61+
#### Project 2 (Generated project: 6.7 MB)
62+
63+
This project is generated with `SourcePopulate.py 200 20 1000` in ./samples/Benchmark.
64+
65+
##### Project size
66+
67+
| Project | Count | Size |
68+
| :------- | ----: | --------: |
69+
| Plugins | 20 | 122,980 |
70+
| Scripts | 1,000 | 5,642,460 |
71+
| Editor | 200 | 1,130,519 |
72+
| Total | 1,220 | 6,895,959 |
73+
74+
##### Compilation time (sec)
75+
76+
| | Mono3 | CSC6 | ICS6 | ICS6@ |
77+
| :-------- | ----: | ----: | ----: | ----: |
78+
| Scripts | 22.82 | 4.72 | 2.68 | 2.68 |
79+
| Editor | 1.88 | 3.08 | 0.59 | 0.17 |
80+
| Total | 24.70 | 7.80 | 3.27 | 2.85 |
81+
| Total (%) | 100% | 31% | 13% | 11% |
82+
83+
After getting mono3 result, compilation time seems to be too long.
84+
Test was repeated several times to make sure but same result.
85+
There should be a corner case for mono compiler.
86+
87+
#### Project 3 (In-house project: 9.4 MB)
88+
89+
This project is my company project and not allowed to be open.
90+
But this is a real project and worth sharing result.
91+
92+
##### Project size
93+
94+
| Project | Count | Size |
95+
| :------- | ----: | --------: |
96+
| Plugins | 10 | 158,858 |
97+
| Scripts | 1,377 | 7,509,543 |
98+
| Editor | 223 | 1,967,076 |
99+
| Total | 1,610 | 9,635,477 |
100+
101+
##### Compilation time (sec)
102+
103+
| | Mono3 | CSC6 | ICS6 | ICS6@ |
104+
| :-------- | ----: | ----: | ----: | ----: |
105+
| Scripts | 8.30 | 5.67 | 3.50 | 3.52 |
106+
| Editor | 2.83 | 3.27 | 1.10 | 0.19 |
107+
| Total | 11.13 | 8.94 | 4.60 | 3.71 |
108+
| Total (%) | 100% | 80% | 41% | 33% |
109+
110+
This result is first place that I consider to make this incremental compiler.
111+
Edit & review workflow doesn't go well with Mono3 compilation time.
112+
Everytime source is modified, I always look hard at loading indicator impatiently.
113+
But incremental compiler with aggresive WhenNoSourceChange gives a way faster compilation
114+
speed and make me happy, still I can feel time fly but bearable.

0 commit comments

Comments
 (0)