Skip to content

Commit 111b391

Browse files
veblushveblush
committed
More docs
1 parent 9c213ef commit 111b391

File tree

1 file changed

+78
-59
lines changed

1 file changed

+78
-59
lines changed

‎docs/UnderTheHood.md‎

Lines changed: 78 additions & 59 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Under the hood
22

3-
IncrementalCompiler for Unity3D was my fun-project for 2016 new year's holiday.
3+
IncrementalCompiler for Unity3D was my fun-project for new year's holiday 2016.
44
At that time I was thinking about slow compilation speed of unity3d-mono and
55
wondering if it is possible to make it faster with the minimum effort.
66
After a couple of hours digging, it turned out feasible to make it.
@@ -10,7 +10,8 @@ After a couple of hours digging, it turned out feasible to make it.
1010
Microsoft / Mono C# compiler doesn't support incremental compilation until now.
1111
They had "/incremental" option once in old times but lost it already.
1212
Because C# compiler is really fast and C# language itself is good to
13-
keep compiler run fast, incremental compilation has not been at the first priority.
13+
keep compiler run fast, implementing incremental compilation has not been
14+
at the first priority.
1415
Also community keep there project small and like to separate big project into smaller ones
1516
to handle this problem indirectly.
1617

@@ -21,6 +22,8 @@ big project into small one. So I decided to make incremental compiler for Unity3
2122
### Unity build process
2223

2324
Unity3D makes three projects from Assets directory if your project has only C# sources.
25+
(If there are .js or .boo sources, additional projects will be created.
26+
more detailed info: [Unity Manual: Special Folders and Script Compilation Order](http://docs.unity3d.com/Manual/ScriptCompileOrderFolders.html))
2427

2528
- Assembly-CSharp-firstpass
2629
- Consists of scripts in Plugins directory.
@@ -37,40 +40,15 @@ Unity3D makes three projects from Assets directory if your project has only C# s
3740
- This project is always built not because these sources are modified but by building
3841
dependent Assembly-CSharp.
3942

40-
[Unity Manual: Special Folders and Script Compilation Order](http://docs.unity3d.com/Manual/ScriptCompileOrderFolders.html)
41-
42-
### How to replace a builtin compiler with a new one.
43-
44-
At first, Replacing mono compiler in unity directory with new one was considered.
45-
But it is not an easy way for users and also intrusive way to affect whole projects.
46-
However [alexzzzz](https://bitbucket.org/alexzzzz/unity-c-5.0-and-6.0-integration/src)
47-
found a smart way to workaround this as following:
48-
49-
```csharp
50-
[InitializeOnLoad]
51-
public static class CSharp60SupportActivator {
52-
static CSharp60SupportActivator() {
53-
var list = GetSupportedLanguages();
54-
list.RemoveAll(language => language is CSharpLanguage);
55-
list.Add(new CustomCSharpLanguage());
56-
}
57-
private static List<SupportedLanguage> GetSupportedLanguages() {
58-
var fieldInfo = typeof(ScriptCompilers).GetField("_supportedLanguages", ...);
59-
var languages = (List<SupportedLanguage>)fieldInfo.GetValue(null);
60-
return languages;
61-
}
62-
}
63-
```
64-
65-
It is an internal feature for Unity3D and cannot be accessed from external DLLs.
66-
But to make it, he renamed plugin DLL to one of internal friend DLL names,
67-
`Unity.PureCSharpTests`.
43+
These projects will be built successively and Unity3d cannot build these concurrently
44+
because they depends on previous projects.
6845

6946
### Roslyn
7047

71-
[Roslyn](https://github.com/dotnet/roslyn) is new open-source project providings
72-
C# and Visual Basic compilers. With this project it's really easy to use features
73-
that compiler can provide like parsing, analysing and even compiling itself.
48+
Incremental compiler is built with [Roslyn](https://github.com/dotnet/roslyn)
49+
which is new open-source project providings C# and Visual Basic compilers.
50+
With this project it's really easy to use features that compiler can
51+
provide like parsing, analysing and even compiling itself.
7452

7553
Just with [Microsoft.CodeAnalysis.CSharp](https://www.nuget.org/packages/Microsoft.CodeAnalysis/),
7654
compiler can be written without any hard work.
@@ -87,65 +65,106 @@ Assembly Build(string[] sourcePaths, string[] referencePaths, string[] defines)
8765
}
8866
```
8967

68+
Not only compiling itself, it can provide new C# 6 and upcoming features.
69+
9070
### Incremental compiler
9171

9272
Roslyn C# compiler does two steps to compile from API view.
93-
First one is parsing step. It loads sources and parse it.
94-
Second one is emitting step. From compiler's view, most of work is done here
95-
such as semantic analysis, optimization and code generation.
73+
74+
1. Parsing step:
75+
It loads sources and build syntax trees by parsing them.
76+
2. Emitting step:
77+
From compiler's view, most of work is done here such as semantic analysis,
78+
optimization and code generation.
79+
9680
Because library users cannot access internal phase in emitting step,
9781
incremental compiler is written in a simple way like:
9882

99-
Full compilation for first time build.
100-
```csharp
101-
var syntaxTrees = sourcePaths.Select(file => /* PARSE */);
102-
var references = referencePaths.Select(file => /* LOAD */);
103-
var compilation = CSharpCompilation.Create(syntaxTrees, references);
104-
compilation.Emit(ms);
105-
```
106-
107-
Incremental compilation for subsequent builds.
108-
```csharp
109-
compliation = compliation.RemoveReferences(oldLoadedReferences)
110-
.AddReferences(load(newReference))
111-
.RemoveSyntaxTrees(oldSourceSyntaxTrees)
112-
.AddSyntaxTrees(parse(newSource))
113-
compilation.Emit(ms);
114-
```
83+
- For the first time, it creates compilation object and compiles whole sources with it:
84+
```csharp
85+
var syntaxTrees = sourcePaths.Select(file => /* PARSE */);
86+
var references = referencePaths.Select(file => /* LOAD */);
87+
var compilation = CSharpCompilation.Create(syntaxTrees, references);
88+
compilation.Emit(ms);
89+
```
90+
- For subsequent builds, it update changes to compilation object and compiles it.
91+
```csharp
92+
compliation = compliation.RemoveReferences(oldLoadedReferences)
93+
.AddReferences(load(newReference))
94+
.RemoveSyntaxTrees(oldSourceSyntaxTrees)
95+
.AddSyntaxTrees(parse(newSource))
96+
compilation.Emit(ms);
97+
```
11598

11699
By telling changes in a project to compilation object, rolsyn can use
117100
pre-parsed syntax trees and some informations that I wish.
118101

102+
TODO: Check information that roslyn uses for next build
103+
119104
### Compile server
120105

121106
Ok. Keeping compilation object and reusing can make an incremental compiler.
122-
But where can we put this object on? Everytime Unity3D want to build DLL, it
123-
invokes C# compiler.
124-
C# compiler is running awhile, exits and leaves built DLL for Unity3D, which means
125-
it should throw away compilation object.
107+
But where can we put this object on? Everytime Unity3D want to build DLL,
108+
it spawns C# compiler.
109+
C# compiler is running awhile, terminates and leaves built DLL for Unity3D,
110+
which means it should throw away compilation object.
126111

127-
We have two options to keep this object permanent:
112+
We have two options to keep this object permanent to reuse:
128113

129114
1. Save all information to disc and load them at next invocation.
130115
1. Make a compiler server. Let it alive while Unity3D is alive and
131116
every compile invocation will be forwared to it.
132117

133118
Because first one involves an IO intensive process which make work slow
134119
and I don't know how to (de)serialize a compilation object of Roslyn,
135-
second one is chosen.
120+
second one was chosen.
136121

137122
When incremental compiler is requested to compile assembly,
138123
it find a compiler server process. If there is no one, it spawns a compile server.
139-
Compiler forwards a compilation request to this server and waits for results.
124+
Compiler forwards a compilation request to this serdver and waits for results.
140125
Compile server processes this request, return it to requester and keep
141126
this intermediate object for subsequent requests.
142127

128+
TODO: Depict process relationship
129+
143130
To communicate between compile client and server,
144131
[WCF on Named pipe](https://msdn.microsoft.com/en-us/library/ms733769%28v=vs.110%29.aspx) is used
145132
to make this tool quickly even Mono doesn't support it.
146133

134+
Compiler server allocates big memory to keep a compilation object.
135+
In my case, for project consisting of 2,000 sources, it takes 300MB.
136+
137+
### How to replace a builtin compiler with a new one.
138+
139+
At first, replacing mono compiler in unity directory with new one was considered.
140+
But it is not an easy way for users and also intrusive way that affects whole projects.
141+
However [alexzzzz](https://bitbucket.org/alexzzzz/unity-c-5.0-and-6.0-integration/src)
142+
found a smart way to workaround this as following:
143+
144+
```csharp
145+
[InitializeOnLoad]
146+
public static class CSharp60SupportActivator {
147+
static CSharp60SupportActivator() {
148+
var list = GetSupportedLanguages();
149+
list.RemoveAll(language => language is CSharpLanguage);
150+
list.Add(new CustomCSharpLanguage());
151+
}
152+
private static List<SupportedLanguage> GetSupportedLanguages() {
153+
var fieldInfo = typeof(ScriptCompilers).GetField("_supportedLanguages", ...);
154+
var languages = (List<SupportedLanguage>)fieldInfo.GetValue(null);
155+
return languages;
156+
}
157+
}
158+
```
159+
160+
It is an internal feature for Unity3D and cannot be accessed from external user DLLs.
161+
But to make it, he renamed plugin DLL to one of internal friend DLL names,
162+
`Unity.PureCSharpTests`.
163+
147164
## Modification for Unity3D/Mono
148165

166+
TODO: Intro
167+
149168
### Reuse prebuilt DLLs
150169

151170
When sources in Assembly-CSharp are edited, Unity3D builds Assembly-CSharp.

0 commit comments

Comments
 (0)