Skip to content

Commit ce99c80

Browse files
dmbrysondmbryson
committed
Add a README describing Tritium
1 parent 2bcaad6 commit ce99c80

File tree

3 files changed

+120
-0
lines changed

3 files changed

+120
-0
lines changed

‎README.tritium.md

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
# Tritium — llbuild3
2+
3+
## Objectives
4+
5+
* Improve graph definition ergonomics
6+
* (Re-)add an intrinsic artifact object, separate from producing rules
7+
* Incremental computation of rules in addition to graphs
8+
* Support dynamically down-scoping rule cache keys
9+
* Enable larger scale graphs
10+
* Recursive engine inter-operability
11+
* Engine as artifact/rule provider
12+
* Fully serializable, cache-able engine state
13+
* Separate and finely control engine parallelism and task scheduling from networking (remove SwiftNIO/Futures use from the core)
14+
* Improve engine resumption performance
15+
* Ability to reconnect to in progress actions
16+
* Facilitate build graph unification
17+
* API adapters/compatibility/on-ramp for existing llbuild and llbuild2fx use cases
18+
19+
## API Overview — Ergonomics
20+
21+
Tritium will incorporate concepts from llbuild1, llbuild2, and llbuild2fx. The constructs are intended to enable a separation of the items that can be produced (Artifacts) from the computations that can produce them (Rules/Tasks), addressing an expression challenge that exists today in llbuild2fx rule definition.
22+
23+
### Dynamic Graph Definition
24+
25+
* Labels
26+
* Hierarchical naming construct used for uniquely identifying items within a build namespace
27+
* Artifacts
28+
* Named items that can be used and produced within and from a build
29+
* Artifacts have concrete types:
30+
* File
31+
* Directory/File-tree
32+
* Blob
33+
* Dictionary
34+
* List
35+
* Can be stored “inline”, or represented as CAS objects
36+
* Rules
37+
* A named individual element of computation that can be configured by the engine to produce one or more artifacts
38+
* Can be abstract, producing a generic set of artifacts that may depend upon details of a configuration context
39+
* Rule Providers
40+
* Own specific rule and artifact label prefixes
41+
* Responsible for constructing rule instances when requested by the engine
42+
* Tasks
43+
* Concrete in-progress computations tracked by the engine that may request inputs, actions, and ultimately produces output artifacts
44+
* Constructed by rules given a specific configuration context by the engine
45+
* Intrinsically representable as functional state machine with serializable task contexts
46+
* Can asynchronously request inputs
47+
* Actions
48+
* Discrete async/parallelizable computations with defined inputs and outputs
49+
* Properties may include
50+
* Functional, cache-able
51+
* Distributable
52+
* Require a specified computation environment
53+
* Build
54+
* Outstanding, in-progress request to produce a specific artifact from the overall build namespace
55+
56+
[Image: docs/tritium.jpg]
57+
58+
### Engine Components
59+
60+
* Action Cache
61+
* Persistent data structure for memoizing task state transitions and action computations
62+
* Executor
63+
* Interface for performing action computations
64+
* Asynchronous, actions may be remote, distributed
65+
66+
## Graph Scale, Engine Performance
67+
68+
Overall goal is to support large operating system scale, multi-project build graphs where fine-grained (individual compile actions or smaller) incremental computation can be performed across thousands of high level projects.
69+
70+
Expected Strategies
71+
72+
* Recursive engine coordination
73+
* Entire sub-graphs can be represented as nodes within a parent graph
74+
* Own specific artifact/rule prefixes within the larger graph
75+
* Can request artifacts outside of the current graph for resolution by parent graph
76+
* Whole engine state that can be serialized and cached for incremental resumption based on requested external inputs
77+
* Sub-graphs/engines that can, under certain circumstances be distributed for remote execution
78+
* Engine control of task/action execution parallelism and priority
79+
* Separation of engine threads from networking
80+
* Graph state introspection
81+
* Defined, serializable, cache-able execution state transitions
82+
* Graph-aware task scheduling
83+
* Resume-able Actions
84+
* Defined API for reconnecting/de-duplicating in-flight action computations within the executor
85+
86+
## Build Graph Unification
87+
88+
The existing ecosystem of llbuild clients have well-tested, reliable implementations of build rules that cover low-level project definitions and higher level multi-project OS relationships. As much as practical, the goal is to provide an incremental path for these existing clients. Although the major benefits in expressivity come from adoption of the new API constructs, providing adapters should enable a smoother on-ramp for existing services and offer the potential for earlier interoperability.
89+
90+
91+
## Implementation Strategy
92+
93+
* Evolve existing llbuild graph execution engine core
94+
* Evolve llbuild2 protobuf data structures for engine protocol and state
95+
* Provide fully Swift-async native API interface for clients
96+
97+
98+
What exists today
99+
100+
* Swift XCTest exercising core engine functionality
101+
* First pass protobuf specs representing engine and task state
102+
* Integration of data structures with the graph execution core
103+
* Working basic, multi-node graph computation using rules, providers that instantiate tasks and produce artifacts
104+
* Bootstrap initialization support enabling dynamic provider registration
105+
106+
Next steps
107+
108+
* Action cache integration
109+
* Executor integration
110+
111+
Code Structure
112+
113+
* Framework Headers
114+
* products/Tritium
115+
* Framework Sources
116+
* src/tritium
117+
* Unit Tests
118+
* unittests/TritiumCoreTests

‎docs/tritium.jpg

205 KB
Loading

‎llbuild.xcodeproj/project.pbxproj

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1622,6 +1622,7 @@
16221622
406A05012162815000EBA895 /* Dockerfile-18.04 */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text; path = "Dockerfile-18.04"; sourceTree = "<group>"; };
16231623
406A05022162A20800EBA895 /* check-all */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text.script.sh; path = "check-all"; sourceTree = "<group>"; };
16241624
406A05032162A5A200EBA895 /* build-and-test */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text.script.sh; path = "build-and-test"; sourceTree = "<group>"; };
1625+
406CE2742C8FA9B30008A97B /* README.tritium.md */ = {isa = PBXFileReference; lastKnownFileType = net.daringfireball.markdown; path = README.tritium.md; sourceTree = "<group>"; };
16251626
4072A3172C51977000B68221 /* ProtocolBuffers_iOS.xcodeproj */ = {isa = PBXFileReference; lastKnownFileType = "wrapper.pb-project"; path = ProtocolBuffers_iOS.xcodeproj; sourceTree = "<group>"; };
16261627
4072A3182C51977000B68221 /* ProtocolBuffers_OSX.xcodeproj */ = {isa = PBXFileReference; lastKnownFileType = "wrapper.pb-project"; path = ProtocolBuffers_OSX.xcodeproj; sourceTree = "<group>"; };
16271628
4072A3192C51977000B68221 /* ProtocolBuffers_tvOS.xcodeproj */ = {isa = PBXFileReference; lastKnownFileType = "wrapper.pb-project"; path = ProtocolBuffers_tvOS.xcodeproj; sourceTree = "<group>"; };
@@ -8959,6 +8960,7 @@
89598960
1484D1DA2094509E00D3830F /* LICENSE.txt */,
89608961
406383D42C76FB5100017979 /* Makefile */,
89618962
E1A223FC19F990E60059043E /* README.md */,
8963+
406CE2742C8FA9B30008A97B /* README.tritium.md */,
89628964
1484D1DB209450A600D3830F /* Vagrantfile */,
89638965
E1A223F219F98F1C0059043E /* Products */,
89648966
);

0 commit comments

Comments
 (0)