You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: rfcs/0017-incremental-build.md
+49-21Lines changed: 49 additions & 21 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -46,11 +46,11 @@ This shall enable the following workflow:
46
46
47
47
1.**Action:** Build is started
48
48
1. Task A, Task B and Task C are executed in sequence, writing their results into individual writer stages.
49
-
1.*Task outputs are written to a content-addressable store and the `cache-info.json` metadata is serialized to disk.*
49
+
1.*Task outputs are written to a content-addressable store and the `build-manifest.json` metadata is serialized to disk.*
50
50
1. Build finishes and the resources of all writer stages and the source reader are combined and written into the target output directory.
51
51
* Resources present in later writer stages (and higher versions) are preferred over competing resources with the same path.
52
52
1.**Action:** A source file is modified and a new build is triggered
53
-
1.*The `cache-info.json` is read from disk, allowing the build to access cached content from the content-addressable store.*
53
+
1.*The `build-manifest.json` is read from disk, allowing the build to access cached content from the content-addressable store.*
54
54
1. The build determines which tasks need to be executed using the imported cache and information about the modified source file.
55
55
* In this example, it is determined that Task A and Task C need to be executed since they requested the modified resource in their previous execution.
56
56
1. Task A is executed. The output is written into a new **version** (v2) of the associated writer stage.
@@ -59,7 +59,7 @@ This shall enable the following workflow:
59
59
* Task A can't access v1 of its writer stage. It can only access the combined resources of all previous writer stages.
60
60
1. The `Project Build Cache` determines whether the resources produced in this latest execution of Task A are relevant for Task B. If yes, the content of those resources is compared to the cached content of the resources Task B has received during its last execution. In this example, the output of Task A is not relevant for Task B and it is skipped.
61
61
1. Task C is called and has access to both versions (v1 and v2) of the writer stage of Task A. Allowing it to access all resources produced in all previous executions of Task A.
62
-
1.*Task outputs are written to the content-addressable store and the `cache-info.json` is updated.*
62
+
1.*Task outputs are written to the content-addressable store and the `build-manifest.json` is updated.*
63
63
1. The build finishes. The combined resources of all writer stages and the source reader are written to the target output directory.
64
64
65
65

@@ -72,14 +72,45 @@ Every project has its own cache metadata. This allows for reuse of a project's c
72
72
73
73
The cache consists of two main parts:
74
74
1. A global **object store (the CAS)** where all file contents are stored, named by a hash of their content.
75
-
2. A per-project `cache-info.json` file which acts as a lightweight **metadata index**, mapping logical file paths to their content hashes in the object store.
75
+
2. A per-project `build-manifest.json` file which acts as a lightweight **metadata index**, mapping logical file paths to their content hashes in the object store.
The concept of a `build-manifest.json` has already been explored in [RFC 0011 Reuse Build Results](https://github.com/SAP/ui5-tooling/pull/612) and found an implementation for consumption of pre-built UI5 framework libraries in [UI5 3.0](https://github.com/UI5/cli/pull/612). The **buildManifest** object is based on that concept.
141
+
113
142
**cacheKey**
114
143
115
144
The cache key can be used to identify the cache for a project. It shall be based on the project's name and package version, as well as a SHA256 hash compiled from the following information:
@@ -141,25 +170,24 @@ The directory structure is flat and efficient. A global `cas/` directory stores
141
170
│ ├── d41d8cd98f00b204e9899998ecf8427e (Content of another file)
Besides the `cas` directory, each project has its own directory named after its cache key. This directory contains only the `cache-info.json` file for that project.
180
+
The `cas` directory contains files named by their SHA256 content hash. Each file contains the raw content of a resource produced during a build. Ideally a library like [`cacache`](https://www.npmjs.com/package/cacache) should be used to manage the content-addressable store.
153
181
154
-
All unique file contents from all projects and their builds are stored **once** in the global `cas` directory, named by their content hash.
182
+
The `manifests` directory contains one build manifest file per project build cache. The filename is derived from the project's namespace, version and cache key.
155
183
156
184

157
185
158
186
### Cache Import
159
187
160
188
Before building a project, UI5 Tooling shall scan for a cache directory with the respective cache key and import the cache if one is found.
161
189
162
-
The import process is very fast, as it only involves reading the lightweight `cache-info.json` file to populate the `Build Task Cache` instances with their metadata. When the build process needs to access a cached resource, it uses the metadata map to find the content hash and reads the corresponding file directly from the global `cas` store.
190
+
The import process is very fast, as it only involves reading the lightweight `build-manifest.json` file to populate the `Build Task Cache` instances with their metadata. When the build process needs to access a cached resource, it uses the metadata map to find the content hash and reads the corresponding file directly from the global `cas` store.
163
191
164
192
This allows executing individual tasks and providing them with the results of all preceding tasks without the overhead of creating numerous file system readers or managing physical copies of files for each build stage.
165
193
@@ -183,7 +211,7 @@ A mechanism to purge unused cache on disk is required. The cache can grow very l
183
211
184
212
This should probably use some sort of LRU-cache to purge unused cache entries dynamically. The same mechanism could be applied to the npm artifacts downloaded by UI5 Tooling.
185
213
186
-
To avoid slowing down core commands, the purge check should run as a non-blocking process after a successful ui5 build or ui5 serve command completes. This process checks if either of the configured thresholds (age or size) has been exceeded. If so, it proceeds with the purge.
214
+
To avoid slowing down core commands, the purge check should run as a non-blocking process after a successful ui5 build or ui5 serve command completes. This process checks if either of the configured thresholds (age or size) has been exceeded. If so, it proceeds with the purge. Some of this functionality might be provided by the underlying content-addressable store library, such as cacache's internal garbage collection.
187
215
188
216
A dedicated command, such as `ui5 cache clean`, should be introduced in addition. This command allows users to manually trigger a cache purge, providing options to specify criteria such as maximum age or size for cache entries to be removed. Similarly, a command `ui5 cache verify` could be provided to check the integrity of the cache.
0 commit comments