feat: Incremental Build (reorganized commits)#1399
Draft
RandomByte wants to merge 15 commits into
Draft
Conversation
File-system primitives used by @ui5/project's incremental build system to track resource changes across build stages. - Resource: calculate integrity on clone, expose tags via getTags(), handle legacy streams without Symbol.asyncIterator, drop the fabricated inode 0 for in-memory resources, replace deprecated getStatInfo() - Adapters: add createBuffer factory in FileSystem to enable buffer reuse for integrity computation; CAS-aware fast path in FileSystem._write() to skip writes when target content matches - New MonitoredReader / MonitoredReaderWriter that report which resources were read/written per task, enabling per-task cache keys - New MonitoredResourceTagCollection that tracks tag mutations, used by the build cache to fold tags into resource hashes - New Proxy reader, plus WriterCollection unique-reader handling and ResourceTagCollection#getAllTagsForResource to support cross-stage resource tracking
Storage layer for the incremental build cache. A single SQLite database per data directory holds both content-addressable resource bodies (CAS) and per-task metadata, replacing the previous file-system layout and an earlier cacache-based attempt. - BuildCacheStorage: unified store with batch transaction support, PRAGMA tuning (WAL, busy_timeout, cache_size), batch existence checks for CAS lookups, gzip compression with a small-resource short-circuit, async compression outside transactions - Cache: high-level API used by ProjectBuildCache/StageCache - CacheManager: process-global singleton with reference counting so multiple concurrently-running builds (e.g. dev server + build) share one DB connection and clean up on the last release - utils: isResourceUnchanged() compares mtime + size + inode to detect in-place file replacement; the size check catches mtime-preserved edits
A Merkle-tree-style index over project resources so cache keys for tasks can be derived from the hashes of their inputs. - HashTree: per-project tree where each leaf hash combines the resource integrity with its tags, and each directory hash combines its children. Shallow rehashing only walks the spine of changed paths - SharedHashTree: composite tree across multiple project trees, used when a task reads from a project plus dependencies - TreeRegistry: deduplicates trees so nodes shared between source and stage indices reuse the same hash tree instances - ResourceIndex: tracks the set of resources behind a tree, with delta upsert/remove and integrity caching
RandomByte
force-pushed
the
feat/incremental-build-5
branch
3 times, most recently
from
6cc60af to
d69f307
Compare
The per-task and per-stage caching layer on top of the storage and hash-tree primitives. - ResourceRequestGraph / ResourceRequestManager: track which resources each task asks for from which projects, so a task's cache key is derived only from the inputs it actually consumed (not the whole project) and re-runs can be skipped when those inputs are unchanged - StageCache: persists post-task resource snapshots so a later run can resume from any prior stage instead of re-running upstream tasks - ProjectBuildCache: per-project facade that owns the source/stage indices, the request graph, and the lifecycle of cached entries - BuildTaskCache: per-task facade that records inputs, outputs, and the resulting cache signature
…nner Wire the build cache, hash-tree index, and request graph into the build pipeline so unchanged tasks are skipped on subsequent builds. - ProjectBuilder/TaskRunner: per-task cache lookup using the resource request graph and shared hash tree. The skip path returns prior outputs from StageCache instead of re-running the task. Source files are stored in CAS and validated after the build finishes - BuildContext / ProjectBuildContext: own the per-project build cache lifecycle, including cleanup hooks, abort-signal handling, and the ui5DataDir for cache isolation - Build definitions / TaskUtil: thread the supportsDifferentialBuilds flag through tasks so the runner knows which can use cached output; apply style transforms to the frozen source reader - ProjectResources / Stage: wrap the project's reader chain so each stage exposes a stable view that the cache can hash and snapshot - Project graph & ProjectGraph: propagate ui5DataDir through graph resolution; correct stage reader priority for ThemeLibrary/Module; recognize build manifest version 1.0 with cached tags - Specifications: add getConfig() and config plumbing needed by the cache; align type-spec tests
A long-running server on top of the differential build pipeline that watches source files and serves freshly-built output to the dev server. - BuildServer: orchestrates watch-mode rebuilds across the project graph. Owns a chokidar watcher per project, a queue that serializes rebuilds while collapsing rapid change bursts, and abort handling so an in-flight build can be cancelled when newer changes arrive - BuildReader: reader exposed to consumers (e.g. serveResources) that blocks reads until any pending rebuild has completed and surfaces build errors as read errors - WatchHandler: emits change/error events used by BuildServer; awaits initial-watch readiness, debounces rapid changes, and waits for the current build to finish before triggering the next - Treat source changes during build as retriable; prevent state corruption when changes arrive during index re-init; handle EBUSY on Windows by closing leaked SQLite handles before re-opening
Standard tasks and processors now participate in differential builds. - Tasks (replaceBuildtime/Copyright/Version, escapeNonAsciiCharacters, buildThemes, minify): return undefined for unchanged resources so the runner can record a cache-hit instead of writing identical output. Filter non-JS resources from minify - Processors (stringReplacer, nonAsciiEscaper, jsdocGenerator): match the same contract. jsdocGenerator additionally fixes JSDoc type expression parse errors that surfaced in cached runs - test/utils/buildCacheIsolation: helper that gives integration tests an isolated cache data dir so they don't interfere with each other or with the user's ~/.ui5
Wire the BuildServer into ui5 serve so requests are answered from a live, incrementally-rebuilt project tree instead of from an in-memory build performed once at startup. - server.js: instantiate BuildServer when the cache option is set, surface build errors via an error callback, destroy the BuildServer on close to prevent process hangs, log error stacks on rebuild failure - serveResources: read from the BuildReader so requests block until the current rebuild finishes; the previous in-server build path is removed
…e configuration (#1367) Previously, bundled-in middlewares like serveThemes were loaded by name through middlewareRepository. They can now be loaded as regular legacy middlewares from node_modules, removing the need for serveThemes to live inside @ui5/server. The serveThemes middleware is removed accordingly and documented in the v5 migration guide. Independent of the incremental build feature, but landed on this branch via #1367.
Surface the incremental build cache through the CLI. - build / serve: new --cache flag with Default/Force/Off modes. Serve defaults to skipping bundling for fast iteration; build keeps prior behaviour by default. CLI accepts lower-case values and validates via yargs coerce - tree: deprecate the old --cache-mode flag in favour of --cache - Maven snapshot resolver: rename CacheMode -> SnapshotCache so the --cache option (which controls the build cache) does not collide with the existing maven snapshot cache in name or docs Closes #1368. Co-authored-by: Merlin Beutlberger <m.beutlberger@sap.com>
ProjectBuild logger emits skipped / differential progress states when a task is served from cache. Console writer renders skipped projects and tasks in grey, so reused work is visually distinct from freshly-executed work.
Integration tests and fixture projects that exercise the incremental build pipeline end-to-end. - ProjectBuilder.integration.js: ~3000 lines covering applications, libraries, theme libraries, modules, components; standard tasks, custom tasks, custom bundling, custom preload configs; cross-project tag changes; dependency add/remove; file add/modify/delete; race conditions; self-contained, JSDoc, and minify builds; cache hits and invalidation - BuildServer.integration.js: ~750 lines covering watch-mode rebuilds for applications and libraries, race conditions when files change during a build, and stability across consecutive change bursts - createBuildManifest.integration.js, graph.integration.js: targeted integration tests for the manifest format and graph traversal - Fixtures: applications, components, modules, theme libraries with various dependency layouts (collection-style nested deps, library.d scenarios), plus task scripts for the race-condition, dependency- change, custom-task and tag-change cases Co-authored-by: Max Reichmann <max.reichmann@sap.com>
- package-lock.json: pulls in chokidar (BuildServer file watching); removes cacache and ssri which were used by an earlier file-based cache implementation - internal/e2e-tests/test/build.js: adapt the end-to-end build test to the new build pipeline Co-authored-by: Max Reichmann <max.reichmann@sap.com>
Repo-local Claude Code skills that document the incremental build subsystem and the @ui5/fs API surface, so future work can pick up the architecture without re-reading the whole branch. - .claude/skills/incremental-build/SKILL.md, architecture.md, and performance-investigation.md cover the cache layout, hash tree, request graph, BuildServer flow, and perf instrumentation - .claude/skills/ui5-fs/SKILL.md and architecture.md cover Resource, readers/writers, adapters, monitoring, tagging - AGENTS.md / CLAUDE.md include the skill triggers and project structure
… bug The "Serve library" test could flake when an FS event from the fixture copy fired after chokidar's ready signal, aborting the initial build and triggering a retry. The retry hit in-memory StageCache entries left behind by the aborted attempt and emitted spurious task-skip events, breaking the first-request assertion. Decouple fixture initialization from serveProject. FixtureTester now exposes a static create() factory that runs the rmrf + fs.cp up front, and serveProject only constructs the graph and starts the build server. The graph computation between cp and watcher attach gives the OS event queue time to flush, so the watcher does not see late events from the copy. Add two test.serial.failing tests that deterministically reproduce the StageCache-on-abort bug by triggering an abort at a known task boundary via _projectResourceChanged. The variants cover early (after task 4) and late (after task 7) abort points to show the leak scales with how far the aborted attempt progressed.
RandomByte
force-pushed
the
feat/incremental-build-5
branch
from
d69f307 to
4314cdb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Same content as #1267 but with better organized commits