Adding incremental Frontend Server-compatible DDC builders + supporting infrastructure

[Markzipan]

Glues FES into build_runner, which is our first step towards hot reload + build_runner.
The high level workflow is:

1. A persistently running frontend server is initialized once when a build starts.
2. build_runner requests JS files based on locally modified/generated dart files (as usual). Builders that collect meta-information about DDC modules also - as a side effect - record the main app entrypoint and any locally modified files.
3. When a JS file is requested, the frontend server receives recompile requests via a proxy queue (to maintain communication order with the frontend server).
4. The frontend server processes compilation requests and serves compiled JS files back to build_runner (hot-reload ready).

Major changes:

• Adds a DdcFrontendServerBuilder to our set of DDC builders (enabled via the web-hot-reload config). This builder keeps a PersistentFrontendServer instance alive across rebuilds. Compile/recompile requests are queued via a FrontendServerProxyDriver resource.
• Uses scratch_space to record both 1) the main app entrypoint and 2) updated local files from the entrypoint_marker builder and the module_builder builder respectively. These are side effects that break certain stateful 'guarantees' of standard build_runner execution. The entrypoint_marker builder runs before any of the downstream DDC builders and finds the web entrypoint, as Frontend Server must receive the same entrypoint on every compilation request.
• Requires that strongly connected components in both the frontend server and build_runner be disabled.

Test changes:

• Extends build_test to permit incremental builds. This involves passing the asset graph + asset reader/writer across build results and only performing cleanup operations after a series of rebuilds.
• build_test doesn't support runs_before and other ordering rules in build.yaml, so the above changes allows a kind of imperative ordering, which is important for testing entrypoint_marker.

Minor changes:

• Added a flag to disable strongly connected components in build_web_compilers (implemented using raw ddc meta-modules over clean ddc meta-modules + enforcing fine module aggregation).
• Added disposal logic to scratch_space so that rebuilds only retain modified files.
• Updated scratch_space package_config.json specs (packageUri and rootUri). The previous values didn't seem to make sense to me, but I'm also not familiar with how that's standardized in scratch_space.
• Added file and uuid deps to build_modules.
• Moved around some helper functions.
• Ported some naming functions from the DDC runtime.

Currently doesn't support live-reloading (functionality appears to have been broken a while ago). This'll be added in an upcoming change and permit webdev-like auto-hot-reload on save (on top of manual).

Enable this by adding the following to a project's build.yaml:

global_options:
  build_web_compilers|sdk_js:
    options:
      web-hot-reload: true
  build_web_compilers|entrypoint:
    options:
      web-hot-reload: true
  build_web_compilers|ddc:
    options:
      web-hot-reload: true
  build_web_compilers|ddc_modules:
    options:
      web-hot-reload: true

[github-actions]

PR Health

Changelog Entry ❗

Package	Changed Files
package:build_modules	build_modules/lib/build_modules.dart build_modules/lib/src/common.dart build_modules/lib/src/ddc_names.dart build_modules/lib/src/frontend_server_driver.dart build_modules/lib/src/kernel_builder.dart build_modules/lib/src/module_builder.dart build_modules/lib/src/modules.dart build_modules/lib/src/scratch_space.dart build_modules/lib/src/workers.dart build_modules/pubspec.yaml
package:build_runner	build_runner/lib/src/constants.dart
package:build_test	build_test/lib/src/internal_test_reader_writer.dart build_test/lib/src/test_builder.dart build_test/lib/src/test_reader_writer.dart
package:build_web_compilers	build_web_compilers/lib/build_web_compilers.dart build_web_compilers/lib/builders.dart build_web_compilers/lib/src/common.dart build_web_compilers/lib/src/ddc_frontend_server_builder.dart build_web_compilers/lib/src/dev_compiler_bootstrap.dart build_web_compilers/lib/src/dev_compiler_builder.dart build_web_compilers/lib/src/sdk_js_compile_builder.dart build_web_compilers/lib/src/sdk_js_copy_builder.dart build_web_compilers/lib/src/web_entrypoint_builder.dart build_web_compilers/lib/src/web_entrypoint_marker_builder.dart build_web_compilers/pubspec.yaml
package:scratch_space	scratch_space/lib/src/scratch_space.dart

Changes to files need to be accounted for in their respective changelogs.

This check can be disabled by tagging the PR with skip-changelog-check.

[davidmorgan]

Thanks!

This will take me a little work to digest :) I guess I can get to it in 1-2 days.

It definitely makes sense to expand support for testing. The details are hard :) ... in particular, I would like to avoid exposing any build_runner internals, including AssetGraph or how it's detecting file changes for follow-on builds.

So I will see if I can come up with a slightly differently suggestion.

Currently testBuilder uses fake build configuration and that's quite awkward; it starts to be weird when you talk about follow-on builds because there is no clear line for what should stay the same in the fake configuration and what's allowed to change.

One possibility would be to go all the way to real builds, I recently added a new type of integration test that I'm pretty happy with

build/build_runner/test/integration_tests/watch_command_test.dart

Line 1 in 17e38a6

// Copyright (c) 2025, the Dart project authors. Please see the AUTHORS file

I'll give it some thought :)

[biggs0125]

Requires that strongly connected components in both the frontend server and build_runner be disabled.
Can you say more about the affect this has on the structure of compilations? Are we doing more work by turning off SCCs? My assumption was that SCCs allowed us to limit the scope of an invalidation.

[Markzipan]

@davidmorgan Thanks! It's a huge change, so please take your time. I wasn't able to configure it in a prettier way since I'm flying out for two weeks.

I'm highly unopinionated wrt testing, but it was definitely a struggle to get testBuilders to cooperate with rebuilds. Please send any followup recs for better ways to represent this test. The important bits to me are: 1) respecting builder order (as per build.yaml) since that matters now, 2) being able to run rebuilds vs new builds (or determine build boundaries), 3) updating files passed to the builders.

@biggs0125 The SCCs here operate on a the import-graph level. DDC's old module system requires that cyclic dependencies be 'unified' into a single module. Both the Frontend Server and build_runner doing this non-deterministically might cause a <--> b to be merged into a in build_runner and b in the Frontend Server, which gives us "import not found" errors.

With SCCs enabled, a.dart would invoke a single builder on module(a, b). Disabling them, two builders, module(a) and module(b), are invoked. The net effect of this on some apps I've tested hasn't been large, as most modules don't end up in an SCC anyway. The largest ramifications would be a hypothetical gigantic app that wants to bundle smaller JS files. But that would ideally be specified statically, so we'd be able to pass it consistently to both build_runner and the Frontend Server. The particularly annoying bit about SCCs is that both systems maintain their own independent algorithm without any configurability. I alternatively could've 1) poked a hole into the Frontend Server to allow per-compile module-to-lib maps and 2) had build_runner serialize its module-to-lib maps and send that per-compile to the Frontend Server. But I figured that was too much of a mess.

Re: failures. I think some tests are failing due to API changes without pinning new versions? I'll add those (maybe in a separate PR) if the current approach looks sound.

[davidmorgan]

This generally looks good, a few comments re: scratch_space.

Re: testing :)

You sort of don't need to pass assetGraph, it's serialized to the readerWriter and the next build will read it if it's there.

But, the next build will discard the asset graph if any builders changed, so you have to pass the exact same builders each time, which means you will get a full build the first time you call it anyway.

You commented with build.yaml ordering is not respected, and that's true. Adding workarounds for that is not too pretty, it's getting very close to a real build but now with quite a lot of configuration that doesn't exactly match a real build.

Would you be up for trying a different way of testing that is a real build?

I added tests recently that make it easy to set up some packages, run build_runner for real and check the output. Since you want persistent processes, watch mode would make sense, so something like:

build/build_runner/test/integration_tests/watch_command_test.dart

Line 1 in 9f07c7e

// Copyright (c) 2025, the Dart project authors. Please see the AUTHORS file

There is an example web build using the same test infra here

https://github.com/dart-lang/build/blob/master/build_runner/test/integration_tests/web_compilers_test.dart

The test infra is currently internal to build_runner, mainly build_runner/test/common/build_runner_tester.dart, if it looks useful let's just hack around that for now and I'll work out how best to clean it up, I guess adding to build_test is one option.

[davidmorgan]

I don't see where this is used, remove?

[davidmorgan]

This doesn't look particularly related to ScratchSpace, which I see is used by a few other builders.

I guess the problem here is of passing configuration between builders?

Maybe a new Resource class for that? It could also go in scratch_space.

[davidmorgan]

Can this be private and expose an Iterable? Maybe a read-only view if Set is needed?

To avoid any possibility of corrupted shared state if a builder thinks it's fun to write to it :)

[davidmorgan]

I suggest making this and the comment for changedFilesInBuild more precise, e.g. here

Assets that have changed since the last time they were seen by ensureAssets are added to changedFilesInBuild.

And for changedFilesInBuild, maybe: Assets that changed between calls to ensureAssets. Cleared at the end of every build.

[davidmorgan]

I think scratch_space needs a new version number and then anything that needs what you're adding here should get a min dep onto that version?