Adding incremental Frontend Server-compatible DDC builders + supporting infrastructure #4240

Markzipan · 2025-10-03T08:23:18Z

Glues FES into build_runner, which is our first step towards hot reload + build_runner.
The high level workflow is:

A persistently running frontend server is initialized once when a build starts.
build_runner requests JS files based on locally modified/generated dart files (as usual). Builders that collect meta-information about DDC modules also - as a side effect - record the main app entrypoint and any locally modified files.
When a JS file is requested, the frontend server receives recompile requests via a proxy queue (to maintain communication order with the frontend server).
The frontend server processes compilation requests and serves compiled JS files back to build_runner (hot-reload ready).

Major changes:

Adds a DdcFrontendServerBuilder to our set of DDC builders (enabled via the web-hot-reload config). This builder keeps a PersistentFrontendServer instance alive across rebuilds. Compile/recompile requests are queued via a FrontendServerProxyDriver resource.
Uses scratch_space to record both 1) the main app entrypoint and 2) updated local files from the entrypoint_marker builder and the module_builder builder respectively. These are side effects that break certain stateful 'guarantees' of standard build_runner execution. The entrypoint_marker builder runs before any of the downstream DDC builders and finds the web entrypoint, as Frontend Server must receive the same entrypoint on every compilation request.
Requires that strongly connected components in both the frontend server and build_runner be disabled.

Test changes:

Extends build_test to permit incremental builds. This involves passing the asset graph + asset reader/writer across build results and only performing cleanup operations after a series of rebuilds.
build_test doesn't support runs_before and other ordering rules in build.yaml, so the above changes allows a kind of imperative ordering, which is important for testing entrypoint_marker.

Minor changes:

Added a flag to disable strongly connected components in build_web_compilers (implemented using raw ddc meta-modules over clean ddc meta-modules + enforcing fine module aggregation).
Added disposal logic to scratch_space so that rebuilds only retain modified files.
Updated scratch_space package_config.json specs (packageUri and rootUri). The previous values didn't seem to make sense to me, but I'm also not familiar with how that's standardized in scratch_space.
Added file and uuid deps to build_modules.
Moved around some helper functions.
Ported some naming functions from the DDC runtime.

Currently doesn't support live-reloading (functionality appears to have been broken a while ago). This'll be added in an upcoming change and permit webdev-like auto-hot-reload on save (on top of manual).

Enable this by adding the following to a project's build.yaml:

global_options:
  build_web_compilers|sdk_js:
    options:
      web-hot-reload: true
  build_web_compilers|entrypoint:
    options:
      web-hot-reload: true
  build_web_compilers|ddc:
    options:
      web-hot-reload: true
  build_web_compilers|ddc_modules:
    options:
      web-hot-reload: true

…r fine modules for hot reload

github-actions · 2025-10-03T08:25:28Z

PR Health

Changelog Entry ✔️

Package	Changed Files

Changes to files need to be accounted for in their respective changelogs.

This check can be disabled by tagging the PR with skip-changelog-check.

davidmorgan · 2025-10-03T14:35:20Z

Thanks!

This will take me a little work to digest :) I guess I can get to it in 1-2 days.

It definitely makes sense to expand support for testing. The details are hard :) ... in particular, I would like to avoid exposing any build_runner internals, including AssetGraph or how it's detecting file changes for follow-on builds.

So I will see if I can come up with a slightly differently suggestion.

Currently testBuilder uses fake build configuration and that's quite awkward; it starts to be weird when you talk about follow-on builds because there is no clear line for what should stay the same in the fake configuration and what's allowed to change.

One possibility would be to go all the way to real builds, I recently added a new type of integration test that I'm pretty happy with

build/build_runner/test/integration_tests/watch_command_test.dart

Line 1 in 17e38a6

I'll give it some thought :)

biggs0125 · 2025-10-03T17:21:06Z

Requires that strongly connected components in both the frontend server and build_runner be disabled.

Can you say more about the affect this has on the structure of compilations? Are we doing more work by turning off SCCs? My assumption was that SCCs allowed us to limit the scope of an invalidation.

Markzipan · 2025-10-03T17:53:18Z

@davidmorgan Thanks! It's a huge change, so please take your time. I wasn't able to configure it in a prettier way since I'm flying out for two weeks.

I'm highly unopinionated wrt testing, but it was definitely a struggle to get testBuilders to cooperate with rebuilds. Please send any followup recs for better ways to represent this test. The important bits to me are: 1) respecting builder order (as per build.yaml) since that matters now, 2) being able to run rebuilds vs new builds (or determine build boundaries), 3) updating files passed to the builders.

@biggs0125 The SCCs here operate on a the import-graph level. DDC's old module system requires that cyclic dependencies be 'unified' into a single module. Both the Frontend Server and build_runner doing this non-deterministically might cause a <--> b to be merged into a in build_runner and b in the Frontend Server, which gives us "import not found" errors.

With SCCs enabled, a.dart would invoke a single builder on module(a, b). Disabling them, two builders, module(a) and module(b), are invoked. The net effect of this on some apps I've tested hasn't been large, as most modules don't end up in an SCC anyway. The largest ramifications would be a hypothetical gigantic app that wants to bundle smaller JS files. But that would ideally be specified statically, so we'd be able to pass it consistently to both build_runner and the Frontend Server. The particularly annoying bit about SCCs is that both systems maintain their own independent algorithm without any configurability. I alternatively could've 1) poked a hole into the Frontend Server to allow per-compile module-to-lib maps and 2) had build_runner serialize its module-to-lib maps and send that per-compile to the Frontend Server. But I figured that was too much of a mess.

Re: failures. I think some tests are failing due to API changes without pinning new versions? I'll add those (maybe in a separate PR) if the current approach looks sound.

davidmorgan

This generally looks good, a few comments re: scratch_space.

Re: testing :)

You sort of don't need to pass assetGraph, it's serialized to the readerWriter and the next build will read it if it's there.

But, the next build will discard the asset graph if any builders changed, so you have to pass the exact same builders each time, which means you will get a full build the first time you call it anyway.

You commented with build.yaml ordering is not respected, and that's true. Adding workarounds for that is not too pretty, it's getting very close to a real build but now with quite a lot of configuration that doesn't exactly match a real build.

Would you be up for trying a different way of testing that is a real build?

I added tests recently that make it easy to set up some packages, run build_runner for real and check the output. Since you want persistent processes, watch mode would make sense, so something like:

build/build_runner/test/integration_tests/watch_command_test.dart

Line 1 in 9f07c7e

There is an example web build using the same test infra here

https://github.com/dart-lang/build/blob/master/build_runner/test/integration_tests/web_compilers_test.dart

The test infra is currently internal to build_runner, mainly build_runner/test/common/build_runner_tester.dart, if it looks useful let's just hack around that for now and I'll work out how best to clean it up, I guess adding to build_test is one option.

build_runner/lib/src/constants.dart

scratch_space/lib/src/scratch_space.dart

…ing scratch space ver

github-actions · 2025-10-24T18:42:42Z

Package publishing

Package	Version	Status	Publish tag (post-merge)
package:build	4.0.2	already published at pub.dev
package:build_config	1.2.0	already published at pub.dev
package:build_daemon	4.1.0	already published at pub.dev
package:build_modules	5.1.0	ready to publish	`build_modules-v5.1.0`
package:build_runner	2.10.1	already published at pub.dev
package:build_test	3.5.1	already published at pub.dev
package:build_web_compilers	4.4.0	ready to publish	`build_web_compilers-v4.4.0`
package:scratch_space	1.2.0	ready to publish	`scratch_space-v1.2.0`

Documentation at https://github.com/dart-lang/ecosystem/wiki/Publishing-automation.

Markzipan · 2025-10-24T18:48:41Z

@davidmorgan Wow, the tests in build/build_runner/test/integration_tests are perfect for this. I've replaced the build_web_compilers FES driver test with additional web_compilers_test.dart tests, added a separate resource for FES-specific state, and bumped scratch_space's ver.

Thanks for the review - back from vacation!

nshahan · 2025-10-27T22:43:54Z

build_modules/lib/src/frontend_server_driver.dart

+
+/// Bundles information associated with a DDC library.
+class LibraryInfo {
+  final String moduleName;


What is the moduleName in the new module system?

nshahan · 2025-10-27T22:50:24Z

build_modules/lib/src/ddc_names.dart

+/// Logic in this file must be synchronized with their namesakes in DDC at:
+/// pkg/dev_compiler/lib/src/compiler/js_names.dart
+
+bool isSdkInternalRuntimeUri(Uri importUri) {


OK to make private?

nshahan · 2025-10-27T22:55:42Z

build_modules/lib/src/ddc_names.dart

+  return importUri.isScheme('dart') && importUri.path == '_runtime';
+}
+
+String libraryUriToJsIdentifier(Uri importUri) {


What is the JsIdentifier used for? I see it being used later in this change for moduleName but I'm getting confused from there.

nshahan · 2025-10-27T23:06:24Z

build_modules/lib/src/frontend_server_driver.dart

+    if (compilerOutput == null) {
+      throw Exception('Frontend Server failed to recompile $entrypoint');
+    }
+    if (compilerOutput.errorCount != 0 || compilerOutput.errorMessage != null) {


Do the reload rejection errors flow here and do these errors get handled somewhere else? We want to make sure the user can recover from them, make some edits and reload again right?

nshahan · 2025-10-27T23:11:52Z

build_modules/lib/src/frontend_server_driver.dart

+// BSD-style license that can be found in the LICENSE file.
+
+// ignore_for_file: constant_identifier_names
+


At some point we need to unify all of the implementations of this type of library right? In creating this, have you seen any common parts we can factor out and reuse?

nshahan · 2025-10-27T23:30:07Z

build_web_compilers/lib/src/common.dart

+}
+
+/// The module name according to ddc for [jsId] which represents the real js
+/// module file.


I can't quite make sense of what the module name according to DDC means. Is this the path to the Javascript file?

nshahan · 2025-10-27T23:31:10Z

build_web_compilers/lib/src/common.dart

+  return jsPath.substring(0, jsPath.length - jsModuleExtension.length);
+}
+
+String ddcLibraryId(AssetId jsId) {


And this is the Dart library import uri? I'm nervous about recreating this from whatever the jsId is. Where does that come from?

davidmorgan · 2025-10-28T15:04:10Z

It looks like you made some changes in response to my comments but I don't see any responses to the comments so I'm not sure if you're done--please let me know if/when I should take another look. Thanks!

davidmorgan

Welcome back!

Looking good so far.

There are some CI failures, could you take a look at those please?

davidmorgan · 2025-10-27T09:22:35Z

build_test/lib/src/internal_test_reader_writer.dart

  );
+
+  @override
+  void reset() {


I think you don't need this any more?

(I think you already removed it, so I guess just close...)

davidmorgan · 2025-10-27T09:22:46Z

build_test/lib/src/test_reader_writer.dart

  /// Deletes [id] from the [TestReaderWriter] in-memory filesystem.
  void delete(AssetId id);
+
+  /// Resets state in this [TestReaderWriter] between rebuilds.


I think you don't need this any more?

davidmorgan · 2025-10-27T09:26:38Z

build_runner/test/integration_tests/web_compilers_test.dart

-  });
+  }, timeout: defaultTimeout);
+
+  test('DDC compiled with the Frontend Server', () async {


Could you please add a TODO for me?

// TODO(davidmorgan): the remaining tests are integration tests of // the web compilers themselves, support testing like this outside the // `build_runner` package.

davidmorgan · 2025-10-27T09:40:11Z

build_runner/test/common/build_runner_tester.dart

  /// defaults to [BuildLog.failurePattern] so that `expect` will stop if the
  /// process reports a build failure.
  ///
+  /// if [expectFailure] is set, then both [pattern] and [failOn] must be


Hmmmm multiple expectations is a bit awkward, I wonder if there's a nicer way.

Would it work to accumulate lines and return them, then the second expectation can be checked on the result?

Looking at how this method is used, I think maybe something like this:

most callers are not using the result, so how about the base method is Future<void> expect.

a few callers are using the single matching line, so how about Future<String> expectAndGetLine for that

and then I think you can do what you want with the full accumulated output until the match, how about Future<List<String>> expectAndGetBlock

?

davidmorgan · 2025-10-28T15:17:08Z

It looks like you made some changes in response to my comments but I don't see any responses to the comments so I'm not sure if you're done--please let me know if/when I should take another look. Thanks!

Whoops, looks like I had unpublished comments that you addressed by coincidence. Sorry about that! Now published.

Markzipan added 25 commits August 28, 2025 00:33

Initial support for DDC + FES with build_runner

cd52372

Adding hot reload flag consistency checks and enforcing module builde…

af089bf

…r fine modules for hot reload

Moving sources to generated scratch dirs

01e2411

Merging

414fc95

Only recompiling and invalidating requested source files

fa6594f

Merge branch 'master' of github.com:dart-lang/build

09a0486

Sample change with a hacky shared repo + digests impl comparisons

efec641

Moving shared state to scratch space

aae24a2

Merging

7aba038

addressing analysis errors

51f1e0d

additional file cleanup

5d309ca

Adding build_modules tests

9a37369

Adding build_modules tests

b2bab91

Merging

cba3f6b

Merge branch 'master' of github.com:dart-lang/build

244dcd4

Adding reset to frontend server driver

12a8af7

Adding a builder for labeling the app entrypoint.

b9e30b3

Removing the entrypoint from builder options

309b11f

Fixing error handling

625a922

Fixing some shutdown race conditions

150a42a

Adding reader-writer resets

c48e015

Adding support for incremental build testing.

9349807

Adding DDC frontend server builder tests

f108cb6

Updating entrypoint marker builder

75f3061

Merge branch 'master' of github.com:dart-lang/build

e248938

Markzipan requested review from biggs0125, davidmorgan, jakemac53 and nshahan October 3, 2025 08:23

davidmorgan reviewed Oct 6, 2025

View reviewed changes

Markzipan added 4 commits October 24, 2025 10:00

Moving ddc fes build_web_compilers tests to web_compilers_test.dart

020956a

Updating scratch_space version and deps

3ceb26f

Merge remote-tracking branch 'refs/remotes/origin/master'

fb5c7ee

creating a frontend server specific resource for holding state + bump…

b235dcd

…ing scratch space ver

Markzipan added 9 commits October 27, 2025 13:42

Adding flag to entrypoint market to avoid extraneous work

d914dfc

extending timeout for daemon test

eeebe86

adding builder options check

0b61746

fixing bootstrap tests

2ebfc54

reverting unused readerwriter updates

0ca833b

bumping versions

6fe34d3

Updating changelog and bumping versions

2635ce3

Merging

ca4156c

fixing ver nums

f61ce53

nshahan reviewed Oct 27, 2025

View reviewed changes

davidmorgan reviewed Oct 28, 2025

View reviewed changes

		// BSD-style license that can be found in the LICENSE file.

		// ignore_for_file: constant_identifier_names

Uh oh!

Adding incremental Frontend Server-compatible DDC builders + supporting infrastructure #4240

Are you sure you want to change the base?

Adding incremental Frontend Server-compatible DDC builders + supporting infrastructure #4240

Conversation

Markzipan commented Oct 3, 2025

Uh oh!

github-actions bot commented Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Health

Uh oh!

davidmorgan commented Oct 3, 2025

Uh oh!

biggs0125 commented Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Markzipan commented Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davidmorgan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Package publishing

Uh oh!

Markzipan commented Oct 24, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidmorgan commented Oct 28, 2025

Uh oh!

davidmorgan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidmorgan commented Oct 28, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions bot commented Oct 3, 2025 •

edited

Loading

biggs0125 commented Oct 3, 2025 •

edited

Loading

Markzipan commented Oct 3, 2025 •

edited

Loading

github-actions bot commented Oct 24, 2025 •

edited

Loading