Add find_additional_modules option to support partial bundling with externals (#3726)

* typo

* move stuff

* remove unnecessary typing

* remove unnecessary `active` property from middleware loader config

* more refactorings

* rename traverseModuleGraph as this is not what it does

* refactor to simplify findAdditionalModules()

* Rename the functions build helpers to make it more clear what their role is.

* invert module collector config - no longer need to return it as it is passed in

* implement module finding in normal bundling if the find_additional_modules flag is on

* allow `__STATIC_CONTENT_MANIFEST` to be imported from any module

* test: improve no-bundle-import tests

The no-bundle-import tests would actually pass if Wrangler actually bundled the code,
since it would identify and inline all the dynamic
imports that were being tested.

This change adds a test that would not pass if we are not capturing the additional modules.

It also moves the declaration of no-bundle to the wrangler.toml to ensure that normal use of `wrangler dev` will get this behavior.

* refactor: convert additional file finding to a generator function

This has a few of benefits:
- we don't risk creating too many file reads in a single go (using up all the available file handles); previously file reads were all done in parallel
- we don't read the files over and over for each rule
- the code is a bit easier to follow as we don't have to create `Promise.all()` objects

* fix: ensure that additional modules appear in the out-dir

When using `find_additional_modules` (or `no_bundle`) we find files that
will be uploaded to be deployed alongside the Worker.

Previously, if an `outDir` was specified, only the Worker code was output
to this directory. Now all additional modules are also output there too.

* pnpm fixups

* test: ignore failure to remove tmp dir on Windows

* test: do not show output in d1 time-travel tests

* refactor: consolidate writing additional modules

* test: add CommonJS lazy import to additional-modules fixture

* test: rename spec to test

* Add debug logging when writing additional modules

* Test additional module failure case and fix message typo

* Display build warnings before updating the bundle

---------

Co-authored-by: bcoll <[email protected]>

Author: petebacondarwin

.changeset/blue-donkeys-pretend.md

@@ -0,0 +1,11 @@
+---
+"wrangler": patch
+---
+
+fix: ensure that additional modules appear in the out-dir
+
+When using `find_additional_modules` (or `no_bundle`) we find files that
+will be uploaded to be deployed alongside the Worker.
+
+Previously, if an `outDir` was specified, only the Worker code was output
+to this directory. Now all additional modules are also output there too.

.changeset/clever-turkeys-leave.md

@@ -0,0 +1,20 @@
+---
+"wrangler": minor
+---
+
+feat: support partial bundling with configurable external modules
+
+Setting `find_additional_modules` to `true` in your configuration file will now instruct Wrangler to look for files in
+your `base_dir` that match your configured `rules`, and deploy them as unbundled, external modules with your Worker.
+`base_dir` defaults to the directory containing your `main` entrypoint.
+
+Wrangler can operate in two modes: the default bundling mode and `--no-bundle` mode. In bundling mode, dynamic imports
+(e.g. `await import("./large-dep.mjs")`) would be bundled into your entrypoint, making lazy loading less effective.
+Additionally, variable dynamic imports (e.g. `` await import(`./lang/${language}.mjs`) ``) would always fail at runtime,
+as Wrangler would have no way of knowing which modules to upload. The `--no-bundle` mode sought to address these issues
+by disabling Wrangler's bundling entirely, and just deploying code as is. Unfortunately, this also disabled Wrangler's
+code transformations (e.g. TypeScript compilation, `--assets`, `--test-scheduled`, etc).
+
+With this change, we now additionally support _partial bundling_. Files are bundled into a single Worker entry-point file
+unless `find_additional_modules` is `true`, and the file matches one of the configured `rules`. See
+https://developers.cloudflare.com/workers/wrangler/bundling/ for more details and examples.

.changeset/hot-deers-return.md

@@ -0,0 +1,8 @@
+---
+"wrangler": patch
+---
+
+fix: allow `__STATIC_CONTENT_MANIFEST` module to be imported anywhere
+
+`__STATIC_CONTENT_MANIFEST` can now be imported in subdirectories when
+`--no-bundle` or `find_additional_modules` are enabled.

.prettierignore

@@ -7,6 +7,9 @@ packages/wrangler/CHANGELOG.md
 packages/jest-environment-wrangler/CHANGELOG.md
 packages/wranglerjs-compat-webpack-plugin/lib
 packages/wrangler-devtools/built-devtools
+packages/wrangler-devtools/.cipd
+packages/wrangler-devtools/depot
+packages/wrangler-devtools/devtools-frontend
 packages/edge-preview-authenticated-proxy/package.json
 packages/format-errors/package.json
 packages/**/dist/**

fixtures/additional-modules/package.json

@@ -0,0 +1,21 @@
+{
+	"name": "additional-modules",
+	"version": "0.0.1",
+	"private": true,
+	"scripts": {
+		"build": "wrangler deploy --dry-run --outdir=dist",
+		"check:type": "tsc",
+		"deploy": "wrangler deploy",
+		"start": "wrangler dev",
+		"test": "vitest run",
+		"test:ci": "vitest run",
+		"test:watch": "vitest",
+		"type:tests": "tsc -p ./test/tsconfig.json"
+	},
+	"devDependencies": {
+		"@cloudflare/workers-tsconfig": "workspace:*",
+		"@cloudflare/workers-types": "^4.20230724.0",
+		"undici": "^5.9.1",
+		"wrangler": "workspace:*"
+	}
+}

fixtures/additional-modules/src/common.cjs

@@ -0,0 +1 @@
+module.exports = "common";

fixtures/additional-modules/src/dep.ts

@@ -0,0 +1 @@
+export default "bundled";

fixtures/additional-modules/src/dynamic.js

@@ -0,0 +1 @@
+export default "dynamic";

fixtures/additional-modules/src/index.ts

@@ -0,0 +1,28 @@
+import dep from "./dep";
+import text from "./text.txt";
+import common from "./common.cjs";
+
+export default <ExportedHandler>{
+	async fetch(request) {
+		const url = new URL(request.url);
+		if (url.pathname === "/dep") {
+			return new Response(dep);
+		}
+		if (url.pathname === "/text") {
+			return new Response(text);
+		}
+		if (url.pathname === "/common") {
+			return new Response(common);
+		}
+		if (url.pathname === "/dynamic") {
+			return new Response((await import("./dynamic.js")).default);
+		}
+		if (url.pathname.startsWith("/lang/")) {
+			// Build the path dynamically to ensure esbuild doesn't inline the import.
+			const language =
+				"./lang/" + url.pathname.substring("/lang/".length) + ".js";
+			return new Response((await import(language)).default.hello);
+		}
+		return new Response("Not Found", { status: 404 });
+	},
+};

fixtures/additional-modules/src/lang/en.js

@@ -0,0 +1 @@
+export default { hello: "hello" };