State of the art for Webpack: Dependency

[h-a-n-a]

Based on Webpack version: 5.73.0.
Some source code is omitted for cleaner demonstration in the example.

Summary

Explain how webpack dependency affects the compilation and what kind of problem that webpack was facing at the moment and the solution to the problem.

Glossary

What's the meaning of a word used to describe a feature?

Why does the Webpack introduce this and what's the background of introducing this? What kind of problem Webpack was facing at the time?

High-level presentations of Dependencies

• Dependency(fileDependency): An existing dependency that is marked as watchable. This is the widely-used type of dependency. CSS Preprocessors like postcss strongly depend on this in order to mark its dependency watchable.
• ContextDependency: Most useful for requests in which Glob and Regexp were used. For real-world usage, see [this](https://webpack.js.org/guides/dependency-management/#require-with-expression).
• MissingDependency: A missing dependency to mark it watchable (handles the creation of files during compilation before watchers are attached correctly.)
• BuildDependency: Related to persistent cache.
• PresentationalDependency: Dependencies that only affect presentation are mostly used with their associated template.

Others

• LoaderContext: Context provided by Webpack loader-runner, which can be accessed through this in each loader function.
• ModuleGraph: A graph to describe the relationship between modules.

Guide-level explanation

Dependency

dependency(fileDependency) stands for the file dependency among missingDependeny and contextDependency, etc. The created dependency will be marked as watchable, which is useful in Hot Module Replacement in developer mode.

The implicit behavior for webpack internally in the case below is to create two dependencies internally.

import foo from "./foo";
import "./style.css";

ContextDependency

contextDependency is mostly used in scenarios where we want to dynamic load some module in runtime. In this case, webpack cannot assure which module it will be included in the final bundle at compile time. In order to make the code runnable in runtime, webpack has to firstly create multiple bundle modules corresponding to the matching filename such as ./components/a.js and ./components/b.js, etc.

// index.js
import("./components" + componentName).then(...)

// components/a.js
...
export default ComponentA;

// components/b.js
...
export default ComponentB;

For loaders, you can access to this.addContextDependency in each loader function.
For plugins, you can access via module.buildInfo.contextDependencies.

Reference-level explanation

The abstraction of Dependency of Webpack was introduced in Webpack version 0.9 with a big refactor. Redirect to the commit

Stakeholders of Dependency

High-level

Low-level

How dependencies affect the creation of module graph?

Duplicated module detection

Each module will have its own identifier, for NormalModule, you can find this in NormalModule#identifier. If the identifier will be duplicated if inserted in this._module, then webpack will directly skip the remaining build process. [source]

Basically, an NormalModule identifier contains these parts:

1. type [string]: The module type of a module. If the type of the module is javascript/auto, this field can be omitted
2. request [string]: Request to the module. All loaders whether it's inline or matched by a config will be stringified. If inline match resource exists, inline loaders will be executed before any normal-loaders after pre-loaders. A module with a different loader passed through will be treated as a different module regardless of its path.
3. layer: applied if provided

Module resolution

getResolve is a loader API on the LoaderContext. Loader developers can pass dependencyType to its option which indicates the category of the module dependency that will be created. Values like esm can be passed, then webpack will use type esm to resolve the dependency.

The resolved dependencies are automatically added to the current module. This is driven by the internal plugin system of enhanced-resolve. Internally, enhanced-resolve uses plugins to handle the dependency registration like FileExistsPlugin [source] to detect whether a file is located on the file system or will add this file to a list of missingDependency and report in respect of the running mode of webpack. The collecting end of Webpack is generated by the getResolveContext in NormalModule [source]

Module dependency in ModuleGraph

Here's a module graph with esm import between modules:

The dependency type introduced by import or require is a derived dependency: ModuleDependency.

A ModuleDependency contains three important fields.

1. category: used to describe the category of dependency. e.g. "esm" | "commonjs"
2. request: see the explanation above.
3. userRequest: Resource and its inline loader syntax will be stringified and applied, but loaders in module.rules will be omitted.

It's also good to note a field we will talk about later:

1. assertions: assertions in import xx from "foo.json" assert { type: "json" }

More fields can be found in abstract class of Dependency and ModuleDependency. source: Dependency source: ModuleDependency

// null -> index.js

EntryDependency {
  category: "esm",
  request: "./index.js",
  type: "entry",
  _parentModule: undefined
}

// index.js -> foo.js

HarmonyImportSideEffectDependency {
  category: "esm",
  request: "./foo",
  type: "harmony side effect evaluation",
  _parentModule: NormalModule { identifier: "index.js" }
}

// index.js -> bar.js

HarmonyImportSideEffectDependency {
  category: "esm",
  request: "./bar",
  type: "harmony side effect evaluation",
  _parentModule: NormalModule { identifier: "index.js" }
}

// bar.js -> foo.js
HarmonyImportSideEffectDependency {
  category: "esm",
  request: "./foo",
  type: "harmony side effect evaluation",
  _parentModule: NormalModule { identifier: "bar.js" }
}

Resolving a module

ModuleDependencies with different dependency category such as esm or commonjs will affect the resolving part. For ECMAScript modules, they may prefer "module" to "main", and for CommonJS modules, they may use "main" in package.json. On top of that, conditional exports are also necessary to be taken into account. doc

Different types of module dependencies

ESM-related derived types

There are a few of ModuleDependencies introduced in ESM imports. A full list of each derived type can be reached at [source]

Import

HarmonyImportDependency

The basic type of harmony-related module dependencies are below. [source]

HarmonyImportSideEffectDependency

import { foo, bar } from "./module"
import * as module from "./module"
import foo from "./module"
import "./module"

Every import statement will come with a HarmonyImportSideEffectDependency, no matter how the specifiers look like. The speicifier will be handled by HarmonyImportSpecifierDendency below.

The field assertions will be stored if any import assertions exist for later consumption.
The field category will be used as dependencyType to resolve modules.

HarmonyImportSpecifierDependency

import { foo, bar } from "./module"
import * as module from "./module"
import foo from "./module"

Example:

import { foo, bar } from "./module"

console.log(foo, bar)

Specifier will be mapped into a specifier dependency if and only if it is used. JavaScript parser will first tag each variable [source], and then create corresponding dependencies on each reading of dependency. [source] and finally be replaced to the generated importVar.

Export(They are not module dependencies to be actual, but I placed here for convienence)

HarmonyExportHeaderDependency

PresentationalDependency

export const foo = "foo";
export default "foo";

This is a presentational dependency. We will take more time on this later.

HarmonyExportSpecifierDependency

export const foo = "foo"; // `foo` is a specifier

HarmonyExportSpecifierDependency {
  id: string;
  name: string;
}

HarmonyExportExpressionDependency

export default "foo"; // "foo" is an expression

HarmonyExportExpressionDependency {
 range: [number, number] // range of the expression
 rangeStatement: [number, number] // range of the whole statement
}

How dependencies affect code generation

Presentational dependency

A type of dependency that only affects code presentation.

ConstDependency

ConstDependency {
  expression: string
  range: [number, number]
  runtimeRequirements: Set<string> | null
}

You can think of the passed expression as a replacement for the corresponding range. For the real world example, you can directly refer to Constant Folding.

Template

Remember the fact that Webpack is an architecture wrapped around source code modifications. Template is the solution that helps Webpack to do the real patch on the source code. Each dependency has its associated template which affects a part of the code generation scoped per dependency. In other words, the effect of each template is strictly scoped to its associated dependency.

There are three types of modification:

• source
• fragments
• runtimeRequirements

A boilerplate of the dependency template looks like this:

class SomeDependency {}

SomeDependency.Template = class SomeDependencyTemplate {
   /**
	 * @param {Dependency} dependency the dependency for which the template should be applied
	 * @param {ReplaceSource} source the current replace source which can be modified
	 * @param {DependencyTemplateContext} templateContext the context object
	 * @returns {void}
	 */
   apply(dependency, source, templateContext) {
      // do code mod here
   }
}

There are three parameters in the function signature:

• dependency: The associated dependency of this template
• source: The source code represent in ReplaceSource, which can be used to replace a snippet of code with a new one, given the start and end position
• templateContext: A context of template, which stores the corresponding module, InitFragments, moduleGraph, runtimeRequirements, etc. (not important in this section)

Source

Again, given an example of ConstDependency, even if you don't have an idea what it is, it doesn't matter. We will cover this in the later sections.

The associated template modifies the code with Source(ReplaceSource to be more specific):

ConstDependency.Template = class ConstDependencyTemplate extends (
	NullDependency.Template
) {
	apply(dependency, source, templateContext) {
		const dep = /** @type {ConstDependency} */ (dependency);
        
        // not necessary code is removed for clearer demostration 

        if (dep.runtimeRequirements) {
			for (const req of dep.runtimeRequirements) {
				templateContext.runtimeRequirements.add(req);
			}
		}

		source.replace(dep.range[0], dep.range[1] - 1, dep.expression);
	}
};

runtimeRequirements

As you can see from the Source section above, there is another modification we talked about: runtimeRequirements, It adds
runtime requirements for the current compilation. We will explain more in the later sections.

Fragments

Essentially, a fragment is a pair of code snippet that to be wrapped around each module source. Note the wording "wrap", it could contain two parts content and endContent [source]. To make it more illustrative, see this:

The order of the fragment comes from two parts:

1. The stage of a fragment: if the stage of two fragments is different, then it will be replaced corresponding to the order define by the stage
2. If two fragments share the same order, then it will be replaced in position order.
   [source]

A real-world example

import { foo } from "./foo"

foo()

Given the example above, here's the code to generate a dependency that replaces import statement with __webpack_require__.

// some code is omitted for cleaner demonstration
parser.hooks.import.tap(
        "HarmonyImportDependencyParserPlugin",
              (statement, source) => {
	              const clearDep = new ConstDependency(
		              "",
		              statement.range
	              );
	              clearDep.loc = statement.loc;
	              parser.state.module.addPresentationalDependency(clearDep);

	              const sideEffectDep = new HarmonyImportSideEffectDependency(
		              source
	              );
	              sideEffectDep.loc = statement.loc;
	              parser.state.module.addDependency(sideEffectDep);

	              return true;
              }
);

Webpack will create two dependencies ConstDependency and HarmonyImportSideEffectDependency while parsing [source].

Let me focus on HarmonyImportSideEffectDependency more, since it uses Fragment to do some patch.

// some code is omitted for cleaner demonstration
HarmonyImportSideEffectDependency.Template = class HarmonyImportSideEffectDependencyTemplate extends (
	HarmonyImportDependency.Template
) {
	apply(dependency, source, templateContext) {
		super.apply(dependency, source, templateContext);
	}
};

As you can see in its associated template [source], the modification to the code is made via its superclass HarmonyImportDependency.Template [source].

// some code is omitted for cleaner demonstration
HarmonyImportDependency.Template = class HarmonyImportDependencyTemplate extends (
	ModuleDependency.Template
) {
	apply(dependency, source, templateContext) {
		const dep = /** @type {HarmonyImportDependency} */ (dependency);
		const { module, chunkGraph, moduleGraph, runtime } = templateContext;

		const referencedModule = connection && connection.module;

		const moduleKey = referencedModule
			? referencedModule.identifier()
			: dep.request;
		const key = `harmony import ${moduleKey}`;
                
                 // 1
		const importStatement = dep.getImportStatement(false, templateContext);
                 // 2
		templateContext.initFragments.push(
			new ConditionalInitFragment(
				importStatement[0] + importStatement[1],
				InitFragment.STAGE_HARMONY_IMPORTS,
				dep.sourceOrder,
				key,
				// omitted for cleaner code
			)
		);
	}
}

As you can see from the simplified source code above, the actual patch made to the generated code is via templateContext.initFragments(2). The import statement generated from dependency looks like this.

/* harmony import */  var _foo__WEBPACK_IMPORTED_MODULE_0__ = __webpack_require__(/*! ./foo */ "./src/foo.js"); //(1)

Note, the real require statement is generated via initFragments, ConditionalInitFragment to be specific. Don't be afraid of the naming, for more information you can see the (background)[https://github.com/webpack/webpack/pull/11802] of this fragment, which let's webpack to change it from InitFragment to ConditionalInitFragment.

How does webpack solve the compatibility issue?

For ESM modules, webpack will additionally call a helper to define _esModule on exports as an hint:

__webpack_require__.r(__webpack_exports__);

The call of a helper is always placed ahead of any require statements. Probably you have already get this as the stage of STAGE_HARMONY_EXPORTS has high priority than STAGE_HARMONY_IMPORTS. Again, this is achieved via initFragments. The logic of the compatibility helper is defined in this file, you can check it out.

Runtime

Runtime generation is based on the previously collected runtimeRequirements in different dependency templates and is done after the code generation of each module. Note: it's not after the renderManifest, but it's after the code generation of each module.

In the first iteration of collection, Sets of runtimeRequirements are collected from the module's code generation results and added to each ChunkGraphModule.

In the second iteration of collection, the collected runtimeRequirements are already stored in ChunkGraphModule, so Webpack again collects them from there and stores the runtimes required by each chunk of ChunkGraphChunk. It's kind of the hoisting procedure of the required runtimes.

Finally, also known as the third iteration of collection, Webpack hoists runtimeRequirements from those chunks that are referenced by the entry chunk and get it hoisted on the ChunkGraphChunk using a different field named runtimeRequirementsInTree which indicates not only does it contains the runtime requirements by the chunk but also it's children runtime requirements.

The referenced source code you can be found it here and these steps are basically done in processRuntimeRequirements. This let me recall the linking procedure of a rollup-like bundler. Anyway, after this procedure, we can finally generate runtime modules. Actually, I lied here, huge thanks to the hook system of Webpack, the creation of runtime modules is done in this method via calls to runtimeRequirementInTree[source]. No doubt, this is all done in the seal step. After that, webpack will process each chunk and create a few code generation jobs, and finally, emit assets.

Hot module replacement

Changes made via hot module replacement is mostly come from HotModuleReplacementPlugin.

Given the code below:

if (module.hot) {
  module.hot.accept(...)
}

Webpack will replace expressions like module.hot and module.hot.accept, etc with ConstDependency as the presentationalDependency as I previously talked about. [source]

With the help of a simple expression replacement is not enough, the plugin also introduce additional runtime modules for each entries. [source]

The plugin is quite complicated, and you should definitely checkout what it actually does, but for things related to dependency, it's enough.

How dependencies affect production optimizations

Constant folding

The logic is defined in ConstPlugin : [source]

Constant folding is a technique that used as an optimization for optimization. For example:

Source

if (process.env.NODE_ENV === "development") {
   ...
} else {
   ...
}

Generated

if (true) {
  ...
}

With mode set to "development", webpack will "fold" the expression process.env.NODE_ENV === "development" into an expression of "true" as you can see for the code generation result.

In the make procedure of webpack, Webpack internally uses an JavaScriptParser for JavaScript parsing. If an ifStatement is encountered, Webpack creates a corresponding ConstDependency. Essentially, for the ifStatement, the ConstDependency looks like this :

ConstDependency {
  expression: "true",
  range: [start, end] // range to replace
}

It's almost the same with else branch, if there is no side effects(refer to source code for more detail), Webpack will create another ConstDependency with expression set to "", which in the end removes the else branch.

In the seal procedure of Webpack, the record of the dependency will be applied to the original source code and generate the final result as you may have already seen above.

Tree shaking & DCE

Tree-shaking is a technique of a bundle-wise DCE(dead code elimination). In the following content, I will use tree-shaking as a wording for bundle-wise and DCE for module-wise code elimination. (I know it's not quite appropriate, but you get the point)

Here's an example:

// webpack configuration
module.exports = {
  optimization: {
    usedExports: true
  }
}

As you can see from the red square, the initFragment is generated based on the usage of the exported symbol in the HarmonyExportSpecifierDependency [source]

If foo is used in the graph, then the generated result will be this:

/* harmony export */ __webpack_require__.d(__webpack_exports__, {
/* harmony export */   "foo": function() { return /* binding */ foo; }
/* harmony export */ });
const foo = "foo";

In the example above, the foo is not used, so it will be excluded in the code generation of the template of HarmonyExportSpecifierDependency and it will be dead-code-eliminated in later steps. For terser plugin, it eliminates all unreachable code in processAssets [source].

Things related to Persistent cache

TODO

Wrap it up!

Let's wrap everything up in a simple example! Isn't it exciting?

Given a module graph that contains three modules, the entry point of this bundle is index.js. To not make this example too complicated, we use normal import statements to reference each module (i.e: only one chunk that bundles everything will be created).

Make

Dependencies after make

seal

References

TODO