module: support require()ing synchronous ESM graphs

This patch adds `require()` support for synchronous ESM graphs under
the flag `--experimental-require-module`

This is based on the the following design aspect of ESM:

- The resolution can be synchronous (up to the host)
- The evaluation of a synchronous graph (without top-level await) is
  also synchronous, and, by the time the module graph is instantiated
  (before evaluation starts), this is is already known.

If `--experimental-require-module` is enabled, and the ECMAScript
module being loaded by `require()` meets the following requirements:

- Explicitly marked as an ES module with a `"type": "module"` field in
  the closest package.json or a `.mjs` extension.
- Fully synchronous (contains no top-level `await`).

`require()` will load the requested module as an ES Module, and return
the module name space object. In this case it is similar to dynamic
`import()` but is run synchronously and returns the name space object
directly.

```mjs
// point.mjs
export function distance(a, b) {
  return (b.x - a.x) ** 2 + (b.y - a.y) ** 2;
}
class Point {
  constructor(x, y) { this.x = x; this.y = y; }
}
export default Point;
```

```cjs
const required = require('./point.mjs');
// [Module: null prototype] {
//   default: [class Point],
//   distance: [Function: distance]
// }
console.log(required);

(async () => {
  const imported = await import('./point.mjs');
  console.log(imported === required);  // true
})();
```

If the module being `require()`'d contains top-level `await`, or the
module graph it `import`s contains top-level `await`,
[`ERR_REQUIRE_ASYNC_MODULE`][] will be thrown. In this case, users
should load the asynchronous module using `import()`.

If `--experimental-print-required-tla` is enabled, instead of throwing
`ERR_REQUIRE_ASYNC_MODULE` before evaluation, Node.js will evaluate the
module, try to locate the top-level awaits, and print their location to
help users fix them.

PR-URL: nodejs#51977
Reviewed-By: Chengzhong Wu <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Guy Bedford <[email protected]>
Reviewed-By: Antoine du Hamel <[email protected]>
Reviewed-By: Geoffrey Booth <[email protected]>

Author: joyeecheung

doc/api/cli.md

@@ -974,6 +974,18 @@ added: v11.8.0
 
 Use the specified file as a security policy.
 
+### `--experimental-require-module`
+
+<!-- YAML
+added: REPLACEME
+-->
+
+> Stability: 1.1 - Active Developement
+
+Supports loading a synchronous ES module graph in `require()`.
+
+See [Loading ECMAScript modules using `require()`][].
+
 ### `--experimental-sea-config`
 
 <!-- YAML
@@ -1695,6 +1707,18 @@ changes:
 
 Identical to `-e` but prints the result.
 
+### `--experimental-print-required-tla`
+
+<!-- YAML
+added: REPLACEME
+-->
+
+This flag is only useful when `--experimental-require-module` is enabled.
+
+If the ES module being `require()`'d contains top-level await, this flag
+allows Node.js to evaluate the module, try to locate the
+top-level awaits, and print their location to help users find them.
+
 ### `--prof`
 
 <!-- YAML
@@ -2642,6 +2666,8 @@ one is included in the list below.
 * `--experimental-network-imports`
 * `--experimental-permission`
 * `--experimental-policy`
+* `--experimental-print-required-tla`
+* `--experimental-require-module`
 * `--experimental-shadow-realm`
 * `--experimental-specifier-resolution`
 * `--experimental-top-level-await`
@@ -3111,6 +3137,7 @@ done
 [ExperimentalWarning: `vm.measureMemory` is an experimental feature]: vm.md#vmmeasurememoryoptions
 [Fetch API]: https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API
 [File System Permissions]: permissions.md#file-system-permissions
+[Loading ECMAScript modules using `require()`]: modules.md#loading-ecmascript-modules-using-require
 [Module customization hooks]: module.md#customization-hooks
 [Module customization hooks: enabling]: module.md#enabling
 [Modules loaders]: packages.md#modules-loaders

doc/api/errors.md

@@ -2521,6 +2521,19 @@ Accessing `Object.prototype.__proto__` has been forbidden using
 [`Object.setPrototypeOf`][] should be used to get and set the prototype of an
 object.
 
+<a id="ERR_REQUIRE_ASYNC_MODULE"></a>
+
+### `ERR_REQUIRE_ASYNC_MODULE`
+
+> Stability: 1 - Experimental
+
+When trying to `require()` a [ES Module][] under `--experimental-require-module`,
+the module turns out to be asynchronous. That is, it contains top-level await.
+
+To see where the top-level await is, use
+`--experimental-print-required-tla` (this would execute the modules
+before looking for the top-level awaits).
+
 <a id="ERR_REQUIRE_ESM"></a>
 
 ### `ERR_REQUIRE_ESM`
@@ -2529,6 +2542,9 @@ object.
 
 An attempt was made to `require()` an [ES Module][].
 
+To enable `require()` for synchronous module graphs (without
+top-level `await`), use `--experimental-require-module`.
+
 <a id="ERR_SCRIPT_EXECUTION_INTERRUPTED"></a>
 
 ### `ERR_SCRIPT_EXECUTION_INTERRUPTED`

doc/api/esm.md

@@ -449,11 +449,10 @@ compatibility.
 
 ### `require`
 
-The CommonJS module `require` always treats the files it references as CommonJS.
+The CommonJS module `require` currently only supports loading synchronous ES
+modules when `--experimental-require-module` is enabled.
 
-Using `require` to load an ES module is not supported because ES modules have
-asynchronous execution. Instead, use [`import()`][] to load an ES module
-from a CommonJS module.
+See [Loading ECMAScript modules using `require()`][] for details.
 
 ### CommonJS Namespaces
 
@@ -1132,6 +1131,7 @@ resolution for ESM specifiers is [commonjs-extension-resolution-loader][].
 [Import Attributes]: #import-attributes
 [Import Attributes proposal]: https://github.com/tc39/proposal-import-attributes
 [JSON modules]: #json-modules
+[Loading ECMAScript modules using `require()`]: modules.md#loading-ecmascript-modules-using-require
 [Module customization hooks]: module.md#customization-hooks
 [Node.js Module Resolution And Loading Algorithm]: #resolution-algorithm-specification
 [Terminology]: #terminology

doc/api/modules.md

@@ -168,16 +168,60 @@ variable. Since the module lookups using `node_modules` folders are all
 relative, and based on the real path of the files making the calls to
 `require()`, the packages themselves can be anywhere.
 
-## The `.mjs` extension
+## Loading ECMAScript modules using `require()`
 
-Due to the synchronous nature of `require()`, it is not possible to use it to
-load ECMAScript module files. Attempting to do so will throw a
-[`ERR_REQUIRE_ESM`][] error. Use [`import()`][] instead.
-
-The `.mjs` extension is reserved for [ECMAScript Modules][] which cannot be
-loaded via `require()`. See [Determining module system][] section for more info
+The `.mjs` extension is reserved for [ECMAScript Modules][].
+Currently, if the flag `--experimental-require-module` is not used, loading
+an ECMAScript module using `require()` will throw a [`ERR_REQUIRE_ESM`][]
+error, and users need to use [`import()`][] instead. See
+[Determining module system][] section for more info
 regarding which files are parsed as ECMAScript modules.
 
+If `--experimental-require-module` is enabled, and the ECMAScript module being
+loaded by `require()` meets the following requirements:
+
+* Explicitly marked as an ES module with a `"type": "module"` field in
+  the closest package.json or a `.mjs` extension.
+* Fully synchronous (contains no top-level `await`).
+
+`require()` will load the requested module as an ES Module, and return
+the module name space object. In this case it is similar to dynamic
+`import()` but is run synchronously and returns the name space object
+directly.
+
+```mjs
+// point.mjs
+export function distance(a, b) { return (b.x - a.x) ** 2 + (b.y - a.y) ** 2; }
+class Point {
+  constructor(x, y) { this.x = x; this.y = y; }
+}
+export default Point;
+```
+
+```cjs
+const required = require('./point.mjs');
+// [Module: null prototype] {
+//   default: [class Point],
+//   distance: [Function: distance]
+// }
+console.log(required);
+
+(async () => {
+  const imported = await import('./point.mjs');
+  console.log(imported === required);  // true
+})();
+```
+
+If the module being `require()`'d contains top-level `await`, or the module
+graph it `import`s contains top-level `await`,
+[`ERR_REQUIRE_ASYNC_MODULE`][] will be thrown. In this case, users should
+load the asynchronous module using `import()`.
+
+If `--experimental-print-required-tla` is enabled, instead of throwing
+`ERR_REQUIRE_ASYNC_MODULE` before evaluation, Node.js will evaluate the
+module, try to locate the top-level awaits, and print their location to
+help users fix them.
+
 ## All together
 
 <!-- type=misc -->
@@ -207,12 +251,24 @@ require(X) from module at path Y
 
 LOAD_AS_FILE(X)
 1. If X is a file, load X as its file extension format. STOP
-2. If X.js is a file, load X.js as JavaScript text. STOP
-3. If X.json is a file, parse X.json to a JavaScript Object. STOP
+2. If X.js is a file,
+    a. Find the closest package scope SCOPE to X.
+    b. If no scope was found, load X.js as a CommonJS module. STOP.
+    c. If the SCOPE/package.json contains "type" field,
+      1. If the "type" field is "module", load X.js as an ECMAScript module. STOP.
+      2. Else, load X.js as an CommonJS module. STOP.
+3. If X.json is a file, load X.json to a JavaScript Object. STOP
 4. If X.node is a file, load X.node as binary addon. STOP
+5. If X.mjs is a file, and `--experimental-require-module` is enabled,
+   load X.mjs as an ECMAScript module. STOP
 
 LOAD_INDEX(X)
-1. If X/index.js is a file, load X/index.js as JavaScript text. STOP
+1. If X/index.js is a file
+    a. Find the closest package scope SCOPE to X.
+    b. If no scope was found, load X/index.js as a CommonJS module. STOP.
+    c. If the SCOPE/package.json contains "type" field,
+      1. If the "type" field is "module", load X/index.js as an ECMAScript module. STOP.
+      2. Else, load X/index.js as an CommonJS module. STOP.
 2. If X/index.json is a file, parse X/index.json to a JavaScript object. STOP
 3. If X/index.node is a file, load X/index.node as binary addon. STOP
 
@@ -1097,6 +1153,7 @@ This section was moved to
 [GLOBAL_FOLDERS]: #loading-from-the-global-folders
 [`"main"`]: packages.md#main
 [`"type"`]: packages.md#type
+[`ERR_REQUIRE_ASYNC_MODULE`]: errors.md#err_require_async_module
 [`ERR_REQUIRE_ESM`]: errors.md#err_require_esm
 [`ERR_UNSUPPORTED_DIR_IMPORT`]: errors.md#err_unsupported_dir_import
 [`MODULE_NOT_FOUND`]: errors.md#module_not_found

doc/api/packages.md

@@ -133,14 +133,15 @@ There is the CommonJS module loader:
   `process.dlopen()`.
 * It treats all files that lack `.json` or `.node` extensions as JavaScript
   text files.
-* It cannot be used to load ECMAScript modules (although it is possible to
-  [load ECMASCript modules from CommonJS modules][]). When used to load a
-  JavaScript text file that is not an ECMAScript module, it loads it as a
-  CommonJS module.
+* It can only be used to [load ECMASCript modules from CommonJS modules][] if
+  the module graph is synchronous (that contains no top-level `await`) when
+  `--experimental-require-module` is enabled.
+  When used to load a JavaScript text file that is not an ECMAScript module,
+  the file will be loaded as a CommonJS module.
 
 There is the ECMAScript module loader:
 
-* It is asynchronous.
+* It is asynchronous, unless it's being used to load modules for `require()`.
 * It is responsible for handling `import` statements and `import()` expressions.
 * It is not monkey patchable, can be customized using [loader hooks][].
 * It does not support folders as modules, directory indexes (e.g.
@@ -623,9 +624,9 @@ specific to least specific as conditions should be defined:
 * `"require"` - matches when the package is loaded via `require()`. The
   referenced file should be loadable with `require()` although the condition
   matches regardless of the module format of the target file. Expected
-  formats include CommonJS, JSON, and native addons but not ES modules as
-  `require()` doesn't support them. _Always mutually exclusive with
-  `"import"`._
+  formats include CommonJS, JSON, native addons, and ES modules
+  if `--experimental-require-module` is enabled. _Always mutually
+  exclusive with `"import"`._
 * `"default"` - the generic fallback that always matches. Can be a CommonJS
   or ES module file. _This condition should always come last._
 
@@ -1371,7 +1372,7 @@ This field defines [subpath imports][] for the current package.
 [entry points]: #package-entry-points
 [folders as modules]: modules.md#folders-as-modules
 [import maps]: https://github.com/WICG/import-maps
-[load ECMASCript modules from CommonJS modules]: modules.md#the-mjs-extension
+[load ECMASCript modules from CommonJS modules]: modules.md#loading-ecmascript-modules-using-require
 [loader hooks]: esm.md#loaders
 [packages folder mapping]: https://github.com/WICG/import-maps#packages-via-trailing-slashes
 [self-reference]: #self-referencing-a-package-using-its-name