build: incremental CJS to ESM migration (Phase 1-3)

[haraldschilly]

Summary

• Add dual CJS + ESM output for 8 packages: util, sync, sync-client, sync-fs, conat, backend, comm, api-client
• Each package now produces both dist/ (CJS, unchanged) and dist-esm/ (ESM) via a second tsconfig-esm.json
• Node.js conditional exports route consumers to the correct format automatically
• Convert all remaining CJS .js files in util to TypeScript
• Include scripts/add-esm-extensions.py utility for future phases requiring .js extension insertion

Strategy: Why Dual Output with Minimal Source Changes

The core constraint is that Node.js require() cannot synchronously load native ESM — it throws ERR_REQUIRE_ESM. Since hub and frontend currently compile to CJS (and have CoffeeScript files blocking full migration), making any dependency pure ESM would break them immediately.

The chosen approach uses module: "ESNext" + moduleResolution: "bundler" in the ESM tsconfig. This combination:

1. Always outputs ESM regardless of the package's type field (unlike NodeNext which reads package.json and may fall back to CJS)
2. Does not require .js extensions on relative imports in source files — this means zero changes to .ts source files across the entire monorepo
3. Works for bundler consumers (Next.js/webpack, rspack, esbuild) which is the primary ESM use case today

The trade-off: dist-esm/ output lacks .js extensions on relative imports, so it cannot be loaded directly by Node.js native ESM (node --input-type=module). This is acceptable because:

• The primary ESM consumers are bundlers (Next.js, webpack) which resolve extensionless imports
• Once hub is CoffeeScript-free and migrated to ESM, the .js extension script (scripts/add-esm-extensions.py) can be used to enable native Node.js ESM support in a later phase

Conditional Exports: "import" / "default"

Each package's package.json uses:

"exports": {
  "./*": {
    "types": "./dist/*.d.ts",
    "import": "./dist-esm/*.js",
    "default": "./dist/*.js"
  }
}

• "types" (first) ensures TypeScript always resolves a single canonical .d.ts from dist/, preventing type identity conflicts between CJS and ESM declarations
• "import" routes ESM consumers to dist-esm/
• "default" routes CJS consumers (and TypeScript moduleResolution: "node") to dist/

Per-Package Changes

Each migrated package gets:

1. tsconfig-esm.json — ESM build config (module: ESNext, moduleResolution: bundler, declaration: false, composite: false)
2. tsconfig.json — Added "dist-esm" to exclude
3. package.json — Conditional exports, updated build/clean scripts, dist-esm/** in files
4. Build script: tsc --build && tsc -p tsconfig-esm.json && echo '{"type":"module"}' > dist-esm/package.json

Additional Fixes

• Convert all CJS .js to TypeScript in util: message, regex-split, mathjax-utils, mathjax-utils-2, immutable-types, smc-version, heartbeat, mathjax-config, upgrades
• Replace underscore with lodash in message.ts; remove underscore dependency
• Update update_version script to emit TypeScript (export const version = ...)
• Add fullySpecified: false rspack rule for dist-esm/ files (tsc doesn't emit .js extensions but "type":"module" requires them)
• Standardize build scripts across all ESM-migrated packages to use pnpm exec tsc instead of ../node_modules/.bin/tsc
• Fix ghost export signal_sent in project/hub/handle-message.ts — was never defined in message.js, replaced with message.success()
• Add regex-split.test.ts with 15 tests

Migration Phases

Phase	Packages	Status
1	@cocalc/util	✅ This PR
2	@cocalc/sync, @cocalc/sync-client, @cocalc/sync-fs, @cocalc/conat, @cocalc/backend	✅ This PR
3	@cocalc/comm, @cocalc/api-client	✅ This PR
4	@cocalc/database	Blocked by CoffeeScript removal: #8687
5	@cocalc/server, @cocalc/project, @cocalc/file-server	Future (after Phase 4)
6	@cocalc/next	No structural changes needed — Next.js/webpack already uses the "import" condition automatically
7	@cocalc/hub	Blocked on CoffeeScript removal (9 .coffee files)
8	@cocalc/frontend	Blocked on CoffeeScript removal (8 .coffee files)

Test plan

• pnpm build-dev — full workspace build passes
• pnpm test — CI passes (depcheck, version-check, tests)
• Each migrated package produces both dist/ and dist-esm/
• dist-esm/package.json contains {"type":"module"}
• ESM output contains export/import (not exports.xxx =)
• rspack static build has zero errors (fullySpecified fix)
• regex-split.test.ts — 15 tests pass

🤖 Generated with Claude Code

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 646346492c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".