Speed up d.ts generation by batching entries into worker processes#34329
Speed up d.ts generation by batching entries into worker processes#34329kasperpeulen wants to merge 3 commits intonextfrom
Conversation
Instead of spawning 27 separate processes (each loading rollup, typescript, and creating a full TS program from scratch), batch entries into 4 worker processes. Each worker processes entries sequentially, amortizing module loading and benefiting from OS file cache warming. Workers run in parallel for multi-core utilization. Includes retry logic for inter-entry dependencies and 8GB memory per worker to prevent OOM. Reduces core package d.ts generation from ~2.82min to ~1.57min (44% faster). https://claude.ai/code/session_01DUhTmj7c91ao6ga2DJPrTB
Use filesystem-based retry: after all batches complete, check which entries are still missing d.ts output and retry them in a single batch. This handles cross-batch dependencies (e.g. mocking-utils depends on babel's d.ts) without fragile timing-based retries. https://claude.ai/code/session_01DUhTmj7c91ao6ga2DJPrTB
…ster d.ts generation Use rolldown (Rust-based bundler) and tsgo (Go-based TypeScript compiler) instead of rollup + rollup-plugin-dts + tsc. tsgo compiles all entries in ~7s (vs ~10s per entry with tsc), and rolldown bundles declarations natively. Total types generation drops from ~2.7min to ~10s for the core package. https://claude.ai/code/session_01DUhTmj7c91ao6ga2DJPrTB
|
📝 WalkthroughWalkthroughAdded new TypeScript tooling and bundling devDependencies. Removed the per-entry Changes
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
✨ Finishing Touches📝 Generate docstrings
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
scripts/build/utils/generate-types.ts (1)
27-33: Consider using normalized path separators for cross-platform compatibility.The third condition in
externalFnusessepfromnode:pathto check fornode_modulesdependencies. However, bundlers like Rolldown often normalize resolved paths to forward slashes (/) internally, regardless of platform. On Windows, this could cause the${sep}node_modules${sep}...pattern to fail if Rolldown passes normalized IDs with forward slashes.To ensure compatibility across platforms, check for both separators:
🛠️ Suggested fix
const externalFn = (id: string) => external.some( (dep: string) => id === dep || id.startsWith(`${dep}/`) || - id.includes(`${sep}node_modules${sep}${dep}${sep}`) + id.includes(`/node_modules/${dep}/`) || + id.includes(`\\node_modules\\${dep}\\`) );🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@scripts/build/utils/generate-types.ts` around lines 27 - 33, The externalFn check can fail on Windows because it compares id against a pattern containing node:path sep; update externalFn to normalize the incoming id to use POSIX separators (or explicitly check both '/' and '\\') before the third condition so the `${sep}node_modules${sep}${dep}${sep}` match works with Rollup-normalized IDs; modify the logic in externalFn (referencing the id parameter and the external array) to use a normalized id variable or a dual-separator check instead of relying solely on sep.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@package.json`:
- Around line 67-77: The package.json lists non-existent versions for
oxc-transform, rolldown, and rolldown-plugin-dts which will break installs;
update the dependency entries for "oxc-transform", "rolldown", and
"rolldown-plugin-dts" to published versions (e.g., change "oxc-transform" from
0.121.0 to an available release like 0.99.0, "rolldown" from 1.0.0-rc.12 to an
existing rc such as 1.0.0-rc.9, and "rolldown-plugin-dts" from 0.23.0 to an
available release like 0.9.9) so npm can resolve them.
In `@scripts/build/utils/generate-types.ts`:
- Around line 43-66: The rolldown dts plugin is currently configured with tsgo:
true which is experimental; update the dts plugin config in the rolldown call
(the dts(...) options used when creating out via rolldown) to either use a
stable option (set oxc: true and remove/disable tsgo) or gate tsgo behind a
deliberate opt-in (e.g., only enable tsgo when a specific env flag is set and
leave oxc as the default for CI/production); adjust the dts(...) options (tsgo,
oxc, emitDtsOnly) and add a short comment explaining the chosen behavior.
---
Nitpick comments:
In `@scripts/build/utils/generate-types.ts`:
- Around line 27-33: The externalFn check can fail on Windows because it
compares id against a pattern containing node:path sep; update externalFn to
normalize the incoming id to use POSIX separators (or explicitly check both '/'
and '\\') before the third condition so the
`${sep}node_modules${sep}${dep}${sep}` match works with Rollup-normalized IDs;
modify the logic in externalFn (referencing the id parameter and the external
array) to use a normalized id variable or a dual-separator check instead of
relying solely on sep.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: e6bf841a-2a45-44ad-a568-da9bca42a96b
⛔ Files ignored due to path filters (1)
yarn.lockis excluded by!**/yarn.lock,!**/*.lock
📒 Files selected for processing (3)
package.jsonscripts/build/utils/dts-process.tsscripts/build/utils/generate-types.ts
💤 Files with no reviewable changes (1)
- scripts/build/utils/dts-process.ts
| "@typescript/native-preview": "^7.0.0-dev.20260325.1", | ||
| "http-server": "^14.1.1", | ||
| "husky": "^9.1.7", | ||
| "jiti": "^2.6.1", | ||
| "kill-port": "^2.0.1", | ||
| "lint-staged": "^16.4.0", | ||
| "nx": "^22.6.1", | ||
| "oxc-transform": "0.121.0", | ||
| "oxfmt": "^0.41.0", | ||
| "rolldown": "^1.0.0-rc.12", | ||
| "rolldown-plugin-dts": "^0.23.0", |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Verify these packages exist and check their latest versions
echo "=== Checking `@typescript/native-preview` ==="
curl -s "https://registry.npmjs.org/@typescript/native-preview" | jq '.["dist-tags"], .versions | keys[-5:]'
echo "=== Checking oxc-transform ==="
curl -s "https://registry.npmjs.org/oxc-transform" | jq '.["dist-tags"], .versions | keys[-5:]'
echo "=== Checking rolldown ==="
curl -s "https://registry.npmjs.org/rolldown" | jq '.["dist-tags"], .versions | keys[-5:]'
echo "=== Checking rolldown-plugin-dts ==="
curl -s "https://registry.npmjs.org/rolldown-plugin-dts" | jq '.["dist-tags"], .versions | keys[-5:]'Repository: storybookjs/storybook
Length of output: 624
Package versions 0.121.0, 1.0.0-rc.12, and 0.23.0 do not exist on npm—this will cause installation to fail.
Three of the four new packages specify versions that are not published:
oxc-transform0.121.0: Latest available is 0.99.0 (off by ~22 versions)rolldown1.0.0-rc.12: Latest available is 1.0.0-rc.9 (off by 3 release candidates)rolldown-plugin-dts0.23.0: Latest available is 0.9.9 (off by ~2.3x)
Update these to versions that actually exist on npm. @typescript/native-preview ^7.0.0-dev.20260325.1 is fine and will resolve correctly.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@package.json` around lines 67 - 77, The package.json lists non-existent
versions for oxc-transform, rolldown, and rolldown-plugin-dts which will break
installs; update the dependency entries for "oxc-transform", "rolldown", and
"rolldown-plugin-dts" to published versions (e.g., change "oxc-transform" from
0.121.0 to an available release like 0.99.0, "rolldown" from 1.0.0-rc.12 to an
existing rc such as 1.0.0-rc.9, and "rolldown-plugin-dts" from 0.23.0 to an
available release like 0.9.9) so npm can resolve them.
| // Use rolldown + rolldown-plugin-dts with tsgo for fast d.ts generation. | ||
| // tsgo (Go-based TypeScript compiler) runs once for all entries (~7s), | ||
| // then rolldown bundles the declarations natively in Rust. | ||
| const out = await rolldown({ | ||
| input: entryMap, | ||
| external: externalFn, | ||
| plugins: [ | ||
| dts({ | ||
| cwd, | ||
| tsconfig: join(cwd, 'tsconfig.json'), | ||
| tsgo: true, | ||
| emitDtsOnly: true, | ||
| }), | ||
| ], | ||
| logLevel: 'warn', | ||
| }); | ||
|
|
||
| if (dtsProcess.exitCode !== 0) { | ||
| if (attempt < MAX_DTS_ATTEMPTS) { | ||
| // Race: parallel DTS can read a .d.ts another process is still writing → invalid. Retry + delay usually fixes (flake in core:compile:production since #33759). | ||
| console.warn( | ||
| `⚠️ DTS failed for ${picocolors.cyan(relative(cwd, entryPoint))}, retrying (${attempt}/${MAX_DTS_ATTEMPTS})...` | ||
| ); | ||
| processes = processes.filter((p) => p !== dtsProcess); | ||
| await new Promise((r) => setTimeout(r, RETRY_DELAY_MS)); | ||
| continue; | ||
| } | ||
| console.error( | ||
| '\n❌ Generating types for', | ||
| picocolors.cyan(relative(cwd, entryPoint)), | ||
| ' failed' | ||
| ); | ||
| // If any fail after all retries, kill all the other processes and exit (bail) | ||
| processes.forEach((p) => p.kill()); | ||
| processes = []; | ||
| process.exit(dtsProcess.exitCode || 1); | ||
| } | ||
| await out.write({ dir: join(cwd, 'dist'), format: 'es' }); | ||
|
|
||
| if (!process.env.CI) { | ||
| console.log('✅ Generated types for', picocolors.cyan(join(DIR_REL, entryPoint))); | ||
| } | ||
| break; | ||
| } | ||
| }); | ||
| }) | ||
| ); | ||
| if (!process.env.CI) { | ||
| for (const entry of dtsEntries) { | ||
| console.log('Generated types for', picocolors.cyan(join(DIR_REL, entry))); | ||
| } | ||
| } |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
What are the configuration options for rolldown-plugin-dts including the tsgo option?
💡 Result:
The rolldown-plugin-dts plugin provides the following configuration options, grouped by category as documented in the official GitHub repository and npm README (latest version 0.22.5 as of March 2026). These enable customization of .d.ts file generation and bundling for Rolldown builds. ### General Options - cwd: Directory to search for tsconfig.json. - dtsInput: boolean, true if entry files are .d.ts (skips generating .d.ts for entry). - emitDtsOnly: boolean, emits only .d.ts files and removes other outputs (useful for CJS .d.ts in separate builds). - tsconfig: Path to tsconfig.json (default: 'tsconfig.json'), or false to ignore. - tsconfigRaw: Raw tsconfig.json object. - compilerOptions: Overrides for compilerOptions from tsconfig.json. - sourcemap: boolean, generates .d.ts.map declaration maps. - resolver: 'oxc' (default, faster) or 'tsc' (more compatible). - cjsDefault: Controls default export emission for CJS. - sideEffects: boolean (default: false), treats .d.ts as having side effects for tree-shaking. ### TSC Options (only when oxc and tsgo are false) - build: boolean, uses tsc --build (true) or plain tsc (false). - vue: boolean, uses vue-tsc for .d.ts generation. - parallel: boolean, launches separate process for parallel tsc/vue-tsc. - emitJs: boolean, emits .d.ts for .js files (auto-enabled if allowJs true). ### Oxc Options - oxc: boolean, uses Oxc for faster .d.ts generation (auto-enabled if isolatedDeclarations true). ### TypeScript Go (tsgo) Option - Experimental - tsgo: boolean | TsgoOptions. Enables experimental DTS generation using tsgo (requires @typescript/native-preview). Ignores tsconfigRaw and compilerOptions. Not recommended for production. - As boolean (true): Uses tsgo from @typescript/native-preview. - As object: e.g., { path: '/path/to/tsgo' } for custom tsgo binary. Example usage: import { dts } from 'rolldown-plugin-dts'; export default { plugins: [dts({ tsgo: true, sourcemap: true })] }; Note: For code splitting, configure groups so .d.ts chunks end with .d (e.g., test: /foo.*.d.[cm]?ts$/).
Citations:
- 1: https://github.com/sxzz/rolldown-plugin-dts
- 2: https://registry.npmjs.org/rolldown-plugin-dts
- 3: https://npmjs.com/package/rolldown-plugin-dts?ref=pkgstats.com
- 4: https://jsr.io/@sxzz/rolldown-plugin-dts/doc/~/Options
- 5: https://jsr.io/%40sxzz/rolldown-plugin-dts
Flag: tsgo is experimental and not recommended for production.
The rolldown configuration uses tsgo: true and emitDtsOnly: true for type generation. While emitDtsOnly correctly ensures only .d.ts files are emitted, the tsgo option has a significant caveat: according to the rolldown-plugin-dts documentation, tsgo is experimental and explicitly not recommended for production use. It requires @typescript/native-preview and may have unexpected behavior in production builds.
Consider either:
- Using
oxc: true(the stable default) instead of tsgo, or - Explicitly documenting that this experimental feature is intentional for local development performance, with clear understanding of the trade-offs.
Error handling is correct—failures will propagate to the caller as expected.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@scripts/build/utils/generate-types.ts` around lines 43 - 66, The rolldown dts
plugin is currently configured with tsgo: true which is experimental; update the
dts plugin config in the rolldown call (the dts(...) options used when creating
out via rolldown) to either use a stable option (set oxc: true and
remove/disable tsgo) or gate tsgo behind a deliberate opt-in (e.g., only enable
tsgo when a specific env flag is set and leave oxc as the default for
CI/production); adjust the dts(...) options (tsgo, oxc, emitDtsOnly) and add a
short comment explaining the chosen behavior.
|
View your CI Pipeline Execution ↗ for commit f31b2b5
☁️ Nx Cloud last updated this comment at |
2 participants
Instead of spawning 27 separate processes (each loading rollup, typescript,
and creating a full TS program from scratch), batch entries into 4 worker
processes. Each worker processes entries sequentially, amortizing module
loading and benefiting from OS file cache warming. Workers run in parallel
for multi-core utilization. Includes retry logic for inter-entry
dependencies and 8GB memory per worker to prevent OOM.
Reduces core package d.ts generation from ~2.82min to ~1.57min (44% faster).
https://claude.ai/code/session_01DUhTmj7c91ao6ga2DJPrTB
Summary by CodeRabbit