why are all platform binaries installed within node_modules?
#59821
Replies: 4 comments 3 replies
|
So (and in reference to my edit above), considering next is doing this on purpose to put binaries in place for all possible archictures, we are left with:
|
|
I took a crack at the
const os = require("node:os")
module.exports = {
hooks: {
readPackage: (pkg, _context) => {
if (pkg.name === "next") {
Object.keys(pkg.optionalDependencies)
.filter(
(optionalDependency) =>
optionalDependency.startsWith("@next/swc-") &&
!optionalDependency.startsWith(`@next/swc-${os.platform()}-${os.arch()}`),
)
.forEach((optionalDependency) => {
console.log(`Omitting optional dependency "${optionalDependency}"`)
delete pkg.optionalDependencies[optionalDependency]
delete pkg.dependencies[optionalDependency]
})
}
return pkg
},
},
}Validation# generate a next app
pnpm create next-app test-reduced-platform-binaries \
--typescript \
--tailwind false \
--app \
--src-dir \
--eslint \
--import-alias "@/*"
cd test-reduced-platform-binaries
# get disk usage of node_modules
du -sh node_modules # 1.3G
# write a `.pnpmfile.cjs` to remove downstream `@next/swc-{platform}-{arch}*`
# `dependencies` + `optionalDependencies`
echo '
const os = require("node:os")
module.exports = {
hooks: {
readPackage: (pkg, _context) => {
if (pkg.name === "next") {
Object.keys(pkg.optionalDependencies)
.filter(
(optionalDependency) =>
optionalDependency.startsWith("@next/swc-") &&
!optionalDependency.startsWith(`@next/swc-${os.platform()}-${os.arch()}`),
)
.forEach((optionalDependency) => {
console.log(`Omitting optional dependency "${optionalDependency}"`)
delete pkg.optionalDependencies[optionalDependency]
delete pkg.dependencies[optionalDependency]
})
}
return pkg
},
},
}
' > .pnpmfile.cjs
# regenerate the dependency tree
rm -rf node_modules pnpm-lock.yaml
pnpm install
# get disk usage of node_modules again
du -sh node_modules # 302M
# still works
pnpm lint
pnpm dev
pnpm build && pnpm start |
|
I tried this out and did get into issues with it - the dependencies for the other platforms won't be stored into the lockfile at all, so once you push into a CI provider running a linux container rather than a local darwin (macos) platform, you'll probably fail to run the build |
|
Yeah this is a huge pain, if anyone finds a solve that would be really nice |
|
Create a .pnpmfile.cjs in your project root: // .pnpmfile.cjs module.exports = { regenerate installrm -rf node_modules pnpm-lock.yaml This approach reduced node_modules from ~1.3G → ~300M in practice. |
|
The problem still exists and the above solution didn't fix the problem. |
|
I've also run into this issue and would like to share some updates which may be helpful.
Most packages put Results for an app initialized with
So I get identical results (and checking the contents of
onlyBuiltDependencies:
- sharp
- unrs-resolver
# Ignore musl glibc dependencies
ignoredOptionalDependencies:
- "*-musl*"
- "*-linuxmusl*"
/**
* Hooks into the pnpm install process by refusing to install `musl` dependencies.
*
* The background for this is that Next.js will install both `glibc` and `musl` dependencies, because it cannot only target one or the other. This is redundant, since we only build for glibc distributions (Debian). By stripping musl dependencies, we save ~200 MB in the final Docker image, as well as install time.
*
* This hook effectively removes any dependency that ends in `musl` or `linuxmusl`, when reading package.json. This means those dependencies will not be present in the lockfile. When building a Docker image, the file MUST be present when running `pnpm install` (by being copied to the image, or by being mounted). Otherwise, pnpm will compute a different checksum for the lockfile and fail to install.
*
* @see For information on pnpmfile, https://pnpm.io/pnpmfile
* @see For information on Next.js, https://github.com/vercel/next.js/discussions/59821
*/
module.exports = {
hooks: {
readPackage(pkg) {
if (!pkg.optionalDependencies) {
return pkg;
}
for (key in pkg.optionalDependencies) {
if (key.includes("-musl") || key.includes("-linuxmusl")) {
delete pkg.optionalDependencies[key];
}
}
return pkg;
},
},
};
FROM node:24 AS base
RUN npm install --global corepack@latest && corepack enable pnpm
WORKDIR /app
# Add `.pnpmfile.cjs` if using. Note: This changes the checksum of the lockfile, so rename or remove it from the repository if you're not using it, or you'll have a mismatch.
# COPY package.json .pnpmfile.cjs pnpm-*.yaml ./
COPY package.json pnpm-*.yaml ./
RUN pnpm install --frozen-lockfile
COPY . .
RUN pnpm build
RUN du -sh node_modules |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Summary
I've created a new next app, and was surprised to observe the local environment's
node_modulesto be over 1 GB. I am glad that optimized prod output with standalone mode is quite trim, but CI and developer machines suffer here. \My team creates corporate guidance around Next.js use, and this a heavy installation cost for users on a capped internet connection, remote users with low internet speeds, or users with very tight disk space. Does anyone know if this is intentional behavior, something that has changed, something that can be optimized, etc?
Re-creation
Info
Size
Research
At one point, @timneutkens stated only the supported version should be installed
#29653 (comment)
Why are all of these platform-specific dependencies installed?
They seem to be direct dependencies now.
(EDIT!: It was pointed out to me that
npm viewwill display the optional dependencies here too - and you can confirm the lockfile representation correctly specifies optionalDependencies - like in https://github.com/vercel/examples/blob/main/starter/next-news/pnpm-lock.yaml#L2192-L2201)(END EDIT)
@felixmosh looked into this at some point too via #33398 (reply in thread)
Moreover, the contributing docs explicitly mention using
optionalDependenciestoo:Is that up to date? Did this change out from under the team at some point? The
package.jsonseems to be programmatically generated, as I don't see any of these swc binaries in https://github.com/vercel/next.js/blob/canary/packages/next/package.json and their inclusion at the end of the list above hints to be they are added perhaps in memory and pnpm never has a chance to save / reorder them.Additional information
No response
Example
No response
All reactions