Migrate to pnpm: 3x Faster Installs, 50% Less Disk Space, Better Dependency Management

[mawburn]

Summary

I'd like to propose migrating LibreChat from npm to pnpm. After extensive experience with both package managers, I believe pnpm offers significant advantages that align perfectly with our project's needs, particularly for our monorepo structure. The migration effort is minimal, while the benefits - faster installations, massive disk savings, and stricter dependency management - are substantial and ongoing.

Why pnpm?

1. Performance: 2-3× Faster Installations

pnpm achieves dramatically faster installation times through its innovative approach:

• Hard links instead of copies: When installing packages, pnpm creates hard links from a global store rather than copying files
• Parallel operations: More efficient parallel processing of package installations
• Intelligent caching: Reuses previously downloaded packages across all projects on your machine

Real-world impact:

• Faster CI/CD pipelines (reducing build times and costs)
• Improved developer experience with quicker dependency updates
• Less time waiting for install commands during development
• Consistent builds across all environments - no more "it works locally but fails in CI" issues

2. Disk Space: Up to 50% Savings

pnpm's content-addressable storage system revolutionizes how packages are stored:

• Global store: All packages are stored once in a global location on your disk
• Deduplication: Different versions of the same package share common files
• Hard linking: Projects reference packages from the global store rather than maintaining copies

Example: If 100 projects use React 18.2.0, npm stores 100 copies, while pnpm stores it once and creates 100 hard links.

3. Dependency Isolation: Preventing "Phantom Dependencies"

What are Phantom Dependencies?

Phantom dependencies occur when your code imports packages that aren't explicitly listed in package.json but happen to be installed by other dependencies.

Real example from LibreChat:

https://github.com/danny-avila/LibreChat/blob/main/api/cache/keyvMongo.js#L4

// Your package.json only lists "mongoose"
// But mongoose internally depends on "mongodb"

// This works with npm (but shouldn't!):
const { GridFSBucket } = require('mongodb');

// With pnpm, this correctly throws an error
// forcing you to explicitly add mongodb@^6.14.0 to package.json

Another example:

http://github.com/danny-avila/LibreChat/blob/main/api/server/services/Files/Audio/STTService.js#L3

// Multiple packages (axios, openai, supertest) bring in "form-data"
// Your code starts using it without explicitly installing it:
const FormData = require('form-data');

// Works in npm due to hoisting, but breaks with pnpm
// The fix: explicitly add form-data@^4.0.0 to package.json

Why this matters:

• Reliability: Code works consistently across different environments
• Transparency: All dependencies are explicitly declared
• Future-proofing: If express removes cookie-parser in a future version, your code won't mysteriously break
• Team consistency: No more "works on my machine" issues from implicit dependencies

4. Superior Monorepo Support

pnpm was designed with monorepos in mind:

• Workspace protocol: Native support for cross-package dependencies
• Filtering commands: Run scripts in specific packages or groups
• Shared dependencies: Automatic hoisting of common dependencies
• Recursive commands: Execute commands across all workspace packages

Example commands:

# Run tests only in packages that changed
pnpm --filter ...[origin/main] test

# Build all packages that depend on @librechat/ui
pnpm --filter ...@librechat/ui build

Pros and Cons Analysis

Pros ✅

1. Performance
   • 2-3× faster installations
   • Reduced CI/CD build times
   • Faster local development cycles
2. Resource Efficiency
   • 50%+ disk space savings
   • Lower bandwidth usage (fewer downloads)
   • Reduced memory footprint during installations
3. Code Quality
   • Enforces explicit dependencies
   • Prevents accidental reliance on transitive dependencies
   • Catches dependency issues early in development
4. Developer Experience
   • Intuitive CLI with helpful error messages
   • Compatible with npm commands
   • Better monorepo tooling
   • Works seamlessly with Bun runtime
5. Ecosystem
   • Growing adoption by major projects
   • Active development and community
   • Excellent documentation

Cons ❌

1. Migration Overhead

   • One-time installation required for all contributors
   • Need to update CI/CD pipelines
   • Documentation updates required
   • New lockfile format to learn

2. Compatibility Considerations

   • Some tools expect flat node_modules structure
   • Tools parsing package-lock.json need updates
   • Potential issues with packages using improper module resolution

3. Learning Curve

   • Team needs to understand new concepts (global store, hard links)
   • Different troubleshooting approaches
   • Muscle memory adjustment from npm to pnpm

4. Strictness

   • May reveal existing dependency issues (though this is ultimately beneficial)
   • Requires more explicit dependency declarations
   • Can initially slow development when fixing phantom dependencies

5. Windows-Specific Challenges

   • Symlink permission issues: Windows requires admin privileges to create symlinks by default
   • Long path limitations: Windows has a 260-character path limit that can cause issues
   • Performance: Hard link operations can be slower on Windows file systems

   Windows Solutions:

   • Enable Developer Mode (Windows 10/11): Settings → Update & Security → For Developers → Toggle "Developer Mode"
   • Or run terminal as Administrator when using pnpm
   • Enable long paths: Run git config --system core.longpaths true and enable in Windows registry
   • Consider using WSL2 for better performance and Unix-like behavior

Comparison with Alternatives

Why not stay with npm?

• While npm v7+ added workspaces, it remains slower and more disk-intensive
• Lacks strict dependency isolation
• No built-in solution for phantom dependencies

Why not Yarn?

• pnpm is faster and more disk-efficient than Yarn
• Better monorepo support with more intuitive filtering
• Stricter dependency resolution
• Yarn 2+ (Berry) introduces a completely different paradigm:
  • Uses Plug'n'Play (PnP) by default instead of node_modules
  • Requires special SDK integrations for IDEs and tools
  • Zero-installs approach stores dependencies in git
  • Many packages and tools don't work without compatibility workarounds
  • Significantly different from npm/Yarn Classic workflows

Bun compatibility

• pnpm works perfectly with Bun runtime
• Developers can choose Node.js or Bun after installing with pnpm
• No lock-in to a specific runtime

Bottom Line

The migration to pnpm represents a strategic investment in our development infrastructure. While there's a one-time migration cost, the ongoing benefits - faster builds, disk savings, and improved dependency management - make this a compelling upgrade for LibreChat's monorepo architecture.

The strict dependency isolation alone will prevent numerous "works on my machine" issues and improve our code quality. Combined with the performance improvements and disk savings, pnpm offers a modern, efficient solution for our package management needs.

I'm happy to address any questions or concerns about this proposal. What are your thoughts on making this transition?

[mawburn]

Here is the timing from a fresh install with all node_modules deleted:

npm install 14.62s user 29.95s system 105% cpu 42.051 total

pnpm install 1.57s user 1.23s system 88% cpu 3.170 total

This should resolve our recurring CI issues where builds randomly fail due to missing peer dependencies. With npm, these dependencies unpredictably appear as unmet requirements even though they should already be available through the dependency tree.

Why this keeps happening with npm (and why pnpm fixes it):

npm's flat node_modules structure creates "phantom dependencies" - packages can accidentally access dependencies they didn't explicitly declare. This works on one developer's machine but breaks in CI or on another teammate's setup. The same package.json can even produce different node_modules structures depending on installation order or npm version.

pnpm solves this through:

• Strict dependency isolation: Packages can only access their declared dependencies, catching missing deps immediately in development
• Deterministic installations: Same dependencies = same structure every time, eliminating "works on my machine" issues
• Better peer dependency handling: More explicit and consistent, preventing those surprise CI failures

[danny-avila]

My team has hit this exact issue with LibreChat installs

It would be good to see how exactly your team is installing/building, so I can compare with npm run reinstall, as I mentioned here:
#7874 (comment)

[mawburn]

I didn't see this first, so I replied there much more thoroughly. Basically, it's hard to really show because it's consistently inconsistent. We are exclusively on Macs for local dev, but these issues have popped up in our image builds which are on Linux. I know most of you work on Windows. But, even across our different machines things would be installed differently or have weird little issues where some dependencies wouldn't be picked up for some reason or another. It's actually something I've been quietly fighting the whole time I've deploying LibreChat for us.

The tl;dr is that pnpm makes things consistent for everyone because npm masks a lot of issues that it shouldn't be. The issues addressed outside the direct pnpm changes and updates to the dependencies are all pre-existing problems that the flat node_modules structure of npm was hiding in one way or another.

The cross section of the issues I had to address in that PR should give you an idea of what I mean by "consistently inconsistent." But migrating to a package manager that forces better consistency should fix these issues for anyone having them.

[mawburn]

Just to be clear, this isn't something I believe is just affecting us. It's likely affecting a lot of people in different ways and either they are being silent about the issues, or don't know the root cause.

[mawburn]

Just documenting what I've done for this.

I created this PR to help with this before making this change. But it was denied due to some big changes that are in flux, but hopefully once those are ironed out we can revisit this. It alleviates most of the major problems when it comes to the pnpm migration, but is just good for the project overall regardless of if the project moves pnpm or not.

• chore: Fix missing dependencies and remove unused ones #8417