Higher memory usage when using Yarn 2 (w/ PnP) vs Yarn 1 in production environments

[bjshort]

Hi there,

We're currently testing rolling out Yarn 2 + PnP for our monorepo which previously used Yarn 1 + Lerna. We have a fairly large monorepo with several packages & independently deployable services (which run inside their own docker containers).

Yarn2 + PnP has worked really well in our development environment and has solved a number of issues for us - initially we only intended to use Yarn 2 + PnP in development but soon realised that we'd probably have to use it in all environments to avoid dependency differences.

After successfully updating our deployment pipelines to work with Yarn 2 (We're using Zero Installs too - we copy the entire monorepo inc .yarn folder into each service's docker image) we noticed that several of our services failed to boot due to running out of memory. Monitoring this showed that in some cases services were now using 3x the memory that they had when using Yarn 1.

We actually realised one of the causes we that we were running the yarn executable twice, once via yarn workspace foo-service start, which then ran yarn node dist/main - this caused an extra yarn process to start on the server. After altering the second command to just be node dist/main the total memory usage reduced by about 30%, but the overall memory usage is still a lot higher than with Yarn1.

We decided we weren't too concerned with this, so we just upped all the memory limits and took the hit. However, we've now noticing that a few of our services are actually running out of memory when under load, which we never had issues with before. One service is particular is throwing OOM errors even when increasing its memory cap to 512mb, which was 256mb previously (it rarely went above 128mb in actuality).

So, I guess it would be really useful is someone could help with the following questions:

• Has anyone else experienced these issues?
• Is Yarn2 + PnP expected to use much more memory? If so, why?
• Is there anything fundamentally wrong with the way we're using Yarn 2 + PnP in production envs? (i.e. almost exactly the same as in dev)

It's worth nothing that we have quite a few packages so Yarn 2 + PnP is using a lot of "virtual" packages to run.

Thanks a lot! Sorry for the long post!

[arcanis]

Is Yarn2 + PnP expected to use much more memory? If so, why?

There's an overhead caused by the Zip layer. To avoid opening/closing archives for each require call, we allow multiple archives to be open at the same time (80 at the moment). Check if decreasing this value (by directly editing the .pnp.cjs file generated) is enough 🤔

Another optimization that could be interesting would be to make the PnP hook a noop when executed within the Yarn CLI environment (since we don't do require calls anyway). It's however a bit tricky because the hook is supposed to be registered before the entry script executes, so it would be hard to reliably detect whether we're about to run Yarn or not.

It's worth nothing that we have quite a few packages so Yarn 2 + PnP is using a lot of "virtual" packages to run.

Having lots of virtual packages may be sign that you have a lot of dependencies that are duplicated. If the virtualized packages are expected to be singletons (for example reeact-dom), you probably should investigate that (use yarn why with its --peers option to get more info).

[bjshort]

Thanks - will try decreasing the number of open archives.

RE virtual dependencies, it seems that packages within the monorepo that depend on each other sometimes refer to each other using workspace:path-to-package but other times by using virtual:hash#workspace:path-to-package. I tried make the interdependent packages peer dependencies of each other and push the direct dependency up to the service (which i guess is also a package), rather than direct dependencies but that actually increases the number of virtual dependencies.

But yes you are right, when i run yarn why foo-package --peers I am seeing several ancestors duplicated (with different hashes but the same version).

[merceyz]

We actually realised one of the causes we that we were running the yarn executable twice, once via yarn workspace foo-service start, which then ran yarn node dist/main - this caused an extra yarn process to start on the server. After altering the second command to just be node dist/main the total memory usage reduced by about 30%, but the overall memory usage is still a lot higher than with Yarn1.

You can (and should) avoid executing yarn in the container by running node -r ./.pnp.cjs ./dist/main.js instead

https://yarnpkg.com/features/pnp#initializing-pnp

[bjshort]

Thanks a lot for pointing this out - using this command reduced memory usage on ours services back down to pretty much the same as when using Yarn 1 / node_modules 🎉 .

I definitely missed this in the docs somehow. It may be useful to position this more clearly as the recommended way to launch things? In general having a section on recommended production use would be super helpful?

[MonsterDeveloper]

Hey! I also encountered the issue of higher memory usage compared to npm.

I’m using Next.js, so my start script is yarn start (next start). Is there a way to run node -r with Next.js start script (or any other script in general)?

[ervin-pactum]

Hello, I do understand that following question might easily be more generic yarn/node question, and do apologise for being on edge of offtopic here, but for my service startup command is to call binary provided by package, /bin/bash -c yarn functions-framework ... . I do also experience about doubled memory usage after upgrading from yarn classic and into pnp zero install mode, but i'm not sure how to call binary from package using something like node -r ./.pnp.cjs (please advice what to put here). package in question is @google-cloud/functions-framework, but i think that answer how to startup using binary from zipped package will be pretty generic.