Feedback on Yarn 3 migration from Yarn classic

[nickserv]

I recently experimented with migrating some projects from Yarn classic/1 to Yarn 3/berry. So far I have noticed significant performance improvements and some workflow improvements, but I've also spent a bit of extra time trying to figure out config options and workflow changes that seemed pretty confusing to me. I greatly appreciate the Yarn team's work on all these new tooling features, but some of these changes are causing friction in me migrating my projects, and I'm hoping that my suggestions on improving the migration process could make it easier for users to switch to them.

1. The docs specifically say that PNP is recommended for new projects, but in the Discord I was recommended to also enable it on existing projects. I think it would be better if the docs kept the separate PNP section but recommended it more directly for example "We recommend PNP for most projects, but if you have issues with it, you can fall back to the node modules linker temporarily and still upgrade yarn". I also think it would be better if running yarn in an existing project automatically switched to PNP. If users don't want to have to make any changes to their code, there's a good change they will continue to use Yarn Classic anyway.
2. I feel like the default behavior of using zero install isn't best for most projects, as it relies on a different workflow than Yarn users are used to, and commits a lot of assets to git (though if you wanted to make this less of an issue you could recommend git-lfs). In my experience most JS package manager users want a global cache for faster installs and smaller dependencies. I would personally prefer to have the zero install file generation off by default, ignore caches in git by default (editing gitignores by hand can be error prone if the user isn't familiar with them, especially if they've already committed the wrong files), and use enableGlobalCache by default. Alternatively instead of adding the configuration locally by default, the default config behavior could be changed to enable enableGlobalCache unless in a zero install project (as to not interfere with those committed files).
3. The migration docs don't mention using loose mode for incompatible dependencies, this could be very helpful for users of create-react-app for example.
4. The overall marketing text on the website doesn't seem to make the distinction between Yarn classic, berry, and latest clear. For example the version 3 docs have a banner saying "This documentation covers Yarn 2. For 1.x docs, see classic.yarnpkg.com.", which isn't really correct as the documentation covers Yarn 3. I think Berry should be used in place of 3 here, and it would be more consistent with how yarn set version berry always updates to the latest version of Berry. Also it doesn't seem to make sense that "Yarn latest" refers to Yarn 3 in some cases, since the latest tag of the yarn npm package is Yarn 1, and Yarn 3 is actually Yarn berry.
5. The output of yarn init should be as similar to the documentation as possible to encourage best practices and reduce confusion. Currently it generates a different gitignore, and adds some files like .editorconfig that aren't necessary on all projects (for example I use Prettier so is an extra file I have to remember to delete).
6. It seems unusual that PNP relies on custom editor configuration. I think it would be better to promote a standard interface for PNP tooling, and then develop patches or extensions for existing editors. Currently it seems like maintainers have to decide between telling users to manually set up their tools, or force them to use specific tools that have their configs committed. Contributors to my packages are used to the Yarn Classic / npm workflow that's supported by default in most editors, and I'd rather not have them learn new commands or use specific tools to continue contributing.

[kmannislands]

I'm also working on migrating a large existing yarn classic + lerna monorepo to yarn 3 and I agree with almost all of these points.

On 2. I agree that git lfs should definitely be called out.

You have this line in the docs:

Yes, very much. To give you an idea, a node_modules folder of 135k uncompressed files (for a total of 1.2GB) gives a Yarn cache of 2k binary archives (for a total of 139MB). Git simply cannot support the former, while the latter is perfectly fine.

I don't think this is accurate. Git may perform okay-ish with 139 MB of zipped yarn cache at a single point in time but as packages are added in subsequent commits, this will definitely compromise git's performance without LFS. Since the files are checked in zipped I don't think git's packing algorithm(s) can do much. I would expect non-shallow clones would become problematic quite quickly.

4. is interesting. I've generally been struggling with the yarn set version berry thing instead of npm i -g [email protected] for example. It's not unprecedented to have tools like yarn bootstrap themselves with the "correct" version but it's definitely not a popular pattern in the js ecosystem AFAIK.

I've seen numerous issues/discussions around this confusion so I think it's something that the maintainers should contemplate.

[arcanis]

Since the files are checked in zipped I don't think git's packing algorithm(s) can do much

@andreialecu made some benchmarks, and using compressionLevel: 0 (ie no compression) was indeed better for Git use-cases, as Git was seemingly able to diff and optimize the changesets in various cases. My objection to making it the default was that it would look counter-intuitive to many people 🤔

4. is interesting. I've generally been struggling with the yarn set version berry thing instead of npm i -g [email protected] for example. It's not unprecedented to have tools like yarn bootstrap themselves with the "correct" version but it's definitely not a popular pattern in the js ecosystem AFAIK.

Funnily the first time I came across this pattern, it was in the JS ecosystem, with grunt shelling out to grunt-cli 😄 but yes, it's not a widely known practice - with good reason, since the package manager is often the gate through which you access your binaries anyway (yarn eslint), which avoids problems with global binaries. Unfortunately, the package managers themselves aren't guarded against this ...

In any case, the fix for this is Corepack, which has been progressing nicely lately (now shipped in Node 16.9+, although still opt-in).

[andreialecu]

@andreialecu made some benchmarks, and using compressionLevel: 0 (ie no compression) was indeed better for Git use-cases, as Git was seemingly able to diff and optimize the changesets in various cases. My objection to making it the default was that it would look counter-intuitive to many people

More about this here:
#3448 (comment)

[arcanis]

Hey Nick, I was in vacations and missed your post -

1. I also think it would be better if running yarn in an existing project automatically switched to PNP

This used to be the case, but we changed it in 3.0 to automatically enable the node_modules linker when porting existing projects. We realized that people are rarely happy about two breaking changes (first the migration from 1 to 2 in general, with the different setting format etc, and then PnP itself which may require some decent time investment). As a result we feel it's better to split this migration in two, which cuts most of the issues people had with the migration.

Yes, PnP is still recommended because it solves many soundness, performance, and stability issues, but not everyone wants that at the same time as they upgrade to the latest "maintained" version.

2. In my experience most JS package manager users want a global cache for faster installs and smaller dependencies. I would personally prefer to have the zero install file generation off by default, ignore caches in git by default, and use enableGlobalCache by default.

It was like this in Yarn 1, and virtually noone knew about the offline mirror - which I truly believe is one of the most essential feature in Yarn, as I saw it countless time let us deploy when the npm registry was down and the yarnpkg twitter account was swamped of support requests. I understand the default may not be for everyone, but at least you know it's there.

That said, there are some improvements that could be done to yarn init to let you disable it by default... not sure yet.

4. The overall marketing text on the website doesn't seem to make the distinction between Yarn classic, berry, and latest clear. For example the version 3 docs have a banner saying "This documentation covers Yarn 2. For 1.x docs, see classic.yarnpkg.com.", which isn't really correct as the documentation covers Yarn 3.

I think this got fixed recently. In any case, we welcome any PR that would help at clarifying those sections (and preparing them for the many Yarn versions to come, since we'll likely start the work on the next major early next year).

6. It seems unusual that PNP relies on custom editor configuration. I think it would be better to promote a standard interface for PNP tooling, and then develop patches or extensions for existing editors.

I don't have the energy to dive into the topic again, but in short: what we do doesn't matter if there's no one picking up the phone. The solution to the IDE problems, to my knowledge, isn't doing X or Y instead of Z, but rather getting someone from those IDE teams actually committed to discuss whether the solution is X, Y, or Z.

As a side note, the JetBrains folks actually came here back when the 2.0 was still in RC, to make sure our softwares were compatible. I enjoyed this interaction very much, and I'm still hoping to see other IDEs knock to the door someday. In the meantime, we'll do what we can do / have to do.

[andreialecu]

I don't think this is accurate. Git may perform okay-ish with 139 MB of zipped yarn cache at a single point in time but as packages are added in subsequent commits, this will definitely compromise git's performance without LFS. Since the files are checked in zipped I don't think git's packing algorithm(s) can do much. I would expect non-shallow clones would become problematic quite quickly.

The community has tested the git LFS approach and it didn't really work out well.

LFS equals LARGE files storage, but the packages are mostly in the several hundred kb range, and there are tons of them.

LFS is very slow in this scenario, and it's very hard to work with because LFS related tooling isn't optimal for such use cases. It was actually meant for actual large files, and ideally a small number of them.

With compressionLevel 0, git can actually store diffs of changes.

This means that you can update a package, and if only a few lines inside a js file from it changes, a new copy of the entire zip file won't be stored. This should be extremely efficient.

Here's a sample script that confirms this:

cd `mktemp -d`
git init
yarn init -2
yarn config set compressionLevel 0
VERSIONS=('1.0.0' '1.0.1' '1.1.0' '1.1.1' '1.2.0')
for version in "${VERSIONS[@]}"
do
  yarn add lodash@$version
  mv .yarn/cache/lodash-*.zip .yarn/cache/lodash.zip
  git add .yarn/cache/lodash.zip
  git commit -m bump
done
git gc
du -sh .git
git verify-pack -v .git/objects/pack/pack-*.pack

Output, size of git: 364K .git. and the other command shows diff sizes between binary deltas.

One additional benefit is that git has its own compression which can be more efficient that zip compression.

By disabling zip compression, git can more efficiently use its own compression and you actually end up with a smaller git repository, when stored.

For more about this there's some related discussion on Discord here:

https://discord.com/channels/226791405589233664/226793713722982400/819593307767177256

[andreialecu]

The berry repository itself is such a monorepo. Admittedly it's huge, over 1GB, and there's a lot of leftover junk in it specifically because it uses compression.

I've made this experimental package a while ago: https://github.com/andreialecu/yarn-cache-remover

It actually nukes .yarn/cache from a git repo, but it requires rewriting history and force pushing. I've been using it successfully to migrate from the default compressionLevel to 0 (which I use everywhere now).

Another idea that was passed around in the community was to store .yarn/cache in a git submodule separately, and that one could be nuked individually when it gets too large and replaced with a fresh one. This should still be a lot better than using LFS, but the tooling around it isn't great. One idea could be to make a yarn plugin that automatically handles the git submodule specifics.

[arcanis]

@arcanis @andreialecu Do you know of any examples of large monorepos using yarn with checked in cache that have a lot of commits? It would be interesting to see what they do.

We used that at both Facebook and Datadog. Hasn't been a problem so far (we however optimize our CI to avoid recloning repositories as much as possible; I don't remember if GH Actions do that out of the box 🤔).

[arcanis]

Oh, and I forgot another option: GitHub and Gitlab added last year support for Partial Clone (not to confuse with --depth 1 clones), which is intended to let repositories deal with large artifacts.

[RuniBr]

@arcanis This is going to be an incredibly cheeky and selfish request from my part. You're probably going to think answering my question requires far too much time and effort explaining details on a highly complex subject.

Me and and my workplace team have begun investigating ways we can improve the performance of our Jenkins pipeline, and one of our interests was related to how we check out the full Git repo for every single build.

Would it be ridiculous to ask if you have some suggestions w.r.t. CI optimisations you found making a noticeable difference?

Thank you for all the great work you and the rest of the Yarn team do, me and my team continue to enjoy the countless benefits and improvements it offers! 🙏

[andreialecu]

@RuniBr I'm using CircleCI but one thing that I found worked was to cache the .git folder, which causes subsequent pulls to only grab what changed lately instead of doing a full clone.

Relevant CircleCI config:

    steps:
      # cache .git once per week
      - run: /bin/date +%V%Y > /tmp/weeknum.txt
      - restore_cache:
          keys:
            - repo-source4-{{ checksum "/tmp/weeknum.txt" }}
            - repo-source4-
      - checkout
      - save_cache:
          key: repo-source4-{{ checksum "/tmp/weeknum.txt" }}
          paths:
            - ".git"