Tighten up nohoist rules to speed up yarn

[willgriffiths]

Summary

Bring Yarn install times down from 30mins to 1 minute.

Details

Hi @sibelius , I'm in the process of writing up a blog post detailing how to improve yarn times and I saw your repro and thought I could help.

The key to the improving the yarn speeds here is to tighten up nohoist rules to speed up yarn.

From the original blog post when nohoist was released:

A word of caution
While nohoist is useful, it does come with drawbacks. The most obvious one is the nohoist modules could be duplicated in multiple locations, denying the benefit of hoisting mentioned above. Therefore, we recommend to keep nohoist scope as small and explicit as possible in your project.
Source: https://yarnpkg.com/blog/2018/02/15/nohoist/

We need the nohoist rules so that react-native and react-native native dependencies can be linked easily and found by cli tools.

In the docs and various github issues the recommended pattern for adding nohoist rules is the way you have it setup:

 "workspaces": {
    "packages": [
      "packages/*"
    ],
    "nohoist": [
      // The wildcard "**" means any package
      // This rule will nohoist every react-native dependency anywhere in the repo. 
      "**/react-native",
      // This rule will nohoist every dependency of every dependency of react-native
      "**/react-native/**"
    ]
  } 

The problem starts to appear when we add modules that depend on react native:

my-app -> react-native
              -> @react-native-community/datetimepicker -> react-native (dev dep)

The nohoist rules mean that we now have two versions of react-native being installed. And because we have the deep nohoist rule we are also nohoisting all of react-native's 64 dependencies for every copy of react native.

The more packages you add that depend on react-native the worse the problem gets.

The solution is to tighten up the nohoist rules. It's a little harder to maintain. And sometimes you have to deal with hoisting issues manually but it's worth it to make install times bare-able (until yarn 2 supports react native 🐈 .

So for the project above the rules become:

    "nohoist": [
      "my-app/react-native",
      "my-app/@react-native-community/datetimepicker"
    ]

Results

Most of the install time is the process of copying files. The best proxy I've found for cutting install times —without having to wait for the install to finish— is the progress bar shown during the yarn install. It shows the number of files being copied/linked out of the total. It's actually shown a few times but we can focus on when it has the largest numbers.

// In your repro you can see that there are 876k files being copied/linked
[############---------------------------------] 231769/876044
Yarn time: ~1889.90s (31.5m)

// After tightening up the nohoist rules we get it down to 92k
[######-----------------------------------------] 11457/91923
Yarn time: ~ 71s

Note: Both yarn times are with a yarn cache and yarn.lock file but without node_modules.

Other notes

In the PR I've added all to the nohoist rules for all react-native packages. This is simpler and does the job but could start to slow down after more packages and apps are added. In our monorepo, we normally only nohoist a package if react native, xcode or gradle complain. Either way works fine. :)

If you have any feedback about this info let me know and I'll incorporate it into the blog post. :)

[sibelius]

thanks to make this work fast

and to provide more information what is happening under the hood

we got our monorepo setup with react-native working without any nohoist

But I think this could be useful to show how to improve when you can't avoid nohoist at all

[willgriffiths]

we got our monorepo setup with react-native working without any nohoist

Hi @sibelius, can I ask what approach you used to get react-native working in a monorepo without nohoist? I'm revisiting this problem at the moment.

[sibelius]

this is the best "documentation" I have of how to setup the monorepo without nohoist

https://twitter.com/sseraphini/status/1202217314638282752

ping me on twitter if you find any trouble

we could write a proper article/post about it

[willgriffiths]

Thanks for the link. It seems pretty straight forward. I'll give it a go.