Feature Request - mutex option with cache clean

[jamesongithub]

Do you want to request a feature or report a bug?

What is the current behavior?
yarn clean act independently of yarn install

If the current behavior is a bug, please provide the steps to reproduce.

What is the expected behavior?
We clean our yarn cache often on our build servers. These happen at the same as our yarn installs. Sometimes we get strange errors, like missing files during builds. Would be nice if these worked together.

Please mention your node.js, yarn and operating system version.

yarn --version
0.21.3
node --version
v6.9.2
cat /proc/version
Linux version 3.13.0-96-generic (buildd@lcy01-09) (gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.3) ) #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016

[bestander]

I like the idea!
Want to send a PR?

[olingern]

@bestander I'd like to jump back in and tackle this and #1845.

[bestander]

Thanks, @olingern, please do!

[olingern]

Looking over install.js's init function, I could add a flag to disable cleaning.

Something like yarn clean && yarn install --noClean seems like it takes care of this use case.

Thoughts @bestander @jamesongithub ?

[bestander]

I think the clean in install is different, that one cleans node_modules from files like README and tests.

This is more like calling cache clean command at the start of install, before resolution maybe.

After thinking about it I think you can solve this for your CI needs another way.
Just provide a unique (temp) folder for caches via --cache-folder argument.
Then caches won't be interfering with each other.

[jamesongithub]

so we already have multiple cache folders on our build servers (one / build slot). this issue is how do clean these while installs are running and using the cache?

[bestander]

Do you want to clean cache after the install to save disk space for example?

[jamesongithub]

we moved to multiple cache folders since running concurrent yarns with mutex was causing waiting times from 20-30 minutes

[jamesongithub]

yeah. after some time, builds servers will run out of space. since we're using 16 cache dirs. it fills up fast.

[jamesongithub]

what would be nice is per package installs and have those shared across yarn installs. maybe using symlinks. then for installs/cleans you could lock individually on those instead. then we wouldn't have to have multiple cache dirs. space would also be less of an issue. in this scenario i think mutex would be on by default.

[bestander]

Yeah, it would be great to make Yarn resilient to concurrent installs and lock individual cache folders.
Then the problem would be solved for many.
That could be a bit complex but you could build something on top of the current locking solution we use in install