Combined/layered/fallback caches

[kornelski]

I would like ability to read from more than one cache in parallel, and store new objects only to one of them.

I have a setup where I can't trust arbitrary build jobs to write to a shared cache (similar to the #845), but I can provide some baseline read-only pre-populated cache. Currently sccache supports a read-only mode, but having only a read-only cache is suboptimal, because objects missing from the read-only cache will remain cache misses forever. Layering another writeable cache on top of it would make it possible to have complete caches for individual build jobs, which effectively behave like pre-populated from the shared cache (without actually copying it).

It would also be useful for primitive CI cache setups that can only download and upload whole tarballs. Currently using sccache's disk cache this way pays a high cost of re-uploading the entire cache instead of only appending new entries. With layered caches, a previous cache could be only downloaded to one directory (read-only), and any newly written objects written to a different directory that will get uploaded, and an occasional job that will periodically merge the new cache into the old read-only cache.

I think it would be relatively easy to implement this in sccache by providing additional Storage implementations that wrap other storage back-ends (like https://rclone.org/combine/)

Wrapping one Storage on top of another may also make it possible to implement some transformations like #1762

I'm happy to implement this. Let me know if you're interested.