Implement registry hierarchy / cache server approach to Cargo

[haydonryan]

Problem

Love cargo!, thank you for all your efforts to make a great tool.

I really like the idea of having a local network cache to improve performance when downloading crates (I do this for pacman packages too on arch).

Currently the caches work by replacing crates.io as the registry in cargo.toml. This works well if the cache server is always up and accessible, however if a user has a laptop,and takes that outside of the network they have to manually change the configuration. Similarly if the server is down, then manual intervention would be required.

eg: current appraoch:

[source]

[source.mirror]
registry = "http://localhost:8080/index"

[source.crates-io]
replace-with = "mirror"

Proposed Solution

Introduce a dedicated cache-server property to cargo under the repository tag. Then add code that it checks if that's accessible and if so uses it, otherwise uses the registry. It could be applied per registry.
eg:

[source.crates-io]
cache-server:  "http://localhost:8080/index"

Using Pacman + Pacoloco as a model here would be awesome. Pacoloco not only proxies the package, but also (at a configurable time) pre-fetches newer versions of the packages that it's cached previously, improving the user's perceived download speed for the first iteration.

Notes

This approach provides a good middle ground vs creating a full mirror (which puts a lot of load on crates.io).

I wouldn't mind having a stab at the PR myself :)

[weihanglo]

Using Pacman + Pacoloco as a model here would be awesome.

Could you share some links / references / docs of them?

[haydonryan]

Using Pacman + Pacoloco as a model here would be awesome.

Could you share some links / references / docs of them?

Sure.

https://wiki.archlinux.org/title/Package_Proxy_Cache

https://github.com/anatol/pacoloco - the pacoloco program / docker image.

anatol/pacoloco#122 - this PR has updates on how to use the cache server, it hasn't been merged as yet.

Currently (using https://github.com/ChrisMacNaughton/cargo-cacher as an example), you have to specify a mirror, by replacing crates.io.

I started working on a PR but have come to somewhat of a design issue. I think this would make a good MVP:

[source.crates-io]
cache-server:  "http://localhost:8080/index"

However I could also see people needing more configuration options than just a single server string.

It might be better to use the reaplce-with approach as a template instead eg:

[source.mirror]
registry = "http://localhost:8080/index"
# additional configuration here

[source.crates-io]
cache-with = "mirror"

Would love to know your thoughts.