Cache/use source tarballs when safe

[Taffer]

problem: huge repos that take a long time to check out (ClickHouse, CUDA and its relatives for example)

request: a way to use source tarballs instead of a checkout

• whitelist repos that generate tarballs from tags/revisions (not ones that have the tarball provided, thus avoiding the xz taint source); initial whitelist would be GitHub, probably GitLab… which probably covers most packages already, others can be added as necessary
• if you need to cherry-pick changes, you'll have to get a diff of that change, then use patch to apply it; unless horrors occur, you'll get the same result

1. if source repo isn't whitelisted: error
2. if tarball in cache and hash matches, return that
3. else get the tarball instead of checkout, add to cache
4. unpack tarball

This would not be automatic; packages that use git-checkout would need to be modified to use this instead.

wins:

• this will reduce network/cpu usage in exchange for cache storage
• we can make a central cache used by the build nodes; presumably builds happen more often than source packages get updated

next problem: how to deal with things that do git submodule update --init as the first step of a build

[Taffer]

There are a few packages (~50) using git submodule; the ClickHouse ones are insane (~250 large submodules in contrib/).

I've added the beginnings of an idea about making this less awful: https://github.com/chainguard-dev/sandbox-of-power/tree/taffer-bits/chris.herborth/submodule-proto (Will merge that to main when it runs and does something useful.)