Download repo archive to reduce rate limit pressure

[juntyr]

Problem

GitHub rate limits unauthenticated repo accesses. When downloading a larger repo repeatedly, it is easy to hit the rate limit.

Proposed Solution

Perhaps we could download the archive of the repo instead and then unpack it. This would drastically reduce the number of API calls, possibly down to one.

Additional context

[brichet]

Thanks @juntyr for opening this.

I tried to download the archive without success at the very beginning.
IIRC I wasn't able to unarchive it, but I don't remember exactly what was the issue.

Did you already try it?

[juntyr]

I haven't tried it myself yet. I was imagining using a tar gz unpacker in JS (nowadays maybe written in WASM) to first get the contents tree, use that to create all directories, then upload each file