Cache xbuildenv releases files

[ryanking13]

Description

We have a pyodide-cross-build-environments.json metadata file that stores information on pyodide cross build environments per version.

pyodide-build fetches this file whenever it needs to install cross-build environments. However, that JSON file is rarely updated (once a quarter), and querying that file every time is unnecessary. (related file)

I think we could update

1. pyodide-build to download that file once, store it in the file system.

2. Reuse the downloaded file instead of querying it every time.

3. If the file is too old, say mtime is 72 hours behind the current time, re-download it (we can make this value configurable)

4. Add a pyodide xbuildenv subcommand that users can use to force re-download the metadata file.

[agriyakhetarpal]

Thanks, @ryanking13 – this sounds good to me. I would only suggest a small change to point 3: instead of checking the last modification time, I'd suggest using the checksum to ensure that we always have the right files. There is a dependency that we can use for this: https://pypi.org/project/pooch/. While it may or may not be a goal, it will improve the security of our supply chain as well, as a by-product.

[ryanking13]

: instead of checking the last modification time, I'd suggest using the checksum to ensure that we always have the right files. There is a dependency that we can use for this: https://pypi.org/project/pooch/

Thanks. Makes sense. I think we should do both? Checking the checksum is for security, and checking the mtime is to avoid using an outdated xbuildenv release file.

[agriyakhetarpal]

No, if the checksum for the xbuildenv is the same, then we are fine to use it, right? If the xbuildenv is outdated, then the checksum would not match.

[ryanking13]

But checking the checksum would require making an HTTP request anyway, right? My intention was to avoid making an HTTP request every time.