HTTP 429 error when doing `pyodide xbuildenv search`

[joerick]

Over in pypa/cibuildwheel#2002 I saw a test failure due to this exception:

  + pyodide xbuildenv search --json --all
  ----------------------------- Captured stderr call -----------------------------
  ╭───────────────────── Traceback (most recent call last) ──────────────────────╮
  │ /tmp/cibw-run-3ba5e_80/cp312-pyodide_wasm32/build/venv/lib/python3.12/site-p │
  │ ackages/pyodide_build/cli/xbuildenv.py:169 in _search                        │
  │                                                                              │
  │   166 │                                                                      │
  │   167 │   # TODO: cache the metadata file somewhere to avoid downloading it  │
  │   168 │   metadata_path = metadata_path or cross_build_env_metadata_url()    │
  │ ❱ 169 │   metadata = load_cross_build_env_metadata(metadata_path)            │
  │   170 │   local = local_versions()                                           │
  │   171 │                                                                      │
  │   172 │   if show_all:                                                       │
  │                                                                              │
  │ /tmp/cibw-run-3ba5e_80/cp312-pyodide_wasm32/build/venv/lib/python3.12/site-p │
  │ ackages/pyodide_build/xbuildenv_releases.py:239 in                           │
  │ load_cross_build_env_metadata                                                │
  │                                                                              │
  │   236 │   │                                                                  │
  │   237 │   │   with _suppress_urllib3_logging():                              │
  │   238 │   │   │   with requests.get(url_or_filename) as response:            │
  │ ❱ 239 │   │   │   │   response.raise_for_status()                            │
  │   240 │   │   │   data = response.json()                                     │
  │   241 │   │                                                                  │
  │   242 │   │   return CrossBuildEnvMetaSpec.model_validate(data)              │
  │                                                                              │
  │ /tmp/cibw-run-3ba5e_80/cp312-pyodide_wasm32/build/venv/lib/python3.12/site-p │
  │ ackages/requests/models.py:1024 in raise_for_status                          │
  │                                                                              │
  │   1021 │   │   │   )                                                         │
  │   1022 │   │                                                                 │
  │   1023 │   │   if http_error_msg:                                            │
  │ ❱ 1024 │   │   │   raise HTTPError(http_error_msg, response=self)            │
  │   1025 │                                                                     │
  │   1026 │   def close(self):                                                  │
  │   1027 │   │   """Releases the connection back to the pool. Once this method │
  ╰──────────────────────────────────────────────────────────────────────────────╯
  HTTPError: 429 Client Error: Too Many Requests for url: 
  https://raw.githubusercontent.com/pyodide/pyodide/main/pyodide-cross-build-envir
  onments.json

Full log

It seems that Github decided to start throttling responses for that URL today.

One easy way to solve this would be to enable Github Pages on the pyodide repo - then the main branch is published to the web as a website, and you'd get a URL that's unlikely to be throttled.

Or, one could try a CDN like jsdelivr. However, it's worth noting that CDNs will not be always up-to-date. I believe jsdelivr is around 12-24 hours behind a moving ref like main.

[agriyakhetarpal]

Thanks for reporting this, @joerick! It's quite strange that this is happening today and has never happened over the years – perhaps GitHub changed their internal rate limits.

I've created https://pyodide.github.io/pyodide/pyodide-cross-build-environments.json, see the above PR for more 😄

[ryanking13]

Thanks for the report. I think we should download and cache the metadata file instead of fetching it from GitHub everytime.

[agriyakhetarpal]

Yes, that's also marked as a TODO. I merged pyodide/pyodide#5642 just now, and we have a new GitHub Pages link to use now onwards: https://pyodide.github.io/pyodide/api/pyodide-cross-build-environments.json

We can leave this open for now, and close it only when we have a cached file ready.

[joerick]

I believe this was fixed in #206.