mulled-build enhancements for dealing with quay.io timeouts/errors#22104
mulled-build enhancements for dealing with quay.io timeouts/errors#22104bgruening wants to merge 6 commits intogalaxyproject:devfrom
Conversation
github-actions
bot
added
area/testing
area/tool-framework
area/tool-dependencies
labels
|
It may be worth using https://pypi.org/project/requests-cache/ ? @bernt-matthias Would you like to review this one? @bgruening Let me know if you need help to fix the mypy errors. |
Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com>
| tag, | ||
| response.status_code, | ||
| ) | ||
| repo_data = quay_repository(namespace, pkg_name, session=session) |
There was a problem hiding this comment.
How likely is that we end up here? Because we risk calculating this twice, once here then again in mull_targets() in the branch if "push" in command:
There was a problem hiding this comment.
Not likely, but I could not find out why a few repos did not return tags.
Any idea how we can refactor it? Or cache the request?
There was a problem hiding this comment.
Caching with https://pypi.org/project/requests-cache/ should be easy, you just need to change how you create the session. I'd put the cache file in Galaxy data_dir .
There was a problem hiding this comment.
What do you think about the latest commit? data_dir does not exist I think if you just run mulled-build from CLI.
Still seems to work on multi-package-containers
Co-authored-by: Björn Grüning <bjoern@gruenings.eu>
| ) from exc | ||
|
|
||
| if not isinstance(data, dict): | ||
| raise QuayApiException(f"Unexpected quay.io response type for {url}: {type(data).__name__}") |
There was a problem hiding this comment.
| raise QuayApiException(f"Unexpected quay.io response type for {url}: {type(data).__name__}") | |
| raise QuayApiException(f"Unexpected quay.io response type for {url}: {type(data)}") |
| assert pkg_name is not None | ||
| assert tag is not None | ||
|
|
||
| url = f"{QUAY_REGISTRY_API_ENDPOINT}/{namespace}/{pkg_name}/manifests/{tag}" |
There was a problem hiding this comment.
Shouldn't GET /api/v1/repository/{namespace}/{repo}/tag/{tag} be used?
3 participants
I was getting more and more timeouts during different build-stages of our containers dealing with quay.io API. My suspicion is that this is due to non-ideal handling of API requests to quay.io.
Before this change,
mulled --check-publishedchecked tagged images by fetching the full Quay repository metadata and inspecting all tags for that repository. This PR switches to a direct existence check for the exact namespace/repo:tag, reducing the lookup to a single tag in the common case. Some repos returned still some strange error codes for this new API route, so I retained a fallback to the old/current method.I also tried to keep one session for all of our queries to quay.io.
I have tested this branch on BioContainers/multi-package-containers#3898 and it seems to work more reliably. But who knows, maybe I was just lucky.