Feat/retry 429 functions #9252
Feat/retry 429 functions #9252
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
Summary of ChangesHello @Zloka, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the reliability of Firebase Functions deployments by introducing an exponential backoff retry mechanism for HTTP 429 "Quota Exceeded" errors. By automatically retrying API calls that hit rate limits, the changes prevent deployment failures, particularly for projects with many functions, leading to more stable CI/CD pipelines. The implementation uses a new utility for HTTP retries, specifically tailored for Cloud Functions operations, ensuring that deployments eventually succeed even under API pressure. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces exponential backoff for Cloud Functions API calls that fail with a 429 (Quota Exceeded) error, which is a great improvement for deployment stability, especially for projects with many functions. The implementation uses the exponential-backoff library and introduces generic helper functions for retrying HTTP requests. The changes are applied to create, update, delete, and generateUploadUrl operations for both GCFv1 and GCFv2. My review focuses on the correctness of the retry logic and its integration. I've found one configuration issue that could unnecessarily slow down deployments, which I've detailed in a comment.
Zloka
force-pushed
the
feat/retry-429-functions
branch
from
cea3ab4 to
d9167c0
Compare
|
Rebased on |
This library seems to be used in other parts of the repo. Thus, it makes sense to stick to it in this case as well, to keep things consistent.
The idea is that this provides a generalized way to retry a 429 error, with exponential backoff.
Zloka
force-pushed
the
feat/retry-429-functions
branch
from
d9167c0 to
32827c8
Compare
Description
#3919, while closed, is still an issue and has been for around 4 years now.
firebase-tools logs that retrying will happen, but instead, an error is actually thrown.
This PR implements retries using
exponential-backoff(a dependency which you seem to use already to some extent), and a couple of helper functions, with what I believe to be some sensible defaults.Opinions wanted on e.g. the delay timers, as I don't actually know your rate limits on these commands, but at least saw mention of a one-minute window somewhere, and these values seemed to provide a good result.
Scenarios Tested
In practice, I have experienced issues with 50-70 functions. I tested creating/updating 200 dummy functions, and after some retries, the deployment succeeded in about 15 minutes. Myself, I'm not too concerned about the speed, and that does not seem to concern those commenting in the issue either - mainly, a stable deployment procedure to avoid CI/CD failures is prefereable.
Here is the tail-end of the output, for reference:
Notes
This is my first PR to this project, don't hesitate to point out any shortcomings and I'll amend it - with that said, I would much appreciate if this PR is considered, as it would help reduce pipeline flakiness for many.