-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Fix race condition in functions discovery shutdown #9453
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Fix race condition in functions discovery shutdown #9453
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
Summary of ChangesHello @tilgovi, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request resolves a critical race condition in the function discovery shutdown mechanism for both Node.js and Python runtimes. By proactively attaching exit and error event listeners to child processes as soon as they are created, the system can now reliably detect and respond to process terminations. This change guarantees that shutdown promises are always honored, leading to a more stable and predictable termination sequence for discovery processes and preventing potential resource leaks or hanging processes. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request aims to fix a race condition in shutting down functions discovery processes by attaching event handlers to subprocesses earlier. The approach of creating the exit promise (pExit) before returning the shutdown function is correct and effectively addresses the race condition.
However, I've found critical issues in both modified files that would lead to runtime errors:
- In
node/index.ts, there's aReferenceErrorbecausequitTimeoutis not defined, along with a syntax error. It seems to be a copy-paste issue. - In
python/index.ts, there's a syntax error in thefinallyblock.
I've left specific comments with suggestions to fix these problems. Once these are addressed, the change should be good to go.
|
Without this, failures of the emulator to start a function (such as do to runtime error in the function's entrypoint) seem to just hang for a number of seconds and then the deploy command exits with code 0. I can't figure out where that timeout is happening or where the error is getting swallowed. With the change, a failure to start the function immediately results in the deploy command exiting with a non-zero exit code. |
Attach event handlers to subprocesses before yielding to calling code to ensure that promises of shutting down discovery processes are fulfilled.
5d91b7e to
867c632
Compare
|
/gemini review |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
The changes in this pull request correctly address a race condition in the shutdown logic for both Node.js and Python function discovery processes. By attaching the 'exit' and 'error' event handlers immediately after the child process is spawned, it ensures that the shutdown promise will always be settled, even if the process terminates unexpectedly early. Additionally, the updated timeout handling, which now correctly clears the SIGKILL timeout when the process exits, is a great improvement that prevents potential resource leaks and fixes a bug in the Python runtime where the timeout was cleared immediately. The code is clean and the fix is well-implemented.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since serveAdmin is an async function, it already returns a Promise. You can remove the explicit Promise.resolve wrapper for simplicity.
| return Promise.resolve(async () => { | |
| try { | |
| await fetch(`http://127.0.0.1:${port}/__/quitquitquit`); | |
| } catch (e) { | |
| logger.debug("Failed to call quitquitquit. This often means the server failed to start", e); | |
| } | |
| const quitTimeout = setTimeout(() => { | |
| if (!childProcess.killed) { | |
| childProcess.kill("SIGKILL"); | |
| } | |
| }, 10_000); | |
| clearTimeout(quitTimeout); | |
| return new Promise<void>((resolve, reject) => { | |
| childProcess.once("exit", resolve); | |
| childProcess.once("error", reject); | |
| return pExit.finally(() => { | |
| clearTimeout(quitTimeout); | |
| }); | |
| }); | |
| return async () => { | |
| try { | |
| await fetch(`http://127.0.0.1:${port}/__/quitquitquit`); | |
| } catch (e) { | |
| logger.debug("Failed to call quitquitquit. This often means the server failed to start", e); | |
| } | |
| const quitTimeout = setTimeout(() => { | |
| if (!childProcess.killed) { | |
| childProcess.kill("SIGKILL"); | |
| } | |
| }, 10_000); | |
| return pExit.finally(() => { | |
| clearTimeout(quitTimeout); | |
| }); | |
| }; |
Attach event handlers to subprocesses before yielding to calling code to ensure that promises of shutting down discovery processes are fulfilled.