Fix GrpcWorkerChannel.StartWorkerProcessAsync timeout #10937
+966
−1,328
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jviau
commented
Mar 21, 2025
| await _rpcWorkerProcess.StartProcessAsync(); | ||
| _state = _state | RpcWorkerChannelState.Initializing; | ||
| await _workerInitTask.Task; | ||
| await _rpcWorkerProcess.StartProcessAsync(cancellationToken); |
There was a problem hiding this comment.
This is the important change - we will now wait on either worker fully initialized (gRPC connection established) or worker exits (in which case, we will re-throw any failures the worker experience).
brettsam
reviewed
Mar 24, 2025
brettsam
reviewed
Mar 24, 2025
|
/azp run host.integration-tests |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run host.public |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
@jviau @mattchenderson I was just investigating this yesterday! Turns out this is the same reason that we are stuck for 60s when trying to exit the CLI during startup (Azure/azure-functions-core-tools#4355). Glad to see there is a already a fix before I opened one :) I'm going to test your changes out with core tools and see if it addresses the issue like I expect. Looks like it works? Before this change, it would hang for 60 seconds waiting on the timeout from PendingItem. For my version of the fix, I was considering passing a CT to PendingItem and registering an edit: more logs - this is outside of my debug session - it takes 5 seconds to shutdown after we cancel: For core tools at least, I would love to reduce this time but I can probably just inject options in the webhost to reduce the flush logs timer to a 1 second or something like that. Anywho not related to this PR |
jviau
force-pushed
the
jviau/abort-channel
branch
from
d499b02 to
d9d1cd6
Compare
There was a problem hiding this comment.
Pull Request Overview
This PR improves the ScriptHost startup experience with faulty workers by implementing timeout optimizations. The key issue addressed is that when a worker crashes or exits immediately after startup, GrpcWorkerChannel.StartWorkerProcessAsync would block on the worker initialization task until it times out, potentially causing host faults during debugging.
Key Changes
- Added
WaitForExitAsyncmethod toIWorkerProcessto wait for process exit with proper exception propagation - Modified
GrpcWorkerChannel.StartWorkerProcessAsyncto race between worker initialization and process exit - Enhanced test coverage for worker process lifecycle and failure scenarios
Reviewed Changes
Copilot reviewed 25 out of 26 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
src/WebJobs.Script/Workers/ProcessManagement/IWorkerProcess.cs |
Added WaitForExitAsync method signature with cancellation token support |
src/WebJobs.Script/Workers/ProcessManagement/WorkerProcess.cs |
Implemented WaitForExitAsync with TaskCompletionSource for custom exception propagation |
src/WebJobs.Script.Grpc/Channel/GrpcWorkerChannel.cs |
Modified StartWorkerProcessAsync to race worker init against process exit |
test/WebJobs.Script.Tests/Workers/Rpc/RpcWorkerProcessTests.cs |
Added comprehensive tests for WaitForExitAsync functionality |
test/WebJobs.Script.Tests/Workers/Rpc/GrpcWorkerChannelTests.cs |
Added tests for process exit scenarios and timeout improvements |
jviau
force-pushed
the
jviau/abort-channel
branch
from
ca68897 to
16650da
Compare
This reverts commit a816a7f.
jviau
commented
Oct 2, 2025
| { | ||
| _processExit.TrySetResult(Process.ExitCode); | ||
| UnregisterFromProcessMonitor(); | ||
| Process.Close(); |
There was a problem hiding this comment.
Non-E2E test step in CI is hanging because some child process is inheriting STDIO and leaving it open. Looking at my PR, the changes to process management seem like the most likely culprit. But I don't understand why -- we were never disposing/closing the process in all cases before. Wouldn't this be a leak in the process restart scenario?
@fabiocav, any thoughts?
jviau
force-pushed
the
jviau/abort-channel
branch
from
d141cb4 to
4f1a2b7
Compare
jviau
force-pushed
the
jviau/abort-channel
branch
from
4f1a2b7 to
de9970f
Compare
brettsam
approved these changes
Oct 7, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue describing the changes in this PR
resolves #issue_for_this_pr
Pull request checklist
IMPORTANT: Currently, changes must be backported to the
in-procbranch to be included in Core Tools and non-Flex deployments.in-procbranch is not requiredrelease_notes.md-- TODOAdditional information
This PR improves the ScriptHost startup experience with a bad worker. Today, if a worker crashes or exits immediately after startup, then the
GrpcWorkerChannel.StartWorkerProcessAsyncwill block on_workerInitTask.Taskuntil it eventually times out. This tends to fault the entire host (at least during debugging).To address this, a
WorkerProcess.WaitForExitAsyncis added andGrpcWorkerChannel.StartWorkerProcessAsyncwill also wait on that, improving the responsiveness to a worker exiting before connecting gRPC events.