Add another platform exception type to OutOfProcMiddleware#3383
Add another platform exception type to OutOfProcMiddleware#3383
Conversation
There was a problem hiding this comment.
Pull request overview
This PR fixes issue #2939 where activities (and orchestrations/entities) were being incorrectly marked as failed when a FunctionTimeoutAbortException was thrown due to another function's timeout causing a worker restart. The fix ensures this exception type is treated as a platform-level exception, triggering a SessionAbortedException for durable retry instead of a permanent failure.
Changes:
- Added
FunctionTimeoutAbortExceptionto theIsPlatformLevelExceptioncheck (used by the orchestrator path) and added explicit checks in the entity and activity paths - Added dedicated unit tests for entity and activity paths with
FunctionTimeoutAbortException - Added
FunctionTimeoutAbortExceptionto thePlatformLevelExceptionstest data for orchestrator tests, plus minor code style cleanups (collection expressions, primary constructors, explicit types)
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
src/WebJobs.Extensions.DurableTask/OutOfProcMiddleware.cs |
Adds FunctionTimeoutAbortException handling in entity/activity dispatch paths and to IsPlatformLevelException |
test/FunctionsV2/OutOfProcMiddlewareTests.cs |
Adds tests for entity/activity FunctionTimeoutAbortException handling, adds SetupEntityTest/SetupActivityTest helpers, and minor style cleanups |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
There was a problem hiding this comment.
Pull request overview
This PR adds handling for FunctionTimeoutAbortException in the OutOfProcMiddleware to fix an issue where activities (and entities) would be permanently failed when another function on the same worker hit a timeout, causing a worker restart. The fix ensures these work items are retried (via SessionAbortedException) instead of being marked as failed.
Changes:
- Added
FunctionTimeoutAbortExceptionchecks in the entity and activity execution paths ofOutOfProcMiddleware, and added it to theIsPlatformLevelExceptionmethod (used by the orchestrator path). - Bumped
Microsoft.Azure.WebJobspackage version from 3.0.39 to 3.0.45 to gain access to theFunctionTimeoutAbortExceptiontype. - Added unit tests for the new exception handling in entity and activity paths, plus added the exception to the existing orchestrator platform-level exception test data.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
src/WebJobs.Extensions.DurableTask/OutOfProcMiddleware.cs |
Added FunctionTimeoutAbortException handling in entity/activity paths and IsPlatformLevelException |
test/FunctionsV2/OutOfProcMiddlewareTests.cs |
Added tests for new exception handling; modernized syntax (collection expressions, explicit types) |
Directory.Packages.props |
Bumped Microsoft.Azure.WebJobs from 3.0.39 to 3.0.45 |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
There was a problem hiding this comment.
Pull request overview
This PR fixes an issue (#2939) where activities (and entities) would permanently fail instead of being retried when FunctionTimeoutAbortException is thrown — an exception that occurs when another function on the same worker exceeds its timeout, causing a worker restart. The fix adds FunctionTimeoutAbortException to the set of recognized platform-level exceptions, ensuring the affected work items are retried via SessionAbortedException.
Changes:
- Added
FunctionTimeoutAbortExceptionchecks to the entity and activity execution paths inOutOfProcMiddleware, and to theIsPlatformLevelExceptionhelper (which covers orchestrators). - Bumped
Microsoft.Azure.WebJobsfrom 3.0.39 to 3.0.45 to gain access to theFunctionTimeoutAbortExceptiontype. - Added unit tests for the new exception handling in entity and activity paths, and added
FunctionTimeoutAbortExceptionto the existing orchestrator platform-level exception test data.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
src/WebJobs.Extensions.DurableTask/OutOfProcMiddleware.cs |
Adds FunctionTimeoutAbortException handling in entity/activity paths and to IsPlatformLevelException |
Directory.Packages.props |
Bumps Microsoft.Azure.WebJobs to 3.0.45 for the new exception type |
test/FunctionsV2/OutOfProcMiddlewareTests.cs |
Adds tests for entity/activity paths, adds FunctionTimeoutAbortException to orchestrator test data, plus minor code modernization |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
Summary
What changed?
This PR adds another exception type (
FunctionTimeoutAbortException) to our check for platform-level exceptions when invoking orchestrations, entities, and notably Activities.Why is this change needed?
This exception is thrown when a worker restarts after another Function has reached its timeout. Currently, since
result.Succeededis false in this case, we will fail the Activity even though it itself did not exceed the Function timeout. This PR changes the behavior to instead throw aSessionAbortedException, such that the Activity (or orchestration, entity, etc.) is retried.Issues / work items
Project checklist
pending_docs.mdrelease_notes.mdWebJobs.Extensions.DurableTaskpackage/src/Worker.Extensions.DurableTask/AssemblyInfo.csEventSourcelogsv2.xbranchWebJobs.Extensions.DurableTaskv3.x and will be retained only in thedevandmainbranchesAI-assisted code disclosure (required)
Was an AI tool used? (select one)
If AI was used:
AI verification (required if AI was used):
Testing
Automated tests
Manual validation (only if runtime/behavior changed)
1.
2.
3.
Notes for reviewers