-
Notifications
You must be signed in to change notification settings - Fork 33
[FSSDK-12153] chore: update event retry strategy #618
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
Implemented synchronous blocking retry strategy with exponential backoff for both DefaultEventDispatcher and OdpEventManager: - Per-batch retry: Each batch gets up to 3 attempts with exponential backoff delays (200ms, 400ms, 800ms) - Global failure counter: Tracks consecutive batch failures across all batches in a flush - Blocking approach: Uses Thread.sleep() for delays and notify.wait() for synchronous operations - Reachability fix: Moved updateNumContiguousFails() outside retry loop to prevent premature network blocking - ODP error handling: Differentiates recoverable errors (retry) vs non-recoverable errors (discard) - JSON determinism: Added .sortedKeys to JSONEncoder for consistent test assertions - OdpConfig fix: Made update() synchronous to prevent race conditions Added comprehensive test suites: - OdpEventManagerRetryTests: 13 tests covering retry scenarios, error handling, and timing - EventDispatcherRetryTests: Tests for DefaultEventDispatcher retry logic - RetryStrategyTests: Unit tests for exponential backoff calculation - Updated existing tests to match new retry behavior expectations All tests passing: OdpEventManagerRetryTests (13/13), EventDispatcherTests_Batch (27/27) Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
… and update tests
219f4ef to
83b0c5d
Compare
jaeopt
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's a major rewrite and I hope it's fully tested with network failure cases against deadlock/livelock.
Other than that, all looks good to me! A couple of nits to clean up.
| guard let batchEvent = batchedEvent else { | ||
| // discard an invalid event that causes batching failure | ||
| // - if an invalid event is found while batching, it batches all the valid ones before the invalid one and sends it out. | ||
| // - when trying to batch next, it finds the invalid one at the header. It discards // Invalid event - discard and continue with next batch |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit - comment format
|
|
||
| // if eventsBatched.count == 1 { | ||
| // return (1, first) | ||
| // } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
intention to clean it up?
Summary
Key Changes
Removed DispatchGroup.wait(): Replaced the blocking synchronization used during network requests with non-blocking asynchronous completion handlers. This ensures that the serial queue threads are not held idle while waiting for I/O.
Non-Blocking close(): Updated DefaultEventDispatcher.close() to use a timed wait on a DispatchGroup. This ensures a graceful shutdown by waiting for pending flushes without risk of hanging the application indefinitely (includes a 10-second safety timeout).
Exponential Backoff: Introduced a formal retry strategy using DispatchQueue.asyncAfter. This replaces the previous "tight-loop" retry behavior (where failed batches were retried immediately) with a non-blocking delay sequence (200ms, 400ms, 800ms).
RetryStrategy Utility: Added a centralized RetryStrategy class to ensure consistent backoff logic across the SDK.
isFlushing Flag: Added state tracking to prevent multiple concurrent flush operations from running on the same dispatcher instance, improving thread safety and resource management.
Test plan
Issues