feat(fortuna): Multiple replica support #2812
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…r modulo filtering Co-Authored-By: Tejas Badadare <[email protected]>
- Add configurable time delay before backup replicas check request status - Backup replicas now wait backup_delay_seconds before attempting fulfillment - Add backup_delay_seconds field to ReplicaConfig with default of 30 seconds - Improves reliability by reducing race conditions between replicas Co-Authored-By: Tejas Badadare <[email protected]>
- Add Multiple Replica Setup section to README.md with modulo assignment explanation - Add replica_config examples to config.sample.yaml for 2, 3, and 5 replica setups - Include deployment considerations, failover behavior, and wallet separation requirements - Add validation for backup_delay_seconds > 0 to prevent race conditions Co-Authored-By: Tejas Badadare <[email protected]>
…git-manager.devin.ai/proxy/github.com/pyth-network/pyth-crosschain into devin/1750801144-fortuna-multiple-replicas
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
m30m
reviewed
Jun 25, 2025
…processing - Updated config.sample.yaml by removing unnecessary blank lines. - Changed backup_delay_seconds in README.md from 45 to 30 for consistency. - Refactored run_keeper_threads to accept KeeperConfig directly instead of private_key. - Enhanced run function to handle keeper configuration more effectively. - Added comments in process_event_with_backoff to clarify primary and backup replica logic.
m30m
approved these changes
Jun 26, 2025
| } | ||
| } | ||
| } | ||
| // If no replica config we are running standalone, process all requests directly |
There was a problem hiding this comment.
I would put this comment at the top of this block. I generally expect comments to describe the next block or line but this seems to be a comment for a non-existent else block
| tracing::debug!("Processing request as primary replica"); | ||
| } else { | ||
| tracing::debug!("Processing request as backup replica"); | ||
|
|
There was a problem hiding this comment.
I think we can merge these 2 logs
…evin/1750801144-fortuna-multiple-replicas
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Implemented multiple Fortuna replica support with modulo-based request assignment and configurable backup delays.
Rationale
We want to improve Fortuna's reliability through redundancy, but naively running multiple Fortuna instances caused race conditions and gas waste. Now we distribute requests to specific replicas via
sequence_number % total_replicas. All replicas still see all requests, so if a replica doesn't fulfill its assigned request, the others will try to fulfill it after the set delay.The replica assignment & delay is implemented in
process_event_with_backoff, which plays well with the block delay and retry mechanisms. It kicks in after the block delay has already elapsed, so the effective latency for requests missed by their assigned replica is block delay + replica delay.How has this been tested?
Tested manually, works well on monad testnet. Stress tested it a bit with the load tester script and both instances were happy. Tested out killing one of them and the other one picks up the requests after the delay as expected.