Prevent Loss of Cluster Quorum Due to Chaos Operations#67
Prevent Loss of Cluster Quorum Due to Chaos Operations#67Nikhil-Manglore wants to merge 4 commits intovalkey-io:mainfrom
Conversation
Signed-off-by: Nikhil Manglore <nmanglor@amazon.com>
Signed-off-by: Nikhil Manglore <nmanglor@amazon.com>
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e14346e874
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if strategy == "primary_only": | ||
| coordination = ChaosCoordination( | ||
| chaos_before_operation=False, | ||
| chaos_during_operation=False, | ||
| chaos_after_operation=True |
There was a problem hiding this comment.
Forcing primary_only scenarios to always set chaos_after_operation=True means chaos is now always deferred, but deferred chaos reuses the node selected before the failover (ChaosCoordinator.coordinate_chaos_with_operation) and injects it later in ParallelExecutor without re-selecting against primary_only. When the selected primary is on the shard being failed over, it is typically demoted to replica before deferred injection, so this path violates the primary_only contract and changes experiment coverage from primary failures to replica failures.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
This is actually true, will look into a better way to do this
1 participant
We have seen scenarios where the fuzzer tests will fail due to loss of cluster quorum. This change makes it so if we only inject chaos after the operations if the chaos strategy chosen is "primary_only"