Fetch up-to-date gcomm members list during a failover #348
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: dciabrin The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
dciabrin
force-pushed
the
failover-gcomm-members
branch
3 times, most recently
from
624dc9e to
2d0fb48
Compare
One chainsaw test consists in abruptly cutting one galera node away from the galera cluster and verify that the active endpoint moves to one of the remaining two galera instances. In doing so, we currently kill -9 the target mysqld server. By design, this can take by default up to 15s for the remaining galera nodes to acknowlege the node went away and react to that. This is a problem for the test as if the pod comes back online before the 15s, the galera cluster won't move the endpoint and the test will fail. To prevent flaky result in the unit test, use the STOP signal instead of the KILL signal. This doesn't kill the pod, and by default galera will mark the node as not responding after 3s, and switch the endpoint. This achieves the same result, which is to make sure that an unexpected disconnection still trigger a endpoint switch.
The operator script that implements service endpoint failover contains internal logic to probe the up-to-date state of the gcomm cluster. This is done when the script starts, or when a command failed and is retried. The list of members was incorrectly extracted from a mysql table which is not guaranteed to be up-to-date when e.g. a node disappears from the cluster due to a network partition. Instead, we must rely on the mysql status, that always exposes the up-to-date gcomm state, in particular the members that are still connected to the primary partition. Jira: OSPRH-18408
dciabrin
force-pushed
the
failover-gcomm-members
branch
from
2d0fb48 to
bfdcd01
Compare
Contributor
Author
|
/retest |
|
/lgtm |
openshift-merge-bot
bot
merged commit 65de595
into
openstack-k8s-operators:main
8 checks passed
|
/cherry-pick 18.0-fr3 |
|
@lmiccini: new pull request created: #349 In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The operator script that implements service endpoint failover
contains internal logic to probe the up-to-date state of the
gcomm cluster. This is done when the script starts, or when
a command failed and is retried.
The list of members was incorrectly extracted from a mysql
table which is not guaranteed to be up-to-date when e.g.
a node disappears from the cluster due to a network partition.
Instead, we must rely on the mysql status, that always exposes
the up-to-date gcomm state, in particular the members that
are still connected to the primary partition.
Jira: OSPRH-18408