[DNM] Simultaneous eip failover ovnkube-node restart #2788
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Periyasamy Palanisamy <[email protected]> (cherry picked from commit e42cf17c6f4e6a0b902781b0fbacbc3de2a9c01a)
Previously, when syncing EgressIP objects, the podAssignment cache was initialized with empty egressStatuses for assigned pods. This commit ensures that the pod state is updated with the correct EIP status during the sync process. Signed-off-by: Periyasamy Palanisamy <[email protected]> (cherry picked from commit 2c8dc6f3fdd6e8cac9085ad063181594ededd767)
When ovnkube-controller and an EgressIP (EIP) failover occur at the same time, a race condition can leave the informer cache empty for the EgressIP watcher. As a result, only the Add event for the EIP with the newly assigned node is triggered, and the controller fails to update the SNAT and LRP configuration for the previously assigned node. This commit leverages the egressStatuses stored in the podAssignment cache to reconcile and update those stale entries correctly. Signed-off-by: Periyasamy Palanisamy <[email protected]> (cherry picked from commit a69e7ceb47baafe47c82264936afab53ccb3f92b)
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: pperiyasamy The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Signed-off-by: Periyasamy Palanisamy <[email protected]>
pperiyasamy
force-pushed
the
simultaneous_eip_failover_ovnkubenode_restart
branch
from
f611325 to
847333f
Compare
|
/assign @huiran0826 |
…odes Previously, we were checking if the next hop IP is valid for the current set of nodes but we werent but every EgressIP is assigned to a subset of the total nodes. Stale LRPs could occur if a node hosted eip pods, ovnkube-controller is down, and the EIP moved to a new Node which said controller is down. Signed-off-by: Martin Kennelly <[email protected]> (cherry picked from commit dad551464861df3906db09fb45ee14a12b9ce755)
For IC mode, there is no expectation we can fetch a remote nodes LSP, therefore, by skipping (continue), it is causing us to skip generating valid next hops for the remote node. Later in sync LRPs, when a valid next hop is inspected, we do not find it valid and remove that valid next hop. Handlers will re-add it shortly later. Signed-off-by: Martin Kennelly <[email protected]> (cherry picked from commit 33dcfdca2b000370791112d76b2ef47e580e5ca5)
Signed-off-by: Martin Kennelly <[email protected]> (cherry picked from commit bc61618e6047fa336fccbb747c968cfc7e054c2c)
Previous to this change, we dont emit log error for stale next hops. Signed-off-by: Martin Kennelly <[email protected]> (cherry picked from commit 62e67e77b24b7ec1d24cd5cd2ed994239d6051a8)
|
@pperiyasamy: The following tests failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
to test stale SNAT and LRP nexthop fixes when eip failover and ovnkube-node restart happens around same time.
cc @jechen0648