NRG: Don't drain response or proposal queues when switching to leader #7703
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The `runAsCandidate` and `runAsFollower` goroutines both try to drain the append entry response and proposal queues if they are still populated from the previous leadership, but we might be blocked in `runAsCandidate` or `runAsFollower` while switching to leader waiting for the state change to complete and accidentally drain incoming proposals. Signed-off-by: Neil Twigg <neil@nats.io>
neilalexander
force-pushed
the
neil/nrgrenotify
branch
from
bc20b68 to
cfd1531
Compare
derekcollison
requested changes
Jan 6, 2026
Member
derekcollison
left a comment
derekcollison
left a comment
There was a problem hiding this comment.
Be good to have a test that shows what this is fixing or accommodating for.
sciascid
reviewed
Jan 7, 2026
Contributor
sciascid
left a comment
sciascid
left a comment
There was a problem hiding this comment.
How about moving these calls to drain before the loop?
| n.votes.popOne() | ||
| case <-n.resp.ch: | ||
| // Ignore append entry responses received from before the state change. | ||
| n.resp.drain() |
Contributor
There was a problem hiding this comment.
How about moving the calls to drain before the loop?
Then we just need to make sure we push into the queues only if the node if the process is the leader. This is already checked for proposals, but appears to be easy to add for appendEntry responses.
Doing so we would avoid the need to re-notify...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The
runAsCandidateandrunAsFollowergoroutines both try to drain the append entry response and proposal queues if they are still populated from the previous leadership, but we might be blocked inrunAsCandidateorrunAsFollowerwhile switching to leader waiting for the state change to complete and accidentally drain incoming proposals.Signed-off-by: Neil Twigg neil@nats.io