raft: ignore setting the lead field from a MsgDeFortify at current term

When receiving a MsgDeFortifyLeader at the same term as we are, we
should not set the lead field to the sender. Mainly for two reasons:

1) By definition, a MsgDeFortifyLeader is sent by an ex-leader until it
hears of a new committed term. If we forgot the leader at the current
term, we shouldn't remember it since we are using the fact that
lead==None to indicate that this replica has been leaderless for
some time. Read the leaderlessWatcher for more details.

2) This could lead to a situation where no replica can win an election
as it could require votes from some replicas that still know who know
the leader is (due to MsgDefortifyLeader), and that have recently
campaigned and lost due to not having the most up-to-date log, which
reset the electionElapsed to 0. Meaning that this replica is in a
heartbeat lease, and will reject MsgVotes.

Fixes: #142994

Release note: None

Author: iskettaneh

pkg/raft/raft.go

@@ -2258,11 +2258,11 @@ func stepCandidate(r *raft, m pb.Message) error {
 
 func stepFollower(r *raft, m pb.Message) error {
 	if IsMsgFromLeader(m.Type) {
-		r.setLead(m.From)
 		if m.Type != pb.MsgDeFortifyLeader {
 			// If we receive any message from the leader except a MsgDeFortifyLeader,
 			// we know that the leader is still alive and still acting as the leader,
 			// so reset the election timer.
+			r.setLead(m.From)
 			r.electionElapsed = 0
 		}
 	}

pkg/raft/testdata/liveness_impacted_after_temporary_loss_of_quorum.txt renamed to pkg/raft/testdata/liveness_recovered_after_temporary_loss_of_quorum.txt

@@ -1,7 +1,5 @@
 # This test ensures that if there is a temporary blip that causes loss of
 # quorum, the liveness of the raft group is restored once the blip is fixed.
-# Right now, there is a bug where no peer will be able to become a leader, and
-# liveness could be stalled indefinitely.
 
 log-level none
 ----
@@ -144,9 +142,6 @@ stabilize
   1->2 MsgDeFortifyLeader Term:1 Log:0/0
 > 2 receiving messages
   1->2 MsgDeFortifyLeader Term:1 Log:0/0
-> 2 handling Ready
-  Ready MustSync=true:
-  HardState Term:1 Vote:1 Commit:11 Lead:1 LeadEpoch:0
 
 # Fix the randomized election timeout to be one tick-election.
 set-randomized-election-timeout 1 timeout=3
@@ -170,13 +165,104 @@ stabilize
   INFO 1 has received 1 MsgPreVoteResp votes and 0 vote rejections
 > 2 receiving messages
   1->2 MsgPreVote Term:2 Log:1/12
-  INFO 2 [logterm: 1, index: 11, vote: 1] ignored MsgPreVote from 1 [logterm: 1, index: 12] at term 1: recently received communication from leader (remaining ticks: 3)
-  
-# At this point we saw that both peers attempted to campaign, but non of them
-# succeeded. Peer 1's request got rejected because 2 recently heard from 1
-# when receiving the MsgDefortifyLeader request.
-# Note that now 2 thinks that 1 is the leader for term 1, even though it isn't.
+  INFO 2 [logterm: 1, index: 11, vote: 1] cast MsgPreVote for 1 [logterm: 1, index: 12] at term 1
+> 2 handling Ready
+  Ready MustSync=false:
+  Messages:
+  2->1 MsgPreVoteResp Term:2 Log:0/0
+> 1 receiving messages
+  2->1 MsgPreVoteResp Term:2 Log:0/0
+  INFO 1 received MsgPreVoteResp from 2 at term 1
+  INFO 1 has received 2 MsgPreVoteResp votes and 0 vote rejections
+  INFO 1 became candidate at term 2
+  INFO 1 [logterm: 1, index: 12] sent MsgVote request to 2 at term 2
+> 1 handling Ready
+  Ready MustSync=true:
+  State:StateCandidate
+  HardState Term:2 Vote:1 Commit:11 Lead:0 LeadEpoch:0
+  Messages:
+  1->2 MsgVote Term:2 Log:1/12
+  INFO 1 received MsgVoteResp from 1 at term 2
+  INFO 1 has received 1 MsgVoteResp votes and 0 vote rejections
+> 2 receiving messages
+  1->2 MsgVote Term:2 Log:1/12
+  INFO 2 [term: 1] received a MsgVote message with higher term from 1 [term: 2], advancing term
+  INFO 2 became follower at term 2
+  INFO 2 [logterm: 1, index: 11, vote: 0] cast MsgVote for 1 [logterm: 1, index: 12] at term 2
+> 2 handling Ready
+  Ready MustSync=true:
+  HardState Term:2 Vote:1 Commit:11 Lead:0 LeadEpoch:0
+  Messages:
+  2->1 MsgVoteResp Term:2 Log:0/0
+> 1 receiving messages
+  2->1 MsgVoteResp Term:2 Log:0/0
+  INFO 1 received MsgVoteResp from 2 at term 2
+  INFO 1 has received 2 MsgVoteResp votes and 0 vote rejections
+  INFO 1 became leader at term 2
+> 1 handling Ready
+  Ready MustSync=true:
+  State:StateLeader
+  HardState Term:2 Vote:1 Commit:11 Lead:1 LeadEpoch:1
+  Entries:
+  2/13 EntryNormal ""
+  Messages:
+  1->2 MsgFortifyLeader Term:2 Log:0/0
+  1->2 MsgApp Term:2 Log:1/12 Commit:11 Entries:[2/13 EntryNormal ""]
+> 2 receiving messages
+  1->2 MsgFortifyLeader Term:2 Log:0/0
+  1->2 MsgApp Term:2 Log:1/12 Commit:11 Entries:[2/13 EntryNormal ""]
+> 2 handling Ready
+  Ready MustSync=true:
+  HardState Term:2 Vote:1 Commit:11 Lead:1 LeadEpoch:1
+  Messages:
+  2->1 MsgFortifyLeaderResp Term:2 Log:0/0 LeadEpoch:1
+  2->1 MsgAppResp Term:2 Log:1/12 Rejected (Hint: 11) Commit:11
+> 1 receiving messages
+  2->1 MsgFortifyLeaderResp Term:2 Log:0/0 LeadEpoch:1
+  2->1 MsgAppResp Term:2 Log:1/12 Rejected (Hint: 11) Commit:11
+> 1 handling Ready
+  Ready MustSync=false:
+  Messages:
+  1->2 MsgApp Term:2 Log:1/11 Commit:11 Entries:[
+    1/12 EntryNormal "data1"
+    2/13 EntryNormal ""
+  ]
+> 2 receiving messages
+  1->2 MsgApp Term:2 Log:1/11 Commit:11 Entries:[
+    1/12 EntryNormal "data1"
+    2/13 EntryNormal ""
+  ]
+> 2 handling Ready
+  Ready MustSync=true:
+  Entries:
+  1/12 EntryNormal "data1"
+  2/13 EntryNormal ""
+  Messages:
+  2->1 MsgAppResp Term:2 Log:0/13 Commit:11
+> 1 receiving messages
+  2->1 MsgAppResp Term:2 Log:0/13 Commit:11
+> 1 handling Ready
+  Ready MustSync=true:
+  HardState Term:2 Vote:1 Commit:13 Lead:1 LeadEpoch:1
+  CommittedEntries:
+  1/12 EntryNormal "data1"
+  2/13 EntryNormal ""
+  Messages:
+  1->2 MsgApp Term:2 Log:2/13 Commit:13
+> 2 receiving messages
+  1->2 MsgApp Term:2 Log:2/13 Commit:13
+> 2 handling Ready
+  Ready MustSync=true:
+  HardState Term:2 Vote:1 Commit:13 Lead:1 LeadEpoch:1
+  CommittedEntries:
+  1/12 EntryNormal "data1"
+  2/13 EntryNormal ""
+  Messages:
+  2->1 MsgAppResp Term:2 Log:0/13 Commit:13
+> 1 receiving messages
+  2->1 MsgAppResp Term:2 Log:0/13 Commit:13
+
 raft-state
 ----
-1: StatePreCandidate (Voter) Term:1 Lead:0 LeadEpoch:0
-2: StateFollower (Voter) Term:1 Lead:1 LeadEpoch:0
+1: StateLeader (Voter) Term:2 Lead:1 LeadEpoch:1
+2: StateFollower (Voter) Term:2 Lead:1 LeadEpoch:1