Merge cockroachdb#107412

107412: kvserver: fix 'observed raft log position' assertion r=irfansharif a=irfansharif

Fixes cockroachdb#107336.
Fixes cockroachdb#106123.
Fixes cockroachdb#107156.
Fixes cockroachdb#106589.

It's possible to hit this assertion under --stress --race when the proposing replica is starved enough for raft ticks that it loses leadership right when it steps proposals through raft. We're relying on undocumented API semantics in the etcd raft library whereby it mutates stepped entries with the term+index its to end up in. But that's only applicable if stepping through entries as a leader. Simply relax this assertion instead.

Release note: None

Co-authored-by: irfan sharif <[email protected]>

Author: craig[bot]

pkg/kv/kvserver/replica_proposal_buf.go

@@ -1041,6 +1041,16 @@ func maybeDeductFlowTokens(
 		if admitHandle.handle == nil {
 			continue // nothing to do
 		}
+		if ents[i].Term == 0 && ents[i].Index == 0 {
+			// It's possible to have lost raft leadership right before stepping
+			// proposals through raft. They'll get forwarded to the new raft
+			// leader, and for flow token purposes, there's no tracking
+			// necessary. The token deductions below asserts on monotonic
+			// observations of log positions, which this empty position would
+			// otherwise violate. There's integration code elsewhere that will
+			// free up all tracked tokens as a result of this leadership change.
+			return
+		}
 		log.VInfof(ctx, 1, "bound index/log terms for proposal entry: %s",
 			raft.DescribeEntry(ents[i], func(bytes []byte) string {
 				return "<omitted>"