kv/kvserver: TestProcessSplitAfterRightHandSideHasBeenRemoved failed

[cockroach-teamcity]

kv/kvserver.TestProcessSplitAfterRightHandSideHasBeenRemoved failed with artifacts on master @ 29d848590e9ee8b57beee1248e94596de05b0e2f:

        	            	Wraps: (2) forced error mark
        	            	  | "originated at breaker breaker"
        	            	  | github.com/cockroachdb/cockroach/pkg/util/circuit/*circuit.breakerErrorMark::
        	            	Wraps: (3) forced error mark
        	            	  | "breaker open"
        	            	  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
        	            	Wraps: (4) replica unavailable: (n1,s1):1 unable to serve request to r80:/System{\x00aaa-testing-/NodeLiveness} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=5, sticky=9223372036.854775807,2147483647]: closed timestamp: 1764020168.928858322,0 (2025-11-24 21:36:08); raft status: {"id":"1","term":7,"vote":"1","commit":32,"lead":"0","leadEpoch":"0","raftState":"StatePreCandidate","applied":32,"progress":{},"leadtransferee":"0"}: have been waiting 60.02s for slow proposal Increment [/System/"�aaa-testinga"]
        	            	Wraps: (5)
        	            	  | (opaque error wrapper)
        	            	  | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
        	            	  | reportable 0:
        	            	  |
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.replicaUnavailableError
        	            	  | 	pkg/kv/kvserver/replica_circuit_breaker.go:268
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).replicaUnavailableErrorRLocked
        	            	  | 	pkg/kv/kvserver/replica_circuit_breaker.go:291
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).replicaUnavailableError
        	            	  | 	pkg/kv/kvserver/replica_circuit_breaker.go:277
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaCircuitBreaker).tripSync
        	            	  | 	pkg/kv/kvserver/replica_circuit_breaker.go:97
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaCircuitBreaker).TripAsync.func1
        	            	  | 	pkg/kv/kvserver/replica_circuit_breaker.go:91
        	            	  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1
        	            	  | 	pkg/util/stop/stopper.go:440
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_arm64.s:1268
        	            	Wraps: (6) closed timestamp: 1764020168.928858322,0 (2025-11-24 21:36:08); raft status: {"id":"1","term":7,"vote":"1","commit":32,"lead":"0","leadEpoch":"0","raftState":"StatePreCandidate","applied":32,"progress":{},"leadtransferee":"0"}
        	            	Wraps: (7)
        	            	  | (opaque error wrapper)
        	            	  | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
        	            	  | reportable 0:
        	            	  |
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).refreshProposalsLocked
        	            	  | 	pkg/kv/kvserver/replica_raft.go:1669
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).tick
        	            	  | 	pkg/kv/kvserver/replica_raft.go:1503
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processTick
        	            	  | 	pkg/kv/kvserver/store_raft.go:786
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
        	            	  | 	pkg/kv/kvserver/scheduler.go:412
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
        	            	  | 	pkg/kv/kvserver/scheduler.go:314
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_arm64.s:1268
        	            	Wraps: (8) have been waiting 60.02s for slow proposal Increment [/System/"�aaa-testinga"]
        	            	Error types: (1) *kvpb.AmbiguousResultError (2) *markers.withMark (3) *markers.withMark (4) *kvpb.ReplicaUnavailableError (5) *errbase.opaqueWrapper (6) *errutil.withPrefix (7) *errbase.opaqueWrapper (8) *errutil.leafError
        	Test:       	TestProcessSplitAfterRightHandSideHasBeenRemoved/lease-type=LeaseLeader/(3)_initial_replica_RHS_partition,_no_restart
--- FAIL: TestProcessSplitAfterRightHandSideHasBeenRemoved/lease-type=LeaseLeader/(3)_initial_replica_RHS_partition,_no_restart (65.57s)
=== RUN   TestProcessSplitAfterRightHandSideHasBeenRemoved/lease-type=LeaseLeader
--- FAIL: TestProcessSplitAfterRightHandSideHasBeenRemoved/lease-type=LeaseLeader (77.39s)

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-57219

[stevendanna]

Part of this test moves the lease away from n1 and then partition n1. From the error it looks like our request went to the partitioned node.

671:I251124 21:36:11.927621 92987 2@kv/kvserver/replica_proposal.go:638 ⋮ [T1,Vsystem,n3,s3,r80/3:/System{‹\x00›‹aaa-test›…-/NodeLive…},raft] 665  acquired system range lease: repl=(n3,s3):3 seq=2 start=1764020170.699040189,0 type=LeaseExpiration exp=1764020171.698986704,0 pro=1764020170.698986704,0 acq=Transfer [acquisition-type=Transfer]
672:I251124 21:36:11.927668 90245 2@kv/kvserver/replica_proposal.go:638 ⋮ [T1,Vsystem,n1,s1,r80/1:/System{‹\x00›‹aaa-test›…-/NodeLive…},raft] 666  acquired system range lease: repl=(n1,s1):1 seq=3 start=1764020171.698986704,1 type=LeaseExpiration exp=1764020172.700647635,0 pro=1764020171.700647635,0 acq=Request [acquisition-type=Request]

This transfer back happens before the partition is installed, which I think explains the error.

I251124 21:36:11.929664 88225 15@kv/kvserver_test/client_raft_test.go:5467 ⋮ [gotest=‹TestProcessSplitAfterRightHandSideHasBeenRemoved/lease-type=LeaseLeader›] 5643  activating LHS partition: ‹partition[rangeID: 80, replicas: [1]; stores: [
1]]›