kv/kvnemesis: TestKVNemesisMultiNode_Partition_Safety failed

[github-actions]

kv/kvnemesis.TestKVNemesisMultiNode_Partition_Safety failed on release-26.1 @ 9a993c881c782c34f28ecfd849f628bdc7d7b40e:

github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.getObjectPrefix({0x710a808, 0xc00b31a1e0}, {{0xc0055e0000, 0xc0091c5500}, {{0x1, 0x0}, {0x0, 0x0, 0x0, 0x0, ...}, ...}}, ...)
	pkg/sql/catalog/descs/helpers.go:91 +0x99
github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.PrefixAndTable({0x710a808, 0xc00b31a1e0}, {{0xc0055e0000, 0xc0091c5500}, {{0x1, 0x0}, {0x0, 0x0, 0x0, 0x0, ...}, ...}}, ...)
	pkg/sql/catalog/descs/helpers.go:128 +0x75
github.com/cockroachdb/cockroach/pkg/sql.(*schemaResolver).LookupObject(0xc01312d710, {0x710a808, 0xc00b31a1e0}, {0x1, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0}, ...)
	pkg/sql/schema_resolver.go:166 +0xe3f
github.com/cockroachdb/cockroach/pkg/sql/catalog/resolver.ResolveExisting({0x710a808, 0xc00b31a1e0}, 0xc005c0b480, {0x7f702e47b528, 0xc01312d710}, {0x1, 0x0, 0x0, 0x0, 0x0, ...}, ...)
	pkg/sql/catalog/resolver/resolver.go:363 +0x30a
github.com/cockroachdb/cockroach/pkg/sql/catalog/resolver.ResolveExistingObject({0x710a808, 0xc00b31a1e0}, {0x7173a08, 0xc01312d710}, 0xc005c0b480, {0x1, 0x0, 0x0, 0x0, 0x0, ...})
	pkg/sql/catalog/resolver/resolver.go:179 +0x16b
github.com/cockroachdb/cockroach/pkg/sql/catalog/resolver.ResolveExistingTableObject({0x710a808, 0xc00b31a1e0}, {0x7173a08, 0xc01312d710}, 0xc01988dc88, {0x1, 0x0, 0x0, 0x0, 0x0, ...})
	pkg/sql/catalog/resolver/resolver.go:103 +0x145
github.com/cockroachdb/cockroach/pkg/sql.(*optCatalog).ResolveDataSource(0xc01988dc70, {0x710a808, 0xc00b31a1e0}, {0x0?, 0x0?, 0x0?, 0x0?}, 0x0?)
	pkg/sql/opt_catalog.go:289 +0x1c5
github.com/cockroachdb/cockroach/pkg/sql/opt.(*Metadata).CheckDependencies(0xc00744f808, {0x710a808, 0xc00b31a1e0}, 0xc01312dcd0, {0x71aa488, 0xc01988dc70})
	pkg/sql/opt/metadata.go:480 +0x1312
github.com/cockroachdb/cockroach/pkg/sql/opt/memo.(*Memo).IsStale(0xc00744f808, {0x710a808, 0xc00b31a1e0}, 0xc01312dcd0, {0x71aa488, 0xc01988dc70})
	pkg/sql/opt/memo/memo.go:531 +0x13e8
github.com/cockroachdb/cockroach/pkg/sql.(*optPlanningCtx).buildExecMemo(0xc01312e690, {0x710a808, 0xc00b31a1e0})
	pkg/sql/plan_opt.go:910 +0x6bc
github.com/cockroachdb/cockroach/pkg/sql.(*planner).makeOptimizerPlanInternal(0xc01312d710, {0x710a808, 0xc00b31a1e0})
	pkg/sql/plan_opt.go:328 +0x2e
github.com/cockroachdb/cockroach/pkg/sql.(*planner).makeOptimizerPlan(0xc01312d710, {0x710a300?, 0xc01988dea0?})
	pkg/sql/plan_opt.go:308 +0x2f6
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).makeExecPlan(0xc01312cf08, {0x710a300, 0xc01988dea0}, 0xc01312d710)
	pkg/sql/conn_executor_exec.go:3246 +0x135
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine(0xc01312cf08, {0x710a300, 0xc01988dea0}, 0xc01312d710, {0x71921b0, 0xc01988ddb0})
	pkg/sql/conn_executor_exec.go:2827 +0xa72
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenStateWithPausablePortal(0xc01312cf08, {0x710a338, 0xc008446e10}, {{0x7133660, 0xc01988dc20}, {0x0, 0x0, 0x0}, {0xc012e9c380, 0xd4}, ...}, ...)
	pkg/sql/conn_executor_exec.go:2075 +0x4830
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1({0x710a338?, 0xc008446e10?})
	pkg/sql/conn_executor_exec.go:171 +0xaa
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling.func1({0x710a338?, 0xc008446e10?})
	pkg/sql/conn_executor_exec.go:4461 +0x2d
github.com/cockroachdb/cockroach/pkg/util/pprofutil.Do({0x710a808?, 0xc0072cfea0?}, 0xc009329728, {0xc009329688?, 0x0?, 0x1?})
	pkg/util/pprofutil/labels.go:38 +0x51
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling(0xc01312cf08, {0x710a808, 0xc0072cfea0}, {0x7133660, 0xc01988dc20}, 0x0, 0xc009329930)
	pkg/sql/conn_executor_exec.go:4460 +0x40a
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt(0xc01312cf08, {0x710a808, 0xc0072cfea0}, {{0x7133660, 0xc01988dc20}, {0x0, 0x0, 0x0}, {0xc012e9c380, 0xd4}, ...}, ...)
	pkg/sql/conn_executor_exec.go:170 +0x6bc
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func1(...)
	pkg/sql/conn_executor.go:2426
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc01312cf08)
	pkg/sql/conn_executor.go:2431 +0x2334
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc01312cf08, {0x710a808, 0xc0072cfd60}, 0x710a808?, 0xc0040b8460?, 0xc00dd3aee8?)
	pkg/sql/conn_executor.go:2339 +0x1f7
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).runWithEx.func2()
	pkg/sql/internal.go:252 +0x13a
created by github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).runWithEx in goroutine 39282
	pkg/sql/internal.go:249 +0x2b6

Parameters:

• attempt=1
• run=1
• shard=1

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-58008

[tbg]

Looks like things are getting stuck at the KV layer acquiring leases. I'll take another look later, but this is probably not a release blocker, since the test introduces partitions.

goroutine 62600 [select, 13 minutes]:github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeWriteBatch(0xc008dc3608, {0x710a808, 0xc008407ae0}, 0xc004320f00, 0xc0061912b0)	pkg/kv/kvserver/replica_write.go:214 +0x7c5github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc008dc3608, {0x710a808, 0xc008407ae0}, 0xc004320f00, 0x62721d8)	pkg/kv/kvserver/replica_send.go:528 +0x388github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes(0xc008dc3608, {0x710a808, 0xc0085fbea0}, 0xc004320f00)	pkg/kv/kvserver/replica_send.go:201 +0x610github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send(0xc004320f00?, {0x710a808?, 0xc0085fbea0?}, 0x0?)	pkg/kv/kvserver/replica_send.go:117 +0x1dgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*pendingLeaseRequest).requestLease(0xc008dc3d50, {0x710a808, 0xc0085fbea0}, {0x7171d18, 0xc004320dc0}, {{{0x18821ab67ab8e3dc, 0x1}, 0xc011219fd0, {0x4, 0x4, ...}, ...}, ...})	pkg/kv/kvserver/replica_range_lease.go:679 +0xb39github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*pendingLeaseRequest).requestLeaseAsync.func1({0x710a808, 0xc0085fbea0})	pkg/kv/kvserver/replica_range_lease.go:501 +0x10bgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1({0x710a808, 0xc0085fbea0})	pkg/util/stop/stopper.go:440 +0x8acreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 61750	pkg/util/stop/stopper.go:438 +0x125

[miraradeva]

I can take this one off your plate @tbg.

I see a few types of stuck goroutines here:

1. Hundreds of goroutines stuck in executeWriteBatch.
2. Goroutine 49796 stuck applying an UpdateZoneConfig.
3. Goroutines 95068 and 40010 stuck in waitForPush, pushing the same transaction (52521d30).

Unfortunately I don't see any logs from this run. I'm pretty sure we should have logs and traces of completed operations even when the test times out. I'll try to repro.
[Edit] Repros fairly easily with the kvvnemesis seed.

[tbg]

Yes, it's odd that there aren't artifacts. I poked around here too but to no avail.

This looks like another instance of #156562.

Thank you for taking a look at this issue. Reassigning to you.