batcheval: investigate latch assertion in DeleteRange #158132
Description
With timestamp-aware latch assertions enabled in #158131, stressing kvnemesis catches the following error. Need to investigate and fix. Reproduces easily with the linked PR: dev test --stress pkg/kv/kvnemesis.
A couple of quirks to be aware:
/{Min-Table/100/"05ee0fb26ba7b41b"}is a[x.Prev(), x)span. Does not actually start atMin, it's just a pretty-printer quirk (need to fix).- The
???key is probably something like a "prevish" key which the pretty-printer doesn't understand well.
Previously, the assertion did not fire either because we weren't stressing kvnemesis under --race, or because it was more liberal (did not take timestamps into account).
What happens here is: in the read global, the ??? is probably the prevish key of "05ee0fb26ba7b41b". It is an MVCC span. Previously, we ignored its timestamp, now we don't. So maybe it doesn't like the non-MVCC SeekLT. Rules are here.
kvnemesis.go:240: error applying x.DelRangeUsingTombstone(ctx, tk(427296273631523867), tk(15658150965789664794) /* @s71 */) // cannot read undeclared span /{Min-Table/100/"05ee0fb26ba7b41b"}
declared:
read global: /Table/100/{???-"d94ced57cb55261a"/NULL} at 1763636207.304303935,0
read local: /Local/RangeID/79/u"rltu" at 0,0
read local: /Local/Range/Table/76/RangeDescriptor at 0,0
read local: /Local/Lock/Table/100/{???-"d94ced57cb55261a"/NULL} at 0,0
read local: /Local/Lock/Local/RangeID/79/u"rltu" at 0,0
read local: /Local/Lock/Local/Range/Table/76/RangeDescriptor at 0,0
write global: /Table/100/"{05ee0fb26ba7b41b"-d94ced57cb55261a"} at 1763636207.304303935,0
write local: /Local/Lock/Table/100/"{05ee0fb26ba7b41b"-d94ced57cb55261a"} at 0,0
stack:
goroutine 187735 [running]:
runtime/debug.Stack()
GOROOT/src/runtime/debug/stack.go:26 +0x64
github.com/cockroachdb/cockroach/pkg/util/debugutil.Stack(...)
pkg/util/debugutil/debugutil.go:63
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset.(*SpanSet).checkAllowed(0x14003cd88c0, 0x0, {{0x0, 0x0, 0x0}, {0x1401274a780, 0x14, 0x18}}, 0x14012a4a628)
pkg/kv/kvserver/spanset/spanset.go:408 +0x134
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset.(*SpanSet).CheckAllowedAt(0x401?, 0x401?, {{0x0, 0x0, 0x0}, {0x1401274a780, 0x14, 0x18}}, {0x14012a4a6e8?, 0x2112f4c?})
pkg/kv/kvserver/spanset/spanset.go:294 +0x90
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset.(*MVCCIterator).checkAllowedValidPos(0x1401446d360, {{0x0, 0x0, 0x0}, {0x1401274a780, 0x14, 0x18}}, 0x1)
pkg/kv/kvserver/spanset/batch.go:142 +0xa4
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset.(*MVCCIterator).checkAllowed(0x1401446d360, {{0x0, 0x0, 0x0}, {0x1401274a780, 0x14, 0x18}}, 0x1)
pkg/kv/kvserver/spanset/batch.go:134 +0x78
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset.(*MVCCIterator).SeekLT(0x1401446d360, {{0x1401274a780, 0x14, 0x18}, {0x0, 0x0}})
pkg/kv/kvserver/spanset/batch.go:88 +0x6c
github.com/cockroachdb/cockroach/pkg/storage.PeekRangeKeysLeft({0x1066ac2c0, 0x1401446d360}, {0x1401274a780, 0x14, 0x18})
pkg/storage/mvcc.go:8336 +0x68
github.com/cockroachdb/cockroach/pkg/storage.MVCCDeleteRangeUsingTombstone({0x1066276f8, 0x140135c8f30}, {0x1066bf950, 0x1400d332fc0}, 0x1401339e630, {0x1401274a780, 0x14, 0x18}, {0x1401274a798, 0x14, ...}, ...)
pkg/storage/mvcc.go:4316 +0x1270
github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval.deleteRangeUsingTombstone({_, _}, {_, _}, {{0x1066c69e8, 0x1400ae10580}, {{0x1879b1c9a9a47011, 0x3}, 0x140127ec700, {0x1879b1c94de988c6, ...}, ...}, ...})
pkg/kv/kvserver/batcheval/cmd_delete_range.go:181 +0x2c0
github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval.DeleteRange({_, _}, {_, _}, {{0x1066c69e8, 0x1400ae10580}, {{0x1879b1c9a9a47011, 0x3}, 0x140127ec700, {0x1879b1c94de988c6, ...}, ...}, ...}, ...)
pkg/kv/kvserver/batcheval/cmd_delete_range.go:123 +0x33c
Jira issue: CRDB-57124