Skip to content

Commit 5d3fb6c

Browse files
craig[bot]tbgspilchen
committed
155176: log,logflags: allow seeding test log config from env r=tbg a=tbg This makes it easier to apply a custom config to various test invocations without having to juggle flags. Many KV-specific logs are not included by default, so they're not in `-show-logs`, which can be annoying. Epic: none 156295: workload/schemachange: add INSPECT operation to random schema workload r=spilchen a=spilchen Adds a new inspect operation to the schema change workload, enabling random generation of INSPECT TABLE and INSPECT DATABASE statements. Features: - Support for TABLE/DB targets, AS OF SYSTEM TIME - Always runs in DETACHED mode so that it can be run inside a transaciton - Results checked post-run via SHOW INSPECT ERRORS Errors reported in JSON, consistent with existing workload logs Closes #155483 Epic: CRDB-55075 Release note: none 156445: asim: reproduce loss of quorum during zone cfg change with suspect nodes r=tbg a=tbg I believe this newly added datadriven test reproduces #152604. The test sets up five nodes with 5x replication, marks n4 and n5 as non-live, and drops the replication factor to 3. We see that the allocator merrily removes replicas from n1-n3 and loses quorum in the process. If it removed any replicas in this scenario, it really ought to be removing from n4 and n5. > next replica action: remove voter > removing voting replica n2,s2 due to over-replication: [1*:2, 2:2, 3:2, 4:2, 5:2] Then: > unable to take action - live voters [(n1,s1):1 (n3,s3):3] don't meet quorum of 3 Informs #152604. Epic: none 156463: kvserver: remove within10s r=stevendanna a=tbg Replace it with a standard invocation of `retry`. This is a follow-up to #156464, which fixed a bug in `within10s`. Epic: none 156516: roachtest: adjust failover tests to changed liveness rangeID r=tbg a=tbg As of #155554, the liveness range has rangeID three, not two. The tests are updated to avoid relying on the particular ID. Instead, we check on the pretty-printed start key of the liveness range, /System/NodeLiveness. Closes #156450. Closes #156449. Closes #156448. Epic: none 156519: asim: rewrite high_cpu_25nodes.txt r=tbg a=tbg The random numbers changed due to the Go upgrade. This didn't fail CI since we run most asim tests nightly only. No big deal. Closes #156411. Epic: none 156563: storage_api: fix TestDecommissionSelf flake r=tbg a=tbg TestDecommissionSelf was flaking with timeouts waiting for decommissioned nodes to observe their own DECOMMISSIONED status. The test would fail when node 4 (one of the decommissioned nodes) never saw its liveness record update to DECOMMISSIONED within the 5-second timeout. The likely root cause is a race condition in how decommission status propagates: when a node is marked as decommissioned, the updated liveness record is gossiped to all nodes. However, if other nodes receive this gossip update before the decommissioned node does, they will write a tombstone to local storage and subsequently reject all RPCs from the decommissioned node, including gossip messages. This can prevent the decommissioned node from ever learning about its own status change. This commit fixes the test by only verifying the cluster state from the perspective of non-decommissioned nodes. We now assert that: 1. Active nodes see themselves as ACTIVE 2. Active nodes see the decommissioned nodes as DECOMMISSIONED We no longer attempt to verify that decommissioned nodes observe their own status, since this is not guaranteed due to the gossip/tombstone race. Fixes #156402. Fixes #156104. Fixes #154474. Release note: None Epic: None 156616: kvserver: deflake TestFlowControlSendQueueRangeFeed r=tbg a=tbg The recently added logging showed that n2 and n3 just weren't even considered for starting the rangefeed. Likely this was because the descriptors weren't gossiped yet (this was corroborated by gossip logging). Rather than waiting for specific preconditions that "likely" fix the problem for the particular version of the code, widen the scope of the retry loop to re-establish the rangefeed until it does get scheduled were it needs to for the test to succeed. Epic: none Co-authored-by: Tobias Grieger <[email protected]> Co-authored-by: Matt Spilchen <[email protected]>
9 parents bedfbd7 + 0e84a7a + dc72c30 + 582e531 + eb38bfc + 2c7aafc + cf52e7f + bc1faf2 + cf1382a commit 5d3fb6c

File tree

12 files changed

+337
-83
lines changed

12 files changed

+337
-83
lines changed

‎pkg/cmd/roachtest/tests/failover.go‎

Lines changed: 20 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,13 @@ var failoverAggregateFunction = func(test string, histogram *roachtestutil.Histo
5656
}, nil
5757
}
5858

59+
// NB: the liveness range's ID has changed in the past. It used to be 2, now
60+
// it's 3 (#155554). Using the start key avoids relying on a specific rangeID.
61+
const (
62+
predAllButLiveness = `start_key != '/System/NodeLiveness'`
63+
predOnlyLiveness = `start_key = '/System/NodeLiveness'`
64+
)
65+
5966
// registerFailover registers a set of failover benchmarks. These tests
6067
// benchmark the maximum unavailability experienced by clients during various
6168
// node failures, and exports them for roachperf graphing. They do not make any
@@ -752,7 +759,7 @@ func runFailoverPartialLeaseLiveness(ctx context.Context, t test.Test, c cluster
752759
// all nodes regardless.
753760
relocateRanges(t, ctx, conn, `database_name = 'kv'`, []int{1, 2, 3, 4}, []int{5, 6, 7})
754761
relocateRanges(t, ctx, conn, `database_name != 'kv'`, []int{5, 6, 7}, []int{1, 2, 3, 4})
755-
relocateRanges(t, ctx, conn, `range_id != 2`, []int{4}, []int{1, 2, 3})
762+
relocateRanges(t, ctx, conn, predAllButLiveness, []int{4}, []int{1, 2, 3})
756763

757764
// Run workload on n8 using n1-n3 as gateways (not partitioned) until test
758765
// ends (context cancels).
@@ -781,8 +788,8 @@ func runFailoverPartialLeaseLiveness(ctx context.Context, t test.Test, c cluster
781788
// them to where they should be.
782789
relocateRanges(t, ctx, conn, `database_name = 'kv'`, []int{1, 2, 3, 4}, []int{5, 6, 7})
783790
relocateRanges(t, ctx, conn, `database_name != 'kv'`, []int{node}, []int{1, 2, 3})
784-
relocateRanges(t, ctx, conn, `range_id = 2`, []int{5, 6, 7}, []int{1, 2, 3, 4})
785-
relocateLeases(t, ctx, conn, `range_id = 2`, 4)
791+
relocateRanges(t, ctx, conn, predOnlyLiveness, []int{5, 6, 7}, []int{1, 2, 3, 4})
792+
relocateLeases(t, ctx, conn, predOnlyLiveness, 4)
786793

787794
// Randomly sleep up to the lease renewal interval, to vary the time
788795
// between the last lease renewal and the failure.
@@ -976,10 +983,10 @@ func runFailoverLiveness(
976983
// do it ourselves. Precreating the database/range and moving it to the
977984
// correct nodes first is not sufficient, since workload will spread the
978985
// ranges across all nodes regardless.
979-
relocateRanges(t, ctx, conn, `range_id != 2`, []int{4}, []int{1, 2, 3})
986+
relocateRanges(t, ctx, conn, predAllButLiveness, []int{4}, []int{1, 2, 3})
980987

981-
// We also make sure the lease is located on n4.
982-
relocateLeases(t, ctx, conn, `range_id = 2`, 4)
988+
// We also make sure the liveness lease is located on n4.
989+
relocateLeases(t, ctx, conn, predOnlyLiveness, 4)
983990

984991
// Run workload on n5 via n1-n3 gateways until test ends (context cancels).
985992
t.L().Printf("running workload")
@@ -1002,8 +1009,8 @@ func runFailoverLiveness(
10021009

10031010
// Ranges and leases may occasionally escape their constraints. Move them
10041011
// to where they should be.
1005-
relocateRanges(t, ctx, conn, `range_id != 2`, []int{4}, []int{1, 2, 3})
1006-
relocateLeases(t, ctx, conn, `range_id = 2`, 4)
1012+
relocateRanges(t, ctx, conn, predAllButLiveness, []int{4}, []int{1, 2, 3})
1013+
relocateLeases(t, ctx, conn, predOnlyLiveness, 4)
10071014

10081015
// Randomly sleep up to the lease renewal interval, to vary the time
10091016
// between the last lease renewal and the failure.
@@ -1018,7 +1025,7 @@ func runFailoverLiveness(
10181025

10191026
t.L().Printf("recovering n%d (%s)", 4, failer)
10201027
failer.Recover(ctx, 4)
1021-
relocateLeases(t, ctx, conn, `range_id = 2`, 4)
1028+
relocateLeases(t, ctx, conn, predOnlyLiveness, 4)
10221029
}
10231030

10241031
sleepFor(ctx, t, time.Minute) // let cluster recover
@@ -1089,9 +1096,9 @@ func runFailoverSystemNonLiveness(
10891096
// n1-n3, so we do it ourselves. Precreating the database/range and moving it
10901097
// to the correct nodes first is not sufficient, since workload will spread
10911098
// the ranges across all nodes regardless.
1092-
relocateRanges(t, ctx, conn, `database_name = 'kv' OR range_id = 2`,
1099+
relocateRanges(t, ctx, conn, `database_name = 'kv' OR `+predOnlyLiveness,
10931100
[]int{4, 5, 6}, []int{1, 2, 3})
1094-
relocateRanges(t, ctx, conn, `database_name != 'kv' AND range_id != 2`,
1101+
relocateRanges(t, ctx, conn, `database_name != 'kv' AND `+predAllButLiveness,
10951102
[]int{1, 2, 3}, []int{4, 5, 6})
10961103

10971104
// Run workload on n7 via n1-n3 as gateways until test ends (context cancels).
@@ -1116,9 +1123,9 @@ func runFailoverSystemNonLiveness(
11161123

11171124
// Ranges may occasionally escape their constraints. Move them
11181125
// to where they should be.
1119-
relocateRanges(t, ctx, conn, `database_name != 'kv' AND range_id != 2`,
1126+
relocateRanges(t, ctx, conn, `database_name != 'kv' AND `+predAllButLiveness,
11201127
[]int{1, 2, 3}, []int{4, 5, 6})
1121-
relocateRanges(t, ctx, conn, `database_name = 'kv' OR range_id = 2`,
1128+
relocateRanges(t, ctx, conn, `database_name = 'kv' OR `+predOnlyLiveness,
11221129
[]int{4, 5, 6}, []int{1, 2, 3})
11231130

11241131
// Randomly sleep up to the lease renewal interval, to vary the time

‎pkg/kv/kvserver/asim/tests/testdata/non_rand/mma/high_cpu_25nodes.txt‎

Lines changed: 26 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -28,31 +28,31 @@ gen_load rate=15000 rw_ratio=0.95 min_block=1 max_block=1 request_cpu_per_access
2828

2929
eval duration=20m samples=1 seed=42 cfgs=(mma-only,mma-count) metrics=(cpu,cpu_util,write_bytes_per_second,replicas,leases)
3030
----
31-
cpu#1: last: [s1=4206019833, s2=4196185516, s3=6307338295, s4=4199573838, s5=4191438687, s6=4198998914, s7=4195902382, s8=4189575907, s9=4205480772, s10=4212518971, s11=4203942143, s12=4208124857, s13=4201922174, s14=4201899513, s15=4186889030, s16=4198293007, s17=4190652522, s18=4201917446, s19=4208931395, s20=4203404170, s21=4211222091, s22=2100653787, s23=4197759491, s24=4209891497, s25=4199679069] (stddev=594952429.35, mean=4201128612.28, sum=105028215307)
32-
cpu#1: thrash_pct: [s1=6%, s2=40%, s3=51%, s4=51%, s5=26%, s6=3%, s7=2%, s8=2%, s9=3%, s10=2%, s11=3%, s12=3%, s13=3%, s14=2%, s15=2%, s16=19%, s17=2%, s18=3%, s19=2%, s20=3%, s21=19%, s22=1%, s23=3%, s24=2%, s25=11%] (sum=263%)
33-
cpu_util#1: last: [s1=0.53, s2=0.52, s3=0.79, s4=0.52, s5=0.52, s6=0.52, s7=0.52, s8=0.52, s9=0.53, s10=0.53, s11=0.53, s12=0.53, s13=0.53, s14=0.53, s15=0.52, s16=0.52, s17=0.52, s18=0.53, s19=0.53, s20=0.53, s21=0.53, s22=0.26, s23=0.52, s24=0.53, s25=0.52] (stddev=0.07, mean=0.53, sum=13)
34-
cpu_util#1: thrash_pct: [s1=6%, s2=40%, s3=51%, s4=51%, s5=26%, s6=3%, s7=2%, s8=2%, s9=3%, s10=2%, s11=3%, s12=3%, s13=3%, s14=2%, s15=2%, s16=19%, s17=2%, s18=3%, s19=2%, s20=3%, s21=19%, s22=1%, s23=3%, s24=2%, s25=11%] (sum=263%)
35-
leases#1: first: [s1=28, s2=14, s3=4, s4=3, s5=3, s6=2, s7=2, s8=2, s9=2, s10=2, s11=3, s12=2, s13=3, s14=3, s15=2, s16=3, s17=3, s18=3, s19=2, s20=2, s21=2, s22=3, s23=3, s24=3, s25=1] (stddev=5.43, mean=4.00, sum=100)
36-
leases#1: last: [s1=3, s2=4, s3=5, s4=4, s5=4, s6=4, s7=4, s8=3, s9=4, s10=4, s11=5, s12=3, s13=5, s14=4, s15=4, s16=4, s17=4, s18=4, s19=4, s20=4, s21=4, s22=3, s23=5, s24=5, s25=3] (stddev=0.63, mean=4.00, sum=100)
37-
leases#1: thrash_pct: [s1=0%, s2=34%, s3=45%, s4=45%, s5=23%, s6=0%, s7=0%, s8=0%, s9=0%, s10=0%, s11=0%, s12=0%, s13=0%, s14=0%, s15=0%, s16=15%, s17=0%, s18=0%, s19=0%, s20=0%, s21=16%, s22=0%, s23=0%, s24=0%, s25=8%] (sum=187%)
38-
replicas#1: first: [s1=56, s2=44, s3=25, s4=16, s5=12, s6=9, s7=8, s8=7, s9=7, s10=7, s11=7, s12=7, s13=7, s14=8, s15=7, s16=8, s17=7, s18=8, s19=7, s20=7, s21=7, s22=7, s23=7, s24=8, s25=7] (stddev=11.97, mean=12.00, sum=300)
39-
replicas#1: last: [s1=54, s2=33, s3=19, s4=11, s5=10, s6=9, s7=10, s8=8, s9=9, s10=9, s11=8, s12=8, s13=9, s14=9, s15=9, s16=9, s17=8, s18=8, s19=9, s20=9, s21=8, s22=7, s23=9, s24=10, s25=8] (stddev=9.99, mean=12.00, sum=300)
40-
replicas#1: thrash_pct: [s1=0%, s2=0%, s3=0%, s4=5%, s5=5%, s6=0%, s7=0%, s8=0%, s9=0%, s10=0%, s11=0%, s12=0%, s13=0%, s14=0%, s15=0%, s16=9%, s17=0%, s18=0%, s19=0%, s20=0%, s21=9%, s22=0%, s23=0%, s24=0%, s25=4%] (sum=31%)
41-
write_bytes_per_second#1: last: [s1=9721, s2=9412, s3=9170, s4=9071, s5=9058, s6=9017, s7=9030, s8=9036, s9=9071, s10=9056, s11=8998, s12=8976, s13=9054, s14=9041, s15=9071, s16=9064, s17=9041, s18=9034, s19=9053, s20=9025, s21=9019, s22=9028, s23=9069, s24=9090, s25=9033] (stddev=151.38, mean=9089.52, sum=227238)
42-
write_bytes_per_second#1: thrash_pct: [s1=461%, s2=452%, s3=399%, s4=395%, s5=411%, s6=380%, s7=378%, s8=353%, s9=336%, s10=337%, s11=330%, s12=360%, s13=357%, s14=364%, s15=341%, s16=349%, s17=382%, s18=375%, s19=355%, s20=354%, s21=400%, s22=345%, s23=426%, s24=381%, s25=348%] (sum=9369%)
43-
artifacts[mma-only]: a914dbff5ba132ec
31+
cpu#1: last: [s1=6305516202, s2=4203121714, s3=4209044827, s4=6316721595, s5=4197368968, s6=2107290360, s7=2099077812, s8=4207644575, s9=4199517611, s10=4195522966, s11=4205024256, s12=4198000753, s13=4202637211, s14=4186734543, s15=4199295119, s16=4199174047, s17=4204480499, s18=6290045651, s19=4193666150, s20=4211828495, s21=4201735135, s22=2096004946, s23=4201402387, s24=4193586357, s25=4193056496] (stddev=1029616922.87, mean=4200699947.00, sum=105017498675)
32+
cpu#1: thrash_pct: [s1=6%, s2=47%, s3=81%, s4=21%, s5=11%, s6=2%, s7=1%, s8=3%, s9=2%, s10=3%, s11=2%, s12=3%, s13=3%, s14=2%, s15=11%, s16=2%, s17=3%, s18=3%, s19=3%, s20=2%, s21=3%, s22=2%, s23=3%, s24=2%, s25=2%] (sum=222%)
33+
cpu_util#1: last: [s1=0.79, s2=0.53, s3=0.53, s4=0.79, s5=0.52, s6=0.26, s7=0.26, s8=0.53, s9=0.52, s10=0.52, s11=0.53, s12=0.52, s13=0.53, s14=0.52, s15=0.52, s16=0.52, s17=0.53, s18=0.79, s19=0.52, s20=0.53, s21=0.53, s22=0.26, s23=0.53, s24=0.52, s25=0.52] (stddev=0.13, mean=0.53, sum=13)
34+
cpu_util#1: thrash_pct: [s1=6%, s2=47%, s3=81%, s4=21%, s5=11%, s6=2%, s7=1%, s8=3%, s9=2%, s10=3%, s11=2%, s12=3%, s13=3%, s14=2%, s15=11%, s16=2%, s17=3%, s18=3%, s19=3%, s20=2%, s21=3%, s22=2%, s23=3%, s24=2%, s25=2%] (sum=222%)
35+
leases#1: first: [s1=29, s2=13, s3=4, s4=2, s5=2, s6=2, s7=3, s8=3, s9=2, s10=3, s11=4, s12=1, s13=2, s14=2, s15=4, s16=3, s17=3, s18=3, s19=2, s20=3, s21=3, s22=2, s23=2, s24=1, s25=2] (stddev=5.56, mean=4.00, sum=100)
36+
leases#1: last: [s1=5, s2=4, s3=4, s4=4, s5=4, s6=3, s7=4, s8=4, s9=4, s10=4, s11=5, s12=3, s13=3, s14=4, s15=5, s16=4, s17=4, s18=6, s19=4, s20=4, s21=4, s22=3, s23=4, s24=3, s25=4] (stddev=0.69, mean=4.00, sum=100)
37+
leases#1: thrash_pct: [s1=0%, s2=40%, s3=71%, s4=15%, s5=8%, s6=0%, s7=0%, s8=0%, s9=0%, s10=0%, s11=0%, s12=0%, s13=0%, s14=0%, s15=8%, s16=0%, s17=0%, s18=0%, s19=0%, s20=0%, s21=0%, s22=0%, s23=0%, s24=0%, s25=0%] (sum=142%)
38+
replicas#1: first: [s1=56, s2=44, s3=25, s4=16, s5=11, s6=9, s7=8, s8=8, s9=7, s10=8, s11=7, s12=7, s13=7, s14=7, s15=7, s16=7, s17=7, s18=7, s19=7, s20=8, s21=8, s22=7, s23=8, s24=7, s25=7] (stddev=11.96, mean=12.00, sum=300)
39+
replicas#1: last: [s1=55, s2=32, s3=15, s4=14, s5=11, s6=9, s7=9, s8=9, s9=9, s10=9, s11=8, s12=9, s13=8, s14=9, s15=8, s16=8, s17=8, s18=10, s19=9, s20=9, s21=8, s22=7, s23=9, s24=9, s25=9] (stddev=9.99, mean=12.00, sum=300)
40+
replicas#1: thrash_pct: [s1=0%, s2=0%, s3=0%, s4=0%, s5=4%, s6=0%, s7=0%, s8=0%, s9=0%, s10=0%, s11=0%, s12=0%, s13=0%, s14=0%, s15=4%, s16=0%, s17=0%, s18=0%, s19=0%, s20=0%, s21=0%, s22=0%, s23=0%, s24=0%, s25=0%] (sum=8%)
41+
write_bytes_per_second#1: last: [s1=9716, s2=9343, s3=9125, s4=9067, s5=9042, s6=9053, s7=9020, s8=9052, s9=9057, s10=9082, s11=9000, s12=9014, s13=9018, s14=9057, s15=9071, s16=9067, s17=9060, s18=9063, s19=9030, s20=9049, s21=9047, s22=9031, s23=9068, s24=9035, s25=9069] (stddev=142.27, mean=9089.44, sum=227236)
42+
write_bytes_per_second#1: thrash_pct: [s1=418%, s2=368%, s3=390%, s4=341%, s5=347%, s6=355%, s7=338%, s8=308%, s9=363%, s10=340%, s11=320%, s12=337%, s13=341%, s14=367%, s15=331%, s16=367%, s17=307%, s18=352%, s19=355%, s20=370%, s21=356%, s22=331%, s23=352%, s24=367%, s25=362%] (sum=8782%)
43+
artifacts[mma-only]: c30c22902539d7de
4444
==========================
45-
cpu#1: last: [s1=4196738540, s2=4209925703, s3=4197642531, s4=4203976925, s5=6310337534, s6=4209549595, s7=4203396226, s8=4200533744, s9=4199232325, s10=4204283723, s11=4202398757, s12=4202236594, s13=4200513264, s14=4198252071, s15=4197887916, s16=4189080432, s17=4200838805, s18=4204416326, s19=4199394659, s20=2104037568, s21=4206078761, s22=4205179750, s23=4193854155, s24=4207461570, s25=4202573370] (stddev=594880102.87, mean=4201992833.76, sum=105049820844)
46-
cpu#1: thrash_pct: [s1=5%, s2=4%, s3=10%, s4=10%, s5=4%, s6=3%, s7=3%, s8=2%, s9=3%, s10=2%, s11=3%, s12=26%, s13=3%, s14=3%, s15=2%, s16=3%, s17=3%, s18=2%, s19=3%, s20=2%, s21=3%, s22=34%, s23=3%, s24=3%, s25=3%] (sum=139%)
47-
cpu_util#1: last: [s1=0.52, s2=0.53, s3=0.52, s4=0.53, s5=0.79, s6=0.53, s7=0.53, s8=0.53, s9=0.52, s10=0.53, s11=0.53, s12=0.53, s13=0.53, s14=0.52, s15=0.52, s16=0.52, s17=0.53, s18=0.53, s19=0.52, s20=0.26, s21=0.53, s22=0.53, s23=0.52, s24=0.53, s25=0.53] (stddev=0.07, mean=0.53, sum=13)
48-
cpu_util#1: thrash_pct: [s1=5%, s2=4%, s3=10%, s4=10%, s5=4%, s6=3%, s7=3%, s8=2%, s9=3%, s10=2%, s11=3%, s12=26%, s13=3%, s14=3%, s15=2%, s16=3%, s17=3%, s18=2%, s19=3%, s20=2%, s21=3%, s22=34%, s23=3%, s24=3%, s25=3%] (sum=139%)
49-
leases#1: first: [s1=28, s2=14, s3=4, s4=3, s5=3, s6=2, s7=2, s8=2, s9=2, s10=2, s11=3, s12=2, s13=3, s14=3, s15=2, s16=3, s17=3, s18=3, s19=2, s20=2, s21=2, s22=3, s23=3, s24=3, s25=1] (stddev=5.43, mean=4.00, sum=100)
50-
leases#1: last: [s1=3, s2=2, s3=2, s4=4, s5=4, s6=4, s7=4, s8=3, s9=4, s10=4, s11=5, s12=4, s13=5, s14=5, s15=4, s16=5, s17=4, s18=4, s19=4, s20=4, s21=4, s22=4, s23=5, s24=5, s25=4] (stddev=0.80, mean=4.00, sum=100)
51-
leases#1: thrash_pct: [s1=0%, s2=0%, s3=8%, s4=8%, s5=8%, s6=0%, s7=0%, s8=0%, s9=0%, s10=0%, s11=0%, s12=23%, s13=0%, s14=0%, s15=0%, s16=0%, s17=0%, s18=0%, s19=0%, s20=0%, s21=0%, s22=30%, s23=0%, s24=0%, s25=0%] (sum=78%)
52-
replicas#1: first: [s1=56, s2=44, s3=25, s4=16, s5=12, s6=9, s7=8, s8=7, s9=7, s10=7, s11=7, s12=7, s13=7, s14=8, s15=7, s16=8, s17=7, s18=8, s19=7, s20=7, s21=7, s22=7, s23=7, s24=8, s25=7] (stddev=11.97, mean=12.00, sum=300)
53-
replicas#1: last: [s1=12, s2=12, s3=13, s4=12, s5=12, s6=14, s7=11, s8=12, s9=12, s10=11, s11=12, s12=12, s13=12, s14=12, s15=12, s16=11, s17=12, s18=12, s19=12, s20=12, s21=11, s22=11, s23=13, s24=13, s25=12] (stddev=0.69, mean=12.00, sum=300)
54-
replicas#1: thrash_pct: [s1=0%, s2=0%, s3=0%, s4=5%, s5=4%, s6=0%, s7=0%, s8=0%, s9=0%, s10=0%, s11=0%, s12=0%, s13=0%, s14=0%, s15=0%, s16=0%, s17=0%, s18=0%, s19=0%, s20=0%, s21=0%, s22=5%, s23=0%, s24=0%, s25=0%] (sum=14%)
55-
write_bytes_per_second#1: last: [s1=1657, s2=179, s3=1671, s4=3136, s5=7606, s6=9091, s7=9045, s8=10574, s9=12089, s10=9083, s11=12027, s12=10531, s13=10575, s14=10567, s15=12086, s16=10579, s17=12085, s18=9095, s19=12056, s20=13503, s21=9063, s22=9088, s23=10619, s24=9136, s25=12089] (stddev=3540.66, mean=9089.20, sum=227230)
56-
write_bytes_per_second#1: thrash_pct: [s1=21%, s2=16%, s3=27%, s4=26%, s5=46%, s6=67%, s7=64%, s8=61%, s9=46%, s10=63%, s11=44%, s12=45%, s13=46%, s14=66%, s15=43%, s16=46%, s17=72%, s18=92%, s19=46%, s20=54%, s21=43%, s22=38%, s23=51%, s24=67%, s25=81%] (sum=1271%)
57-
artifacts[mma-count]: f4a8d3922318a8d0
45+
cpu#1: last: [s1=4195620275, s2=4196803898, s3=4201453881, s4=4207023762, s5=4205406711, s6=4208677351, s7=4214607012, s8=4200332000, s9=4197470437, s10=4195295352, s11=4213016495, s12=4204698413, s13=4207405981, s14=4200445833, s15=4195950476, s16=4206976540, s17=4201083041, s18=4195551669, s19=4194449158, s20=4199778225, s21=4200014668, s22=4201038712, s23=4199242495, s24=4197916137, s25=4199865623] (stddev=5412725.77, mean=4201604965.80, sum=105040124145)
46+
cpu#1: thrash_pct: [s1=4%, s2=4%, s3=18%, s4=3%, s5=2%, s6=10%, s7=2%, s8=11%, s9=11%, s10=10%, s11=2%, s12=3%, s13=10%, s14=3%, s15=3%, s16=11%, s17=11%, s18=11%, s19=11%, s20=11%, s21=2%, s22=11%, s23=2%, s24=2%, s25=3%] (sum=172%)
47+
cpu_util#1: last: [s1=0.52, s2=0.52, s3=0.53, s4=0.53, s5=0.53, s6=0.53, s7=0.53, s8=0.53, s9=0.52, s10=0.52, s11=0.53, s12=0.53, s13=0.53, s14=0.53, s15=0.52, s16=0.53, s17=0.53, s18=0.52, s19=0.52, s20=0.52, s21=0.53, s22=0.53, s23=0.52, s24=0.52, s25=0.52] (stddev=0.00, mean=0.53, sum=13)
48+
cpu_util#1: thrash_pct: [s1=4%, s2=4%, s3=18%, s4=3%, s5=2%, s6=10%, s7=2%, s8=11%, s9=11%, s10=10%, s11=2%, s12=3%, s13=10%, s14=3%, s15=3%, s16=11%, s17=11%, s18=11%, s19=11%, s20=11%, s21=2%, s22=11%, s23=2%, s24=2%, s25=3%] (sum=172%)
49+
leases#1: first: [s1=29, s2=13, s3=4, s4=2, s5=2, s6=2, s7=3, s8=3, s9=2, s10=3, s11=4, s12=1, s13=2, s14=2, s15=4, s16=3, s17=3, s18=3, s19=2, s20=3, s21=3, s22=2, s23=2, s24=1, s25=2] (stddev=5.56, mean=4.00, sum=100)
50+
leases#1: last: [s1=3, s2=3, s3=2, s4=3, s5=4, s6=4, s7=5, s8=4, s9=4, s10=4, s11=5, s12=5, s13=3, s14=5, s15=5, s16=4, s17=4, s18=6, s19=4, s20=4, s21=4, s22=4, s23=4, s24=3, s25=4] (stddev=0.85, mean=4.00, sum=100)
51+
leases#1: thrash_pct: [s1=0%, s2=9%, s3=15%, s4=0%, s5=0%, s6=8%, s7=0%, s8=7%, s9=8%, s10=7%, s11=0%, s12=0%, s13=7%, s14=0%, s15=0%, s16=7%, s17=14%, s18=8%, s19=8%, s20=7%, s21=0%, s22=8%, s23=0%, s24=0%, s25=0%] (sum=114%)
52+
replicas#1: first: [s1=56, s2=44, s3=25, s4=16, s5=11, s6=9, s7=8, s8=8, s9=7, s10=8, s11=7, s12=7, s13=7, s14=7, s15=7, s16=7, s17=7, s18=7, s19=7, s20=8, s21=8, s22=7, s23=8, s24=7, s25=7] (stddev=11.96, mean=12.00, sum=300)
53+
replicas#1: last: [s1=14, s2=12, s3=11, s4=12, s5=12, s6=12, s7=12, s8=12, s9=12, s10=12, s11=12, s12=13, s13=12, s14=12, s15=12, s16=12, s17=11, s18=11, s19=12, s20=12, s21=12, s22=11, s23=11, s24=12, s25=14] (stddev=0.75, mean=12.00, sum=300)
54+
replicas#1: thrash_pct: [s1=0%, s2=16%, s3=6%, s4=5%, s5=0%, s6=0%, s7=5%, s8=0%, s9=5%, s10=0%, s11=0%, s12=0%, s13=0%, s14=0%, s15=0%, s16=5%, s17=9%, s18=5%, s19=5%, s20=0%, s21=0%, s22=0%, s23=0%, s24=0%, s25=0%] (sum=60%)
55+
write_bytes_per_second#1: last: [s1=1687, s2=4653, s3=1641, s4=1668, s5=9057, s6=9096, s7=10547, s8=10574, s9=10578, s10=10615, s11=10528, s12=13516, s13=10549, s14=13531, s15=12095, s16=12095, s17=7615, s18=12030, s19=9073, s20=9093, s21=9109, s22=10572, s23=9099, s24=9085, s25=9144] (stddev=3277.31, mean=9090.00, sum=227250)
56+
write_bytes_per_second#1: thrash_pct: [s1=23%, s2=95%, s3=22%, s4=19%, s5=39%, s6=66%, s7=66%, s8=43%, s9=73%, s10=43%, s11=40%, s12=59%, s13=125%, s14=53%, s15=43%, s16=50%, s17=90%, s18=47%, s19=67%, s20=42%, s21=62%, s22=42%, s23=42%, s24=97%, s25=42%] (sum=1391%)
57+
artifacts[mma-count]: e82e4e9c52d587b
5858
==========================

‎pkg/kv/kvserver/asim/tests/testdata/non_rand/sma/downreplicate_recently_down_node_issue152604.txt‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# This test reproduces the issue observed in #152604, where
2+
# a zone config change prefers removing live replicas despite
3+
# the presence of recently down replicas. This leads to loss
4+
# of quorum.
5+
#
6+
# See:
7+
# - https://github.com/cockroachdb/cockroach/issues/152604
8+
# - https://github.com/cockroachdb/cockroach/issues/155734
9+
10+
gen_cluster nodes=5
11+
----
12+
13+
# Place ranges, replicated across all five nodes.
14+
gen_ranges ranges=100 repl_factor=5 min_key=1 max_key=10000
15+
----
16+
17+
# Mark n4 and n5 as NodeLivenessStatus_UNAVAILABLE, which is the status
18+
# stores have when down but not down for long enough to be marked as dead.
19+
# The range doesn't lose quorum as a result of this, since three replicas
20+
# are still around.
21+
set_liveness node=4 liveness=unavailable
22+
----
23+
24+
set_liveness node=5 liveness=unavailable
25+
----
26+
27+
# Trigger down-replication to three replicas.
28+
29+
set_span_config
30+
[0,10000): num_replicas=3 num_voters=3
31+
----
32+
33+
# Note how s4 and s5 retain their replicas, while replicas are being
34+
# remved from live nodes s1-s3. This leads to a loss of quorum that
35+
# isn't immediately obvious since this is an asim test, but the logs
36+
# show that the allocator itself realizes (when trying to make the
37+
# next change), but that is too late.
38+
#
39+
# In the real world, as of #156464, these dangerous replication changes
40+
# would be blocked, but it is far from ideal that they are attempted
41+
# in the first place.
42+
eval duration=10m cfgs=(sma-count) metrics=(replicas)
43+
----
44+
replicas#1: first: [s1=101, s2=101, s3=101, s4=101, s5=101] (stddev=0.00, mean=101.00, sum=505)
45+
replicas#1: last: [s1=68, s2=67, s3=68, s4=101, s5=101] (stddev=16.33, mean=81.00, sum=405)
46+
replicas#1: thrash_pct: [s1=0%, s2=0%, s3=0%, s4=0%, s5=0%] (sum=0%)
47+
artifacts[sma-count]: ff4c6613afd4b749
48+
==========================

0 commit comments

Comments
 (0)