Merge #153484 #153503 #153586

153484: asim: use queues with LBRebalancingMultiMetricOnly r=tbg a=wenyihu6

Epic: none
Release note: none

---
**asim: use queues with LBRebalancingMultiMetricOnly**

This commit updates the mma-only mode to enable replicate and lease queues while
using LBRebalancingMultiMetricOnly. In this mode, constraint repairs in the
replicate and lease queues should still occur with mma enabled, but range
count–based rebalancing is disabled.

It’s a bit surprising that this diff didn’t cause any datadriven test changes. I
think this is likely because the mma tests currently don’t exercise constraint
repair. Tests that do cover constraint repair and lease preference likely don’t
run in mma-only mode. From a range/lease count rebalancing perspective, this
makes sense, as this mode previously disabled both replicate and lease queues.
Future commits will add test coverage to verify that this mode supports constraint
repair with mma.

---
**asim: rename mode from both to mma-and-count**

This commit renames the mode from both to mma-and-count to clarify that it
enables the replica and lease count based rebalancing.

---
**asim: schedule event with default**

Previously, the default configuration did not schedule an event. This would
cause issues in cases like cfg(mma-only,default), where mma-only would schedule
an event to change the cluster setting, but default would not reset it in the
next round, leaving the cluster setting incorrectly persisted. This commit fixes
the issue by ensuring that default also sets the cluster setting as an event.
The persistence of settings and cluster settings across simulations is a bit
troublesome. For now, this fix is sufficient since we only allow rebalance_mode
to be a delay event. We may need to revisit this in the future.

---
**asim: add mma-only mode to some existing asim tests**

This commit adds mma-only mode to some existing asim tests that exercise
constraint conformance to demonstrate that mma-only mode can pass assertions
with the replicate and lease queues enabled. Note that some assertions on
QPS/lease fail which worth investigating further. I took a quick pass - CPU
usage was balanced, but QPS was not.

Note that this commit removes the default mode for these tests so that changes
can be compared more clearly with git diff, avoiding side-by-side comparisons
between two modes. A follow-up commit will reintroduce the default mode
alongside mma-only.

---
**asim: add the default mode back for asim tests**

This commit adds the default mode for some asim tests.

---
**kvserverbase: update validation check for LoadBasedRebalancingMode**

Previously, we introduced LBRebalancingMultiMetricOnly and renamed
LBRebalancingMultiMetric to LBRebalancingMultiMetricAndCount. However, a
hardcoded enum string check in the cluster setting was overlooked and not
updated. This commit fixes that by updating the check correctly.



153503: server: always enable DRPC listener irrespective of the setting r=cthumuluru-crdb a=shubhamdhama

Previously, the DRPC server was only started when the experimental DRPC
setting was enabled. This created a problem where enabling DRPC after a
node had already started would not work, as the listener would not be
running and would reject all DRPC requests until the node was restarted.

To address this, this patch always enables the DRPC listener regardless of
the experimental setting. The cost of leaving the DRPC listener on all the
time is low, and this change simplifies the codebase by removing the need
to check cluster settings when creating clients (see #153449). This could even help us
avoid the need for a restart to enable DRPC entirely, if we can also figure
out how to swap our gRPC clients for DRPC clients in various subsystems.

Fixes #153450
Release note: None
Epic: CRDB-52378

153586: release: released CockroachDB version v25.4.0-alpha.1. Next version: v25.4.0-alpha.2 r=rail a=cockroach-teamcity


Release note: None
Epic: None
Release justification: non-production (release infra) change.


Co-authored-by: wenyihu6 <[email protected]>
Co-authored-by: Shubham Dhama <[email protected]>
Co-authored-by: Justin Beaver <[email protected]>

Author: 4 people

docs/generated/settings/settings.html

@@ -87,7 +87,7 @@
 <tr><td><div id="setting-jobs-retention-time" class="anchored"><code>jobs.retention_time</code></div></td><td>duration</td><td><code>336h0m0s</code></td><td>the amount of time for which records for completed jobs are retained</td><td>Basic/Standard/Advanced/Self-Hosted</td></tr>
 <tr><td><div id="setting-kv-allocator-lease-rebalance-threshold" class="anchored"><code>kv.allocator.lease_rebalance_threshold</code></div></td><td>float</td><td><code>0.05</code></td><td>minimum fraction away from the mean a store&#39;s lease count can be before it is considered for lease-transfers</td><td>Advanced/Self-Hosted</td></tr>
 <tr><td><div id="setting-kv-allocator-load-based-lease-rebalancing-enabled" class="anchored"><code>kv.allocator.load_based_lease_rebalancing.enabled</code></div></td><td>boolean</td><td><code>true</code></td><td>set to enable rebalancing of range leases based on load and latency</td><td>Advanced/Self-Hosted</td></tr>
-<tr><td><div id="setting-kv-allocator-load-based-rebalancing" class="anchored"><code>kv.allocator.load_based_rebalancing</code></div></td><td>enumeration</td><td><code>leases and replicas</code></td><td>whether to rebalance based on the distribution of load across stores [off = 0, leases = 1, leases and replicas = 2, multi-metric = 3, multi-metric and count = 4]</td><td>Advanced/Self-Hosted</td></tr>
+<tr><td><div id="setting-kv-allocator-load-based-rebalancing" class="anchored"><code>kv.allocator.load_based_rebalancing</code></div></td><td>enumeration</td><td><code>leases and replicas</code></td><td>whether to rebalance based on the distribution of load across stores [off = 0, leases = 1, leases and replicas = 2, multi-metric only = 3, multi-metric and count = 4]</td><td>Advanced/Self-Hosted</td></tr>
 <tr><td><div id="setting-kv-allocator-load-based-rebalancing-objective" class="anchored"><code>kv.allocator.load_based_rebalancing.objective</code></div></td><td>enumeration</td><td><code>cpu</code></td><td>what objective does the cluster use to rebalance; if set to `qps` the cluster will attempt to balance qps among stores, if set to `cpu` the cluster will attempt to balance cpu usage among stores [qps = 0, cpu = 1]</td><td>Advanced/Self-Hosted</td></tr>
 <tr><td><div id="setting-kv-allocator-load-based-rebalancing-interval" class="anchored"><code>kv.allocator.load_based_rebalancing_interval</code></div></td><td>duration</td><td><code>1m0s</code></td><td>the rough interval at which each store will check for load-based lease / replica rebalancing opportunities</td><td>Advanced/Self-Hosted</td></tr>
 <tr><td><div id="setting-kv-allocator-qps-rebalance-threshold" class="anchored"><code>kv.allocator.qps_rebalance_threshold</code></div></td><td>float</td><td><code>0.1</code></td><td>minimum fraction away from the mean a store&#39;s QPS (such as queries per second) can be before it is considered overfull or underfull</td><td>Advanced/Self-Hosted</td></tr>

pkg/build/version.txt

@@ -1 +1 @@
-v25.4.0-alpha.1
+v25.4.0-alpha.2

pkg/kv/kvserver/asim/tests/datadriven_simulation_test.go

@@ -463,7 +463,7 @@ func TestDataDriven(t *testing.T) {
 				if len(cfgs) == 0 {
 					// TODO(tbg): force each test to specify the configs it wants to run
 					// under.
-					cfgs = []string{"default"}
+					cfgs = []string{"sma-count"}
 				}
 
 				metricsMap := map[string]struct{}{}
@@ -475,33 +475,26 @@ func TestDataDriven(t *testing.T) {
 				require.NotZero(t, rangeGen)
 
 				knownConfigurations := map[string]func(eg *gen.StaticEvents){
-					// In default mode, the test manages its rebalancer-related settings
-					// manually.
-					"default": func(*gen.StaticEvents) {},
-					// 'mma-only' runs with the multi-metric allocator and turns off the
-					// replicate and lease queues.
-					"sma-only": func(eg *gen.StaticEvents) {
+					"sma-count": func(eg *gen.StaticEvents) {
 						eg.ScheduleEvent(settingsGen.Settings.StartTime, 0,
 							event.SetSimulationSettingsEvent{
 								IsClusterSetting: true,
 								Key:              "LBRebalancingMode",
 								Value:            int64(kvserverbase.LBRebalancingLeasesAndReplicas),
 							})
 					},
+					// 'mma-only' runs with the multi-metric allocator and turns off the
+					// replicate and lease queues.
 					"mma-only": func(eg *gen.StaticEvents) {
-						settingsGen.Settings.ReplicateQueueEnabled = false
-						settingsGen.Settings.LeaseQueueEnabled = false
 						eg.ScheduleEvent(settingsGen.Settings.StartTime, 0,
 							event.SetSimulationSettingsEvent{
 								IsClusterSetting: true,
 								Key:              "LBRebalancingMode",
-								Value:            int64(kvserverbase.LBRebalancingMultiMetricAndCount),
+								Value:            int64(kvserverbase.LBRebalancingMultiMetricOnly),
 							})
 					},
 					// Both the replicate/lease queues and the MMA are enabled.
-					"both": func(eg *gen.StaticEvents) {
-						settingsGen.Settings.ReplicateQueueEnabled = true
-						settingsGen.Settings.LeaseQueueEnabled = true
+					"mma-and-count": func(eg *gen.StaticEvents) {
 						eg.ScheduleEvent(settingsGen.Settings.StartTime, 0,
 							event.SetSimulationSettingsEvent{
 								IsClusterSetting: true,

pkg/kv/kvserver/asim/tests/testdata/generated/example_rebalancing/example_rebalancing_mma-only_1.json

@@ -0,0 +1,3 @@
+AU_EAST
+  AU_EAST_1
+    └── [1 2 3 4 5 6 7]