Merge #153067 #155554

153067: opt: make outside-of-histogram estimates more pessimistic r=DrewKimball a=DrewKimball

#### opt: improve outside-histogram optimizer tests

This commit makes the following improvements to the `outside-histogram`
optimizer test:
* Fixes a minor mistake in the test, where the upper bounds on a histogram
  for a unique column were incorrect.
* Simplifies the test code for setting session variable defaults.
* Adds comments explaining each test case.

Epic: None

Release note: None

#### opt: add unique and distinct column to outside-histogram test

This commit adds a column `u` with a unique index, and a column `d`
with a non-unique index to the outside-histogram test table. Both
columns use the same histogram as the primary key `k` (and therefore
have no duplicate values).

This commit also adds tests that are similar to `Q1`, but using the
new columns instead of `k`. This shows the optimizer's behavior when
the "good" plan requires an index join, and when filtering on `d`,
shows how the optimizer behaves when no plan limits the max cardinality
of the query.

Epic: None

Release note: None

#### opt: add outside-histogram test for missing enum value

This commit adds a pair of `outside-histogram` optimizer test cases that
simulate a query against an enum column for which one of the values was not
sampled.

Epic: None

Release note: None

#### opt: add outside-histogram tests with large table

This commit adds to the existing optimizer tests for filters that select
values outside of histograms. The new tests are against a large table
to show how the optimizer's trust of low-selectivity estimates varies
with table size.

Epic: None

Release note: None

#### opt: make outside-of-histogram estimates more pessimistic

This commit makes rowcount estimation fall back on distinct count estimates
when a constraint includes zero histogram values. This biases the optimizer
toward less risky plans when less information is known about the filtered
values.

The pessimistic logic is triggered when the estimate derived from a histogram
is smaller than `table_row_count / 10,000`. This threshold is chosen because
we choose samples such that we expect to sample *nearly* every value with
multiplicity down to `table_row_count / 10,000` (see computeNumberSamples).
Selecivity estimates from a histogram below this resolution are suspect,
since there is increasing likelihood that a value was missed either due
to being omitted from the sample, or due to staleness.

Informs #130201

Release note (sql change): Added a clamp for row-count estimates over very
large tables so that the optimizer assumes that at least one distinct value
will be scanned. This reduces the chances of a catastrophic underestimate.
The new logic is off by default, gated by a session setting
`optimizer_clamp_low_histogram_selectivity`.

#### opt: use pessimistic estimates for inequality filters

This commit changes row count estimates for inequality filters, so that
we expect at least `rowCount / (bucketCount * 100)` rows to "survive" the
filter. This is in line with Postgres, which clamps inequality rowcount
estimates in a similar fashion.

Informs #130201

Release note (sql change): Added a clamp for the estimated selectivity of
inequality predicates that are unbounded on one or both sides (ex: `x > 5`).
This reduces the chances of a catastrophic understimate causing the optimizer
to choose a poorly-constrained scan. The new logic is off by default, gated by
the session setting `optimizer_clamp_inequality_selectivity`.

#### sql/opt: add telemetry counters for selectivity clamping

This commit adds telemetry counters for the new histogram selectivity
clamping behavior, as well as log messages to indicate when the new
behavior applies in a query's trace.

Epic: None

Release note: None

155554: kvserver: split meta1 and meta2 r=iskettaneh a=iskettaneh

Previously, we used to start meta1 and meta2 in one range at bootstrap, and rely on load-based splitting to split them if needed. However, in some cases, load-based splitting doesn't work when it decides to split a point in meta1 (meta1 is not allowed to split).

This PR does two things:
1) At bootstrap, we create two separate ranges for meta1 and meta2.
2) Use spanconfig to install a split point at the start of meta2. This will prevent the two ranges from getting merged together, also, it will split meta1 and meta2 ranges for clusters that were bootstrapped before this PR.


In order to ensure that we get the descriptors correct, I started a
cluster (when meta1 and meta2 were on the same range), and manually
split them and captured the descriptors:

```
[1] Meta Key: /Meta1/""
    Range r1:
      StartKey: /Min
      EndKey:   /Meta2/""
      Replicas: (n1,s1):1
      Generation: 1

[2] Meta Key: /Meta1/Max
    Range r79:
      StartKey: /Meta2/""
      EndKey:   /System/NodeLiveness
      Replicas: (n1,s1):1
      Generation: 1

[3] Meta Key: /Meta2/System/NodeLiveness
    Range r79:
      StartKey: /Meta2/""
      EndKey:   /System/NodeLiveness
      Replicas: (n1,s1):1
      Generation: 1

[4] Meta Key: /Meta2/System/NodeLivenessMax
    Range r2:
      StartKey: /System/NodeLiveness
      EndKey:   /System/NodeLivenessMax
      Replicas: (n1,s1):1
      Generation: 0
```

After this commit, the first few ranges are bootstrapped with
the following descriptors:

```
[1] Meta Key: /Meta1/""
    Range r1:
      StartKey: /Min
      EndKey:   /Meta2/""
      Replicas: (n1,s1):1
      Generation: 0

[2] Meta Key: /Meta1/Max
    Range r2:
      StartKey: /Meta2/""
      EndKey:   /System/NodeLiveness
      Replicas: (n1,s1):1
      Generation: 0

[3] Meta Key: /Meta2/System/NodeLiveness
    Range r2:
      StartKey: /Meta2/""
      EndKey:   /System/NodeLiveness
      Replicas: (n1,s1):1
      Generation: 0

[4] Meta Key: /Meta2/System/NodeLivenessMax
    Range r3:
      StartKey: /System/NodeLiveness
      EndKey:   /System/NodeLivenessMax
      Replicas: (n1,s1):1
      Generation: 0
```

Fixes: #119421

Release note: None

Co-authored-by: Drew Kimball <[email protected]>
Co-authored-by: iskettaneh <[email protected]>

Author: 3 people

pkg/ccl/logictestccl/testdata/logic_test/crdb_internal_tenant

@@ -251,12 +251,12 @@ txn_id  txn_fingerprint_id  query  implicit_txn  session_id  start_time  end_tim
 query ITTI
 SELECT range_id, start_pretty, end_pretty, lease_holder FROM crdb_internal.ranges
 ----
-79  /Tenant/10  /Tenant/11  1
+80  /Tenant/10  /Tenant/11  1
 
 query ITT
 SELECT range_id, start_pretty, end_pretty FROM crdb_internal.ranges_no_leases
 ----
-79  /Tenant/10  /Tenant/11
+80  /Tenant/10  /Tenant/11
 
 query IT
 SELECT zone_id, target FROM crdb_internal.zones ORDER BY 1

pkg/ccl/logictestccl/testdata/logic_test/multi_region_remote_access_error

@@ -471,7 +471,7 @@ skipif config multiregion-9node-3region-3azs-vec-off
 query I retry
 SELECT DISTINCT range_id FROM [SHOW RANGES FROM TABLE messages_rbr]
 ----
-84
+85
 
 # Update does not fail when accessing all rows in messages_rbr because lookup
 # join does not error out the lookup table in phase 1.

pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_insert_fast_path

@@ -200,7 +200,7 @@ query T rowsort
 SELECT message FROM [SHOW KV TRACE FOR SESSION]
 WHERE message LIKE '%batch%' AND message LIKE '%Scan%'
 ----
-r78: sending batch 4 Scan to (n1,s1):1
+r79: sending batch 4 Scan to (n1,s1):1
 
 # Regression test for #115377.
 statement ok

pkg/ccl/spanconfigccl/spanconfigreconcilerccl/testdata/basic

@@ -7,7 +7,8 @@ reconcile
 
 mutations
 ----
-upsert /{Min-System/NodeLiveness}          ttl_seconds=3600 num_replicas=5
+upsert /M{in-eta2}                         ttl_seconds=3600 num_replicas=5
+upsert /{Meta2-System/NodeLiveness}        ttl_seconds=3600 num_replicas=5
 upsert /System/NodeLiveness{-Max}          ttl_seconds=600 num_replicas=5
 upsert /System/{NodeLivenessMax-tsd}       range system
 upsert /System{/tsd-tse}                   range default
@@ -117,7 +118,7 @@ upsert /Table/10{7-8}                      num_replicas=7
 delete /Table/11{2-3}
 upsert /Table/11{2-3}                      num_replicas=7
 
-state offset=47
+state offset=48
 ----
 ...
 /Table/4{6-7}                              database system (host)
@@ -288,7 +289,7 @@ upsert /Table/7{5-6}                       ttl_seconds=100 ignore_strict_gc=true
 delete /Table/7{6-7}
 upsert /Table/7{6-7}                       ttl_seconds=100 ignore_strict_gc=true num_replicas=5 rangefeed_enabled=true
 
-state offset=5 limit=42
+state offset=6 limit=42
 ----
 ...
 /Table/{0-4}                               ttl_seconds=100 ignore_strict_gc=true num_replicas=5 rangefeed_enabled=true

pkg/ccl/spanconfigccl/spanconfigreconcilerccl/testdata/indexes

@@ -18,7 +18,7 @@ mutations
 ----
 upsert /Table/10{6-7}                      range default
 
-state offset=47
+state offset=48
 ----
 ...
 /Table/4{6-7}                              database system (host)
@@ -69,7 +69,7 @@ delete /Table/10{6-7}
 upsert /Table/106/{2-3}                    num_replicas=7 num_voters=5
 upsert /Table/10{6/3-7}                    num_replicas=7
 
-state offset=47
+state offset=48
 ----
 ...
 /Table/4{6-7}                              database system (host)
@@ -122,7 +122,7 @@ upsert /Table/106/{2-3}                    ttl_seconds=25 num_replicas=7 num_vot
 delete /Table/10{6/3-7}
 upsert /Table/10{6/3-7}                    ttl_seconds=3600 num_replicas=7
 
-state offset=47
+state offset=48
 ----
 ...
 /Table/4{6-7}                              database system (host)
@@ -165,7 +165,7 @@ exec-sql
 ALTER TABLE db.t CONFIGURE ZONE USING num_replicas = 9
 ----
 
-state offset=47
+state offset=48
 ----
 ...
 /Table/4{6-7}                              database system (host)
@@ -221,7 +221,7 @@ delete /Table/106{-/2}
 delete /Table/106/{2-3}
 delete /Table/10{6/3-7}
 
-state offset=46
+state offset=47
 ----
 ...
 /Table/4{5-6}                              ttl_seconds=7200 ignore_strict_gc=true num_replicas=5 rangefeed_enabled=true exclude_data_from_backup=true

pkg/ccl/spanconfigccl/spanconfigreconcilerccl/testdata/multitenant/basic

@@ -19,7 +19,7 @@ mutations
 
 # We should observe placeholder entries for both tenants (installed when
 # creating tenant records).
-state offset=47
+state offset=48
 ----
 ...
 /Table/4{6-7}                              database system (host)
@@ -130,7 +130,7 @@ upsert /Tenant/10/Table/7{4-5}             database system (tenant)
 upsert /Tenant/10/Table/7{5-6}             database system (tenant)
 upsert /Tenant/10/Table/7{6-7}             database system (tenant)
 
-state offset=47
+state offset=48
 ----
 ...
 /Table/4{6-7}                              database system (host)
@@ -250,7 +250,7 @@ upsert /Tenant/10/Table/10{7-8}            rangefeed_enabled=true
 upsert /Tenant/10/Table/11{2-3}            rangefeed_enabled=true
 upsert /Tenant/10/Table/11{3-4}            rangefeed_enabled=true
 
-state offset=81
+state offset=82
 ----
 ...
 /Tenant/10/Table/{7-8}                     database system (tenant)

pkg/ccl/spanconfigccl/spanconfigreconcilerccl/testdata/multitenant/protectedts

@@ -16,7 +16,7 @@ mutations
 
 # We should observe placeholder entries for both tenants (installed when
 # creating tenant records).
-state offset=47
+state offset=48
 ----
 ...
 /Table/4{6-7}                              database system (host)
@@ -204,9 +204,10 @@ mutations tenant=10
 delete {source=10,target=10}
 
 # All system span config targets should have been removed at this point.
-state limit=4
+state limit=5
 ----
-/{Min-System/NodeLiveness}                 ttl_seconds=3600 num_replicas=5
+/M{in-eta2}                                ttl_seconds=3600 num_replicas=5
+/{Meta2-System/NodeLiveness}               ttl_seconds=3600 num_replicas=5
 /System/NodeLiveness{-Max}                 ttl_seconds=600 num_replicas=5
 /System/{NodeLivenessMax-tsd}              range system
 /System{/tsd-tse}                          range default

pkg/ccl/spanconfigccl/spanconfigreconcilerccl/testdata/multitenant/range_tenants

@@ -35,7 +35,7 @@ initialize tenant=11
 # We should observe placeholder entries for both tenants (installed when
 # creating tenant records). tenant=11 should start off with whatever RANGE
 # TENANT was at the time.
-state offset=47
+state offset=48
 ----
 ...
 /Table/4{6-7}                              database system (host)
@@ -81,7 +81,7 @@ reconcile tenant=10
 mutations discard tenant=10
 ----
 
-state offset=47
+state offset=48
 ----
 ...
 /Table/4{6-7}                              database system (host)
@@ -214,7 +214,7 @@ reconcile tenant=11
 mutations discard tenant=11
 ----
 
-state offset=81
+state offset=82
 ----
 ...
 /Tenant/10/Table/{7-8}                     database system (tenant)

pkg/ccl/spanconfigccl/spanconfigreconcilerccl/testdata/multitenant/tenant_end_key_split

@@ -14,7 +14,7 @@ initialize tenant=11
 
 # A record IS written for a key that logically belongs to the next tenant,
 # tenant=12, because tenant=12 DOES NOT exist.
-state offset=59
+state offset=60
 ----
 ...
 /Table/{59-60}                             database system (host)
@@ -49,7 +49,7 @@ reconcile tenant=11
 # Peek near the start of the span_configurations table where tenant=11's records
 # are stored. The first one is from the start of its keyspace to start of
 # table with ID=4: /Tenant/11{-/Table/4}.
-state offset=60 limit=3
+state offset=61 limit=3
 ----
 ...
 /Table/6{0-1}                              database system (host)
@@ -60,7 +60,7 @@ state offset=60 limit=3
 # Peek near the end of the span_configurations table where tenant=11's records
 # are stored. The last one is for its last system table. Right now the split is
 # at /Tenant/12. Which is fine.
-state offset=103
+state offset=104
 ----
 ...
 /Tenant/11/Table/3{5-6}                    database system (tenant)
@@ -186,7 +186,7 @@ initialize tenant=10
 
 # A record IS NOT written for a key that logically belongs to the next tenant,
 # tenant=11, because tenant=11 DOES exist.
-state offset=59 limit=5
+state offset=60 limit=5
 ----
 ...
 /Table/{59-60}                             database system (host)

pkg/ccl/spanconfigccl/spanconfigreconcilerccl/testdata/named_zones

@@ -7,9 +7,10 @@ reconcile
 mutations discard
 ----
 
-state limit=5
+state limit=6
 ----
-/{Min-System/NodeLiveness}                 ttl_seconds=3600 num_replicas=5
+/M{in-eta2}                                ttl_seconds=3600 num_replicas=5
+/{Meta2-System/NodeLiveness}               ttl_seconds=3600 num_replicas=5
 /System/NodeLiveness{-Max}                 ttl_seconds=600 num_replicas=5
 /System/{NodeLivenessMax-tsd}              range system
 /System{/tsd-tse}                          range default
@@ -51,9 +52,10 @@ upsert /System/{NodeLivenessMax-tsd}       range default
 delete /System{tse-/SystemSpanConfigKeys}
 upsert /System{tse-/SystemSpanConfigKeys}  range default
 
-state limit=5
+state limit=6
 ----
-/{Min-System/NodeLiveness}                 ttl_seconds=3600 num_replicas=5
+/M{in-eta2}                                ttl_seconds=3600 num_replicas=5
+/{Meta2-System/NodeLiveness}               ttl_seconds=3600 num_replicas=5
 /System/NodeLiveness{-Max}                 ttl_seconds=600 num_replicas=7
 /System/{NodeLivenessMax-tsd}              range default
 /System{/tsd-tse}                          ttl_seconds=42
@@ -69,14 +71,17 @@ ALTER RANGE timeseries CONFIGURE ZONE DISCARD;
 
 mutations
 ----
-delete /{Min-System/NodeLiveness}
-upsert /{Min-System/NodeLiveness}          range default
+delete /M{in-eta2}
+upsert /M{in-eta2}                         range default
+delete /{Meta2-System/NodeLiveness}
+upsert /{Meta2-System/NodeLiveness}        range default
 delete /System{/tsd-tse}
 upsert /System{/tsd-tse}                   range default
 
-state limit=5
+state limit=6
 ----
-/{Min-System/NodeLiveness}                 range default
+/M{in-eta2}                                range default
+/{Meta2-System/NodeLiveness}               range default
 /System/NodeLiveness{-Max}                 ttl_seconds=600 num_replicas=7
 /System/{NodeLivenessMax-tsd}              range default
 /System{/tsd-tse}                          range default
@@ -100,8 +105,10 @@ ALTER RANGE default CONFIGURE ZONE USING gc.ttlseconds = 50;
 
 mutations
 ----
-delete /{Min-System/NodeLiveness}
-upsert /{Min-System/NodeLiveness}          ttl_seconds=50
+delete /M{in-eta2}
+upsert /M{in-eta2}                         ttl_seconds=50
+delete /{Meta2-System/NodeLiveness}
+upsert /{Meta2-System/NodeLiveness}        ttl_seconds=50
 delete /System/{NodeLivenessMax-tsd}
 upsert /System/{NodeLivenessMax-tsd}       ttl_seconds=50
 delete /System{/tsd-tse}
@@ -111,16 +118,17 @@ upsert /System{tse-/SystemSpanConfigKeys}  ttl_seconds=50
 delete /Table/10{6-7}
 upsert /Table/10{6-7}                      ttl_seconds=50
 
-state limit=5
+state limit=6
 ----
-/{Min-System/NodeLiveness}                 ttl_seconds=50
+/M{in-eta2}                                ttl_seconds=50
+/{Meta2-System/NodeLiveness}               ttl_seconds=50
 /System/NodeLiveness{-Max}                 ttl_seconds=600 num_replicas=7
 /System/{NodeLivenessMax-tsd}              ttl_seconds=50
 /System{/tsd-tse}                          ttl_seconds=50
 /System{tse-/SystemSpanConfigKeys}         ttl_seconds=50
 ...
 
-state offset=46
+state offset=47
 ----
 ...
 /Table/4{5-6}                              ttl_seconds=7200 ignore_strict_gc=true num_replicas=5 rangefeed_enabled=true exclude_data_from_backup=true
@@ -165,7 +173,7 @@ mutations
 ----
 upsert /Table/10{7-8}                      ttl_seconds=50
 
-state offset=46
+state offset=47
 ----
 ...
 /Table/4{5-6}                              ttl_seconds=7200 ignore_strict_gc=true num_replicas=5 rangefeed_enabled=true exclude_data_from_backup=true
@@ -213,9 +221,10 @@ upsert /System/{NodeLivenessMax-tsd}       ttl_seconds=100
 delete /System{tse-/SystemSpanConfigKeys}
 upsert /System{tse-/SystemSpanConfigKeys}  ttl_seconds=100
 
-state limit=5
+state limit=6
 ----
-/{Min-System/NodeLiveness}                 ttl_seconds=50
+/M{in-eta2}                                ttl_seconds=50
+/{Meta2-System/NodeLiveness}               ttl_seconds=50
 /System/NodeLiveness{-Max}                 ttl_seconds=600 num_replicas=7
 /System/{NodeLivenessMax-tsd}              ttl_seconds=100
 /System{/tsd-tse}                          ttl_seconds=50