want CockroachDB cluster/replication status in metrics, inventory?

[davepacheco]

CockroachDB exposes a bunch of metrics that are important for understanding the cluster's status and particularly when it's at reduced fault tolerance. I'm using our original testing notes as a reference and seeing these useful metrics:

• Each type of "Bad ranges" -- either broken out by reason or else separate metrics?
  • raft leaders that are not also leaseholders
  • unavailable
  • under-replicated
• Count of nodes
• Replicas per node
• Leaseholders per node
• Range operations (adds, removes, splits, merges)

I'm not sure if the basic query metrics there came from CockroachDB too but if they did that'd be useful to pull in, too.

Some of this is critical if we ever do graceful removal but that's not currently planned. Still, this seems very valuable to be able to have in Clickhouse if we're ever debugging an issue related to CockroachDB availability.

I'm not sure if some of this should also go into inventory but we could defer that until the point comes (if ever) that we want to act on these metrics programmatically.

[smklein]

I'm only half-starting on this -- got some additional support bundle thing to wrap up first -- but here are my notes from digging through the docs:

The crdb_internal system catalog seems to be the interface for accessing a lot of this info. Additionally, SHOW ZONE CONFIGURATION exposes a fair bit of this too.

here's some info I pulled from dogfood:

root@[fd00:1122:3344:105::3]:32221/omicron> SHOW ZONE CONFIGURATION FROM DATABASE omicron;
     target     |              raw_config_sql
----------------+-------------------------------------------
  RANGE default | ALTER RANGE default CONFIGURE ZONE USING
                |     range_min_bytes = 134217728,
                |     range_max_bytes = 536870912,
                |     gc.ttlseconds = 90000,
                |     num_replicas = 5,
                |     constraints = '[]',
                |     lease_preferences = '[]'
(1 row)

This information identifies the "num_replicas" desired, but it's a little tricky to parse -- it's hard to pull that value out as a column. It's also a little tricky, because the internal ranges might have different "num_replica" goals than the omicron DB.

From the CRDB internal tables:

I can view ranges with the following:

root@[fd00:1122:3344:105::3]:32221/omicron> SELECT range_id,replicas,voting_replicas,range_size from crdb_internal.ranges;
  range_id |  replicas   | voting_replicas | range_size
-----------+-------------+-----------------+-------------
         1 | {1,2,3,4,5} | {2,1,3,5,4}     |      36809
         2 | {1,2,3,4,5} | {3,2,5,1,4}     |      51581
         3 | {1,2,3,4,5} | {3,5,2,4,1}     |     295707
         4 | {1,2,3,4,5} | {5,4,3,2,1}     |  298639248
      1086 | {1,2,3,4,5} | {5,4,3,1,2}     |  305354661
       243 | {1,2,3,4,5} | {5,3,2,1,4}     |  319919901
      1107 | {1,2,3,4,5} | {5,4,3,1,2}     |  373695494
      1110 | {1,2,3,4,5} | {5,4,3,1,2}     |    6002810
       166 | {1,2,3,4,5} | {5,1,2,4,3}     |  325367684
      1100 | {1,2,3,4,5} | {5,4,3,2,1}     |  366771853

I can also observe the nodes:

root@[fd00:1122:3344:105::3]:32221/omicron> SELECT node_id,address,server_version FROM crdb_internal.kv_node_status;
  node_id |            address            | server_version
----------+-------------------------------+-----------------
        1 | [fd00:1122:3344:109::3]:32221 | 22.1
        2 | [fd00:1122:3344:105::3]:32221 | 22.1
        3 | [fd00:1122:3344:10b::3]:32221 | 22.1
        4 | [fd00:1122:3344:107::3]:32221 | 22.1
        5 | [fd00:1122:3344:108::3]:32221 | 22.1
(5 rows)

root@[fd00:1122:3344:105::3]:32221/omicron> SELECT * FROM crdb_internal.kv_node_liveness;
  node_id | epoch |       expiration       | draining | membership
----------+-------+------------------------+----------+-------------
        1 |   522 | 1746217524.845013471,0 |  false   | active
        2 |   778 | 1746217524.915877789,0 |  false   | active
        3 |   597 | 1746217525.269190907,0 |  false   | active
        4 |   564 | 1746217524.814026350,0 |  false   | active
        5 |   345 | 1746217526.206910969,0 |  false   | active
(5 rows)

And the "storage" for those nodes:

root@[fd00:1122:3344:105::3]:32221/omicron> SELECT node_id,store_id,range_count,lease_count from crdb_internal.kv_store_status;
  node_id | store_id | range_count | lease_count
----------+----------+-------------+--------------
        1 |        1 |         302 |          56
        2 |        2 |         302 |          58
        3 |        3 |         302 |          63
        4 |        4 |         302 |          65
        5 |        5 |         302 |          60

Putting this all together:

WITH range_replication AS (
  SELECT
    r.range_id,
    unnest(r.replicas) AS store_id,
    array_length(replicas, 1) as replica_count
  FROM
    crdb_internal.ranges r
)
SELECT
  s.node_id,
  replica_count,
  count(*) AS ranges_on_node_with_this_replica_count
FROM
  range_replication rr
JOIN
  crdb_internal.kv_store_status s
ON
  rr.store_id = s.store_id
GROUP BY
  s.node_id, replica_count
ORDER BY
  ranges_on_node_with_this_replica_count DESC;

On dogfood, this gives me the following info:

  node_id | replica_count | ranges_on_node_with_this_replica_count
----------+---------------+-----------------------------------------
        1 |             5 |                                    302
        2 |             5 |                                    302
        3 |             5 |                                    302
        4 |             5 |                                    302
        5 |             5 |                                    302
(5 rows)

[bnaecker]

So I've been reading a bunch about the zone configuration stuff during multirack research. I think we'll probably avoid changing that from the default, which is explicitly discouraged by their documentation here. Not that you're doing that here, just that the data in it might be less valuable initially.

It does seem really useful to go after the punchlist on the list of metrics to track in self-hosted deployments. It includes things like under-replicated ranges, for example, which shows us when a chunk of a table (key range) is not fully-replicated (e.g., only 4 / 5 replicas are online).

[smklein]

It does seem really useful to go after the punchlist on the list of metrics to track in self-hosted deployments. It includes things like under-replicated ranges, for example, which shows us when a chunk of a table (key range) is not fully-replicated (e.g., only 4 / 5 replicas are online).

Do these stats exist on version 22.1 of CRDB? I'm only seeing them in more recent versions

[davepacheco]

The crdb_internal system catalog seems to be the interface for accessing a lot of this info.

If I remember right, for the testing we did in the past, we used a built-in Prometheus endpoint for this.

I think it's probably important that we leave it to CockroachDB to determine something like "under-replicated ranges" rather than trying to work it out ourselves from the data it provides. I suspect we can probably work it out too but there's risk that we've missed something in the (complex) accounting and the impact of getting it wrong is high.

[bnaecker]

Do these stats exist on version 22.1 of CRDB?

Oh, very good point. I didn't notice that. Looks like this list is available in our version. Here is the Prometheus guide for that version too.

[smklein]

Mkay, the cockroachDB http API is hard-coded to 127.0.0.1:8080 within smf/cockroachdb/method_script.sh, so we can query it from the cockroach admin server (which makes sense as a place to parse / return this info).

Within a Cockroach zone on dogfood, I can use:

curl http://127.0.0.1:8080/_status/vars

To query a ton of info - for example:

# HELP ranges_underreplicated Number of ranges with fewer live replicas than the replication target
# TYPE ranges_underreplicated gauge
ranges_underreplicated{store="2"} 0

(When queried on "node 2")

It's unclear to me that we actually want to stand up a Prometheus server for the sake of parsing this data and receiving alerts - the format is pretty straightforward, so it may be more useful for us to manually query this prometheus-formatted data, and hand it off to our own metrics collection systems.

[davepacheco]

Agreed. This format is well-established and I'm pretty sure Rust parsers exist. The inventory collector could just make this request, parse it, and store the stuff Reconfigurator cares about.

We may also want an Oximeter producer that's backed by this but agreed we don't need a Prometheus server on the scene here.

[smklein]

This is mostly a note for myself next week: This issue should strongly consider our operations experience with #8239. That issue may be a CRDB bug, but it contains a lot of information about "how to observe and track range replication, especially across node movement".

Of note:

• Tracking node status (membership, decommissioning) was critical
• Identifying under-replicated ranges was critical - associating them with schema-layer tables was also nice to understand impact
• Pulling the logs was helpful for debugging. Some info, e.g., "node 5 kicked out node 1 because it thought a range was over-replicated", was critical info, but I'm not sure this will be exposed through the prometheus API. (If so: great. If not, we may want to consider the developer flow for "what to do when a range is underreplicated and cannot repair. Service bundles should include the appropriate cockroachdb logs).

[smklein]

Initial version will be targeting:

liveness.livenodes = 5 + ranges.underreplicated = 0 as a criteria for "is it safe to bounce a node"