From 11b96f29e9193cdb51f6edd468d6b7d02f95ae84 Mon Sep 17 00:00:00 2001
From: Josh <jm.carp@gmail.com>
Date: Wed, 8 Oct 2025 15:40:39 -0400
Subject: [PATCH] clickhouse: prevent replicated tables from starting in
 read-only mode.

On start, ClickHouse compares the local state of each distributed table to its
distributed state. If it finds a discrepancy, it starts the table in read-only
mode. When this happens, oximeter can't write new records to the relevant
table(s). In the past, we've worked around this by manually instructing
ClickHouse using the `force_restore_data` sentinel file, but this requires
manual detection and intervention each time a table starts up in read-only
mode. This patch sets the `replicated_max_ratio_of_wrong_parts` flag to 1.0 so
that ClickHouse always accepts local state, and never starts tables in
read-only mode.

As described in https://github.com/ClickHouse/ClickHouse/issues/66527, this
appears to be a bug, or at least an ergonomic flaw, in ClickHouse. One replica
of a table can routinely fall behind the others, e.g. due to restart or network
partition, and shouldn't require manual intervention to start back up.

Part of #8595.
---
 clickhouse-admin/types/src/config.rs | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/clickhouse-admin/types/src/config.rs b/clickhouse-admin/types/src/config.rs
index 742b442bf3e..aa506843631 100644
--- a/clickhouse-admin/types/src/config.rs
+++ b/clickhouse-admin/types/src/config.rs
@@ -201,9 +201,13 @@ impl ReplicaConfig {
         <max_tasks_in_queue>1000</max_tasks_in_queue>
      </distributed_ddl>
 
-     <!-- Disable sparse column serialization, which we expect to not need -->
      <merge_tree>
+        <!-- Disable sparse column serialization, which we expect to not need -->
         <ratio_of_defaults_for_sparse_serialization>1.0</ratio_of_defaults_for_sparse_serialization>
+
+        <!-- Prevent ClickHouse from setting distributed tables to read-only. -->
+        <!-- See https://github.com/oxidecomputer/omicron/issues/8595 for details. -->
+        <replicated_max_ratio_of_wrong_parts>1.0</replicated_max_ratio_of_wrong_parts>
     </merge_tree>
 {macros}
 {remote_servers}