clickhouse: prevent replicated tables from starting in read-only mode. #9183
+5
−1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
On start, ClickHouse compares the local state of each distributed table to its distributed state. If it finds a discrepancy, it starts the table in read-only mode. When this happens, oximeter can't write new records to the relevant table(s). In the past, we've worked around this by manually instructing ClickHouse using the `force_restore_data` sentinel file, but this requires manual detection and intervention each time a table starts up in read-only mode. This patch sets the `replicated_max_ratio_of_wrong_parts` flag to 1.0 so that ClickHouse always accepts local state, and never starts tables in read-only mode. As described in ClickHouse/ClickHouse#66527, this appears to be a bug, or at least an ergonomic flaw, in ClickHouse. One replica of a table can routinely fall behind the others, e.g. due to restart or network partition, and shouldn't require manual intervention to start back up. Part of #8595.
jmcarp
force-pushed
the
jmcarp/clickhouse-readonly-workaround
branch
from
d84e49b to
11b96f2
Compare
bnaecker
reviewed
Oct 9, 2025
| <!-- Disable sparse column serialization, which we expect to not need --> | ||
| <ratio_of_defaults_for_sparse_serialization>1.0</ratio_of_defaults_for_sparse_serialization> | ||
| <!-- Prevent ClickHouse from setting distributed tables to read-only. --> |
There was a problem hiding this comment.
Doc nit, this is setting the local replica to read-only. I.e., it avoids updating the local state with the shared state in the Raft cluster.
bnaecker
approved these changes
Oct 9, 2025
There was a problem hiding this comment.
Looks good. As far as testing, I think we should figure out how to check if this is being used on the Dogfood rack. That is, can we see in the logs that it would have set the table to read-only mode, but is not because of this setting?
Dogfood is the only place we run the replicated cluster.
Just a nit here, it's so that ClickHouse always accepts the shared state, not the local state. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
On start, ClickHouse compares the local state of each distributed table to its distributed state. If it finds a discrepancy, it starts the table in read-only mode. When this happens, oximeter can't write new records to the relevant table(s). In the past, we've worked around this by manually instructing ClickHouse using the
force_restore_datasentinel file, but this requires manual detection and intervention each time a table starts up in read-only mode. This patch sets thereplicated_max_ratio_of_wrong_partsflag to 1.0 so that ClickHouse always accepts local state, and never starts tables in read-only mode.As described in ClickHouse/ClickHouse#66527, this appears to be a bug, or at least an ergonomic flaw, in ClickHouse. One replica of a table can routinely fall behind the others, e.g. due to restart or network partition, and shouldn't require manual intervention to start back up.
Part of #8595.
Note: I'm not sure now best to test this. It sounds like we have reasonably high confidence that the fix will work, so we could just merge and deploy to dogfood, and revert if necessary. Or is clickhouse cluster running on another rack that we can test?