feat(health): allow clickhouse check bypass (#7600)

We've had a lot of issues over the years with the health check being an
issue that takes Snuba API pods down in a way that reduces stability
rather than increasing it. Recently, there's an issue where a too-slow
health check response, not being caught by our own timeout handling
behavior, causes API pods to be taken out of rotation even when we have
no evidence that new pods would behave any better.

Rather than scale down when we have connection issues to ClickHouse,
this essentially adds an option to keep accepting requests on the pod
for as long as uwsgi has capacity. This still is some degradation to the
customer (at this time, some of their requests will time out) but
assuming some kind of network or envoy issue is causing bad behavior
that should recover faster than a pod restart. This option
(`health_check_ignore_clickhouse`) is going to be disabled by default,
but I want to try enabling this behavior in `us` for a while and see how
it goes.

Internal doc on this:
https://www.notion.so/sentry/investigation-pod-availability-drop-2ca8b10e4b5d808796c6ecd2e3ad1700

Author: onewland

snuba/utils/health_info.py

@@ -23,7 +23,7 @@
 from snuba.datasets.storage import Storage
 from snuba.datasets.storages.factory import get_all_storage_keys, get_storage
 from snuba.environment import setup_logging
-from snuba.state import get_float_config
+from snuba.state import get_float_config, get_int_config
 from snuba.utils.metrics.wrapper import MetricsWrapper
 
 metrics = MetricsWrapper(environment.metrics, "api")
@@ -94,11 +94,16 @@ def get_health_info(thorough: Union[bool, str]) -> HealthInfo:
         "thorough": str(thorough),
     }
 
-    clickhouse_health = (
-        check_all_tables_present(metric_tags=metric_tags)
-        if thorough
-        else sanity_check_clickhouse_connections()
-    )
+    if get_int_config("health_check_ignore_clickhouse", 0) == 1:
+        clickhouse_health = True
+        metric_tags["skipped_clickhouse_check"] = "true"
+    else:
+        clickhouse_health = (
+            check_all_tables_present(metric_tags=metric_tags)
+            if thorough
+            else sanity_check_clickhouse_connections()
+        )
+
     metric_tags["clickhouse_ok"] = str(clickhouse_health)
 
     body: Mapping[str, Union[str, bool]]