[destination-clickhouse] Clickhouse cluster support

[rsitrifork]

Connector Name

destination-clickhouse

Connector Version

1.0.0

What step the error happened?

Configuring a new connector

Relevant information

Clickhouse is installed on AKS as a cluster with 3 shards and 2 replicas. Airbyte is installed in the same cluster.

The "airbyte_internal" table that is created is lacking the "on cluster default" specifier and as such is only created on the current replica. When connecting to clickhouse using a loadbalancer (in my case a kubernetes service) this means that sometimes the database does not exist.

Relevant log output

No response

Contribute

• Yes, I want to contribute

[jesperbagge]

+1 on this! Connections to ClickHouse client managed clusters are always made to one of the nodes. Basically all SQL statements sent to that node needs to have the “on cluster”-clause to ensure that databases and tables are propagated to all nodes.