Destination Clickhouse - Add Typing and Deduping

[jbfbell]

We removed (PR )"normalization" from Destination Clickhouse. Normalization was a process which would extract data from the raw json format and cast the data into tables with the appropriate types. For other destinations, we implemented Typing and Deduping to replace normalization and we will bring typing and deduplication to Clickhouse if there is sufficient interest

The purpose of this discussion is to gauge interest in bringing this functionality back to Clickhouse.

Without typing and deduping, you would need to integrate another tool, such as dbt to accomplish the same result. While integrating another tool gives you greater customization over the data format, it comes with additional operational overhead.

Should Airbyte bring Typing and Deduping to Clickhouse?

Update

Please check the new connector Clickhouse and the beta test discussion happening here

[asimumba]

How can I revert to pre 1.0.0 upgrade of the Clickhouse destination?

[kvosha]

Definitely we need normalization for clickhouse! Is there any way to stay on old version of clickhouse connection until normalization will be implemented?

[alberk8]

Yes, bring Typing and Deduping ClickHouse

[Riya-gupta-8]

Yes . You can degrade to v0.2.5 version for clickhouse

[nailele]

airbyte should bring typing and deduping for clickhouse 🥹

[nailele]

[MaxMusing]

Strong +1 on this, we would benefit greatly from typing and deduping on ClickHouse as a destination.

[asimumba]

In support to have this feature implemented.

[machariamuguku]

FYI: The airbyte team published the connector docker images on dockerhub. You can 'revert' to an older connector version (with normalization for example) by uploading it's older image as a custom connector. See clickhouse connector images here cc @asimumba @kvosha

[asimumba]

Hi, I actually resolved this already. Thanks.

[Aryanshaw]

how did u solve this problem can u tell me

[AlexisSerneels]

I've spent hours debugging why the final tables were not created before finding this discussion.
We definitely need this feature ;)

[vietpm]

I can't understand why this feature is removed. Please bring this back in destination v2.

For anyone wants this feature, you can go back to version 0.2.5 of clickhouse destination. Note that you'll be switched back to destination v1.
1, Settings -> Destinations -> Clickhouse. Change version to 0.2.5 and click "Change".
2, Enter your connection, choose Transformation, change to "Normalized tabular data", click "Save".
3, (optional) If you're using Incremental sync mode, "Incremental | Append + Deduped" is available now so you may be want to change.
4, Reset data, resync from scratch.

[shakalandy]

@vietpm does this still work for you?
The "Transformation" tab is not shown anymore, not sure if it's because of a new ClickHouse Version?
Downgrading to ClickHouse Connector 0.2.4 and then back to 0.2.5 does not bring transformation back anymore...

[shakalandy]

ok, Downgraded Airbyte to 0.63.3, works again - with 0.63.9 the "transformation" tab was lost despite the downgrade of the clickhouse Destination.

[Aryanshaw]

Hey @shakalandy does it still work for you?

[shakalandy]

Migrated to MySQL as Destination in this case.

[edwardysun]

+1 need this

[cedstrom]

Holy hell, please bring normalization back +1000.

[luishdosreis]

Pretty please bring normalisation back :) 🍰

[zowen-ch]

We need the normalization for our workflows. Please keep supporting this in Clickhouse!

[georgeemr]

Please bring it back!

[jesperbagge]

A big heartfelt YES! Please bring Typing and Deduping to ClickHouse!

[Lawchak-Fadhil]

Please bring it back, I've spent countless hours trying to figure out what is wrong with my setup! We really need this for ClickHouse.

[mert-ozturk-lgs]

Yes, please integrate Typing and Deduping to Clickhouse. It is essential. Without the functionality, Airbyte becomes less attractive.

[Aryanshaw]

checkout my article

[mert-ozturk-lgs]

Yes, I saw your article, it's good! But still it would be better to avoid the additional effort in future.

[blarghmatey]

Is there any information available about whether and when the Clickhouse destination will be updated to support the typing and deduping functionality? I think this is a rather critical feature across Airbyte and should be the default behavior for all supported destinations.

[philippevezina]

I just saw that it's on their roadmap for Feb. - May 2025 so I am guessing it's reasonable to exepect more information around that time.

[zackrspv]

any other good alternatives until this is actually working again -.-

[Aryanshaw]

I made a solution on how to transform the raw airbyte file using DBT and it is 100% dynamic just provide the db name and run the script and everything will be done

ELT pipeline between source to airbyte to clickhouse
https://medium.com/@aryanrot234/airbyte-raw-data-transformation-using-dbt-dynamically-221de752733b

[geekbleek]

Any update here from the Airbyte team? Great that there's lots of other scripted solutions, but if we're setting up dbt etc and running them in schedulers like Airflow, might as well setup us dlt etc, and no longer use Airbyte if Clickhouse is a destination.

[jesperbagge]

That’s exactly what I did, in the end. Running dlt and dbt on Airflow gave us less tools to work with and roughly a 500% increase in performance due to how fast dlt runs with pyarrow as backend - and in parallel! - compared to old Java-connectors that does one table at a time.

[Segfaultd]

Any chance this get attention? No way for us to keep using Airbyte if its not handled.

[alexworkingcats]

Yes please this would be great !

[john-griffin]

Please add support for ClickHouse.

[Aryanshaw]

Hey guys, did you find any other alternatives for airbyte? (free open source)

[Segfaultd]

Not using AirByte at all. They don't seem to care about something broken for months that was perfectly working before.
Made the move to Meltano self-deployed.

[jesperbagge]

We migrated to dlt. Orders of magnitude faster and can move multiple tables in parallel.

[john-griffin]

Also moving to Meltano

[Aryanshaw]

tell me is meltano faster? and does it provide rest API access like airbyte? or is it just cli based?

[Segfaultd]

https://docs.meltano.com/

[yalattas]

+1

[mikich-trintech]

+1

I'm surprised and disappointed to see Airbyte abandoning its efforts in supporting ClickHouse!

ClickHouse engine is gaining significant traction within the community.
Meanwhile, ClickHouse is experiencing rapid growth, with an increasing number of users and contributions.

[ian-at-airbyte]

Hi, everyone! We're working on ClickHouse this cycle. This is one of our most upvoted requests ever and we definitely haven't forgotten about it! I won't speculate on a delivery date at this early stage, but I do hope to be able to do said speculation soon.

👨‍💻 👩‍💻 🧑‍💻

[ian-at-airbyte]

It seems like we're currently on track for a delivery window sometime in June or July. I'll try to make that more specific shortly.

[havrilyukv]

It seems like we're currently on track for a delivery window sometime in June or July. I'll try to make that more specific shortly.

thanks for update. really want to stick with your software but it is must have for us.

[ian-at-airbyte]

Thanks for your patience. This one has been a long time coming but it was important to us to get it done.

[lerminou]

First official version available: https://docs.airbyte.com/integrations/destinations/clickhouse-v2
I will go try it, thank you for your work

[havrilyukv]

First official version available: https://docs.airbyte.com/integrations/destinations/clickhouse-v2 I will go try it, thank you for your work

do not see it in my Airbyte Open Source marketplace (((((

[lerminou]

Hi, do you plan to support connecting to a ClickHouse cluster in your roadmap? @ian-at-airbyte
an old issue: #9134

[ian-at-airbyte]

Hi @lerminou. This is not planned as part of the current typing and deduping project, but I've flagged it with the dev team so they're aware.

[marcosmarxm]

Please check the new connector Clickhouse and the beta test discussion happening here