When does Airbyte create tables in the destination, and how does it decide the schema? #28309
Unanswered
rahilsondhi
asked this question in
Connector Questions
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Airbyte created this table in Snowflake using the wrong schema.
created_at
andupdated_at
should be timestamps, but they're varchar. Also, why did it cluster it by_airbyte_emitted_at
? It should not be manually clustered.When I click on Source > Fields, it shows the correct schema:
So I'm just confused why Airbyte created the table that way.
I tried dropping the table in Snowflake, and I triggered a new sync in Airbyte, hoping that it would re-create the table with the correct schema, but it did not, even though the sync "succeeded."
I wanted to try the
Reset your data
button, but the docs say it will delete all the rows in my destination tables. Is that in my entire Snowflake database, or just the active streams within this connection? Because I only have one stream/table in this connection, thewebhooks
table. It's fine if Airbyte truncates that table, but I'm terrified that it will truncate all tables in the Snowflake database.To summarize, my questions are:
a. If so, when does it do it? Does it verify the table exists before starting a sync?
b. How does it come up with the schema? Why did it pick varchar for timestamp columns?
c. How does it pick the
cluster by
?Reset your data
does? I hope it only truncates the active tables for the current connection, not all tables in the destinationThank you!
Beta Was this translation helpful? Give feedback.
All reactions