Cleaning up old data from SCD tables #34648
Unanswered
Borks
asked this question in
Connector Questions
Replies: 1 comment
-
I looked into the setting TEMPORAL_HISTORY_RETENTION_IN_DAYS but I understand that's only for logs on the airbyte machine. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Hi.
I'm using airbyte for a Snowflake to postgres migration.
In Incremental | Append + dedupe mode.
Due to how the data is presented to me I have to download a large chunk of duplicated data, and I didn't realize the scd tables are taking up most of my space now (With most of the data having _airbyte_active_row=0). Is Airbyte not supposed to clean those up itself? Can I clean them safely myself?
Because of this each incremental sync is starting to take longer and longer..
Beta Was this translation helpful? Give feedback.
All reactions