-
Notifications
You must be signed in to change notification settings - Fork 4.9k
Docs snowflake 4 detailed migration guide #69728
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from 2 commits
d83fda1
222dd69
1c67397
96825e5
9607f34
6cf894e
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,16 +1,58 @@ | ||
| import MigrationGuide from '@site/static/_migration_guides_upgrade_guide.md'; | ||
|
|
||
| # Snowflake Migration Guide | ||
|
|
||
| ## Upgrading to 4.0.0 | ||
|
|
||
| This version upgrades Destination Snowflake to the [Direct-Load](/platform/using-airbyte/core-concepts/direct-load-tables) paradigm, which improves performance and reduces warehouse spend. If you have unusual requirements around record visibility or schema evolution, read that document for more information about how direct-load differs from Typing and Deduping. | ||
| This version upgrades the Snowflake destination from using [typing and deduping](/platform/using-airbyte/core-concepts/typing-deduping) to [direct loading](/platform/using-airbyte/core-concepts/direct-load-tables). This upgrade improves performance and reduces warehouse spend. If you have unusual requirements around record visibility or schema evolution, read the documentation for those methodologies for more information about how direct loading differs from typing and deduping. | ||
ian-at-airbyte marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| This version also adds an option to enable CDC deletions as soft deletes. | ||
|
|
||
| ### Decide how to handle raw tables | ||
|
|
||
| The exact steps to begin the migration depend on whether you interact with Airbyte's raw tables in Snowflake. | ||
|
|
||
| #### If you don't interact with raw tables | ||
|
|
||
| If you don't interact with the raw tables, you can safely upgrade. There's no breaking change for you. | ||
|
|
||
| #### If you only interact with raw tables | ||
|
|
||
| If you _only_ interact with the raw tables, enable the `Disable Final Tables` option before upgrading. This enables the `Legacy raw tables` option after upgrading. | ||
|
|
||
| 1. In the navigation bar, click **Destinations**. | ||
|
|
||
| 2. Open your **Snowflake** connector. | ||
|
|
||
| 3. <!-- Devin: fill out the rest of these steps to use Disable Final Tables from Snowflake v3 --> | ||
|
|
||
| #### If you interact with both raw and final tables | ||
|
|
||
| If you interact with both the raw _and_ final tables, this use case is no longer supported. Instead, create two connectors. One with `Disable Final Tables` turned on, and one with it turned off. Starting now, you must run the two connections in parallel. | ||
|
|
||
| 1. In the navigation bar, click **Destinations**. | ||
|
|
||
| 2. For each Snowflake destination you have, add an identical second Snowflake destination. | ||
|
|
||
| 3. Ensure each pair of Snowflake connectors have opposite settings for <!-- Devin: fill out the rest of this step to use Disable Final Tables from Snowflake v3 --> | ||
|
|
||
| 4. <!-- Devin: fill out the rest of this process --> | ||
|
|
||
| <MigrationGuide /> | ||
|
|
||
| ### Optional: clean up legacy raw tables | ||
|
|
||
| The version 4.0 connector doesn't automatically remove tables created by earlier versions. After upgrading to version 4 and verifying your data, you can optionally remove the old raw tables. | ||
|
|
||
| This version also adds an option to enable CDC deletions as soft-deletes. | ||
| For most users, You can find the raw tables <!-- Devin: where? -->. The names match the pattern <!-- Devin: what are the tables and schemas called? --> | ||
|
|
||
| If you do not interact with the raw tables, you can safely upgrade. There is no breakage for this usecase. | ||
| :::note | ||
| Version 4 of the Snowfalke destination uses the `airbyte_internal` database for temporary scratch space (for example, streams running in dedup mode, truncate refreshes, and overwrite syncs). Dropping the entire `airbyte_internal database` can interrupt active syncs and cause data loss. Only drop the specific raw tables you no longer need. | ||
|
||
| ::: | ||
|
|
||
| If you _only_ interact with the raw tables, make sure that you have the `Disable Final Tables` option enabled before upgrading. This will automatically enable the `Legacy raw tables` option after upgrading. | ||
| To remove the old raw tables: | ||
|
|
||
| If you interact with both the raw _and_ final tables, this usecase will no longer be directly supported. You must create two connectors (one with `Disable Final Tables` enabled, and one with it disabled) and run two connections in parallel. | ||
| <!-- Devin: SQL query to delete a raw table in Snowflake --> | ||
|
|
||
| ## Upgrading to 3.0.0 | ||
|
|
||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.