Skip to content

Commit 1c67397

Browse files
docs: expand Snowflake v4 migration guide with detailed instructions (#69729)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: [email protected] <[email protected]>
1 parent 222dd69 commit 1c67397

File tree

1 file changed

+57
-6
lines changed

1 file changed

+57
-6
lines changed

docs/integrations/destinations/snowflake-migrations.md

Lines changed: 57 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,15 @@ If you _only_ interact with the raw tables, enable the `Disable Final Tables` op
2424

2525
2. Open your **Snowflake** connector.
2626

27-
3. <!-- Devin: fill out the rest of these steps to use Disable Final Tables from Snowflake v3 -->
27+
3. Open the **Advanced** section.
28+
29+
4. Turn on **Disable Final Tables**.
30+
31+
5. Click **Test and save**.
32+
33+
:::note
34+
After upgrading to version 4, this setting appears as **Legacy raw tables** and remains enabled.
35+
:::
2836

2937
#### If you interact with both raw and final tables
3038

@@ -34,25 +42,68 @@ If you interact with both the raw _and_ final tables, this use case is no longer
3442

3543
2. For each Snowflake destination you have, add an identical second Snowflake destination.
3644

37-
3. Ensure each pair of Snowflake connectors have opposite settings for <!-- Devin: fill out the rest of this step to use Disable Final Tables from Snowflake v3 -->
45+
3. Ensure each pair of Snowflake connectors have opposite settings for **Disable Final Tables**. One connector should have this setting turned on, and the other should have it turned off.
3846

39-
4. <!-- Devin: fill out the rest of this process -->
47+
4. Configure distinct default schemas for each destination to avoid table name collisions:
48+
- For the destination that will create final tables, set a distinct **Schema** in the Snowflake destination configuration (for example, `ANALYTICS_V4`). This is where final tables will be written.
49+
- For the raw-only destination (with **Disable Final Tables** turned on), set a distinct **Airbyte Internal Table Dataset Name** under the **Advanced** section (for example, `AIRBYTE_INTERNAL_RAW`). This is where raw tables will be written.
50+
- Example configuration:
51+
- Destination A (final tables): Schema = `ANALYTICS_V4`, Airbyte Internal Table Dataset Name = `AIRBYTE_INTERNAL`
52+
- Destination B (raw-only): Airbyte Internal Table Dataset Name = `AIRBYTE_INTERNAL_RAW`
53+
- Using distinct schemas prevents table name collisions when running both destinations in parallel.
54+
55+
5. Update your connections to point to the appropriate destination:
56+
- Connections that need raw tables only should target the destination with **Disable Final Tables** turned on.
57+
- Connections that need final tables should target the destination with this setting turned off.
58+
59+
6. Run test syncs on both destinations to verify outputs:
60+
- The raw-only destination should write only to the internal schema (default `airbyte_internal`).
61+
- The standard destination should write only final tables to the target schema.
62+
63+
7. After verifying that both destinations work correctly, continue running both connections in parallel going forward.
4064

4165
<MigrationGuide />
4266

4367
### Optional: clean up legacy raw tables
4468

4569
The version 4.0 connector doesn't automatically remove tables created by earlier versions. After upgrading to version 4 and verifying your data, you can optionally remove the old raw tables.
4670

47-
For most users, You can find the raw tables <!-- Devin: where? -->. The names match the pattern <!-- Devin: what are the tables and schemas called? -->
71+
For most users, you can find the raw tables in the schema configured as **Airbyte Internal Table Dataset Name** (which defaults to `airbyte_internal`). If you customized this setting, look in that schema instead.
72+
73+
The table names match these patterns depending on which version created them:
74+
75+
- **Version 2/3 (Typing and Deduping)**: `raw_{namespace}__{stream}` (for example, `airbyte_internal.raw_public__users`)
76+
- **Version 4 (Legacy raw tables mode)**: `{namespace}_raw__stream_{stream}` (for example, `airbyte_internal.public_raw__stream__users`)
77+
78+
Note: The number of underscores between `raw` and `stream` may vary depending on the longest underscore sequence in your namespace and stream names.
4879

4980
:::note
50-
Version 4 of the Snowfalke destination uses the `airbyte_internal` database for temporary scratch space (for example, streams running in dedup mode, truncate refreshes, and overwrite syncs). Dropping the entire `airbyte_internal database` can interrupt active syncs and cause data loss. Only drop the specific raw tables you no longer need.
81+
Version 4 of the Snowflake destination uses the `airbyte_internal` schema for temporary scratch space (for example, streams running in dedup mode, truncate refreshes, and overwrite syncs). Dropping the entire `airbyte_internal` schema can interrupt active syncs and cause data loss. Only drop the specific raw tables you no longer need.
5182
:::
5283

5384
To remove the old raw tables:
5485

55-
<!-- Devin: SQL query to delete a raw table in Snowflake -->
86+
1. **Pause or allow active syncs to complete** before dropping any tables to avoid interrupting data transfers.
87+
88+
2. **List candidate raw tables** to identify which tables to remove:
89+
90+
```sql
91+
-- For Version 2/3 raw tables:
92+
SHOW TABLES IN SCHEMA <DATABASE>.<INTERNAL_SCHEMA> LIKE 'RAW\_%';
93+
94+
-- For Version 4 legacy raw tables:
95+
SHOW TABLES IN SCHEMA <DATABASE>.<INTERNAL_SCHEMA> LIKE '%_RAW__STREAM_%';
96+
```
97+
98+
Replace `<DATABASE>` with your Snowflake database name and `<INTERNAL_SCHEMA>` with your internal schema name (default `airbyte_internal`).
99+
100+
3. **Drop specific raw tables** you no longer need:
101+
102+
```sql
103+
DROP TABLE IF EXISTS <DATABASE>.<INTERNAL_SCHEMA>.<TABLE_NAME>;
104+
```
105+
106+
Replace `<TABLE_NAME>` with the specific table name you want to remove. Use fully qualified names (database.schema.table) to avoid ambiguity.
56107

57108
## Upgrading to 3.0.0
58109

0 commit comments

Comments
 (0)