You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/integrations/destinations/snowflake-migrations.md
+57-6Lines changed: 57 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -24,7 +24,15 @@ If you _only_ interact with the raw tables, enable the `Disable Final Tables` op
24
24
25
25
2. Open your **Snowflake** connector.
26
26
27
-
3.<!-- Devin: fill out the rest of these steps to use Disable Final Tables from Snowflake v3 -->
27
+
3. Open the **Advanced** section.
28
+
29
+
4. Turn on **Disable Final Tables**.
30
+
31
+
5. Click **Test and save**.
32
+
33
+
:::note
34
+
After upgrading to version 4, this setting appears as **Legacy raw tables** and remains enabled.
35
+
:::
28
36
29
37
#### If you interact with both raw and final tables
30
38
@@ -34,25 +42,68 @@ If you interact with both the raw _and_ final tables, this use case is no longer
34
42
35
43
2. For each Snowflake destination you have, add an identical second Snowflake destination.
36
44
37
-
3. Ensure each pair of Snowflake connectors have opposite settings for <!-- Devin: fill out the rest of this step to use Disable Final Tables from Snowflake v3 -->
45
+
3. Ensure each pair of Snowflake connectors have opposite settings for **Disable Final Tables**. One connector should have this setting turned on, and the other should have it turned off.
38
46
39
-
4.<!-- Devin: fill out the rest of this process -->
47
+
4. Configure distinct default schemas for each destination to avoid table name collisions:
48
+
- For the destination that will create final tables, set a distinct **Schema** in the Snowflake destination configuration (for example, `ANALYTICS_V4`). This is where final tables will be written.
49
+
- For the raw-only destination (with **Disable Final Tables** turned on), set a distinct **Airbyte Internal Table Dataset Name** under the **Advanced** section (for example, `AIRBYTE_INTERNAL_RAW`). This is where raw tables will be written.
50
+
- Example configuration:
51
+
- Destination A (final tables): Schema = `ANALYTICS_V4`, Airbyte Internal Table Dataset Name = `AIRBYTE_INTERNAL`
52
+
- Destination B (raw-only): Airbyte Internal Table Dataset Name = `AIRBYTE_INTERNAL_RAW`
53
+
- Using distinct schemas prevents table name collisions when running both destinations in parallel.
54
+
55
+
5. Update your connections to point to the appropriate destination:
56
+
- Connections that need raw tables only should target the destination with **Disable Final Tables** turned on.
57
+
- Connections that need final tables should target the destination with this setting turned off.
58
+
59
+
6. Run test syncs on both destinations to verify outputs:
60
+
- The raw-only destination should write only to the internal schema (default `airbyte_internal`).
61
+
- The standard destination should write only final tables to the target schema.
62
+
63
+
7. After verifying that both destinations work correctly, continue running both connections in parallel going forward.
40
64
41
65
<MigrationGuide />
42
66
43
67
### Optional: clean up legacy raw tables
44
68
45
69
The version 4.0 connector doesn't automatically remove tables created by earlier versions. After upgrading to version 4 and verifying your data, you can optionally remove the old raw tables.
46
70
47
-
For most users, You can find the raw tables <!-- Devin: where? -->. The names match the pattern <!-- Devin: what are the tables and schemas called? -->
71
+
For most users, you can find the raw tables in the schema configured as **Airbyte Internal Table Dataset Name** (which defaults to `airbyte_internal`). If you customized this setting, look in that schema instead.
72
+
73
+
The table names match these patterns depending on which version created them:
74
+
75
+
-**Version 2/3 (Typing and Deduping)**: `raw_{namespace}__{stream}` (for example, `airbyte_internal.raw_public__users`)
76
+
-**Version 4 (Legacy raw tables mode)**: `{namespace}_raw__stream_{stream}` (for example, `airbyte_internal.public_raw__stream__users`)
77
+
78
+
Note: The number of underscores between `raw` and `stream` may vary depending on the longest underscore sequence in your namespace and stream names.
48
79
49
80
:::note
50
-
Version 4 of the Snowfalke destination uses the `airbyte_internal`database for temporary scratch space (for example, streams running in dedup mode, truncate refreshes, and overwrite syncs). Dropping the entire `airbyte_internal database` can interrupt active syncs and cause data loss. Only drop the specific raw tables you no longer need.
81
+
Version 4 of the Snowflake destination uses the `airbyte_internal`schema for temporary scratch space (for example, streams running in dedup mode, truncate refreshes, and overwrite syncs). Dropping the entire `airbyte_internal` schema can interrupt active syncs and cause data loss. Only drop the specific raw tables you no longer need.
51
82
:::
52
83
53
84
To remove the old raw tables:
54
85
55
-
<!-- Devin: SQL query to delete a raw table in Snowflake -->
86
+
1.**Pause or allow active syncs to complete** before dropping any tables to avoid interrupting data transfers.
87
+
88
+
2.**List candidate raw tables** to identify which tables to remove:
89
+
90
+
```sql
91
+
-- For Version 2/3 raw tables:
92
+
SHOW TABLES IN SCHEMA <DATABASE>.<INTERNAL_SCHEMA>LIKE'RAW\_%';
93
+
94
+
-- For Version 4 legacy raw tables:
95
+
SHOW TABLES IN SCHEMA <DATABASE>.<INTERNAL_SCHEMA>LIKE'%_RAW__STREAM_%';
96
+
```
97
+
98
+
Replace `<DATABASE>` with your Snowflake database name and `<INTERNAL_SCHEMA>` with your internal schema name (default `airbyte_internal`).
99
+
100
+
3.**Drop specific raw tables** you no longer need:
101
+
102
+
```sql
103
+
DROPTABLE IF EXISTS <DATABASE>.<INTERNAL_SCHEMA>.<TABLE_NAME>;
104
+
```
105
+
106
+
Replace `<TABLE_NAME>` with the specific table name you want to remove. Use fully qualified names (database.schema.table) to avoid ambiguity.
0 commit comments