Skip to content
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 53 additions & 6 deletions docs/integrations/destinations/snowflake-migrations.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,17 @@ If you _only_ interact with the raw tables, enable the `Disable Final Tables` op

2. Open your **Snowflake** connector.

3. <!-- Devin: fill out the rest of these steps to use Disable Final Tables from Snowflake v3 -->
3. Click **Edit configuration**.

4. Open the **Advanced** section.

5. Turn on **Disable Final Tables**.

6. Click **Save**.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you check these UI steps? They don't appear to be correct to me.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this! I based the UI steps on the v4 spec placement, but I don't have access to verify the exact v3 UI flow. Could you confirm:

  1. Is "Edit configuration" the correct button text, or should it be something else (e.g., "Edit", "Settings")?
  2. Is "Disable Final Tables" located under an "Advanced" section in v3, or is it in a different location?
  3. Are there any other UI differences I should be aware of?

I can update the steps once I know the correct v3 UI flow, or I can make them more location-agnostic if the exact path varies.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. It should be enough to open the Connector. It defaults to the "Settings" tab and you don't need to click "Edit Configuration". I don't even think that label exists.
  2. Unclear, but you should be able to determine this by examining the connector's historical source code for a commit prior to version 4.0.
  3. The "Save" button is always "Test and save" in the UI in Airbyte.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clarification! I've updated the steps to:

  1. Remove "Click Edit configuration" (since it defaults to Settings tab)
  2. Keep "Open the Advanced section" (verified by checking the v3 spec at commit 2ea4544^ - the disable_type_dedupe field had "group": "advanced")
  3. Change "Save" to "Test and save"

The updated steps are now:

  1. In the navigation bar, click Destinations.
  2. Open your Snowflake connector.
  3. Open the Advanced section.
  4. Turn on Disable Final Tables.
  5. Click Test and save.

Pushing this change now.


:::note
After upgrading to version 4, this setting appears as **Legacy raw tables** and should remain enabled.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't use "should". Just say it remains enabled and assume success.

:::

#### If you interact with both raw and final tables

Expand All @@ -34,25 +44,62 @@ If you interact with both the raw _and_ final tables, this use case is no longer

2. For each Snowflake destination you have, add an identical second Snowflake destination.

3. Ensure each pair of Snowflake connectors have opposite settings for <!-- Devin: fill out the rest of this step to use Disable Final Tables from Snowflake v3 -->
3. Ensure each pair of Snowflake connectors have opposite settings for **Disable Final Tables** (in version 3) or **Legacy raw tables** (in version 4). One connector should have this setting enabled, and the other should have it disabled.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] reported by reviewdog 🐶
[Google.WordList] Use 'turn off' or 'off' instead of 'disabled'.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should assume at this point they haven't upgraded and use the V3 instructions only.


4. Configure distinct default schemas for each destination to avoid table name collisions.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you elaborate on this a bit more? I'm not convinced everyone knows how to do this.


5. Update your connections to point to the appropriate destination:
- Connections that need raw tables only should target the destination with **Disable Final Tables** or **Legacy raw tables** enabled.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto about not using V4 instructions before they upgrade

- Connections that need final tables should target the destination with this setting disabled.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] reported by reviewdog 🐶
[Google.WordList] Use 'turn off' or 'off' instead of 'disabled'.


4. <!-- Devin: fill out the rest of this process -->
6. Run test syncs on both destinations to verify outputs:
- The raw-only destination should write only to the internal schema (default `airbyte_internal`).
- The standard destination should write only final tables to the target schema.

7. After verifying that both destinations work correctly, continue running both connections in parallel going forward.

<MigrationGuide />

### Optional: clean up legacy raw tables

The version 4.0 connector doesn't automatically remove tables created by earlier versions. After upgrading to version 4 and verifying your data, you can optionally remove the old raw tables.

For most users, You can find the raw tables <!-- Devin: where? -->. The names match the pattern <!-- Devin: what are the tables and schemas called? -->
For most users, you can find the raw tables in the schema configured as **Airbyte Internal Table Dataset Name** (which defaults to `airbyte_internal`). If you customized this setting, look in that schema instead.

The table names match these patterns depending on which version created them:

- **Version 2/3 (Typing and Deduping)**: `raw_{namespace}__{stream}` (for example, `airbyte_internal.raw_public__users`)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'Deduping'?

- **Version 4 (Legacy raw tables mode)**: `{namespace}_raw__stream_{stream}` (for example, `airbyte_internal.public_raw__stream__users`)

Note: The number of underscores between `raw` and `stream` may vary depending on the longest underscore sequence in your namespace and stream names.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] reported by reviewdog 🐶
[Google.Colons] ': T' should be in lowercase.


:::note
Version 4 of the Snowfalke destination uses the `airbyte_internal` database for temporary scratch space (for example, streams running in dedup mode, truncate refreshes, and overwrite syncs). Dropping the entire `airbyte_internal database` can interrupt active syncs and cause data loss. Only drop the specific raw tables you no longer need.
Version 4 of the Snowflake destination uses the `airbyte_internal` schema for temporary scratch space (for example, streams running in dedup mode, truncate refreshes, and overwrite syncs). Dropping the entire `airbyte_internal` schema can interrupt active syncs and cause data loss. Only drop the specific raw tables you no longer need.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'dedup'?

:::

To remove the old raw tables:

<!-- Devin: SQL query to delete a raw table in Snowflake -->
1. **Pause or allow active syncs to complete** before dropping any tables to avoid interrupting data transfers.

2. **List candidate raw tables** to identify which tables to remove:

```sql
-- For Version 2/3 raw tables:
SHOW TABLES IN SCHEMA <DATABASE>.<INTERNAL_SCHEMA> LIKE 'RAW\_%';

-- For Version 4 legacy raw tables:
SHOW TABLES IN SCHEMA <DATABASE>.<INTERNAL_SCHEMA> LIKE '%_RAW__STREAM_%';
```

Replace `<DATABASE>` with your Snowflake database name and `<INTERNAL_SCHEMA>` with your internal schema name (default `airbyte_internal`).

3. **Drop specific raw tables** you no longer need:

```sql
DROP TABLE IF EXISTS <DATABASE>.<INTERNAL_SCHEMA>.<TABLE_NAME>;
```

Replace `<TABLE_NAME>` with the specific table name you want to remove. Use fully qualified names (database.schema.table) to avoid ambiguity.

## Upgrading to 3.0.0

Expand Down
Loading