Hybrid table materialization #1296
Conversation
|
Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA. In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR. CLA has not been signed by users: @AkhilGurrapu |
|
Did you want help with this? |
|
considering this is code I wrote I am sure I can help here |
Add Snowflake Hybrid Table MaterializationDescriptionThis PR introduces a new materialization for Snowflake Hybrid Tables, enabling dbt users to leverage the benefits of both regular and dynamic tables in their data models. The implementation provides support for incremental processing, primary keys, secondary indexes, and advanced merge control while maintaining dbt's idiomatic approach to data transformations. Features
Basic Usage
Advanced Configuration OptionsMerge Column Control
Configuration Parameters
When to Use Hybrid Tables✅ Good for:
Implementation DetailsMaterialization Logic
Error Handling
Performance Optimizations
Requirements
Cost ConsiderationsHybrid tables use Snowflake's hybrid engine which has different pricing:
|
|
I made some changes to your code couldn't push it, but figured I would share this as I keep getting emails about this {% materialization hybrid_table, adapter='snowflake' %}
{% set query_tag = set_query_tag() %}
{% set existing_relation = load_cached_relation(this) %}
{% set target_relation = this.incorporate(type='table') %}
{{ run_hooks(pre_hooks) }}
{% set column_definitions = config.get('column_definitions', {}) %}
{% set primary_key = config.get('primary_key', []) %}
{% set indexes = config.get('indexes', []) %}
{% set force_ctas = config.get('force_ctas', false) %}
{% set merge_exclude_columns = config.get('merge_exclude_columns', []) %}
{% set merge_update_columns = config.get('merge_update_columns', []) %}
{# Validation #}
{% if column_definitions | length == 0 %}
{{ exceptions.raise_compiler_error("Hybrid table materialization requires 'column_definitions' in model config") }}
{% endif %}
{% if existing_relation and not force_ctas and primary_key | length == 0 %}
{{ exceptions.raise_compiler_error("Incremental hybrid table updates require 'primary_key' in model config") }}
{% endif %}
{% if existing_relation is none or force_ctas %}
{# Create new hybrid table #}
{% call statement('main') %}
CREATE OR REPLACE HYBRID TABLE {{ target_relation }} (
{% for column, definition in column_definitions.items() %}
{{ column }} {{ definition }}{% if not loop.last %},{% endif %}
{% endfor %}
{% if primary_key %}
, PRIMARY KEY ({{ primary_key | join(', ') }})
{% endif %}
{% for index in indexes %}
, INDEX {{ index.name }} ({{ index.columns | join(', ') }})
{% endfor %}
) AS (
{{ sql }}
)
{% endcall %}
{% else %}
{# Merge into existing hybrid table #}
{% if merge_update_columns | length > 0 %}
{% set update_columns = merge_update_columns %}
{% else %}
{% set update_columns = column_definitions.keys() | reject('in', primary_key) | reject('in', merge_exclude_columns) | list %}
{% endif %}
{% call statement('main') %}
MERGE INTO {{ target_relation }} t
USING ({{ sql }}) s
ON {% for pk in primary_key %}
t.{{ pk }} = s.{{ pk }}{% if not loop.last %} AND {% endif %}
{% endfor %}
{% if update_columns | length > 0 %}
WHEN MATCHED THEN
UPDATE SET
{% for column in update_columns %}
t.{{ column }} = s.{{ column }}{% if not loop.last %},{% endif %}
{% endfor %}
{% endif %}
WHEN NOT MATCHED THEN
INSERT ({{ column_definitions.keys() | join(', ') }})
VALUES ({{ column_definitions.keys() | map('prefix', 's.') | join(', ') }})
{% endcall %}
{% endif %}
{{ run_hooks(post_hooks) }}
{% do unset_query_tag(query_tag) %}
{% set grant_config = config.get('grants') %}
{% if grant_config %}
{% do apply_grants(target_relation, grant_config, should_revoke=should_revoke(existing_relation, full_refresh_mode=force_ctas)) %}
{% endif %}
{% do persist_docs(target_relation, model) %}
{{ return({'relations': [target_relation]}) }}
{% endmaterialization %} |
|
Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA. In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR. CLA has not been signed by users: @AkhilGurrapu |
|
@AkhilGurrapu thanks for the proposal! You'll need to sign the CLA and migrate this PR to dbt-adapters as we have moved dbt-snowflake development there Beyond that some high level feedback:
|
4 participants
Add Snowflake Hybrid Table Materialization
Description
This PR introduces a new materialization for Snowflake Hybrid Tables, enabling dbt users to leverage the benefits of both regular and dynamic tables in their data models. The implementation provides support for incremental processing, primary keys, and secondary indexes while maintaining dbt's idiomatic approach to data transformations.
Features
Example Usage
{{config( materialized='hybrid_table', column_definitions={ 'customer_id': 'VARCHAR NOT NULL', 'order_id': 'VARCHAR NOT NULL', 'amount': 'NUMBER', 'created_at': 'TIMESTAMP' }, primary_key=['customer_id', 'order_id'], indexes=[ {'name': 'idx_customer', 'columns': ['customer_id']}, {'name': 'idx_created', 'columns': ['created_at']} ] )}} SELECT customer_id, order_id, amount, created_at FROM source_tableImplementation Details
Requirements
Testing
Documentation
Breaking Changes
None. This is an additive feature that doesn't affect existing materializations.
Checklist