Add redshift support

[CoderCookE]

Add Amazon Redshift Support

Summary

This PR adds full Amazon Redshift support to automate-dv, enabling users to build Data Vault 2.0 warehouses on Redshift using all table types (hubs, links, satellites, PITs, bridges, etc.).

Motivation

Amazon Redshift is a widely-used cloud data warehouse platform, and many organizations need to implement Data Vault patterns on Redshift. This implementation provides native Redshift support with optimizations for Redshift's SQL dialect and performance characteristics.

Implementation Details

Supported Features

✅ All 10 Data Vault Table Types:

• Hubs (hub.sql)
• Links (link.sql)
• Satellites (sat.sql)
• Effectivity Satellites (eff_sat.sql)
• Multi-Active Satellites (ma_sat.sql)
• Point-in-Time Tables (pit.sql)
• Bridge Tables (bridge.sql)
• Extended Tracking Satellites (xts.sql)
• Non-Historized Links (nh_link.sql)
• Reference Tables (ref_table.sql)

✅ Hash Algorithms:

• MD5: Full support via native MD5() function → VARCHAR(32)
• SHA1: Full support via native SHA1() function → VARCHAR(40)
• SHA256: Full support via native SHA2(string, 256) function → VARCHAR(64)

Key Design Decisions

1. VARCHAR Hex Strings for Hash Storage

Decision: Store hashes as VARCHAR hex strings instead of binary types.

Rationale:

• Simplifies the PIT (Point-in-Time) macro significantly - no need for PostgreSQL-specific ENCODE/DECODE operations
• Makes hashes human-readable for debugging
• Follows proven pattern from BigQuery and Databricks implementations
• Storage overhead is negligible for most use cases

Implementation:

• MD5: VARCHAR(32) - 32 hex characters
• SHA1: VARCHAR(40) - 40 hex characters
• SHA256: VARCHAR(64) - 64 hex characters

2. QUALIFY Clause for Performance

Decision: Require Redshift version with QUALIFY support (July 2023+).

Rationale:

• QUALIFY provides significant performance improvements over subquery patterns
• Most production Redshift clusters are kept current for security and performance
• Cleaner, more maintainable SQL

Example:

SELECT columns
FROM table
QUALIFY ROW_NUMBER() OVER(PARTITION BY pk ORDER BY ldts) = 1

3. Inheritance Strategy

Approach: Mix of custom implementations and inheritance from existing adapters:

• Custom: hub.sql (QUALIFY pattern), pit.sql (VARCHAR hex simplification)
• PostgreSQL-based: link.sql, sat.sql (compatible window functions)
• Default: ref_table.sql, xts.sql, eff_sat.sql, ma_sat.sql, nh_link.sql, bridge.sql

Rationale: Maximize code reuse while optimizing for Redshift-specific features.

Files Changed

Supporting Macros (7 files modified)

• macros/internal/metadata_processing/get_escape_characters.sql - Double quotes like PostgreSQL
• macros/internal/metadata_processing/concat_ws.sql - Native CONCAT_WS support
• macros/supporting/data_types/type_binary.sql - VARCHAR(32/64) for hashes
• macros/supporting/data_types/type_string.sql - VARCHAR type
• macros/supporting/data_types/type_timestamp.sql - TIMESTAMP (no timezone)
• macros/supporting/hash_components/select_hash_alg.sql - MD5, SHA256, SHA1 handling
• macros/supporting/casting/cast_binary.sql - Dynamic VARCHAR casting

Table Macros (10 files created)

• macros/tables/redshift/*.sql - All 10 Data Vault table types

Documentation (1 file created)

• macros/tables/redshift/README.md - Comprehensive implementation guide

Total: 17 files changed, 560 insertions(+)

Configuration

Users can configure their hash algorithm in dbt_project.yml:

vars:
  # Use MD5 (default)
  hash: 'md5'

  # Or use SHA256 for stronger hashing
  hash: 'sha'

Requirements

• Minimum Redshift Version: July 2023 or later (QUALIFY support required)
• dbt Version: >=1.0.0, <3.0.0
• dbt-redshift adapter: Latest version recommended

Usage Example

-- models/raw_vault/hubs/hub_customer.sql
{{- config(
    materialized='incremental',
    schema='raw_vault'
) -}}

{%- set src_pk = 'CUSTOMER_PK' -%}
{%- set src_nk = 'CUSTOMER_ID' -%}
{%- set src_ldts = 'LOAD_DATETIME' -%}
{%- set src_source = 'RECORD_SOURCE' -%}

{{ automate_dv.hub(src_pk=src_pk,
                   src_nk=src_nk,
                   src_ldts=src_ldts,
                   src_source=src_source,
                   source_model='stg_customer') }}

Migration Notes

For users migrating from PostgreSQL to Redshift:

1. Hash Storage: Redshift uses VARCHAR hex strings vs PostgreSQL's BYTEA binary. MD5 hash values remain compatible when compared as strings.

2. Data Type Mapping:

   • PostgreSQL BYTEA → Redshift VARCHAR(32) or VARCHAR(64)
   • PostgreSQL TIMESTAMP → Redshift TIMESTAMP
   • PostgreSQL VARCHAR → Redshift VARCHAR

3. Performance: QUALIFY clause provides better performance than PostgreSQL's DISTINCT ON for many queries.

Documentation Updates Needed

This PR includes in-code documentation (macros/tables/redshift/README.md). External documentation updates needed:

• Update platform support matrix at https://automate-dv.readthedocs.io/en/latest/platform_support/
• Add Redshift configuration guide to official docs
• Add Redshift to platform comparison tables
• Update installation instructions with Redshift example

References

• AWS Redshift SQL Reference: https://docs.aws.amazon.com/redshift/latest/dg/cm_chap_SQLCommandRef.html
• QUALIFY Clause (July 2023): https://docs.aws.amazon.com/redshift/latest/dg/r_QUALIFY_clause.html
• SHA2 Function: https://docs.aws.amazon.com/redshift/latest/dg/SHA2.html
• MD5 Function: https://docs.aws.amazon.com/redshift/latest/dg/r_MD5.html

---

Note: This PR is ready for review and testing. I'm happy to make adjustments based on feedback and add any additional test coverage or documentation as needed.