Skip to content

[CT-1710] [CT-1703] [Feature] on_schema_change should handle non-top-level schema changes #599

@anaghshineh

Description

@anaghshineh

Is this your first time submitting a feature request?

  • I have read the expectations for open source contributors
  • I have searched the existing issues, and I could not find an existing issue for this feature
  • I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

Currently, on_schema_change tracks schema changes involving top-level columns. This excludes nested changes, like the addition of a column in a STRUCT (BigQuery). It would be great to extend the functionality of on_schema_change to track changes in the schema for top-level columns as well as nested ones, too.

Describe alternatives you've considered

We've had to update schemas manually to accommodate nested changes. Or, we write the data to new tables and then migrate those new tables to a production context.

Who will this benefit?

This type of feature will benefit those with meaty incremental models who do not want to or cannot afford to do a full refresh. This benefits those with models that change somewhat frequently - more holistic, automated handling of schema changes would be fantastic!

Are you interested in contributing this feature?

Sure! If you all see this as a feasible feature extension, I am happy to dig into the code and contribute.

Anything else?

At the very least, I think the documentation around on_schema_change should be updated to make it very clear that it deals with top-level schema changes.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions