Skip to content

Conversation

@andtrott
Copy link

@andtrott andtrott commented Aug 19, 2025

Summary

This PR adds support for unique suffixes on test failure tables to prevent table name conflicts during parallel test execution (e.g., in Airflow DAGs with hourly/daily runs).

Problem

When running dbt tests with store_failures: true in parallel environments:

  • Multiple concurrent test runs overwrite the same failure table
  • Historical failure data gets lost
  • Debugging becomes difficult as failure records are replaced

Solution

This PR works in conjunction with dbt-labs/dbt-core#11939 to implement:

  1. Optional unique suffixes for test failure tables via new configuration:

    • store_failures_unique: Boolean flag to enable suffixes
    • store_failures_suffix: Strategy for suffix generation
  2. Multiple suffix strategies:

    • invocation_id: Uses first 8 chars of invocation ID (default)
    • timestamp: Full timestamp (YYYYMMDD_HHMMSS)
    • date: Date only (YYYYMMDD)
    • hour: Date and hour (YYYYMMDD_HH)
    • Custom string literals or templates
  3. Template support for advanced use cases:

    • Allows Jinja templates in suffix values
    • Enables integration with Airflow variables like data_interval_start

Configuration Example

tests:
  +store_failures: true
  +store_failures_unique: true
  +store_failures_suffix: 'hour'  # or 'invocation_id', 'timestamp', etc.

Key Changes

  • Modified src/dbt/include/global_project/macros/materializations/tests/test.sql
  • Added suffix generation logic after identifier assignment
  • Ensures table names don't exceed platform limits (e.g., 1024 chars for BigQuery)
  • Maintains backward compatibility when store_failures_unique is false

Testing

  • Tested locally with various suffix strategies
  • Verified backward compatibility
  • Confirmed table name uniqueness in parallel execution

Related Work

Breaking Changes

None. The feature is fully backward compatible and opt-in.

Checklist

  • I have read the contributing guide
  • I have added an entry to CHANGELOG.md
  • I have added tests to cover my changes (pending - will add after core PR approval)

Adds configuration options to append unique suffixes to test failure tables,
enabling parallel test execution and preserving historical test data.

New configurations:
- store_failures_unique: Enable unique table names (default: false)
- store_failures_suffix: Suffix strategy (invocation_id, timestamp, date, hour, custom)

This is fully backward compatible as it's opt-in and doesn't affect existing workflows.

Fixes dbt-labs#1276
@andtrott andtrott requested a review from a team as a code owner August 19, 2025 19:00
@cla-bot
Copy link

cla-bot bot commented Aug 19, 2025

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR.

CLA has not been signed by users: @andtrott

@cla-bot cla-bot bot added the cla:yes The PR author has signed the CLA label Aug 19, 2025
- Add unit tests for all suffix strategies (invocation_id, timestamp, date, hour, custom)
- Add tests for backward compatibility (feature disabled by default)
- Add tests for parallel execution scenarios
- Add integration test template for database verification
- Add changelog entry per contributing guidelines

Tests cover:
- Different suffix strategies working correctly
- Backward compatibility maintained
- Parallel runs creating different tables
- Default behavior unchanged when feature not enabled
@andtrott
Copy link
Author

Hi maintainers! 👋

This PR adds optional unique suffixes to test failure table names to enable parallel test execution and preserve historical test data.

The CI requires the ci:approve-public-fork-ci label to run since this is from a public fork. Could someone please add the label to trigger the test suite?

What's included:

  • ✅ Core feature implementation in test materialization
  • ✅ Comprehensive test coverage in test_unique_store_failures.py
  • ✅ Changelog entry per contributing guidelines
  • ✅ Fully backward compatible (opt-in feature)

Key benefits:

  • Solves parallel test execution conflicts (critical for hourly/daily DAGs)
  • Preserves historical test failure data for auditing
  • No breaking changes - feature is disabled by default

Happy to address any feedback or questions. Thanks for reviewing!

@andtrott
Copy link
Author

Closing this PR as the solution has been fully implemented in dbt-core PR dbt-labs/dbt-core#11939.

By modifying the node.alias field in dbt-core's compilation process, the suffix is properly applied before the materialization uses it. This means no changes are needed in dbt-adapters.

The dbt-core solution:

  1. Adds suffix configuration options to TestConfig
  2. Applies the suffix to node.alias during compilation
  3. The modified alias flows through to create_from() which generates the relation_name
  4. Materializations use model['alias'] which now includes the suffix

This approach is cleaner and keeps all the logic in dbt-core where it belongs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla:yes The PR author has signed the CLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant