Skip to content

Require DuckDB DDL on every SQL profiler step - #2597

Open
dgomez04 wants to merge 4 commits into
mainfrom
feat/2460-required-sql-ddl-source
Open

Require DuckDB DDL on every SQL profiler step#2597
dgomez04 wants to merge 4 commits into
mainfrom
feat/2460-required-sql-ddl-source

Conversation

@dgomez04

Copy link
Copy Markdown
Contributor

Summary

  • Make ddl_source required on every type: sql profiler step so DuckDB schemas are always explicit (no inferred CTAS path).
  • Fold Oracle/MSSQL/Teradata paired ddl+sql steps into a single SQL step; add DuckDB DDL for Snowflake, Redshift, and legacy Synapse.
  • SQL overwrite now always recreates the typed table from DDL before insert; empty results still land an empty typed table.

Resolves #2460

Test plan

  • pytest tests/unit/assessment/ -q (213 passed)
  • Smoke one formerly paired profiler (Oracle or MSSQL) same-day overwrite
  • Smoke one formerly inferred profiler (Snowflake or Redshift) same-day overwrite with wide numeric values
  • Confirm source_ddl Redshift steps still run unchanged
  • Confirm Python profilers (Synapse/BigQuery) unaffected

Fold paired ddl+sql extracts into a single sql step with ddl_source so schemas are always explicit and overwrite recreates typed tables before insert.
@dgomez04
dgomez04 requested a review from a team as a code owner July 28, 2026 17:02
@codecov

codecov Bot commented Jul 28, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.00000% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.54%. Comparing base (cd24783) to head (db4b75a).

Files with missing lines Patch % Lines
...databricks/labs/lakebridge/assessments/pipeline.py 85.29% 3 Missing and 2 partials ⚠️
...databricks/labs/lakebridge/assessments/profiler.py 83.33% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2597      +/-   ##
==========================================
+ Coverage   71.45%   71.54%   +0.08%     
==========================================
  Files         108      108              
  Lines        9796     9802       +6     
  Branches     1087     1089       +2     
==========================================
+ Hits         7000     7013      +13     
+ Misses       2572     2565       -7     
  Partials      224      224              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions

github-actions Bot commented Jul 28, 2026

Copy link
Copy Markdown

✅ 181/181 passed, 9 flaky, 2 skipped, 1h10m38s total

Flaky tests:

  • 🤪 test_installs_and_runs_local_bladebridge (12.277s)
  • 🤪 test_installs_and_runs_pypi_bladebridge (23.199s)
  • 🤪 test_recon_for_report_type_is_data (50.055s)
  • 🤪 test_transpiles_informatica_to_sparksql (23.442s)
  • 🤪 test_transpiles_informatica_to_sparksql_non_interactive[False] (4.061s)
  • 🤪 test_transpile_teradata_sql_non_interactive[True] (23.945s)
  • 🤪 test_transpile_teradata_sql_non_interactive[False] (5.84s)
  • 🤪 test_transpiles_informatica_to_sparksql_non_interactive[True] (22.181s)
  • 🤪 test_transpile_teradata_sql (27.048s)

Running from acceptance #5214

@m-abulazm m-abulazm added the feat/profiler Issues related to profilers label Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feat/profiler Issues related to profilers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add optional DDL to SQL pipeline step

2 participants