Skip to content

feat: add logic to skip empty array columns when no columns or --skip-add-source-columns#327

Open
LuigiCerone wants to merge 1 commit intoz3z1ma:mainfrom
LuigiCerone:fix/empty-columns
Open

feat: add logic to skip empty array columns when no columns or --skip-add-source-columns#327
LuigiCerone wants to merge 1 commit intoz3z1ma:mainfrom
LuigiCerone:fix/empty-columns

Conversation

@LuigiCerone
Copy link
Contributor

Problem

_sync_doc_section in sync_operations.py always wrote the columns key unconditionally:

doc_section["columns"] = incoming_columns

When a node has no columns (e.g. a source with --skip-add-source-columns enabled), incoming_columns is [] and the result is columns: [] being written to every source table osmosis touches — even though the user explicitly opted out of column management.

The two behaviors were fully decoupled: --skip-add-source-columns prevented osmosis from querying the database for columns, but the empty list was still written unconditionally by _sync_doc_section regardless.


Fix

Only write the columns key when there are actual columns to document; remove it otherwise:

if incoming_columns:
    doc_section["columns"] = incoming_columns
else:
    doc_section.pop("columns", None)

This is safe — dbt treats a missing columns key and columns: [] identically.


Tests

Two unit tests added to test_sync_operations.py, using lightweight mocks (no DuckDB fixture needed):

Test Description
test_sync_doc_section_no_columns_key_not_added Verifies columns is never added when the node has no columns
test_sync_doc_section_existing_empty_columns_removed Verifies a pre-existing columns: [] is cleaned up on the next run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant