feat: add fusion_compat mode for dbt 1.9.6+ YAML output#317
Open
RicardoAGL wants to merge 5 commits intoz3z1ma:mainfrom
Open
feat: add fusion_compat mode for dbt 1.9.6+ YAML output#317RicardoAGL wants to merge 5 commits intoz3z1ma:mainfrom
RicardoAGL wants to merge 5 commits intoz3z1ma:mainfrom
Conversation
Auto-detect dbt version and produce Fusion-compatible YAML when column-level config is supported (dbt >= 1.9.6). When enabled, meta/tags are nested under config blocks instead of top-level. Adds --fusion-compat/--no-fusion-compat CLI flags for override. Also preserves anchors, data_tests, and semantic_models in YAML.
Fix 17 pre-existing test failures: - Pass DbtProjectContext (yaml_context.project) instead of YamlRefactorContext to generator functions that expect the project context directly - Fix mock.patch paths for lazy imports: mock at dbt_core_interface.* instead of dbt_osmosis.core.generators.* for classes imported inside functions - Use tmp_path instead of absolute /custom/staging to avoid read-only filesystem
Detect dbt Fusion manifests (schema version > v12) in the target directory before parsing, so teams running both Fusion and dbt-core get fusion_compat=True automatically even with older dbt-core versions. Detection also checks for dbtf/dbt-fusion binaries on PATH as a fallback when no manifest exists. The explicit --fusion-compat flag still takes priority over all auto-detection.
- Document --fusion-compat/--no-fusion-compat in CLI reference, settings reference, and configuration tutorial - Add disclaimer that dbt-osmosis requires dbt-core and does not run on dbt Fusion directly, with hybrid workflow instructions - Anchor schema version regex to path segment to avoid false matches - Optimize manifest reading to parse only first 4KB instead of full JSON - Downgrade Fusion binary detection log from info to debug - Remove anchors from parser allowed_keys (preserved via writer only) - Simplify single-element tuple filters to != comparisons - Remove unused json import from config.py
The fusion-compat setting is only available via CLI flag or auto-detection. The project vars path is not wired up in YamlRefactorContext, so the example was misleading.
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
fusion_compatmode that produces Fusion-compatible YAML output where column-levelmetaandtagsare nested insideconfig:blocks, as required by dbt >= 1.9.6 and dbt Fusion.fusion_compatis not explicitly set, it auto-detects from: (1) Fusion manifest intarget/(schema version > v12), (2) Fusion binary on PATH, (3) dbt-core version >= 1.9.6--fusion-compat/--no-fusion-compatonyaml refactor,yaml organize, andyaml documentWhat changed
settings.pyfusion_compatfield + resolved property onYamlRefactorContextconfig.pyis_dbt_v1_9_6_or_greaterversion flag +_detect_fusion_manifest()for Fusion manifest/binary detectioncli/main.py--fusion-compat/--no-fusion-compatClick flagsync_operations.pymeta/tagsINTOconfigblock. When disabled: pullsconfig.metaUP to top-levelrestructuring.py_generate_minimal_model_yaml/_generate_minimal_source_yamlrespectfusion_compatinheritance.pymetaandconfig.metapaths transparentlyschema/parser.pydata_testsas a top-level keyschema/writer.pyanchorssection during writes (via_PRESERVED_KEYS)schema/validation.pytestsanddata_testscolumn keysdocs/Issues addressed
Test coverage
yaml organize,yaml document,yaml refactor)Test plan
--fusion-compatproducesconfig.meta/config.tagsat column level--no-fusion-compatkeepsmeta/tagsat top level (classic)pytest- 660 passed, 6 skippedbash demo_duckdb/integration_tests.sh- all passed