feat: add external YAML formatter integration (--formatter)#323
Open
RicardoAGL wants to merge 8 commits intoz3z1ma:mainfrom
Open
feat: add external YAML formatter integration (--formatter)#323RicardoAGL wants to merge 8 commits intoz3z1ma:mainfrom
RicardoAGL wants to merge 8 commits intoz3z1ma:mainfrom
Conversation
Auto-detect dbt version and produce Fusion-compatible YAML when column-level config is supported (dbt >= 1.9.6). When enabled, meta/tags are nested under config blocks instead of top-level. Adds --fusion-compat/--no-fusion-compat CLI flags for override. Also preserves anchors, data_tests, and semantic_models in YAML.
Fix 17 pre-existing test failures: - Pass DbtProjectContext (yaml_context.project) instead of YamlRefactorContext to generator functions that expect the project context directly - Fix mock.patch paths for lazy imports: mock at dbt_core_interface.* instead of dbt_osmosis.core.generators.* for classes imported inside functions - Use tmp_path instead of absolute /custom/staging to avoid read-only filesystem
Detect dbt Fusion manifests (schema version > v12) in the target directory before parsing, so teams running both Fusion and dbt-core get fusion_compat=True automatically even with older dbt-core versions. Detection also checks for dbtf/dbt-fusion binaries on PATH as a fallback when no manifest exists. The explicit --fusion-compat flag still takes priority over all auto-detection.
- Document --fusion-compat/--no-fusion-compat in CLI reference, settings reference, and configuration tutorial - Add disclaimer that dbt-osmosis requires dbt-core and does not run on dbt Fusion directly, with hybrid workflow instructions - Anchor schema version regex to path segment to avoid false matches - Optimize manifest reading to parse only first 4KB instead of full JSON - Downgrade Fusion binary detection log from info to debug - Remove anchors from parser allowed_keys (preserved via writer only) - Simplify single-element tuple filters to != comparisons - Remove unused json import from config.py
The fusion-compat setting is only available via CLI flag or auto-detection. The project vars path is not wired up in YamlRefactorContext, so the example was misleading.
Allow running an external formatter (prettier, yamlfmt, yq, etc.) on YAML files after osmosis writes them, reducing CI to a single step. - Add --formatter CLI option to refactor/organize/document commands - Track written files via YamlRefactorContext._written_files - New formatting.py module with run_external_formatter() (non-fatal) - Config precedence: CLI flag > dbt-osmosis.yml > None - 29 new tests covering formatter execution, settings, and wiring
- Add subprocess timeout (default 120s) to prevent hung formatters - Log warning instead of silently swallowing dbt-osmosis.yml parse errors - Reuse cached _written_file_tracker variable in restructuring.py - Add 3 tests for timeout behavior
Relative paths caused formatter to fail when cwd differed from the invoking directory. Resolving to absolute in register_written_file() ensures formatters can always find the files. Discovered during integration testing with prettier and yq against the demo_duckdb project.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
--formatterCLI option torefactor,organize, anddocumentcommands that runs an external YAML formatter on files written by osmosis. This lets teams use their preferred formatter (prettier, yamlfmt, yq, etc.) in a single command instead of the common 3-step CI pattern (osmosis → formatter → git diff).--formatterflag accepts any command that takes file paths as trailing argsYamlRefactorContext._written_filesdbt-osmosis.yml> None (backward compatible)Example usage
CI simplification
Before (3-step):
After (2-step):
Files changed (12 files, +713 lines)
src/dbt_osmosis/core/formatting.pyrun_external_formatter()with shlex parsing, timeout, non-fatal error handlingsrc/dbt_osmosis/core/settings.pyformatterfield,_written_filestracking,resolved_formatterpropertysrc/dbt_osmosis/core/schema/writer.pywritten_file_trackercallback in_write_yaml()/commit_yamls()src/dbt_osmosis/core/sync_operations.py_write_yaml()src/dbt_osmosis/core/restructuring.py_write_yaml()/commit_yamls()src/dbt_osmosis/cli/main.py--formatterClick option +_run_formatter_if_configured()src/dbt_osmosis/core/osmosis.pyrun_external_formattertests/core/test_formatting.pytests/core/test_settings.pytests/core/test_schema.pytests/core/test_cli.pyTest plan
--formatter, behavior is identical to currentDependencies