Fusion Diaries: Local Preview Edition! (2024-08-21)

[dataders]

New versioning scheme: who dis?

Announcement from yesterday: Fusion and the dbt VS Code extension are now in Preview for local development

Read the blog if you haven't already! Not only are there feature announcements, and there's context on how we're tracking Fusion readiness for teams using Core today, as well as guidance on which projects are a great fit for Fusion today. There are also guidelines detailing who the project works best for.

If you haven't tried Fusion yet, I highly encourage you to try dbt init's new --fusion-upgrade

There are many killer features landing this week that I'll go into more detail on. First some highlights and an abbreviated changelog.

• 39 issues closed across the public and internal repos
• 5 new releases
• strict mode is here!
• 3 new custom VS Code themes!
• an officially supported Windows experience!
• dozens of packages are now officially Fusion compatible
• incremental compilation
• polished deps and logs experience

(Abridged) Release notes

FYI, any links that are just numbers are links to our internal private repo, not the public-facing dbt-fusion repo

2.0.0-preview.3 - Schema compatibility and hydration refinements

• Add deps detection for schema compatibility - Proactive dependency compatibility checking #5205
• Fix identifier resolution for hydrating sharded inline sources - Complex scenario handling #5203
• Normalize NUMBER type in unit test rendering - Type system consistency #5194
• Refactor LSP to use parquet cache nodes - Performance architecture improvement #5076

2.0.0-preview.2 - Strict mode by default and bug fixes

• Turn strict mode on by default - Enhanced validation and error checking #5103
• Add package compatibility check and bring back suggestion - Package validation framework #5205
• Merge LSP and CLI call path - Unified architecture #5140
• Fix defer progress bars - UI improvement for defer operations #5197
• Update dbt Fusion name and final link - Branding consistency #5183
• Update Windows install script - Installation improvements #5160

2.0.0-preview.1 - incremental compilation, `dbt init --fusion-upgrade`, better logs, relation caching

🎯 Major Features:

• Interactive onboarding flow - Add new dbt onboard command with step-by-step guidance #5104
• LSP incremental compile - Revolutionary IDE performance improvement #5131
• Version awareness - Log when newer version of fusion available #4827
• Source navigation - Support goto_def for source() references #4982

📊 Relation Cache & Performance:

• [BigQuery] Relation cache pre-hydration - Dramatic performance boost #5046
• [Databricks] Relation cache pre-hydration - Performance parity #5102
• Catalog.json generation - Fetch Snowflake catalog info from warehouse adapter #4950

🛠️ Developer Experience:

• Package deprecation grouping - Cleaner output with consolidated warnings #4879
• Legal project names - Validation in dbt init command #5143
• Enhanced logging - New structured logging system #5181
• LSP channels - Upgrade from locking to channel-based communication #5137

🔧 SQL Engine Enhancements:

• Snowflake functions - Execute dateadd, NVL, NVL2 functions
• Type casting - CAST from TIMESTAMP_TZ to DATE support
• Date arithmetic - Execute date ± decimal operations
• BigQuery arrays - Typecheck ARRAY_FIRST, LAST, SLICE functions #5180

🐛 Critical Fixes:

• Databricks compatibility - Fix adapter support for DBR version < 16.2 #4886
• Windows stability - Fix bridge_adapter TLS deadlock #5149
• Lineage accuracy - Fix union of MOD and COPY dependencies #5182
• Seed hydration - Fix when --static-analysis is off #5150

2.0.0-beta.66 - Multi-resource selection and compilation improvements

• Multi-resource selection - Allow resource-type and exclude-resource-type to accept multiple inputs dbt-fusion#5148
• LSP compilation worker - Message passing architecture improvements #5137
• Context macro support - Support getting macros from "context" in run-operation #5081
• Binary update notifications - VSCE offers user to update when out of date #4800
• dbt-external-tables compatibility - Improved integration #5129
• Onboarding integration - Merge onboard into dbt init with --fusion-upgrade flag #5151
• Project name validation - Enforce legal project names during init #5143
• Snowflake type fix - Fix TRY_CAST from BOOLEAN to BOOLEAN #5139

2.0.0-beta.65 - VS Code themes, better spans, bug fixes

🎯 Core Features:

• YAML value fixes - Comprehensive fixes for dunders & serde_yaml handling #5094
• Duplicate test rejection - Fusion should reject duplicate test names dbt-fusion#5086
• Profile improvements - Enhanced dbt init profile handling #5018
• VS Code themes - Add dbt theme support #4932
• Get Support CTA - Add support call-to-action for error scenarios #4977

⚡ Performance & Architecture:

• Status reporter channels - Use channels instead of locking for performance #5090
• TaskRunner optimization - Limit TaskRunnerCtx size & ensure hydrate sources #5113
• Structured tracing - Enhanced telemetry with parquet support #4915
• LSP incremental optimization - Remove init_adapter call #5123

🔧 Adapter & SQL Improvements:

• [Databricks] List schemas - Implement list_schemas method with tests #5057
• [BigQuery] Metadata optimization - Use limit 0 on information schema views #5051
• Snowflake auth overhaul - Keypair authentication improvements #5052
• SQL function execution - Support for date operations, NVL functions, and type casting
• Unit test improvements - Enhanced testing framework #5075

Big rocks

Strict mode

If you've tried Fusion, you've likely seen warnings that start with:
warning: (will error post beta).

With today’s preview announcement we’re officially post-beta, so these warnings will now become errors.

For more context on why you’re seeing errors, check out the "What is the dbt language?" section of the May 2025 dbt Core roadmap post and dbt-fusion#401, a more detailed guide for package maintainers.

dbt-autofix can remediate virtually all of these errors and the changes made should be backwards compatible with dbt Core and the latest version on the dbt platform.

In the event that you cannot resolve an error—as might be the case for parsing errors encountered in packages—you can set the below environment variable to revert the errors back to non-blocking warnings. However, this is not a recommended long-term approach.

SET DBT_ENGINE_BETA_PARSING=1

Please reach out in the event that you are unable to resolve the parsing errors!

Package compatibility

Given the above-described strict mode, a large concern was that users would be blocked by conformance errors stemming from packages. Not a great scenario when users cannot unblock themselves!

dbt-autofix deprecations --include-packages is a great stopgap, but isn't durable, as the changes are overwritten every time deps are installed.

Fortunately, we've made a lot of headway towards getting the most popular packages conformant with the new authoring layer.

The real heroes here are the Fivetran Quickstart Data Models Team. A huge shoutout to @fivetran-joemarkiewicz @fivetran-jamie and the rest of the team to ensure users of their dozens of packages have a smooth experience on Fusion. We couldn't have done it without y'all.

Polish on dbt deps

Related to the above-mentioned new package versions. deps will now tell you if there's a newer version of a package available.

If you see newer versions, you should upgrade ASAP to ensure your project works with Fusion local preview!

Incremental compilation!

This is my personal favorite new feature.

To see what causes me to make the below face, check out this unboxing video of sorts where I explain the purpose of incremental compilation and try it for the first time. It works exactly as expected! What a time to be alive!

Hugely improved logs (with more polish to come!)

We've entered a new world where Fusion logs have both:

• a higher signal-to-noise ratio
• the information parity with dbt Core

If you have anything more you'd like to add, please check out this discussion and add your feedback if it hasn't yet been mentioned there.
dbt-fusion#584: UX Feedback on Logging and stdout

a new onboarding flow with dbt init --fusion-upgrade

We have a very slick new onboarding wizard that goes beyond dbt Core's init in that it also:

1. will generate a profiles.yml for you if a dbt_cloud.yml is detected for you
2. guide you through progressively more complex Fusion operations to make sure things are in good working order

VS Code Extension: Go To Source, a new sidebar, and dbt color themes!

Reporting exact locations (i.e. spans) in YAML files is one of the big reasons as to why we needed to evolve dbt's authoring layer.

The payoff? You can now "Go to Definition" on source() macros in models just as you can with ref(). Check out the below recording!

go_to_source.mov

Windows officially supported!

All summer Windows users have been patiently awaiting the announcement that it should be ready for them!

Good news, Tyler S (and all other Windows users) -- give it a go and shout if something is amiss!

Last, there's 3 custom dbt color themes that now ship with the extension. Check them out! I'm personally a big fan of the "Fusion theme"

🐉 Here be dragons

Dragon vanquished: spammed Oauth browser tabs in Snowflake!

Our adapters and driver team has done a huge amount of work zeroing out issues with External Browser authentication in Snowflake.

Community member Danielle had a long-standing issue that she reports that today is finally resolved thanks in large part to @VersusFacit and @felipecrv's dogged efforts to stabilize the upstream Snowflake Go SDK.

Dragon: spans spans spans

We've been using the word "span" to mean the location in your project where errors occur.

In the below error, the span says that the error happens on line 5, on the 5th character from the start of the line.

dbt0101: mismatched input 'selectt' expecting one of 'SELECT', 'TABLE', 'WITH', '('
  --> models/staging/okta/stg_okta__users.sql:5:5

We've put a lot of work in the past few weeks to improving spans in addition to stack traces of errors as well.

It's still not perfect, so please shout when you see something amiss.

A great example is dbt-fusion#555 reported by @pempey where his error message was decidedly unhelpful!

He was getting the below error message which was hard to figure out! We've got a PR up to improve this so it tells users exactly the place where dbt Fusion fails to parse.

error: dbt1502: Failed to compile Jinja expression syntax error: unexpected input after expression
(in <expression>:1:1)
  --> <expression>:1:1

Dragon: Sensemaking added complexity

Fusion now has a compile cache, a relation cache, and fancy logging by default. These are all features that simplify your life when they work as expected, but they can be hard to debug when things go awry. We plan to focus our efforts on improving this experience over the coming weeks. Reach out in Slack if you see something strange.

🤔 Looking for feedback

Have you been thrown by the lack of explicit support in Fusion for snapshot's target_database and target_schema configs? Please respond to Elias's thread in Slack! We want to make this right, but there's some nuance.

Also, I want to reiterate that we're looking for feedback on logging UX!
dbt-fusion#584: UX Feedback on Logging and stdout

👓 Stuff you should read watch

Check out this great talk from another, more educated Anders on how modern compilers work. Not only is incremental compilation now a requirement, but it also must integrate seamlessly into a developer's tooling. With the dbt VSCE & LSP, this means features like "Preview CTE" and Go to Definition.

My own brain told me that this video would go way beyond my education/intelligence, but I found it very informative and was able to make many connections to Fusion's ability to "compile" and understand SQL.

Anders Hejlsberg on Modern compiler construction

🏁 Made it to the meme

It’s one thing to dream about a world where analytics engineers have SQL understanding at their fingertips. It’s another to spend days months squashing hard bugs to make this dream a reality.

But you know what makes it all worth it? That's right: it's you, dear reader! Improving the DX of data practitioners is why I get out of bed every morning. It's a tremendous privilege, and all of here at dbt Labs are so grateful for the opportunity.

Until next week!

[pempey]

@dataders That you for you update and your enthuseasum. I would ask for one point of clarity in the comunicacion about dbt Fusion capabilities: clearly label and identify enterprise features and non enterprise features. From the licensing FAQs:

Do I need to be a paying dbt customer in order to use Fusion?
No.
Do I need to be a paying dbt customer in order to access some enterprise features of Fusion?
Yes.

As a team that is currently on dbt Core and do not know when or if we would be a paying dbt customer it is imperative that we are able to clearly know what features we would have access to with Fusion so that we can make a determination of value. As we have a team greater than 15 I would put the VSC Extension and its features into the category of enterprise features and while Fusion enables awesome features like incremental compilation, it appears that it is the extension that provides them.