feat: introduce asset caching for sqlmesh related assets

[ravenac95]

This currently required opensource-observer/oso-dagster# and that should be reviewed post-hoc.

We now create a new way to create cacheable dagster asset factories. Originally, I wanted a much cleaner interface but because of the way that dagster does things, it's not really possible to provide an interface that is nearly identical to dagster's interface. So we must do things in two steps minimally. The doc explains it (and needs a little bit of work) but essentially you need to do whatever operation is necessary to generate some kind cacheable intermediary asset. Then you need to provide a rehydrator function that rehydrates the intermediary asset into an actual dagster asset, job, sensor, check, etc.

Other things in this:

• Added a cache system to oso_core that can cache and load any pydantic BaseModel

Things I'm choosing to do later:

• We need to refactor the oso_dagster.factories.* package. It's got quite a few things that are not needed and the module organization has gotten out of hand.

This PR closes the following:

• Close dagster-sqlmesh: SQLMesh Translator is incorrectly translating assets in dagster #4667
• Close Dagster asset caching #4727

TODO:

• Update dockerfile to use uv run oso_dagster build to build the cache for the sqlmesh assets

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
kariba-network	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Aug 8, 2025 8:29pm
oso-www	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Aug 8, 2025 8:29pm

[ravenac95]

(edit: This comment is now out of date)

I'm actually going to cannibalize this PR and I think I'm going to combine this and also the opensource-observer/dagster-sqlmesh#48 to provide a better solution.

In opensource-observer/dagster-sqlmesh#48 we are using a solution that provides a way to Cache Assets just for the sqlmesh. Instead, I think we update the EarlyResourceAssetFactory and also the AssetFactoryResponse and include a way to respond with any of the following CacheableAssetDefinition, CacheableJobDefinition, CacheableAssetCheckDefinition, and CacheableSensor. With these cachable definitions we will be able to rehydrate those definitions from a cache.

I still need to figure out exactly how I can get that to work.