dbt-athena - High availability full-refresh "snapshot"?

[FredericoCoelhoNunes]

Recently we've run into some issues with the table materialization using the dbt-athena connector (Iceberg tables).

Our setup is the following: we are building a few models on s3/Glue with dbt-athena, as Iceberg tables. And then, on another dbt project, we use these Iceberg tables as sources for our Redshift models. Redshift reads the Iceberg. tables using Redshift Spectrum.

The issue: the build of the Redshift and Athena models can't overlap if the Athena models are not incremental. This is because the full-refresh involves completely rewriting the data on s3 (including a new catalog), and so halfway through building the Redshift models, they fail with Spectrum scan error - file not found (since the original files no longer exist on s3).

This can be solved with careful scheduling, but sometimes the use cases are more complicated and this isn't so easily achieved.

It would be great if there was some sort of "high-availability" full refresh, meaning that the Iceberg table is never fully deleted, but simply a new "snapshot" of the table is added to the manifest.

Has anyone faced a similar issue before? Your opinions are very welcome!

[mehdiat22]

Hi Frederico,

Thanks for bringing this up — we’ve faced a similar issue when Redshift reads Iceberg tables from dbt-athena. During a full-refresh, replacing the S3 files often breaks Spectrum scans mid-process with "file not found" errors.

One approach we tested was writing to versioned S3 paths and updating the Glue catalog only after the refresh is complete, which avoids downtime and broken reads. It simulates a snapshot switch without fully deleting the table.

I agree — having native support for high-availability full-refresh (via Iceberg snapshots) would be a great enhancement. Would be happy to follow or contribute if something’s being explored.

Best,

[borjagonzal]

Hi @mehdiat22,
I'm interested in the strategy you mentioned, because it can be a work around we could use in my team. Could you elaborate please?

Borja