Reduce DuckLake table materialization conflicts

[leonardovida]

We found that the real concurrency issue is dbt persist_docs on DuckLake, specifically the COMMENT ON TABLE / COMMENT ON COLUMN path under concurrent model builds.

Concurrent CTAS alone was fine, and CTAS+rename was also fine, but conflicts appeared when comments were part of the same write flow. I simplified the fix that I originally posted by keeping DuckLake retries for transient TransactionExceptions and deferring persist_docs until after commit.

Initially I created a PR that was explicitly retrying on failures, now this PR is solely focusing on persist_docs

[jwills]

so the fix is to hack the persist_docs operation for Ducklake tables to occur outside of the main transaction?

[leonardovida]

so the fix is to hack the persist_docs operation for Ducklake tables to occur outside of the main transaction?

We are working on the "real" fix, that will happen on the DuckLake side. In the meantime this might be a possible approach, though it assumes customers do not relay on the invariant of comments being in the same transactions as the CTAS (so possibly having the CTAS go through but not the comment part). What is your stance on this @jwills ?