[dbt-athena] Migration from Glue to dbt Python models: Lake Formation compatibility

[moritzsanne]

Background

We're currently using Airflow for orchestration and dbt for SQL-based transformations. Some of our transformations require
custom code written in Python, which we've been running via AWS Glue. We're looking to consolidate our workflow by migrating these Python transformations to dbt Python models.

Issue

While exploring dbt Python models for Athena (per the documentation), we've encountered a potential blocker: Python models require "a Spark-enabled workgroup created in Athena," but Amazon Athena for Apache Spark doesn't support AWS Lake Formation.
This is problematic as we're currently migrating from IAM-based access control to Lake Formation-based access control for our data lake.

Questions

We're looking for ideas and guidance on:

• Is there any workaround for using dbt Python models with Athena while using Lake Formation for access control?
• Has anyone successfully implemented hybrid access patterns, where a Spark-enabled workgroup uses IAM while still maintaining Lake Formation governance?
• If we create tables via dbt using IAM roles in a Spark-enabled workgroup, would these tables properly inherit Lake Formation tags from their parent database?
• Could dbt Python models be run on alternative AWS Spark backends (EMR, Glue, etc.) that have better Lake Formation compatibility while still integrating with our Athena-based dbt workflow?

Any insights, examples, or alternative approaches would be greatly appreciated. We're trying to determine the best path forward that balances our needs for Python transformations with our Lake Formation governance requirements.