Associating LakeFormation tags with source tables

[rstml]

Hi,

We have source tables that are created using Glue Crawlers. We also have LakeFormation tags that are created outside of dbt.

We would like to manage all LF-Tags associations centrally inside of the dbt project, including LF-Tags for the source tables.

What would be the best way to achieve that?

[dacreify]

We use Terraform to manage tags on source tables, FWIW. I believe this discussion has taken place for dbt-athena and the conclusion was to manage source table tags outside of it.

[rstml]

Thanks for sharing this.

What I'm trying to achieve is to have a single place to describe tables and permissions. So if tables are already defined elsewhere (e.g. terraform) then it makes sense to define lf-tags there as well.

However, in the case of Glue Crawlers, the definition of those tables is not in terraform. Terraform knows nothing about them.

dbt on the other hand is where I describe my source tables and I'd like to define lftags as a part of the same metadata.

[dacreify]

Tables are created automatically in our catalog as well by Glue Crawlers and other processes. Nevertheless we tag them using aws_lakeformation_resource_lf_tags as all it cares about is the presence of a table of a given name. DBT doesn't manage the lifecycle of these tables so tagging them outside of it seems to make sense.

[rstml]

Just an update: I've implemented this through dbt-external-tables macro. If anyone is interested, here's the source code: https://github.com/rstml/dbt-external-tables-athena

We can now store all metadata, including for source tables, in a dbt project.

[nicor88]

You can consider to include your changes here https://github.com/dbt-athena/dbt-athena-external-tables, it will benefit other users too.