possibility to use templated properties to apply by default on file update

[david-whimsical]

hey, first of all, thanks for building this package, it's been a pleasure using it!

---

Our team recently started to use what's in the meta field, most notably to set the meta.owner property:

version: 2
models:
- name: dim_user
  description: >
    Some description
  meta:
    owner: '@data-team'
  columns:
  - name: user_id
    description: ''
    tests:
    - not_null
    - unique
  - name: some_col
    description: ''
  - name: some_other_col
    description: ''

In this case, @data-team is our default, which we sometimes replace with an individual owner, eg. @david, if the data model is very particular. This default value, we want it on every single data models that we create, and it was a little tedious to update all newly created .yml file with the property.

This PR adds an optional dbt_invoke_template.yml that will add default properties to the files updated by dbt-invoke.

What changes:

1. load the contents of dbt_invoke_template.yml in _util.py,
2. update the _structure_property_file_dict to apply the template's default properties.

I've also updated the README.md file so you can better understand how this works, and I've tested this by updating my local dbt-invoke package.

(feel free to edit anything on this; I'd love to have this functionality, so I'm saving the frustration of manually adding meta.owner)

[robastel]

Thanks for your PR!

And thanks also for the context you already provided. I added a few initial comments/questions so that I can further develop my understanding of how this feature may be used.

[robastel]

I might be nitpicking a bit here, but I think we should change model, seed, snapshot, and analysis to their plural versions; models, seeds, snapshots, and analyses so that dbt_invoke_template.yml matches the convention used in property files. Thoughts?

[david-whimsical]

I agree with you! I think it's better to align to the used convention, I'll update properties.py to use plural versions instead.

[robastel]

What if the key is already in the property_dict, but we had altered dbt_invoke_template.yml between the previous run and the current run?

[david-whimsical]

We would keep the value already in the file – else I think it could lead to frustrating behaviors, where dbt-invoke overwrites things.

In my opinion, The template should only add new lines, and not update any existing ones, if that makes sense!

[robastel]

Given that this feature is totally new to dbt-invoke, we are going to need to add some tests, most likely to the tests/test_properties.py file.

A few test scenarios that immediately come to mind and that we should ensure work (both for the custom default resource properties and custom default column properties) are:

• That everything still works as is when there is no dbt_invoke_template.yml file
• That the output matches the expected output when there is a dbt_invoke_template.yml
• If the value of a key in dbt_invoke_template.yml is updated, and that key already exists in the property file, then the value of that key should be updated in property file next time dbt-invoke properties.update is run

Were there any interesting scenarios or issues you noticed during your testing or any other scenarios you can think of that should be tested?

Something else I am pondering: What happens if the dbt_invoke_template.yml file contains invalid yml, is empty, or contains none of the supported resource types? In your opinion, what should happen in these cases?