Skip to content

Commit f6666ca

Browse files
bruno-szdlBruno Souza de Limadbeatty10
authored
adding generate_unit_test_template macro (#251)
* adding generate_unit_test_template_macro * added a relation exists check * adding arg to controle inline/multiline columns * Removing duplicated `get_resource_from_unique_id` macro * Add newline to end of file * Add a simple incremental model for testing * CI tests * Use dispatch, etc. * Usage documentation for `generate_unit_test_template` macro --------- Co-authored-by: Bruno Souza de Lima <[email protected]> Co-authored-by: Doug Beatty <[email protected]>
1 parent 95bb093 commit f6666ca

10 files changed

+347
-2
lines changed

‎README.md‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,9 @@ Macros that generate dbt code, and log it to the command line.
2626
- [generate_model_import_ctes (source)](#generate_model_import_ctes-source)
2727
- [Arguments:](#arguments-5)
2828
- [Usage:](#usage-5)
29+
- [generate_unit_test_template (source)](#generate_unit_test_template-source)
30+
- [Arguments:](#arguments-6)
31+
- [Usage:](#usage-6)
2932
- [Contributing](#contributing)
3033

3134
# Installation instructions
@@ -394,6 +397,45 @@ select * from final
394397

395398
4. Replace the contents of the model's current SQL file with the compiled or logged code
396399

400+
## generate_unit_test_template ([source](macros/generate_unit_test_template.sql))
401+
402+
This macro generates the unit testing YAML for a given model with all references included as `given` inputs (along with their columns), plus the columns within the expected output.
403+
404+
### Arguments:
405+
406+
- `model_name` (required): The model you wish to generate unit testing YAML for.
407+
- `inline_columns` (optional, default=False): Whether you want all columns on the same line.
408+
409+
### Usage:
410+
411+
1. Create a model with your original SQL query
412+
2. Call the macro as an [operation](https://docs.getdbt.com/docs/using-operations):
413+
414+
```
415+
$ dbt run-operation generate_unit_test_template --args '{"model_name": "order_items", "inline_columns": true}'
416+
```
417+
418+
3. The new YAML - with all given inputs included - will be logged to the command line
419+
420+
```yaml
421+
unit_tests:
422+
- name: unit_test_order_items
423+
model: order_items
424+
425+
given:
426+
- input: ref("stg_order_items")
427+
rows:
428+
- col_a:
429+
col_b:
430+
431+
expect:
432+
rows:
433+
- id:
434+
```
435+
436+
4. Create a new YAML file with the compiled or logged code.
437+
5. Add column values for the given inputs and expected output.
438+
397439
## Contributing
398440
399441
To contirbute code to this package, please follow the steps outlined in the `integration_tests` directory's [README](https://github.com/dbt-labs/dbt-codegen/blob/main/integration_tests/README.md) file.

‎integration_tests/models/model_incremental.sql‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{{ config(
2+
materialized='incremental'
3+
) }}
4+
5+
select 1 as id

‎integration_tests/tests/test_generate_unit_test_template.sql‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
{% set actual_model_yaml = codegen.generate_unit_test_template(
2+
model_name='child_model',
3+
inline_columns=False
4+
)
5+
%}
6+
7+
-- depends_on: {{ ref('model_data_a') }}
8+
-- depends_on: {{ ref('child_model') }}
9+
10+
{% set expected_model_yaml %}
11+
unit_tests:
12+
- name: unit_test_child_model
13+
model: child_model
14+
15+
given:
16+
- input: ref("model_data_a")
17+
rows:
18+
- col_a:
19+
col_b:
20+
21+
expect:
22+
rows:
23+
- col_a:
24+
col_b:
25+
26+
{% endset %}
27+
28+
{{ assert_equal (actual_model_yaml | trim, expected_model_yaml | trim) }}

‎integration_tests/tests/test_generate_unit_test_template_incremental.sql‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
{% set actual_model_yaml = codegen.generate_unit_test_template(
2+
model_name='model_incremental',
3+
)
4+
%}
5+
6+
-- depends_on: {{ ref('model_incremental') }}
7+
8+
{% set expected_model_yaml %}
9+
unit_tests:
10+
- name: unit_test_model_incremental
11+
model: model_incremental
12+
13+
overrides:
14+
macros:
15+
is_incremental: true
16+
17+
given:
18+
- input: this
19+
rows:
20+
- id:
21+
22+
expect:
23+
rows:
24+
- id:
25+
26+
{% endset %}
27+
28+
{{ assert_equal (actual_model_yaml | trim, expected_model_yaml | trim) }}

‎integration_tests/tests/test_generate_unit_test_template_inline_columns.sql‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
{% set actual_model_yaml = codegen.generate_unit_test_template(
2+
model_name='child_model',
3+
inline_columns=True
4+
)
5+
%}
6+
7+
-- depends_on: {{ ref('model_data_a') }}
8+
-- depends_on: {{ ref('child_model') }}
9+
10+
{% set expected_model_yaml %}
11+
unit_tests:
12+
- name: unit_test_child_model
13+
model: child_model
14+
15+
given:
16+
- input: ref("model_data_a")
17+
rows:
18+
- {col_a: , col_b: }
19+
20+
expect:
21+
rows:
22+
- {col_a: , col_b: }
23+
24+
{% endset %}
25+
26+
{{ assert_equal (actual_model_yaml | trim, expected_model_yaml | trim) }}

‎integration_tests/tests/test_generate_unit_test_template_model_from_source.sql‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
{% set actual_model_yaml = codegen.generate_unit_test_template(
2+
model_name='model_from_source',
3+
)
4+
%}
5+
6+
-- depends_on: {{ ref('model_from_source') }}
7+
8+
{% set expected_model_yaml %}
9+
unit_tests:
10+
- name: unit_test_model_from_source
11+
model: model_from_source
12+
13+
given:
14+
- input: source("codegen_integration_tests__data_source_schema", "codegen_integration_tests__data_source_table")
15+
rows:
16+
- my_integer_col:
17+
my_bool_col:
18+
19+
expect:
20+
rows:
21+
- my_integer_col:
22+
my_bool_col:
23+
24+
{% endset %}
25+
26+
{{ assert_equal (actual_model_yaml | trim, expected_model_yaml | trim) }}

‎integration_tests/tests/test_generate_unit_test_template_no_inputs.sql‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
{% set actual_model_yaml = codegen.generate_unit_test_template(
2+
model_name='data__a_relation',
3+
inline_columns=False
4+
)
5+
%}
6+
7+
-- depends_on: {{ ref('data__a_relation') }}
8+
9+
{% set expected_model_yaml %}
10+
unit_tests:
11+
- name: unit_test_data__a_relation
12+
model: data__a_relation
13+
14+
given: []
15+
16+
expect:
17+
rows:
18+
- col_a:
19+
col_b:
20+
21+
{% endset %}
22+
23+
{{ assert_equal (actual_model_yaml | trim, expected_model_yaml | trim) }}

‎integration_tests/tests/test_helper_get_models.sql‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
-- depends_on: {{ ref('model_data_a') }}
2+
-- depends_on: {{ ref('model_incremental') }}
23
-- depends_on: {{ ref('model_struct') }}
34
-- depends_on: {{ ref('model_without_import_ctes') }}
45
-- depends_on: {{ ref('model_without_any_ctes') }}
@@ -7,6 +8,6 @@
78
{% set actual_list = codegen.get_models(prefix='model_')|sort %}
89
{% endif %}
910

10-
{% set expected_list = ['model_data_a', 'model_from_source', 'model_repeated', 'model_struct', 'model_without_any_ctes', 'model_without_import_ctes'] %}
11+
{% set expected_list = ['model_data_a', 'model_from_source', 'model_incremental', 'model_repeated', 'model_struct', 'model_without_any_ctes', 'model_without_import_ctes'] %}
1112

1213
{{ assert_equal (actual_list, expected_list) }}

‎macros/generate_unit_test_template.sql‎

Lines changed: 151 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,151 @@
1+
{% macro generate_unit_test_template(model_name, inline_columns=false) %}
2+
{{ return(adapter.dispatch('generate_unit_test_template', 'codegen')(model_name, inline_columns)) }}
3+
{% endmacro %}
4+
5+
{% macro default__generate_unit_test_template(model_name, inline_columns=false) %}
6+
7+
{%- set ns = namespace(depends_on_list = []) -%}
8+
9+
{%- if execute -%}
10+
11+
-- getting inputs and materialization info
12+
{%- for node in graph.nodes.values()
13+
| selectattr("resource_type", "equalto", "model")
14+
| selectattr("name", "equalto", model_name) -%}
15+
{%- set ns.depends_on_list = ns.depends_on_list + node.depends_on.nodes -%}
16+
{%- set ns.this_materialization = node.config['materialized'] -%}
17+
{%- endfor -%}
18+
19+
{%- endif -%}
20+
21+
-- getting the input columns
22+
{%- set ns.input_columns_list = [] -%}
23+
{%- for item in ns.depends_on_list -%}
24+
{%- set input_columns_list = [] -%}
25+
{%- set item_dict = codegen.get_resource_from_unique_id(item) -%}
26+
{%- if item_dict.resource_type == 'source' %}
27+
{%- set columns = adapter.get_columns_in_relation(source(item_dict.source_name, item_dict.identifier)) -%}
28+
{%- else -%}
29+
{%- set columns = adapter.get_columns_in_relation(ref(item_dict.alias)) -%}
30+
{%- endif -%}
31+
{%- for column in columns -%}
32+
{{ input_columns_list.append(column.name) }}
33+
{%- endfor -%}
34+
{{ ns.input_columns_list.append(input_columns_list) }}
35+
{%- endfor -%}
36+
37+
-- getting 'this' columns
38+
{% set relation_exists = load_relation(ref(model_name)) is not none %}
39+
{% if relation_exists %}
40+
{%- set ns.expected_columns_list = [] -%}
41+
{%- set columns = adapter.get_columns_in_relation(ref(model_name)) -%}
42+
{%- for column in columns -%}
43+
{{ ns.expected_columns_list.append(column.name) }}
44+
{%- endfor -%}
45+
{% endif %}
46+
47+
{%- set unit_test_yaml_template -%}
48+
unit_tests:
49+
- name: unit_test_{{ model_name }}
50+
model: {{ model_name }}
51+
{% if ns.this_materialization == 'incremental' %}
52+
overrides:
53+
macros:
54+
is_incremental: true
55+
{% else -%}
56+
57+
{%- endif %}
58+
given: {%- if ns.depends_on_list|length == 0 and ns.this_materialization != 'incremental' %} []{% endif %}
59+
{%- for i in range(ns.depends_on_list|length) -%}
60+
{%- set item_dict = codegen.get_resource_from_unique_id(ns.depends_on_list[i]) -%}
61+
{% if item_dict.resource_type == 'source' %}
62+
- input: source("{{item_dict.source_name}}", "{{item_dict.identifier}}")
63+
rows:
64+
{%- else %}
65+
- input: ref("{{item_dict.alias}}")
66+
rows:
67+
{%- endif -%}
68+
{%- if inline_columns -%}
69+
{%- set ns.column_string = '- {' -%}
70+
{%- for column_name in ns.input_columns_list[i] -%}
71+
{%- if not loop.last -%}
72+
{%- set ns.column_string = ns.column_string ~ column_name ~ ': , ' -%}
73+
{%- else -%}
74+
{%- set ns.column_string = ns.column_string ~ column_name ~ ': }' -%}
75+
{%- endif -%}
76+
{% endfor %}
77+
{%- else -%}
78+
{%- set ns.column_string = '' -%}
79+
{%- for column_name in ns.input_columns_list[i] -%}
80+
{%- if loop.first -%}
81+
{%- set ns.column_string = ns.column_string ~ '- ' ~ column_name ~ ': ' -%}
82+
{%- else -%}
83+
{%- set ns.column_string = ns.column_string ~ '\n ' ~ column_name ~ ': ' -%}
84+
{%- endif -%}
85+
{% endfor %}
86+
{%- endif %}
87+
{{ns.column_string}}
88+
{%- endfor %}
89+
90+
{%- if ns.this_materialization == 'incremental' %}
91+
- input: this
92+
rows:
93+
{%- if relation_exists -%}
94+
{%- if inline_columns -%}
95+
{%- set ns.column_string = '- {' -%}
96+
{%- for column_name in ns.expected_columns_list -%}
97+
{%- if not loop.last -%}
98+
{%- set ns.column_string = ns.column_string ~ column_name ~ ': , ' -%}
99+
{%- else -%}
100+
{%- set ns.column_string = ns.column_string ~ column_name ~ ': }' -%}
101+
{%- endif -%}
102+
{% endfor %}
103+
{%- else -%}
104+
{%- set ns.column_string = '' -%}
105+
{%- for column_name in ns.expected_columns_list -%}
106+
{%- if loop.first -%}
107+
{%- set ns.column_string = ns.column_string ~ '- ' ~ column_name ~ ': ' -%}
108+
{%- else -%}
109+
{%- set ns.column_string = ns.column_string ~ '\n ' ~ column_name ~ ': ' -%}
110+
{%- endif -%}
111+
{% endfor %}
112+
{%- endif %}
113+
{{ns.column_string}}
114+
{%- endif %}
115+
{%- endif %}
116+
117+
expect:
118+
rows:
119+
{%- if relation_exists -%}
120+
{%- if inline_columns -%}
121+
{%- set ns.column_string = '- {' -%}
122+
{%- for column_name in ns.expected_columns_list -%}
123+
{%- if not loop.last -%}
124+
{%- set ns.column_string = ns.column_string ~ column_name ~ ': , ' -%}
125+
{%- else -%}
126+
{%- set ns.column_string = ns.column_string ~ column_name ~ ': }' -%}
127+
{%- endif -%}
128+
{% endfor %}
129+
{%- else -%}
130+
{%- set ns.column_string = '' -%}
131+
{%- for column_name in ns.expected_columns_list -%}
132+
{%- if loop.first -%}
133+
{%- set ns.column_string = ns.column_string ~ '- ' ~ column_name ~ ': ' -%}
134+
{%- else -%}
135+
{%- set ns.column_string = ns.column_string ~ '\n ' ~ column_name ~ ': ' -%}
136+
{%- endif -%}
137+
{% endfor %}
138+
{%- endif %}
139+
{{ns.column_string}}
140+
{%- endif -%}
141+
142+
{%- endset -%}
143+
144+
{% if execute %}
145+
146+
{{ print(unit_test_yaml_template) }}
147+
{% do return(unit_test_yaml_template) %}
148+
149+
{% endif %}
150+
151+
{% endmacro %}

‎macros/helpers/helpers.sql‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,7 @@
4848
{% set model_path = "/".join(model.path.split("/")[:-1]) %}
4949
{% if model_path == directory and model.name.startswith(prefix) %}
5050
{% do model_names.append(model.name) %}
51-
{% endif %}
51+
{% endif %}
5252
{% endfor %}
5353
{% elif directory %}
5454
{% for model in models %}
@@ -88,3 +88,18 @@
8888
{% set formatted = codegen.format_column(column) %}
8989
{{ return(formatted['data_type'] | lower) }}
9090
{% endmacro %}
91+
92+
{# retrieve entire resource dictionary based on unique id #}
93+
{% macro get_resource_from_unique_id(resource_unique_id) %}
94+
{% set resource_type = resource_unique_id.split('.')[0] %}
95+
{% if resource_type == 'source' %}
96+
{% set resource = graph.sources[resource_unique_id] %}
97+
{% elif resource_type == 'exposure' %}
98+
{% set resource = graph.exposure[resource_unique_id] %}
99+
{% elif resource_type == 'metric' %}
100+
{% set resource = graph.metrics[resource_unique_id] %}
101+
{% else %}
102+
{% set resource = graph.nodes[resource_unique_id] %}
103+
{% endif %}
104+
{{ return(resource) }}
105+
{% endmacro %}

0 commit comments

Comments
 (0)