Skip to content

Commit 5da079b

Browse files
denikshreyas-goenka
denik
and
shreyas-goenka
authored
Resolve $resources refs before deployment if set by user (#3423)
## Changes Resolve ${resources} references internally, before the deployment. This is v2 of #3370 which was reverted in #3421 because it interacted poorly with other resources transformations that are done by DABs. By moving this resolution later we ensure that all transformation are visible when we do reference replacement. ## Why Currently we resolve these via terraform, but that's not going to work with direct deployment. To minimize the changes between TF and direct and surface possible issues with this approach, we do this for both backends. --------- Co-authored-by: shreyas-goenka <[email protected]>
1 parent 9e3d3fd commit 5da079b

File tree

18 files changed

+246
-7
lines changed

18 files changed

+246
-7
lines changed

‎acceptance/bundle/resource_deps/resources_var/databricks.yml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
resources:
2+
volumes:
3+
bar:
4+
catalog_name: mycatalog
5+
schema_name: myschema
6+
name: barname
7+
foo:
8+
catalog_name: ${resources.volumes.bar.catalog_name}
9+
schema_name: ${resources.volumes.bar.schema_name}
10+
name: myname
11+
pipelines:
12+
mypipeline:
13+
name: pipeline for ${resources.volumes.foo.catalog_name}.${resources.volumes.foo.schema_name}.${resources.volumes.foo.name}
14+
15+
targets:
16+
dev:
17+
mode: development

‎acceptance/bundle/resource_deps/resources_var/out.test.toml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
Local = true
2+
Cloud = false
3+
4+
[EnvMatrix]
5+
DATABRICKS_CLI_DEPLOYMENT = ["terraform", "direct-exp"]

‎acceptance/bundle/resource_deps/resources_var/output.txt‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
2+
>>> [CLI] bundle validate -t dev -o json
3+
{
4+
"pipelines": {
5+
"mypipeline": {
6+
"channel": "CURRENT",
7+
"deployment": {
8+
"kind": "BUNDLE",
9+
"metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/dev/state/metadata.json"
10+
},
11+
"development": true,
12+
"edition": "ADVANCED",
13+
"name": "[dev [USERNAME]] pipeline for ${resources.volumes.foo.catalog_name}.${resources.volumes.foo.schema_name}.${resources.volumes.foo.name}",
14+
"permissions": [],
15+
"tags": {
16+
"dev": "[USERNAME]"
17+
}
18+
}
19+
},
20+
"volumes": {
21+
"bar": {
22+
"catalog_name": "mycatalog",
23+
"name": "barname",
24+
"schema_name": "myschema",
25+
"volume_type": "MANAGED"
26+
},
27+
"foo": {
28+
"catalog_name": "${resources.volumes.bar.catalog_name}",
29+
"name": "myname",
30+
"schema_name": "${resources.volumes.bar.schema_name}",
31+
"volume_type": "MANAGED"
32+
}
33+
}
34+
}
35+
36+
>>> [CLI] bundle deploy -t dev
37+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/dev/files...
38+
Deploying resources...
39+
Updating deployment state...
40+
Deployment complete!
41+
42+
>>> jq -s .[] | select(.path=="/api/2.0/pipelines") | .body.name out.requests.txt
43+
"[dev [USERNAME]] pipeline for mycatalog.myschema.myname"

‎acceptance/bundle/resource_deps/resources_var/script‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
trace $CLI bundle validate -t dev -o json | jq .resources
2+
trace $CLI bundle deploy -t dev
3+
trace jq -s '.[] | select(.path=="/api/2.0/pipelines") | .body.name' out.requests.txt
4+
rm out.requests.txt

‎acceptance/bundle/resource_deps/resources_var/test.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
RecordRequests = true

‎acceptance/bundle/resource_deps/resources_var_presets/databricks.yml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# this tests presets that modify schema are visible in $resources references
2+
resources:
3+
schemas:
4+
bar:
5+
catalog_name: mycatalog
6+
name: myschema
7+
foo:
8+
catalog_name: ${resources.schemas.bar.catalog_name}
9+
name: ${resources.schemas.bar.name}
10+
pipelines:
11+
mypipeline:
12+
# When creating the pipeline we should see dev prefix in schema_name here
13+
name: pipeline for ${resources.schemas.foo.catalog_name}.${resources.schemas.foo.name}
14+
15+
targets:
16+
dev:
17+
mode: development

‎acceptance/bundle/resource_deps/resources_var_presets/out.test.toml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
Local = true
2+
Cloud = false
3+
4+
[EnvMatrix]
5+
DATABRICKS_CLI_DEPLOYMENT = ["terraform", "direct-exp"]

‎acceptance/bundle/resource_deps/resources_var_presets/output.txt‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
2+
>>> [CLI] bundle validate -t dev -o json
3+
{
4+
"pipelines": {
5+
"mypipeline": {
6+
"channel": "CURRENT",
7+
"deployment": {
8+
"kind": "BUNDLE",
9+
"metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/dev/state/metadata.json"
10+
},
11+
"development": true,
12+
"edition": "ADVANCED",
13+
"name": "[dev [USERNAME]] pipeline for ${resources.schemas.foo.catalog_name}.${resources.schemas.foo.name}",
14+
"permissions": [],
15+
"tags": {
16+
"dev": "[USERNAME]"
17+
}
18+
}
19+
},
20+
"schemas": {
21+
"bar": {
22+
"catalog_name": "mycatalog",
23+
"name": "dev_[USERNAME]_myschema"
24+
},
25+
"foo": {
26+
"catalog_name": "${resources.schemas.bar.catalog_name}",
27+
"name": "dev_[USERNAME]_${resources.schemas.bar.name}"
28+
}
29+
}
30+
}
31+
32+
>>> [CLI] bundle deploy -t dev
33+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/dev/files...
34+
Deploying resources...
35+
Updating deployment state...
36+
Deployment complete!
37+
38+
>>> jq -s .[] | select(.path=="/api/2.0/pipelines") | .body.name out.requests.txt
39+
"[dev [USERNAME]] pipeline for mycatalog.dev_[USERNAME]_dev_[USERNAME]_myschema"

‎acceptance/bundle/resource_deps/resources_var_presets/script‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
trace $CLI bundle validate -t dev -o json | jq .resources
2+
trace $CLI bundle deploy -t dev
3+
trace jq -s '.[] | select(.path=="/api/2.0/pipelines") | .body.name' out.requests.txt
4+
rm out.requests.txt

‎acceptance/bundle/resource_deps/resources_var_presets/test.toml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
RecordRequests = true
2+
3+
#EnvMatrix.DATABRICKS_CLI_DEPLOYMENT = ["terraform"] # direct does not support non-id references yet
4+
# Error: reading config: failed to read references from config for volumes.foo: parsing refs: cannot process reference resources.schemas.bar.name: ${resources...} can only refer to field in the config or 'id'

0 commit comments

Comments
 (0)