Skip to content

Commit 0f6ea36

Browse files
denikdenik
authored
direct: Detailed bundle plan (#3636)
## Changes The JSON plan (output of “bundle debug plan”) is now input to “bundle deploy” internally. This means it contains full information about resources about to be deployed. The JSON plan stores configs as well as all unresolved fields as map (path -> reference). There is new module, structvar, that supports resolving these references. It does it quickly and without copying the full config for every resolution, like the previous implementation did. The JSON plan also stores per-field action. This is purely informational, it gives an idea why a certain action for the resource was chosen. During plan and apply, variable resolution is now done in the node that contains the reference. Previously, it was updated by the node that was referenced. This results in better error messages when resolution does not work and it does not fail resources that themselves have no issues (but have incorrect references pointing to them). References can now refer to missing field and it works like in terraform (missing_string_field test). There is slight simplification in how $resources are resolved. All local resolutions (from config) are now done in plan phase. All remote resolutions (from remote state) are now done in apply phase. Previously we could resolve some remote references in plan stage if action was “skip”. This might result in slower deploys, so this is a potential perf regression, to be revisited in the future. Depends on #3650 & #3646 Other changes: - dagrun.Graph now exports its fields. dagrun.OutgoingLabels() is removed. - new function structaccess.Set() to set any field in a struct by path (used by structvar). - dresources.ClassifyByTriggers([]Change) is replaced by dresources.ClassifyByTriggers(Change), aggregation logic now lives in bundle_plan.go ## Why Having the plan as input to deploy provides a view into deployment that we did not have before, allowing to see exactly what is going to be deployed and why. This is useful for: - debugging (why is my pipeline being recreated?) - see diff between deployed infra and config - deploying exactly what was in the diff and nothing else: "bundle deploy -readplan plan.json" can be implemented easily after this change.
1 parent 34ce1a1 commit 0f6ea36

File tree

130 files changed

+4244
-914
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

130 files changed

+4244
-914
lines changed

‎acceptance/bundle/artifacts/whl_dynamic/out.plan_create.direct-exp.json‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
{
2+
"plan": {
3+
"resources.jobs.test_job": {
4+
"action": "create",
5+
"new_state": {
6+
"config": {
7+
"deployment": {
8+
"kind": "BUNDLE",
9+
"metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/metadata.json"
10+
},
11+
"edit_mode": "UI_LOCKED",
12+
"environments": [
13+
{
14+
"environment_key": "test_env",
15+
"spec": {
16+
"client": "1",
17+
"dependencies": [
18+
"/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal/my_test_code-0.0.1+[UNIX_TIME_NANOS][0]-py3-none-any.whl"
19+
]
20+
}
21+
}
22+
],
23+
"format": "MULTI_TASK",
24+
"max_concurrent_runs": 1,
25+
"name": "[default] My Wheel Job",
26+
"queue": {
27+
"enabled": true
28+
},
29+
"tasks": [
30+
{
31+
"environment_key": "test_env",
32+
"python_wheel_task": {
33+
"entry_point": "run",
34+
"package_name": "my_test_code"
35+
},
36+
"task_key": "ServerlessTestTask"
37+
},
38+
{
39+
"existing_cluster_id": "0717-132531-5opeqon1",
40+
"for_each_task": {
41+
"inputs": "[1]",
42+
"task": {
43+
"existing_cluster_id": "0717-132531-5opeqon1",
44+
"libraries": [
45+
{
46+
"whl": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal/my_test_code-0.0.1+[UNIX_TIME_NANOS][0]-py3-none-any.whl"
47+
}
48+
],
49+
"python_wheel_task": {
50+
"entry_point": "run",
51+
"package_name": "my_test_code"
52+
},
53+
"task_key": "SubTask"
54+
}
55+
},
56+
"libraries": [
57+
{
58+
"whl": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal/my_test_code-0.0.1+[UNIX_TIME_NANOS][0]-py3-none-any.whl"
59+
},
60+
{
61+
"whl": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal/other_test_code-0.0.1+[UNIX_TIME_NANOS][1]-py3-none-any.whl"
62+
}
63+
],
64+
"python_wheel_task": {
65+
"entry_point": "run",
66+
"package_name": "my_test_code"
67+
},
68+
"task_key": "TestTask"
69+
}
70+
]
71+
}
72+
}
73+
}
74+
}
75+
}

‎acceptance/bundle/artifacts/whl_dynamic/out.plan_create.terraform.json‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
{
2+
"plan": {
3+
"resources.jobs.test_job": {
4+
"action": "create"
5+
}
6+
}
7+
}

‎acceptance/bundle/artifacts/whl_dynamic/out.plan_update.direct-exp.json‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
{
2+
"plan": {
3+
"resources.jobs.test_job": {
4+
"action": "update",
5+
"new_state": {
6+
"config": {
7+
"deployment": {
8+
"kind": "BUNDLE",
9+
"metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/metadata.json"
10+
},
11+
"edit_mode": "UI_LOCKED",
12+
"environments": [
13+
{
14+
"environment_key": "test_env",
15+
"spec": {
16+
"client": "1",
17+
"dependencies": [
18+
"/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal/my_test_code-0.0.1+[UNIX_TIME_NANOS][0]-py3-none-any.whl"
19+
]
20+
}
21+
}
22+
],
23+
"format": "MULTI_TASK",
24+
"max_concurrent_runs": 1,
25+
"name": "[default] My Wheel Job",
26+
"queue": {
27+
"enabled": true
28+
},
29+
"tasks": [
30+
{
31+
"environment_key": "test_env",
32+
"python_wheel_task": {
33+
"entry_point": "run",
34+
"package_name": "my_test_code"
35+
},
36+
"task_key": "ServerlessTestTask"
37+
},
38+
{
39+
"existing_cluster_id": "0717-132531-5opeqon1",
40+
"for_each_task": {
41+
"inputs": "[1]",
42+
"task": {
43+
"existing_cluster_id": "0717-132531-5opeqon1",
44+
"libraries": [
45+
{
46+
"whl": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal/my_test_code-0.0.1+[UNIX_TIME_NANOS][0]-py3-none-any.whl"
47+
}
48+
],
49+
"python_wheel_task": {
50+
"entry_point": "run",
51+
"package_name": "my_test_code"
52+
},
53+
"task_key": "SubTask"
54+
}
55+
},
56+
"libraries": [
57+
{
58+
"whl": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal/my_test_code-0.0.1+[UNIX_TIME_NANOS][0]-py3-none-any.whl"
59+
},
60+
{
61+
"whl": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal/other_test_code-0.0.1+[UNIX_TIME_NANOS][1]-py3-none-any.whl"
62+
}
63+
],
64+
"python_wheel_task": {
65+
"entry_point": "run",
66+
"package_name": "my_test_code"
67+
},
68+
"task_key": "TestTask"
69+
}
70+
]
71+
}
72+
},
73+
"changes": {
74+
"local": {
75+
"environments[0].spec.dependencies[0]": {
76+
"action": "update"
77+
},
78+
"tasks[1].for_each_task.task.libraries[0].whl": {
79+
"action": "update"
80+
},
81+
"tasks[1].libraries[0].whl": {
82+
"action": "update"
83+
}
84+
}
85+
}
86+
}
87+
}
88+
}

‎acceptance/bundle/artifacts/whl_dynamic/out.plan_update.terraform.json‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
{
2+
"plan": {
3+
"resources.jobs.test_job": {
4+
"action": "update"
5+
}
6+
}
7+
}

‎acceptance/bundle/artifacts/whl_dynamic/output.txt‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@
2323
"type": "whl"
2424
}
2525
}
26+
Building my_test_code...
2627

2728
>>> [CLI] bundle deploy
2829
Building my_test_code...
@@ -103,6 +104,8 @@ my_test_code-0.0.1+[UNIX_TIME_NANOS][1].dist-info/RECORD
103104
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/prebuilt/other_test_code-0.0.1-py3-none-any.whl"
104105

105106
=== Updating the local wheel and deploying again
107+
Building my_test_code...
108+
106109
>>> [CLI] bundle deploy
107110
Building my_test_code...
108111
Uploading .databricks/bundle/default/patched_wheels/my_prebuilt_whl_other_test_code/other_test_code-0.0.1+[UNIX_TIME_NANOS][0]-py3-none-any.whl...

‎acceptance/bundle/artifacts/whl_dynamic/script‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ cp -r $TESTDIR/../whl_prebuilt_multiple/dist/lib/other_test_code-0.0.1-py3-none-
66

77
trace $CLI bundle validate -o json | jq .artifacts
88

9+
$CLI bundle debug plan > out.plan_create.$DATABRICKS_BUNDLE_ENGINE.json
910
trace $CLI bundle deploy
1011

1112
title "There are 2 original wheels and 2 patched ones"
@@ -22,8 +23,9 @@ trace jq .path < out.requests.txt | grep import | grep whl | sort
2223

2324
rm out.requests.txt
2425

25-
title "Updating the local wheel and deploying again"
26+
title "Updating the local wheel and deploying again\n"
2627
touch my_test_code/src/new_module.py
28+
$CLI bundle debug plan > out.plan_update.$DATABRICKS_BUNDLE_ENGINE.json
2729
trace $CLI bundle deploy
2830

2931
title "Verify contents, it should now have new_module.py"

‎acceptance/bundle/bundle_tag/url_ref/out.deploy.direct-exp.txt‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,5 @@
11
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
2-
Error: cannot resolve "${resources.jobs.foo.url}": schema mismatch: url: field "url" not found in resources.Job; url: field "url" not found in jobs.Job
3-
4-
Error: cannot plan resources.jobs.bar: dependency failed: resources.jobs.foo
2+
Error: cannot plan resources.jobs.bar: cannot resolve "${resources.jobs.foo.url}": schema mismatch: url: field "url" not found in jobs.JobSettings; url: field "url" not found in jobs.Job
53

64
Error: planning failed
75

‎acceptance/bundle/bundle_tag/url_ref/out.plan.direct-exp.txt‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,4 @@
1-
Error: cannot resolve "${resources.jobs.foo.url}": schema mismatch: url: field "url" not found in resources.Job; url: field "url" not found in jobs.Job
2-
3-
Error: cannot plan resources.jobs.bar: dependency failed: resources.jobs.foo
1+
Error: cannot plan resources.jobs.bar: cannot resolve "${resources.jobs.foo.url}": schema mismatch: url: field "url" not found in jobs.JobSettings; url: field "url" not found in jobs.Job
42

53
Error: planning failed
64

‎acceptance/bundle/deploy/models/basic/out.plan_update_description.direct-exp.json‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
{
2+
"plan": {
3+
"resources.models.my_model": {
4+
"action": "update",
5+
"new_state": {
6+
"config": {
7+
"description": "new-description-[UNIQUE_NAME]",
8+
"name": "original-name-[UNIQUE_NAME]",
9+
"tags": [
10+
{
11+
"key": "key1",
12+
"value": "value1"
13+
}
14+
]
15+
}
16+
},
17+
"changes": {
18+
"local": {
19+
"description": {
20+
"action": "update"
21+
}
22+
}
23+
}
24+
}
25+
}
26+
}

‎acceptance/bundle/deploy/models/basic/out.plan_update_description.terraform.json‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
{
2+
"plan": {
3+
"resources.models.my_model": {
4+
"action": "update"
5+
}
6+
}
7+
}

0 commit comments

Comments
 (0)