Skip to content

Commit e1d9bf4

Browse files
denikdenik
authored
Support "bundle plan -o json"; refactor to use Plan internally (#3598)
## Changes - New struct deployplan.Plan to carry the deployment plan. Today it contains only actions per resource but eventually it will have all info necessary for deployment. - Use this struct in place of []deployplan.Action everywhere. - Add "bundle plan -o json" option to dump this plan to stdout. - Switched "bundle plan" to stdout as well. ## Why Eventually for direct deployment the plan will become executable and the only source of deployment info (we will no read the config of resources anymore once we have the plan). This will make plan the true state of deployment and will aid in the debugging, it will also allow implementing "bundle deploy --readplan myplan.json" feature. The plan will also include per field details, explaining why an action was taken. This PR prepares for that, at this moment plan contains only action per resource. ## Tests Some acceptance tests expanded to include json plan in the output.
1 parent ab83bd4 commit e1d9bf4

File tree

24 files changed

+450
-205
lines changed

24 files changed

+450
-205
lines changed

‎acceptance/bundle/deploy/mlops-stacks/output.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -129,9 +129,9 @@ Warning: unknown field: description
129129
in resources/ml-artifacts-resource.yml:21:7
130130

131131
The following resources will be deleted:
132+
delete experiment experiment
132133
delete job batch_inference_job
133134
delete job model_training_job
134-
delete experiment experiment
135135
delete model model
136136

137137
All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/project_name_[UNIQUE_NAME]/dev

‎acceptance/bundle/resources/jobs/update/output.txt‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,15 @@
22
>>> [CLI] bundle plan
33
create jobs.foo
44

5+
>>> [CLI] bundle plan -o json
6+
{
7+
"plan": {
8+
"resources.jobs.foo": {
9+
"action": "create"
10+
}
11+
}
12+
}
13+
514
>>> [CLI] bundle deploy
615
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
716
Deploying resources...
@@ -10,6 +19,11 @@ Deployment complete!
1019

1120
>>> [CLI] bundle plan
1221

22+
>>> [CLI] bundle plan -o json
23+
{
24+
"plan": {}
25+
}
26+
1327
>>> print_requests
1428
{
1529
"body": {
@@ -52,6 +66,15 @@ jobs foo id='[JOB_ID]' name='foo'
5266
>>> [CLI] bundle plan
5367
update jobs.foo
5468

69+
>>> [CLI] bundle plan -o json
70+
{
71+
"plan": {
72+
"resources.jobs.foo": {
73+
"action": "update(id_stable)"
74+
}
75+
}
76+
}
77+
5578
>>> [CLI] bundle deploy
5679
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
5780
Deploying resources...
@@ -60,6 +83,11 @@ Deployment complete!
6083

6184
>>> [CLI] bundle plan
6285

86+
>>> [CLI] bundle plan -o json
87+
{
88+
"plan": {}
89+
}
90+
6391
>>> print_requests
6492
{
6593
"body": {

‎acceptance/bundle/resources/jobs/update/script‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,9 @@
11
echo "*" > .gitignore
22
trace $CLI bundle plan
3+
trace $CLI bundle plan -o json
34
trace $CLI bundle deploy
45
trace $CLI bundle plan
6+
trace $CLI bundle plan -o json
57

68
print_requests() {
79
jq --sort-keys 'select(.method != "GET" and (.path | contains("/jobs")))' < out.requests.txt
@@ -14,8 +16,10 @@ trace print_requests
1416
title "Update trigger.periodic.unit and re-deploy"
1517
trace update_file.py databricks.yml DAYS HOURS
1618
trace $CLI bundle plan
19+
trace $CLI bundle plan -o json
1720
trace $CLI bundle deploy
1821
trace $CLI bundle plan
22+
trace $CLI bundle plan -o json
1923
trace print_requests
2024

2125
title "Fetch job ID and verify remote state"

‎acceptance/bundle/resources/pipelines/recreate/_script‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ trace cat databricks.yml
22
touch foo.py
33
touch bar.py
44
trace $CLI bundle plan # should show 'create'
5+
trace $CLI bundle plan -o json
56
trace $CLI bundle deploy
67

78
ppid1=`read_id.py pipelines my`
@@ -16,6 +17,7 @@ trace print_requests
1617

1718
trace update_file.py databricks.yml $CONFIG_UPDATE
1819
trace $CLI bundle plan # should show 'recreate'
20+
trace $CLI bundle plan -o json
1921
trace $CLI bundle deploy --auto-approve
2022
trace print_requests
2123

‎acceptance/bundle/resources/pipelines/recreate/change-catalog/output.txt‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,15 @@ resources:
1717
>>> [CLI] bundle plan
1818
create pipelines.my
1919

20+
>>> [CLI] bundle plan -o json
21+
{
22+
"plan": {
23+
"resources.pipelines.my": {
24+
"action": "create"
25+
}
26+
}
27+
}
28+
2029
>>> [CLI] bundle deploy
2130
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/acc-[UNIQUE_NAME]/default/files...
2231
Deploying resources...
@@ -51,6 +60,15 @@ Deployment complete!
5160
>>> [CLI] bundle plan
5261
recreate pipelines.my
5362

63+
>>> [CLI] bundle plan -o json
64+
{
65+
"plan": {
66+
"resources.pipelines.my": {
67+
"action": "recreate"
68+
}
69+
}
70+
}
71+
5472
>>> [CLI] bundle deploy --auto-approve
5573
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/acc-[UNIQUE_NAME]/default/files...
5674

‎acceptance/bundle/resources/pipelines/recreate/change-ingestion-definition/output.txt‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,15 @@ resources:
1717
>>> [CLI] bundle plan
1818
create pipelines.my
1919

20+
>>> [CLI] bundle plan -o json
21+
{
22+
"plan": {
23+
"resources.pipelines.my": {
24+
"action": "create"
25+
}
26+
}
27+
}
28+
2029
>>> [CLI] bundle deploy
2130
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/acc-[UNIQUE_NAME]/default/files...
2231
Deploying resources...
@@ -56,6 +65,15 @@ Deployment complete!
5665
>>> [CLI] bundle plan
5766
recreate pipelines.my
5867

68+
>>> [CLI] bundle plan -o json
69+
{
70+
"plan": {
71+
"resources.pipelines.my": {
72+
"action": "recreate"
73+
}
74+
}
75+
}
76+
5977
>>> [CLI] bundle deploy --auto-approve
6078
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/acc-[UNIQUE_NAME]/default/files...
6179

‎acceptance/bundle/resources/pipelines/recreate/change-storage/output.txt‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,15 @@ resources:
1717
>>> [CLI] bundle plan
1818
create pipelines.my
1919

20+
>>> [CLI] bundle plan -o json
21+
{
22+
"plan": {
23+
"resources.pipelines.my": {
24+
"action": "create"
25+
}
26+
}
27+
}
28+
2029
>>> [CLI] bundle deploy
2130
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/acc-[UNIQUE_NAME]/default/files...
2231
Deploying resources...
@@ -51,6 +60,15 @@ Deployment complete!
5160
>>> [CLI] bundle plan
5261
recreate pipelines.my
5362

63+
>>> [CLI] bundle plan -o json
64+
{
65+
"plan": {
66+
"resources.pipelines.my": {
67+
"action": "recreate"
68+
}
69+
}
70+
}
71+
5472
>>> [CLI] bundle deploy --auto-approve
5573
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/acc-[UNIQUE_NAME]/default/files...
5674

‎acceptance/bundle/resources/volumes/change-name/out.plan.direct-exp.txt‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
{
2+
"plan": {
3+
"resources.volumes.volume1": {
4+
"action": "update(id_changes)"
5+
}
6+
}
7+
}

‎acceptance/bundle/resources/volumes/change-name/out.plan.terraform.txt‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
{
2+
"plan": {
3+
"resources.volumes.volume1": {
4+
"action": "update(id_stable)"
5+
}
6+
}
7+
}

‎acceptance/bundle/resources/volumes/change-name/output.txt‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,8 @@ Deployment complete!
3939
>>> [CLI] bundle plan
4040
update volumes.volume1
4141

42+
>>> [CLI] bundle plan -o json
43+
4244
>>> [CLI] bundle deploy
4345
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
4446
Deploying resources...

0 commit comments

Comments
 (0)