Skip to content

Commit 8f66fb8

Browse files
denikdenik
authored
direct: Support "bundle deploy --plan" to deploy previously generated plans (#4134)
## Changes New `--readplan` option to `bundle deploy` that allows executing previously saved plan (`bundle plan -o json > myplan.json`). In this case, configuration to deploy is taken from the plan, not from databricks.yml. ## Why Enable workflow where you can review the changes before that are deployed. It's also faster, does not repeat plan operation as part of deploy. Note, currently it still requires to pass -t/--target. It also parses databricks.yml and goes through Initialize phase because that where get host, remote folders etc. In the future we might store those on plan so that we can run this without databricks.yml at all. Implements #4094 ## Tests Some tests are modified to run with and without `--readplan`. There is env var matrix variable READPLAN that controls this. Additional test helper readplanarg that expands into --readplan based on env var. Test runner knows that terraform does not support it and skips that configuration.
1 parent 5d64303 commit 8f66fb8

File tree

170 files changed

+1202
-161
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

170 files changed

+1202
-161
lines changed

‎NEXT_CHANGELOG.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,9 +9,10 @@
99

1010
### Bundles
1111
* Add support for configurable catalog/schema for dashboards ([#4130](https://github.com/databricks/cli/pull/4130))
12-
* engine/direct: Fix dependency-ordered deletion by persisting depends_on in state ([#4105](https://github.com/databricks/cli/pull/4105))
13-
* Pass SYSTEM_ACCESSTOKEN from env to the Terraform provider ([#4135](https://github.com/databricks/cli/pull/4135)
12+
* Pass SYSTEM\_ACCESSTOKEN from env to the Terraform provider ([#4135](https://github.com/databricks/cli/pull/4135))
1413
* `bundle deployment migrate`: when running `bundle plan` propagate `-var` arguments.
14+
* engine/direct: New option --plan to `bundle deploy` to deploy previously saved plan (saved plan with `bundle plan -o json`) ([#4134](https://github.com/databricks/cli/pull/4134))
15+
* engine/direct: Fix dependency-ordered deletion by persisting depends\_on in state ([#4105](https://github.com/databricks/cli/pull/4105))
1516

1617
### Dependency updates
1718
* Upgrade Go SDK to 0.94.0 ([#4148](https://github.com/databricks/cli/pull/4148))

‎acceptance/acceptance_test.go‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -336,7 +336,7 @@ func testAccept(t *testing.T, inprocessMode bool, singleTest string) int {
336336
if testdiff.OverwriteMode && len(expanded) > 1 {
337337
// All variants of the test are producing the same output,
338338
// there is no need to run the concurrently when updating.
339-
// Exception: if EnvVaryOutput is configured with multiple values, we must
339+
// Exception: if EnvVaryOutput is configured, we must
340340
// run all variants to record variant-specific outputs.
341341
if config.EnvVaryOutput == nil || len(config.EnvMatrix[*config.EnvVaryOutput]) <= 1 {
342342
expanded = expanded[0:1]
@@ -351,6 +351,9 @@ func testAccept(t *testing.T, inprocessMode bool, singleTest string) int {
351351
runTest(t, dir, 0, coverDir, repls.Clone(), config, expanded[0], envFilters)
352352
} else {
353353
for ind, envset := range expanded {
354+
if forbiddenEnvSet(envset) {
355+
continue
356+
}
354357
envname := strings.Join(envset, "/")
355358
t.Run(envname, func(t *testing.T) {
356359
if !inprocessMode {
@@ -368,6 +371,23 @@ func testAccept(t *testing.T, inprocessMode bool, singleTest string) int {
368371
return selectedDirs - skippedDirs
369372
}
370373

374+
func forbiddenEnvSet(envset []string) bool {
375+
hasTerraform := false
376+
hasReadplan := false
377+
378+
for _, pair := range envset {
379+
if pair == "DATABRICKS_BUNDLE_ENGINE=terraform" {
380+
hasTerraform = true
381+
}
382+
if pair == "READPLAN=1" {
383+
hasReadplan = true
384+
}
385+
}
386+
387+
// Do not run terraform tests with --plan option:
388+
return hasTerraform && hasReadplan
389+
}
390+
371391
func getEnvFilters(t *testing.T) []string {
372392
envFilterValue := os.Getenv(EnvFilterVar)
373393
if envFilterValue == "" {

‎acceptance/bundle/artifacts/whl_dynamic/out.plan_update.direct.json‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
{
22
"plan_version": 1,
33
"cli_version": "[DEV_VERSION]",
4+
"lineage": "[UUID]",
5+
"serial": 1,
46
"plan": {
57
"resources.jobs.test_job": {
68
"action": "update",

‎acceptance/bundle/deploy/readplan/basic/databricks.yml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
bundle:
2+
name: test-bundle
3+
4+
resources:
5+
jobs:
6+
job:
7+
name: test-job

‎acceptance/bundle/deploy/readplan/basic/out.plan_create.json‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
{
2+
"plan_version": 1,
3+
"cli_version": "[DEV_VERSION]",
4+
"plan": {
5+
"resources.jobs.job": {
6+
"action": "create",
7+
"new_state": {
8+
"value": {
9+
"deployment": {
10+
"kind": "BUNDLE",
11+
"metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/metadata.json"
12+
},
13+
"edit_mode": "UI_LOCKED",
14+
"format": "MULTI_TASK",
15+
"max_concurrent_runs": 1,
16+
"name": "test-job",
17+
"queue": {
18+
"enabled": true
19+
}
20+
}
21+
}
22+
}
23+
}
24+
}

‎acceptance/bundle/deploy/readplan/basic/out.plan_skip.json‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
{
2+
"plan_version": 1,
3+
"cli_version": "[DEV_VERSION]",
4+
"lineage": "[UUID]",
5+
"serial": 1,
6+
"plan": {
7+
"resources.jobs.job": {
8+
"action": "skip",
9+
"remote_state": {
10+
"created_time": [UNIX_TIME_MILLIS],
11+
"creator_user_name": "[USERNAME]",
12+
"job_id": [NUMID],
13+
"run_as_user_name": "[USERNAME]",
14+
"settings": {
15+
"deployment": {
16+
"kind": "BUNDLE",
17+
"metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/metadata.json"
18+
},
19+
"edit_mode": "UI_LOCKED",
20+
"email_notifications": {},
21+
"format": "MULTI_TASK",
22+
"max_concurrent_runs": 1,
23+
"name": "test-job",
24+
"queue": {
25+
"enabled": true
26+
},
27+
"timeout_seconds": 0,
28+
"webhook_notifications": {}
29+
}
30+
},
31+
"changes": {
32+
"remote": {
33+
"email_notifications": {
34+
"action": "skip",
35+
"reason": "server_side_default"
36+
},
37+
"timeout_seconds": {
38+
"action": "skip",
39+
"reason": "server_side_default"
40+
},
41+
"webhook_notifications": {
42+
"action": "skip",
43+
"reason": "server_side_default"
44+
}
45+
}
46+
}
47+
}
48+
}
49+
}

‎acceptance/bundle/deploy/readplan/basic/out.test.toml‎

Lines changed: 5 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎acceptance/bundle/deploy/readplan/basic/output.txt‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
2+
>>> [CLI] bundle plan -o json
3+
4+
>>> print_requests.py --get //jobs
5+
6+
>>> [CLI] bundle deploy --plan out.plan_create.json
7+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
8+
Deploying resources...
9+
Updating deployment state...
10+
Deployment complete!
11+
12+
>>> print_requests.py --get //jobs
13+
{
14+
"method": "POST",
15+
"path": "/api/2.2/jobs/create",
16+
"body": {
17+
"deployment": {
18+
"kind": "BUNDLE",
19+
"metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/metadata.json"
20+
},
21+
"edit_mode": "UI_LOCKED",
22+
"format": "MULTI_TASK",
23+
"max_concurrent_runs": 1,
24+
"name": "test-job",
25+
"queue": {
26+
"enabled": true
27+
}
28+
}
29+
}
30+
31+
>>> [CLI] bundle plan -o json
32+
33+
>>> print_requests.py --get //jobs
34+
{
35+
"method": "GET",
36+
"path": "/api/2.2/jobs/get",
37+
"q": {
38+
"job_id": "[NUMID]"
39+
}
40+
}
41+
42+
>>> [CLI] bundle deploy --plan out.plan_skip.json
43+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
44+
Deploying resources...
45+
Updating deployment state...
46+
Deployment complete!
47+
48+
>>> print_requests.py --get //jobs
49+
50+
>>> [CLI] bundle destroy --auto-approve
51+
The following resources will be deleted:
52+
delete resources.jobs.job
53+
54+
All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default
55+
56+
Deleting files...
57+
Destroy complete!

‎acceptance/bundle/deploy/readplan/basic/script‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
# Generate initial plan
2+
trace $CLI bundle plan -o json > out.plan_create.json
3+
trace print_requests.py --get //jobs | contains.py "!GET" "!POST"
4+
5+
# First deploy to create the resource
6+
trace $CLI bundle deploy --plan out.plan_create.json
7+
trace print_requests.py --get //jobs | contains.py "!GET" "POST"
8+
9+
# Generate a plan (should show "skip" since nothing changed)
10+
trace $CLI bundle plan -o json > out.plan_skip.json
11+
trace print_requests.py --get //jobs | contains.py "GET"
12+
13+
trace $CLI bundle deploy --plan out.plan_skip.json
14+
# Show what requests were made (should not create/update job again; should not read remote state)
15+
trace print_requests.py --get //jobs | contains.py "!GET" "!POST"
16+
17+
trace $CLI bundle destroy --auto-approve
18+
rm out.requests.txt

‎acceptance/bundle/deploy/readplan/basic/test.toml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# Test basic --plan functionality
2+
EnvMatrix.DATABRICKS_BUNDLE_ENGINE = ["direct"]
3+
RecordRequests = true

0 commit comments

Comments
 (0)