Skip to content

Commit 1a39e26

Browse files
denikdenik
committed
update
1 parent d00f343 commit 1a39e26

File tree

17 files changed

+454
-30
lines changed

17 files changed

+454
-30
lines changed

‎acceptance/bundle/deployment/migrate/basic/databricks.yml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
bundle:
2+
name: migrate-basic-test
3+
4+
resources:
5+
jobs:
6+
test_job:
7+
name: "Test Migration Job"
8+
tasks:
9+
- task_key: "main"
10+
notebook_task:
11+
notebook_path: "./notebook.py"
12+
#permissions:
13+
# - level: CAN_VIEW
14+
# user_name: [email protected]
15+
16+
pipelines:
17+
test_pipeline:
18+
name: "Test Migration Pipeline"
19+
libraries:
20+
- notebook:
21+
path: "./pipeline.py"
22+
#permissions:
23+
# - level: CAN_MANAGE
24+
# user_name: [email protected]

‎acceptance/bundle/deployment/migrate/basic/notebook.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# Databricks notebook source
2+
print("Hello from test migration job")

‎acceptance/bundle/deployment/migrate/basic/out.test.toml‎

Lines changed: 5 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎acceptance/bundle/deployment/migrate/basic/output.txt‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
2+
>>> musterr [CLI] bundle deployment migrate
3+
Error: no existing state found
4+
5+
>>> DATABRICKS_BUNDLE_ENGINE=terraform [CLI] bundle deploy
6+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/migrate-basic-test/default/files...
7+
Deploying resources...
8+
Updating deployment state...
9+
Deployment complete!
10+
11+
>>> head -n 5 .databricks/bundle/default/terraform/terraform.tfstate
12+
{
13+
"version": 4,
14+
"terraform_version": "1.5.5",
15+
"serial": 3,
16+
"lineage": "[UUID]",
17+
18+
>>> [CLI] bundle deployment migrate
19+
Migrated 2 resources to direct engine state file: [TEST_TMP_DIR]/.databricks/bundle/default/resources.json
20+
deployment.json
21+
resources.json
22+
sync-snapshots
23+
terraform
24+
25+
>>> cat .databricks/bundle/default/resources.json
26+
{
27+
"lineage": "[UUID]",
28+
"serial": 4,
29+
"state": {
30+
"resources.jobs.test_job": {
31+
"__id__": "[NUMID]",
32+
"state": {
33+
"deployment": {
34+
"kind": "BUNDLE",
35+
"metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/migrate-basic-test/default/state/metadata.json"
36+
},
37+
"edit_mode": "UI_LOCKED",
38+
"format": "MULTI_TASK",
39+
"max_concurrent_runs": 1,
40+
"name": "Test Migration Job",
41+
"queue": {
42+
"enabled": true
43+
},
44+
"tasks": [
45+
{
46+
"notebook_task": {
47+
"notebook_path": "/Workspace/Users/[USERNAME]/.bundle/migrate-basic-test/default/files/notebook"
48+
},
49+
"task_key": "main"
50+
}
51+
]
52+
}
53+
},
54+
"resources.pipelines.test_pipeline": {
55+
"__id__": "[UUID]",
56+
"state": {
57+
"channel": "CURRENT",
58+
"deployment": {
59+
"kind": "BUNDLE",
60+
"metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/migrate-basic-test/default/state/metadata.json"
61+
},
62+
"edition": "ADVANCED",
63+
"libraries": [
64+
{
65+
"notebook": {
66+
"path": "/Workspace/Users/[USERNAME]/.bundle/migrate-basic-test/default/files/pipeline"
67+
}
68+
}
69+
],
70+
"name": "Test Migration Pipeline"
71+
}
72+
}
73+
}
74+
}
75+
>>> [CLI] bundle plan
76+
Plan: 0 to add, 0 to change, 0 to delete, 2 unchanged
77+
78+
>>> print_requests.py --get //jobs/get
79+
{
80+
"headers": {
81+
"User-Agent": [
82+
"cli/[DEV_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_plan cmd-exec-id/[UUID] engine/direct auth/pat"
83+
]
84+
},
85+
"method": "GET",
86+
"path": "/api/2.2/jobs/get",
87+
"q": {
88+
"job_id": "[NUMID]"
89+
}
90+
}
91+
92+
>>> [CLI] bundle deploy
93+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/migrate-basic-test/default/files...
94+
Deploying resources...
95+
Updating deployment state...
96+
Deployment complete!
97+
98+
>>> print_requests.py --get //jobs/get
99+
"cli/[DEV_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_deploy cmd-exec-id/[UUID] engine/direct auth/pat"
100+
101+
=== Should show that it's already migrated
102+
>>> musterr [CLI] bundle deployment migrate
103+
Error: already using direct engine

‎acceptance/bundle/deployment/migrate/basic/pipeline.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# Databricks notebook source
2+
# This is a test pipeline notebook

‎acceptance/bundle/deployment/migrate/basic/script‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# Migrating without state, should fail
2+
trace musterr $CLI bundle deployment migrate
3+
4+
trace DATABRICKS_BUNDLE_ENGINE=terraform $CLI bundle deploy
5+
trace head -n 5 .databricks/bundle/default/terraform/terraform.tfstate
6+
7+
trace $CLI bundle deployment migrate
8+
9+
ls -1 .databricks/bundle/default/ | contains.py "resources.json"
10+
trace cat .databricks/bundle/default/resources.json
11+
12+
rm out.requests.txt
13+
trace $CLI bundle plan
14+
trace print_requests.py --get //jobs/get | contains.py 'engine/direct'
15+
16+
trace $CLI bundle deploy
17+
trace print_requests.py --get //jobs/get | jq '.headers["User-Agent"][0]' | contains.py 'engine/direct'
18+
19+
title "Should show that it's already migrated"
20+
trace musterr $CLI bundle deployment migrate
21+
rm out.requests.txt

‎acceptance/bundle/deployment/migrate/basic/test.toml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
RecordRequests = true
2+
IncludeRequestHeaders = ["User-Agent"]

‎acceptance/bundle/help/bundle-deployment-migrate/out.test.toml‎

Lines changed: 5 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎acceptance/bundle/help/bundle-deployment-migrate/output.txt‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
2+
>>> [CLI] bundle deployment migrate --help
3+
Migrate bundle resources from Terraform deployment to Direct deployment.
4+
5+
This command converts your bundle from using Terraform for deployment to using
6+
the Direct deployment engine. It reads resource IDs from the existing Terraform
7+
state and creates a Direct deployment state file (resources.json) with the same
8+
lineage and incremented serial number.
9+
10+
The command will:
11+
1. Read the current Terraform state file (terraform.tfstate)
12+
2. Extract resource IDs and lineage information
13+
3. Create a Direct deployment state file (resources.json)
14+
4. Preserve the same lineage with serial number incremented by 1
15+
16+
After migration:
17+
- The CLI will automatically detect and use Direct deployment based on the
18+
presence of resources.json
19+
- Use 'bundle deploy' as usual - it will use Direct deployment engine
20+
- Your existing resources will be managed through Direct deployment
21+
22+
WARNING: After migration, don't switch back to Terraform deployment without
23+
proper state management as it may cause resource conflicts.
24+
25+
Usage:
26+
databricks bundle deployment migrate [flags]
27+
28+
Flags:
29+
--auto-approve Automatically approve the migration
30+
--force-lock Force acquisition of deployment lock.
31+
-h, --help help for migrate
32+
33+
Global Flags:
34+
--debug enable debug logging
35+
-o, --output type output type: text or json (default text)
36+
-p, --profile string ~/.databrickscfg profile
37+
-t, --target string bundle target to use (if applicable)
38+
--var strings set values for variables defined in bundle config. Example: --var="foo=bar"

‎acceptance/bundle/help/bundle-deployment-migrate/script‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
trace $CLI bundle deployment migrate --help

0 commit comments

Comments
 (0)