Skip to content

Commit c4e1b83

Browse files
denikdenik
authored
direct: New plan format and logic re remote changes (#4201)
## Changes - When remote value matches new config value, we now assign "skip" action to this field (with reason="remote_already_set"). Previously if there were changes in the local config we would do 'update' regardless of what remote value is. - New plan format for changes. Instead of separate "local" and "remote" blocks we know record change per path with 3 values: old (prior state), new (config), remote. - Note, this is breaking change if you parse plan json. Version will be updated to 2 in follow up PR to make it easier to review. Internally, there is new resource API for overriding diff logic. New function OverrideChangeDesc allows updating ChangeDesc (which now includes Old/New/Remote fields) after built-in logic and FieldTriggers() were applied. Additionally, - fixed some error messages’s prefixes to describe action, not include “failed” or “error”. - fieldTriggers isLocal parameter is removed. It’s not needed in most cases and if complex logic is needed, OverrideChangeDesc is a better place. - calladapt allows passing nil to parameters requiring *T. In this case, nil will be converted to typed nil. ## Plan format ## Old format (plan_version: 1) Assuming field has value “A” in the state and “B” in the config and “C” in the remote state. ``` changes: local: field: old: A new: B remote: field: old: B new: C ``` (old format has old & new swapped in remote, which was somewhat confusing). ### New format (plan_version: 2) ``` changes: field: old: A new: B remote: C ``` ## Why Matches terraform and user expectations. Plan format is easier to understand. ## Tests There is a new acceptance test showing remote_already_set behaviour. Existing tests show format change.
1 parent 04d744a commit c4e1b83

File tree

108 files changed

+2008
-1559
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

108 files changed

+2008
-1559
lines changed

‎NEXT_CHANGELOG.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
## Release v0.282.0
44

55
### Notable Changes
6+
* engine/direct: Plan format changed to 2, see ([#4201](https://github.com/databricks/cli/pull/4201))
67

78
### CLI
89
* Skip non-exportable objects (e.g., `MLFLOW_EXPERIMENT`) during `workspace export-dir` instead of failing ([#4081](https://github.com/databricks/cli/issues/4081))
@@ -11,10 +12,11 @@
1112
### Bundles
1213
* Pass SYSTEM_ACCESSTOKEN from env to the Terraform provider ([#4135](https://github.com/databricks/cli/pull/4135))
1314
* Added missing schema grants privileges ([#4139](https://github.com/databricks/cli/pull/4139))
14-
* Fix app deployment failure when app is in `DELETING` state ([#4176](https://github.com/databricks/cli/pull/4176))
1515
* Add `ipykernel` to the `default` template to enable Databricks Connect notebooks in Cursor/VS Code ([#4164](https://github.com/databricks/cli/pull/4164))
1616
* Add interactive SQL warehouse picker to `default-sql` and `dbt-sql` bundle templates ([#4170](https://github.com/databricks/cli/pull/4170))
1717
* Add `name`, `target` and `mode` fields to the deployment metadata file ([#4180](https://github.com/databricks/cli/pull/4180))
18+
* engine/direct: Fix app deployment failure when app is in `DELETING` state ([#4176](https://github.com/databricks/cli/pull/4176))
19+
* engine/direct: Changes in config that match remote changes no longer trigger an update ([#4201](https://github.com/databricks/cli/pull/4201))
1820

1921
### Dependency updates
2022

‎acceptance/bundle/artifacts/whl_dynamic/out.plan_update.direct.json‎

Lines changed: 31 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -150,36 +150,38 @@
150150
}
151151
},
152152
"changes": {
153-
"local": {
154-
"environments[0].spec.dependencies[0]": {
155-
"action": "update",
156-
"old": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal/my_test_code-0.0.1+[UNIX_TIME_NANOS][2]-py3-none-any.whl",
157-
"new": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal/my_test_code-0.0.1+[UNIX_TIME_NANOS][0]-py3-none-any.whl"
158-
},
159-
"tasks[task_key='TestTask'].for_each_task.task.libraries[0].whl": {
160-
"action": "update",
161-
"old": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal/my_test_code-0.0.1+[UNIX_TIME_NANOS][2]-py3-none-any.whl",
162-
"new": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal/my_test_code-0.0.1+[UNIX_TIME_NANOS][0]-py3-none-any.whl"
163-
},
164-
"tasks[task_key='TestTask'].libraries[0].whl": {
165-
"action": "update",
166-
"old": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal/my_test_code-0.0.1+[UNIX_TIME_NANOS][2]-py3-none-any.whl",
167-
"new": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal/my_test_code-0.0.1+[UNIX_TIME_NANOS][0]-py3-none-any.whl"
168-
}
153+
"email_notifications": {
154+
"action": "skip",
155+
"reason": "server_side_default",
156+
"remote": {}
169157
},
170-
"remote": {
171-
"email_notifications": {
172-
"action": "skip",
173-
"reason": "server_side_default"
174-
},
175-
"timeout_seconds": {
176-
"action": "skip",
177-
"reason": "server_side_default"
178-
},
179-
"webhook_notifications": {
180-
"action": "skip",
181-
"reason": "server_side_default"
182-
}
158+
"environments[0].spec.dependencies[0]": {
159+
"action": "update",
160+
"old": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal/my_test_code-0.0.1+[UNIX_TIME_NANOS][2]-py3-none-any.whl",
161+
"new": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal/my_test_code-0.0.1+[UNIX_TIME_NANOS][0]-py3-none-any.whl",
162+
"remote": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal/my_test_code-0.0.1+[UNIX_TIME_NANOS][2]-py3-none-any.whl"
163+
},
164+
"tasks[task_key='TestTask'].for_each_task.task.libraries[0].whl": {
165+
"action": "update",
166+
"old": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal/my_test_code-0.0.1+[UNIX_TIME_NANOS][2]-py3-none-any.whl",
167+
"new": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal/my_test_code-0.0.1+[UNIX_TIME_NANOS][0]-py3-none-any.whl",
168+
"remote": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal/my_test_code-0.0.1+[UNIX_TIME_NANOS][2]-py3-none-any.whl"
169+
},
170+
"tasks[task_key='TestTask'].libraries[0].whl": {
171+
"action": "update",
172+
"old": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal/my_test_code-0.0.1+[UNIX_TIME_NANOS][2]-py3-none-any.whl",
173+
"new": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal/my_test_code-0.0.1+[UNIX_TIME_NANOS][0]-py3-none-any.whl",
174+
"remote": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal/my_test_code-0.0.1+[UNIX_TIME_NANOS][2]-py3-none-any.whl"
175+
},
176+
"timeout_seconds": {
177+
"action": "skip",
178+
"reason": "server_side_default",
179+
"remote": 0
180+
},
181+
"webhook_notifications": {
182+
"action": "skip",
183+
"reason": "server_side_default",
184+
"remote": {}
183185
}
184186
}
185187
}

‎acceptance/bundle/deploy/readplan/basic/out.plan_skip.json‎

Lines changed: 14 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -29,19 +29,20 @@
2929
}
3030
},
3131
"changes": {
32-
"remote": {
33-
"email_notifications": {
34-
"action": "skip",
35-
"reason": "server_side_default"
36-
},
37-
"timeout_seconds": {
38-
"action": "skip",
39-
"reason": "server_side_default"
40-
},
41-
"webhook_notifications": {
42-
"action": "skip",
43-
"reason": "server_side_default"
44-
}
32+
"email_notifications": {
33+
"action": "skip",
34+
"reason": "server_side_default",
35+
"remote": {}
36+
},
37+
"timeout_seconds": {
38+
"action": "skip",
39+
"reason": "server_side_default",
40+
"remote": 0
41+
},
42+
"webhook_notifications": {
43+
"action": "skip",
44+
"reason": "server_side_default",
45+
"remote": {}
4546
}
4647
}
4748
}

‎acceptance/bundle/migrate/basic/out.plan_update.json‎

Lines changed: 43 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -61,30 +61,31 @@
6161
}
6262
},
6363
"changes": {
64-
"local": {
65-
"name": {
66-
"action": "update",
67-
"old": "Test Migration Job",
68-
"new": "Test Migrated Job"
69-
}
64+
"email_notifications": {
65+
"action": "skip",
66+
"reason": "server_side_default",
67+
"remote": {}
7068
},
71-
"remote": {
72-
"email_notifications": {
73-
"action": "skip",
74-
"reason": "server_side_default"
75-
},
76-
"tasks[task_key='main'].notebook_task.source": {
77-
"action": "skip",
78-
"reason": "server_side_default"
79-
},
80-
"timeout_seconds": {
81-
"action": "skip",
82-
"reason": "server_side_default"
83-
},
84-
"webhook_notifications": {
85-
"action": "skip",
86-
"reason": "server_side_default"
87-
}
69+
"name": {
70+
"action": "update",
71+
"old": "Test Migration Job",
72+
"new": "Test Migrated Job",
73+
"remote": "Test Migration Job"
74+
},
75+
"tasks[task_key='main'].notebook_task.source": {
76+
"action": "skip",
77+
"reason": "server_side_default",
78+
"remote": "WORKSPACE"
79+
},
80+
"timeout_seconds": {
81+
"action": "skip",
82+
"reason": "server_side_default",
83+
"remote": 0
84+
},
85+
"webhook_notifications": {
86+
"action": "skip",
87+
"reason": "server_side_default",
88+
"remote": {}
8889
}
8990
}
9091
},
@@ -177,23 +178,22 @@
177178
"state": "IDLE"
178179
},
179180
"changes": {
180-
"local": {
181-
"name": {
182-
"action": "update",
183-
"old": "Test Migration Pipeline",
184-
"new": "Test Migrated Pipeline"
185-
},
186-
"tags['myjob_name']": {
187-
"action": "update",
188-
"old": "Test Migration Job",
189-
"new": "Test Migrated Job"
190-
}
181+
"name": {
182+
"action": "update",
183+
"old": "Test Migration Pipeline",
184+
"new": "Test Migrated Pipeline",
185+
"remote": "Test Migration Pipeline"
191186
},
192-
"remote": {
193-
"storage": {
194-
"action": "skip",
195-
"reason": "server_side_default"
196-
}
187+
"storage": {
188+
"action": "skip",
189+
"reason": "server_side_default",
190+
"remote": "dbfs:/pipelines/[UUID]"
191+
},
192+
"tags['myjob_name']": {
193+
"action": "update",
194+
"old": "Test Migration Job",
195+
"new": "Test Migrated Job",
196+
"remote": "Test Migration Job"
197197
}
198198
}
199199
},
@@ -212,11 +212,10 @@
212212
"volume_type": "MANAGED"
213213
},
214214
"changes": {
215-
"remote": {
216-
"storage_location": {
217-
"action": "skip",
218-
"reason": "server_side_default"
219-
}
215+
"storage_location": {
216+
"action": "skip",
217+
"reason": "server_side_default",
218+
"remote": "s3://deco-uc-prod-isolated-aws-us-east-1/metastore/[UUID]/volumes/[UUID]"
220219
}
221220
}
222221
}

‎acceptance/bundle/migrate/dashboards/out.plan_after_migrate.json‎

Lines changed: 9 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -20,18 +20,16 @@
2020
"warehouse_id": "123456"
2121
},
2222
"changes": {
23-
"local": {
24-
"etag": {
25-
"action": "skip",
26-
"old": "[NUMID]"
27-
}
23+
"etag": {
24+
"action": "skip",
25+
"old": "[NUMID]",
26+
"remote": "[NUMID]"
2827
},
29-
"remote": {
30-
"serialized_dashboard": {
31-
"action": "skip",
32-
"old": "{\"pages\":[{\"name\":\"02724bf2\",\"displayName\":\"Dashboard test bundle-deploy-dashboard\"}]}\n",
33-
"new": "{\"pages\":[{\"displayName\":\"Dashboard test bundle-deploy-dashboard\",\"name\":\"02724bf2\",\"pageType\":\"PAGE_TYPE_CANVAS\"}]}\n"
34-
}
28+
"serialized_dashboard": {
29+
"action": "skip",
30+
"old": "{\"pages\":[{\"name\":\"02724bf2\",\"displayName\":\"Dashboard test bundle-deploy-dashboard\"}]}\n",
31+
"new": "{\"pages\":[{\"name\":\"02724bf2\",\"displayName\":\"Dashboard test bundle-deploy-dashboard\"}]}\n",
32+
"remote": "{\"pages\":[{\"displayName\":\"Dashboard test bundle-deploy-dashboard\",\"name\":\"02724bf2\",\"pageType\":\"PAGE_TYPE_CANVAS\"}]}\n"
3533
}
3634
}
3735
}

‎acceptance/bundle/migrate/default-python/out.plan_after_deploy.json‎

Lines changed: 34 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -217,35 +217,37 @@
217217
}
218218
},
219219
"changes": {
220-
"local": {
221-
"tasks[task_key='notebook_task'].libraries[0].whl": {
222-
"action": "update",
223-
"old": "/Workspace/Users/[USERNAME]/.bundle/my_default_python/dev/artifacts/.internal/my_default_python-0.0.1+[UNIX_TIME_NANOS][1]-py3-none-any.whl",
224-
"new": "/Workspace/Users/[USERNAME]/.bundle/my_default_python/dev/artifacts/.internal/my_default_python-0.0.1+[UNIX_TIME_NANOS][0]-py3-none-any.whl"
225-
},
226-
"tasks[task_key='python_wheel_task'].libraries[0].whl": {
227-
"action": "update",
228-
"old": "/Workspace/Users/[USERNAME]/.bundle/my_default_python/dev/artifacts/.internal/my_default_python-0.0.1+[UNIX_TIME_NANOS][1]-py3-none-any.whl",
229-
"new": "/Workspace/Users/[USERNAME]/.bundle/my_default_python/dev/artifacts/.internal/my_default_python-0.0.1+[UNIX_TIME_NANOS][0]-py3-none-any.whl"
230-
}
220+
"email_notifications": {
221+
"action": "skip",
222+
"reason": "server_side_default",
223+
"remote": {}
231224
},
232-
"remote": {
233-
"email_notifications": {
234-
"action": "skip",
235-
"reason": "server_side_default"
236-
},
237-
"tasks[task_key='notebook_task'].notebook_task.source": {
238-
"action": "skip",
239-
"reason": "server_side_default"
240-
},
241-
"timeout_seconds": {
242-
"action": "skip",
243-
"reason": "server_side_default"
244-
},
245-
"webhook_notifications": {
246-
"action": "skip",
247-
"reason": "server_side_default"
248-
}
225+
"tasks[task_key='notebook_task'].libraries[0].whl": {
226+
"action": "update",
227+
"old": "/Workspace/Users/[USERNAME]/.bundle/my_default_python/dev/artifacts/.internal/my_default_python-0.0.1+[UNIX_TIME_NANOS][1]-py3-none-any.whl",
228+
"new": "/Workspace/Users/[USERNAME]/.bundle/my_default_python/dev/artifacts/.internal/my_default_python-0.0.1+[UNIX_TIME_NANOS][0]-py3-none-any.whl",
229+
"remote": "/Workspace/Users/[USERNAME]/.bundle/my_default_python/dev/artifacts/.internal/my_default_python-0.0.1+[UNIX_TIME_NANOS][1]-py3-none-any.whl"
230+
},
231+
"tasks[task_key='notebook_task'].notebook_task.source": {
232+
"action": "skip",
233+
"reason": "server_side_default",
234+
"remote": "WORKSPACE"
235+
},
236+
"tasks[task_key='python_wheel_task'].libraries[0].whl": {
237+
"action": "update",
238+
"old": "/Workspace/Users/[USERNAME]/.bundle/my_default_python/dev/artifacts/.internal/my_default_python-0.0.1+[UNIX_TIME_NANOS][1]-py3-none-any.whl",
239+
"new": "/Workspace/Users/[USERNAME]/.bundle/my_default_python/dev/artifacts/.internal/my_default_python-0.0.1+[UNIX_TIME_NANOS][0]-py3-none-any.whl",
240+
"remote": "/Workspace/Users/[USERNAME]/.bundle/my_default_python/dev/artifacts/.internal/my_default_python-0.0.1+[UNIX_TIME_NANOS][1]-py3-none-any.whl"
241+
},
242+
"timeout_seconds": {
243+
"action": "skip",
244+
"reason": "server_side_default",
245+
"remote": 0
246+
},
247+
"webhook_notifications": {
248+
"action": "skip",
249+
"reason": "server_side_default",
250+
"remote": {}
249251
}
250252
}
251253
},
@@ -289,11 +291,10 @@
289291
"state": "IDLE"
290292
},
291293
"changes": {
292-
"remote": {
293-
"storage": {
294-
"action": "skip",
295-
"reason": "server_side_default"
296-
}
294+
"storage": {
295+
"action": "skip",
296+
"reason": "server_side_default",
297+
"remote": "dbfs:/pipelines/[UUID]"
297298
}
298299
}
299300
}

0 commit comments

Comments
 (0)