Add "bundle deployment migrate" command

[denik]

Changes

New migrate command to convert terraform state to direct engine state. This currently only works locally (needs bundle deploy to sync changes) and does not support permissions.

Depends on #3875

Tests

Acceptance tests.

[eng-dev-ecosystem-bot]

Run: 19166282007

	Env	❌​FAIL	🟨​KNOWN	🔄​flaky	💚​RECOVERED	🙈​SKIP	✅​pass	🙈​skip
💚​	aws linux				1	1	356	599
💚​	aws windows				1	1	357	598
💚​	aws-ucws linux				1	1	479	494
🔄​	aws-ucws windows			1	1	1	479	493
💚​	azure linux				1	1	356	598
💚​	azure windows				1	1	357	597
🔄​	azure-ucws linux			3		1	473	493
💚​	azure-ucws windows				1	1	476	492
❌​	gcp linux	21	1			1	334	600
❌​	gcp windows	23	1			1	333	599

48 failing tests:

Test Name	aws linux	aws windows	aws-ucws linux	aws-ucws windows	azure linux	azure windows	azure-ucws linux	azure-ucws windows	gcp linux	gcp windows
TestAccept	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	🔄​f	💚​R	🟨​K	🟨​K
TestAccept/bundle/deploy/mlops-stacks	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F
TestAccept/bundle/deployment/bind/dashboard	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestAccept/bundle/deployment/bind/dashboard/recreation	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F
TestAccept/bundle/deployment/bind/experiment	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	❌​F
TestAccept/bundle/deployment/bind/job/generate-and-bind	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestAccept/bundle/deployment/bind/job/job-abort-bind	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestAccept/bundle/deployment/bind/model-serving-endpoint	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestAccept/bundle/deployment/unbind/permissions	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestAccept/bundle/resources/clusters/deploy/data_security_mode	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	🔄​f	✅​p	✅​p	✅​p
TestAccept/bundle/resources/clusters/deploy/data_security_mode/DATABRICKS_BUNDLE_ENGINE=direct	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	🔄​f	✅​p	✅​p	✅​p
TestAccept/bundle/resources/dashboards/change-embed-credentials	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F
TestAccept/bundle/resources/dashboards/change-embed-credentials/DATABRICKS_BUNDLE_ENGINE=direct	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F
TestAccept/bundle/resources/dashboards/change-parent-path	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F
TestAccept/bundle/resources/dashboards/change-parent-path/DATABRICKS_BUNDLE_ENGINE=direct	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F
TestAccept/bundle/resources/dashboards/detect-change	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F
TestAccept/bundle/resources/dashboards/detect-change/DATABRICKS_BUNDLE_ENGINE=terraform	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F
TestAccept/bundle/run/app-with-job	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
TestAccept/bundle/templates/default-python/integration_classic	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=direct/UV_PYTHON=3.12	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=direct/UV_PYTHON=3.13	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestFsCpDir	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestFsCpDir/dbfs_to_dbfs	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestFsCpDir/local_to_dbfs	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestFsCpDirToDirFileNotOverwritten	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestFsCpDirToDirFileNotOverwritten/dbfs_to_dbfs	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestFsCpDirToDirFileNotOverwritten/dbfs_to_local	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestFsCpDirToDirFileNotOverwritten/local_to_dbfs	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestFsCpFileToDir	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestFsCpFileToDir/dbfs_to_dbfs	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestFsCpFileToDir/dbfs_to_local	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestFsCpFileToDirFileNotOverwritten	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestFsCpFileToDirFileNotOverwritten/dbfs_to_local	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestSyncIncrementalFileOverwritesFolder	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F
TestFilerRecursiveDelete	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F
TestFilerRecursiveDelete/workspace_files	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F
TestFilerWorkspaceFilesExtensionsDelete	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F
TestFilerWorkspaceFilesExtensionsRead	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F
TestFilerWorkspaceFilesExtensionsReadDir	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F
TestWorkspaceFilesExtensions_ExportFormatIsPreserved	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F
TestWorkspaceFilesExtensions_ExportFormatIsPreserved/jupyter_python	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F
TestWorkspaceFilesExtensions_ExportFormatIsPreserved/jupyter_r	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F
TestWorkspaceFilesExtensions_ExportFormatIsPreserved/jupyter_scala	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F
TestWorkspaceFilesExtensions_ExportFormatIsPreserved/jupyter_sql	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F
TestWorkspaceFilesExtensions_ExportFormatIsPreserved/source_python	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F
TestWorkspaceFilesExtensions_ExportFormatIsPreserved/source_sql	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F
TestFetchRepositoryInfoAPI_FromRepo	✅​p	✅​p	✅​p	🔄​f	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p
TestLock	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F

[pietern]

Nice! Less code than I had thought.

[pietern]

This error should be more actionable; e.g. revert to previous configuration, or run a non-migration deploy first to achieve a clean slate, etc.

[pietern]

But doesn't this open up the probability for drift, given that we go off of the local config to determine if we deploy or not. If the previous deploy was done with TF, now we changed 1 field in some config, ran deploy, we may persist a config snapshot that we never materialized.

[denik]

Yes, that's why migration needs to happen after clean deploy with terraform.

[pietern]

Nice hybrid between bools and enums, without the need for unnecessary consts.

[pietern]

Only downside is that it shows up at every callsite.

Potential opportunity is to make .Migrate a dedicated function. Not blocking.

[pietern]

We can suggest running "bundle plan" as well, to surface differences/updates.

[pietern]

... or leave it empty to infer the right engine.

[pietern]

Suggested change

	To start using direct engine, deploy with DATABRICKS_BUNDLE_ENGINE=direct env var or bundle.engine="direct" in databricks.yml.`)
	To start using the direct deployment engine, deploy with the environment variable DATABRICKS_BUNDLE_ENGINE=direct or bundle.engine="direct" in databricks.yml.`)

[andrewnester]

Should it be false instead, so we abort the migration?

[andrewnester]

Either one of the comments is confusing, or I am misunderstanding something, but we don't seem to execute the deploy in this branch, only in the others.

[andrewnester]

Suggested change

	return fmt.Errorf("temporary state file %s already exists, another migration is in progress or was interrupated. In the latter case, delete the file", tempStatePath)
	return fmt.Errorf("temporary state file %s already exists, another migration is in progress or was interrupted. In the latter case, delete the file", tempStatePath)