Set engine based on state; refactor commands (#3797)

## Changes

### Engine selection
Engine is now selected based on available state, rather than env var.
The env var is still consulted if there are no local or remote state
files.

Since we need remote state, this is done during state-pull. This means
we don't know engine in bundle validate/summary and in deploy/destroy
until we reached that stage.

Since we don't know if remote is migrated, we pull both terraform state
and direct state all the time and decide based on serial number which to
use.

### Command refactoring
There are many commands that needed refactoring, so I extracted common
bundle steps in cmd/bundle/utils/process.go. This allows to enforce
certain order on how things are run and encode assumptions in one place.
For example, you cannot pull state until you called phases.Initialize()
because certain paths are not initialized.

## Why

This makes bundle engine setting sticky, once migrated to direct it'll
stay on direct. This will be important for subsequent 'bundle deployment
migrate' command.

Author: denik

acceptance/bundle/python/experimental-compatibility-both-error/output.txt

@@ -12,8 +12,39 @@ Error: failed to apply python mutator: 'experimental/python' and 'python' config
   },
   "resources": null,
   "__locations": {
-    "files": null,
-    "locations": null
+    "files": [
+      "databricks.yml"
+    ],
+    "locations": {
+      "bundle": [
+        [
+          0,
+          2,
+          3
+        ]
+      ],
+      "experimental": [
+        [
+          0,
+          9,
+          3
+        ]
+      ],
+      "python": [
+        [
+          0,
+          14,
+          3
+        ]
+      ],
+      "sync": [
+        [
+          0,
+          4,
+          7
+        ]
+      ]
+    }
   }
 }
 

acceptance/bundle/resource_deps/bad_ref_string_to_int/out.requests.txt

@@ -10,6 +10,14 @@
     "return_export_info": "true"
   }
 }
+{
+  "method": "GET",
+  "path": "/api/2.0/workspace/get-status",
+  "q": {
+    "path": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/resources.json",
+    "return_export_info": "true"
+  }
+}
 {
   "method": "GET",
   "path": "/api/2.0/workspace/get-status",

acceptance/bundle/resource_deps/missing_ingestion_definition/out.requests.txt

@@ -35,6 +35,14 @@
     "return_export_info": "true"
   }
 }
+{
+  "method": "GET",
+  "path": "/api/2.0/workspace/get-status",
+  "q": {
+    "path": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/resources.json",
+    "return_export_info": "true"
+  }
+}
 {
   "method": "GET",
   "path": "/api/2.0/workspace/get-status",

acceptance/bundle/state/bad_json_local/databricks.yml

@@ -0,0 +1,8 @@
+bundle:
+  name: test-bundle
+
+resources:
+  schemas:
+    foo:
+      catalog_name: c
+      name: s

acceptance/bundle/state/out.test.toml …undle/state/bad_json_local/out.test.tomlacceptance/bundle/state/out.test.toml renamed to acceptance/bundle/state/bad_json_local/out.test.toml acceptance/bundle/state/out.test.toml renamed to acceptance/bundle/state/bad_json_local/out.test.toml

@@ -0,0 +1,8 @@
+
+>>> musterr [CLI] bundle plan
+Error: parsing [TEST_TMP_DIR]/.databricks/bundle/default/terraform/terraform.tfstate: unexpected end of JSON input
+
+
+>>> musterr [CLI] bundle plan
+Error: parsing [TEST_TMP_DIR]/.databricks/bundle/default/resources.json: unexpected end of JSON input
+

acceptance/bundle/state/bad_json_local/output.txt

@@ -0,0 +1,8 @@
+export DATABRICKS_BUNDLE_ENGINE=""
+mkdir -p .databricks/bundle/default/terraform
+echo "{" > .databricks/bundle/default/terraform/terraform.tfstate
+trace musterr $CLI bundle plan 2>&1 | contains.py 'Error' 'parsing' 'terraform.tfstate' 'unexpected end'
+
+rm .databricks/bundle/default/terraform/terraform.tfstate
+echo "{" > .databricks/bundle/default/resources.json
+trace musterr $CLI bundle plan 2>&1 | contains.py 'Error' 'parsing' 'resources.json' 'unexpected end'