direct: support remote state lookups (#3531)

## Changes
- Change resource API to support remote state.
- Update all resources to implement that.
- Use remote state is used to resolve ${resource.jobs.foo.some.field}
references when they cannot be resolved via config.

Based on top of #3532

## Design

There is a new required method DoRefresh() that returns remote state for
existing resource. The return type defines the schema for the remote
looks up. The method is implemented by performing get() on the resource.

A number of resources return full object as part of normal
create/update/wait operations so extra get() is wasteful. For those
cases there is alternative "WithRefresh" interface where methods return
remote state as part of normal operation. Implementation can pick for
every case individually (create/update/waitcreate/waitupdate) whether to
implement NoRefresh or WithRefresh variant.

Note, there is no attempt to match terraform field format. That would be
possible by defining custom struct for remote state that follows TF
rather than using SDK type. I think using SDK type is more natural there
and for cases where needed we can add a translation layer later.

## Tests
A couple new acceptance tests checking remote state for apps and
pipelines.

Expanded unit tests in bundle/terranova for quick sanity check of
resources against test server.

Author: denik

acceptance/bundle/bundle_tag/url_ref/out.deploy.direct-exp.txt

@@ -1,5 +1,5 @@
 Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
-Error: schema mismatch for ${resources.jobs.foo.url}: url: field "url" not found in resources.Job
+Error: cannot resolve "${resources.jobs.foo.url}": schema mismatch: url: field "url" not found in resources.Job; url: field "url" not found in jobs.Job
 
 Error: cannot plan jobs.bar: dependency failed: jobs.foo
 

acceptance/bundle/bundle_tag/url_ref/out.plan.direct-exp.txt

@@ -1,4 +1,4 @@
-Error: schema mismatch for ${resources.jobs.foo.url}: url: field "url" not found in resources.Job
+Error: cannot resolve "${resources.jobs.foo.url}": schema mismatch: url: field "url" not found in resources.Job; url: field "url" not found in jobs.Job
 
 Error: cannot plan jobs.bar: dependency failed: jobs.foo
 

acceptance/bundle/bundle_tag/url_ref/test.toml

@@ -1 +1,2 @@
+Badness = "Error message in direct is developer-focussed, includes struct names that were checked; could be made more user friendly and point to the right doc and maybe even give a list of allowed fields"
 RecordRequests = true

acceptance/bundle/resource_deps/missing_ingestion_definition/out.plan.direct-exp.txt

@@ -1,4 +1,4 @@
-Error: field not set ${resources.pipelines.foo.ingestion_definition.connection_name}: ingestion_definition.connection_name: cannot access key "connection_name" on invalid
+Error: cannot resolve "${resources.pipelines.foo.ingestion_definition.connection_name}": field not set: ingestion_definition.connection_name: cannot access key "connection_name" on invalid
 
 Error: cannot plan pipelines.bar: dependency failed: pipelines.foo
 

acceptance/bundle/resource_deps/missing_map_key/out.deploy.direct-exp.txt

@@ -1,5 +1,5 @@
 Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/state/default/files...
-Error: field not set ${resources.jobs.test.tasks[0].new_cluster.custom_tags.missing_tag}: tasks[0].new_cluster.custom_tags.missing_tag: key "missing_tag" not found in map
+Error: cannot resolve "${resources.jobs.test.tasks[0].new_cluster.custom_tags.missing_tag}": field not set: tasks[0].new_cluster.custom_tags.missing_tag: key "missing_tag" not found in map
 
 Error: cannot plan jobs.bar: dependency failed: jobs.test
 

acceptance/bundle/resource_deps/missing_map_key/out.plan.direct-exp.txt

@@ -1,4 +1,4 @@
-Error: field not set ${resources.jobs.test.tasks[0].new_cluster.custom_tags.missing_tag}: tasks[0].new_cluster.custom_tags.missing_tag: key "missing_tag" not found in map
+Error: cannot resolve "${resources.jobs.test.tasks[0].new_cluster.custom_tags.missing_tag}": field not set: tasks[0].new_cluster.custom_tags.missing_tag: key "missing_tag" not found in map
 
 Error: cannot plan jobs.bar: dependency failed: jobs.test
 

acceptance/bundle/resource_deps/missing_map_key/test.toml

@@ -1,2 +1,2 @@
-Badness = "Uploading files even though bundle is wrong; bundle validate does not detect the issue; direct fails but with wrong error message"
+Badness = "Uploading files even though bundle is wrong; bundle validate does not detect the issue"
 RecordRequests = false

acceptance/bundle/resource_deps/missing_map_key_tffix/out.deploy.direct-exp.txt

@@ -1,5 +1,5 @@
 Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/state/default/files...
-Error: schema mismatch for ${resources.jobs.test.task[0].new_cluster[0].custom_tags.missing_tag}: task: field "task" not found in resources.Job
+Error: cannot resolve "${resources.jobs.test.task[0].new_cluster[0].custom_tags.missing_tag}": schema mismatch: task: field "task" not found in resources.Job; task: field "task" not found in jobs.Job
 
 Error: cannot plan jobs.bar: dependency failed: jobs.test
 

acceptance/bundle/resource_deps/missing_map_key_tffix/out.plan.direct-exp.txt

@@ -1,4 +1,4 @@
-Error: schema mismatch for ${resources.jobs.test.task[0].new_cluster[0].custom_tags.missing_tag}: task: field "task" not found in resources.Job
+Error: cannot resolve "${resources.jobs.test.task[0].new_cluster[0].custom_tags.missing_tag}": schema mismatch: task: field "task" not found in resources.Job; task: field "task" not found in jobs.Job
 
 Error: cannot plan jobs.bar: dependency failed: jobs.test
 

acceptance/bundle/resource_deps/missing_string_field/out.deploy-requests.direct-exp.txt

@@ -13,16 +13,3 @@
   "method": "POST",
   "path": "/api/2.0/pipelines"
 }
-{
-  "body": {
-    "channel": "CURRENT",
-    "deployment": {
-      "kind": "BUNDLE",
-      "metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/metadata.json"
-    },
-    "edition": "ADVANCED",
-    "name": "pbar"
-  },
-  "method": "POST",
-  "path": "/api/2.0/pipelines"
-}