Add support for models in direct deployment

[shreyas-goenka]

Changes

This PR adds support for direct deployment for MLflow models, matching the existing Terraform behaviour today.

Why

To make direct deployments possible.

Tests

Test asserting that TF and direct behaviour match for all relevent fields.

[eng-dev-ecosystem-bot]

Run: 18083105839

	Env	✅​pass	🔄​flaky	🙈​skip
🔄​	aws linux	313	2	531
✅​	aws windows	316		530
🔄​	aws-ucws linux	421	7	428
🔄​	aws-ucws windows	426	3	427
🔄​	azure linux	312	3	530
✅​	azure windows	316		529
✅​	azure-ucws linux	428		427
🔄​	azure-ucws windows	427	2	426
✅​	gcp linux	314		532
✅​	gcp windows	315		531

14 failing tests:

Test Name	aws linux	aws-ucws linux	aws-ucws windows	azure linux	azure-ucws windows
TestAccept	✅​pass	✅​pass	✅​pass	🔄​flaky	✅​pass
TestAccept/bundle/deploy/files/no-snapshot-sync	✅​pass	✅​pass	✅​pass	🔄​flaky	✅​pass
TestAccept/bundle/deploy/files/no-snapshot-sync/DATABRICKS_BUNDLE_ENGINE=direct-exp	✅​pass	✅​pass	✅​pass	🔄​flaky	✅​pass
TestAccept/bundle/deploy/pipeline/auto-approve/DATABRICKS_BUNDLE_ENGINE=direct-exp	✅​pass	✅​pass	✅​pass	✅​pass	🔄​flaky
TestAccept/bundle/deploy/pipeline/auto-approve/DATABRICKS_BUNDLE_ENGINE=terraform	✅​pass	✅​pass	✅​pass	✅​pass	🔄​flaky
TestAccept/bundle/deployment/bind/job/job-spark-python-task	✅​pass	🔄​flaky	✅​pass	✅​pass	✅​pass
TestAccept/bundle/deployment/bind/model-serving-endpoint	✅​pass	🔄​flaky	🔄​flaky	✅​pass	✅​pass
TestAccept/bundle/deployment/bind/registered-model	🙈​skip	🔄​flaky	🔄​flaky	🙈​skip	✅​pass
TestAccept/bundle/deployment/bind/schema	🙈​skip	🔄​flaky	🔄​flaky	🙈​skip	✅​pass
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=no/NBOOK=no/PY=no	🔄​flaky	✅​pass	✅​pass	✅​pass	✅​pass
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=no/NBOOK=yes/PY=yes	🔄​flaky	✅​pass	✅​pass	✅​pass	✅​pass
TestSparkJarTaskDeployAndRunOnVolumes/Databricks_Runtime_13.3_LTS	🙈​skip	🔄​flaky	✅​pass	🙈​skip	✅​pass
TestSparkJarTaskDeployAndRunOnVolumes/Databricks_Runtime_14.3_LTS	🙈​skip	🔄​flaky	✅​pass	🙈​skip	✅​pass
TestSparkJarTaskDeployAndRunOnVolumes/Databricks_Runtime_15.4_LTS	🙈​skip	🔄​flaky	✅​pass	🙈​skip	✅​pass

[denik]

Looks good, minor suggestion in testserver.

[denik]

Would it make sense to store ml.ModelDatabricks in the state directly and then use MapGet() instead of custom method with remapping?

[shreyas-goenka]

Yes, but the benefit is marginal, so I'll skip it for now.

[denik]

the comments says "this should be a no-op" but the plan shows an update.

[shreyas-goenka]

We show a change even though field is annotated as skip right? I observed the same behaviour in models.

[denik]

Discussed offline - TF does make a call but the call does not have "tags" in the request, so the request is essentially noop.

Direct should not make a request at all since the field is marked as 'skip', it's a bug that it does. Fixed in #3636

[denik]

The json plan makes it more clear -- the field shows up but it's marked as skip and the whole resource is marked as skip.

{
  "plan": {
    "resources.models.my_model": {
      "action": "skip",
      "changes": {
        "local": {
          "tags": {
            "action": "skip"
          }
        }
      }
    }
  }
}