Support serverless workspaces in databricks_mws_workspaces (#4670)

## Changes
A new field `compute_mode` has been added to
`databricks_mws_workspaces`. This can be set to `SERVERLESS` to indicate
that a workspace should be serverless. Serverless workspaces are
documented at
https://docs.databricks.com/aws/en/admin/workspace/serverless-workspaces.
These workspaces need neither `credentials_id` or
`storage_configuration_id`, so the setup should be much easier.

## Tests
Added an integration test for creating serverless workspaces in AWS.

## Todos
- [x] Need to verify if serverless workspaces work in GCP. EDIT: not yet
supported.
- [x] Need to verify which regions users can specify for serverless
workspaces. EDIT: this will be documented shortly. I'll include a link.
- [x] The test does not work right now, I think our E2 account still
needs to be onboarded. EDIT: our account was onboarded and we verified
that this works.

Author: mgyucht

NEXT_CHANGELOG.md

@@ -6,6 +6,7 @@
 
  * Add support for `power_bi_task` in jobs ([#4647](https://github.com/databricks/terraform-provider-databricks/pull/4647))
  * Add support for `dashboard_task` in jobs ([#4646](https://github.com/databricks/terraform-provider-databricks/pull/4646))
+ * Add `compute_mode` to `databricks_mws_workspaces` to support creating serverless workspaces ([#4670](https://github.com/databricks/terraform-provider-databricks/pull/4670)).
 
 ### Bug Fixes
 

docs/resources/mws_workspaces.md

@@ -18,7 +18,22 @@ This resource allows you to set up [workspaces on AWS](https://docs.databricks.c
 
 ## Example Usage
 
-### Creating a Databricks on AWS workspace
+### Creating a serverless workspace in AWS
+
+Creating a serverless workspace does not require any prerequisite resources. Simply specify `compute_mode = "SERVERLESS"` when creating the workspace. Serverless workspaces must not include `credentials_id` or `storage_configuration_id`.
+
+To use serverless workspaces, you must enroll in the [Default Storage preview](https://docs.databricks.com/aws/en/storage/express-storage).
+
+```hcl
+resource "databricks_mws_workspaces" "serverless_workspace" {
+  account_id     = "" # Your Databricks account ID
+  workspace_name = "serverless-workspace"
+  aws_region     = "us-east-1"
+  compute_mode   = "SERVERLESS"
+}
+```
+
+### Creating a workspace on AWS
 
 ![Simplest multiworkspace](https://raw.githubusercontent.com/databricks/terraform-provider-databricks/main/docs/simplest-multiworkspace.png)
 
@@ -85,11 +100,11 @@ output "databricks_token" {
 }
 ```
 
-### Creating a Databricks on AWS workspace with Databricks-Managed VPC
+### Creating a workspace on AWS with Databricks-Managed VPC
 
 ![VPCs](https://docs.databricks.com/_images/customer-managed-vpc.png)
 
-By default, Databricks creates a VPC in your AWS account for each workspace. Databricks uses it for running clusters in the workspace. Optionally, you can use your VPC for the workspace, using the feature customer-managed VPC. Databricks recommends that you provide your VPC with [databricks_mws_networks](mws_networks.md) so that you can configure it according to your organization’s enterprise cloud standards while still conforming to Databricks requirements. You cannot migrate an existing workspace to your VPC. Please see the difference described through IAM policy actions [on this page](https://docs.databricks.com/administration-guide/account-api/iam-role.html).
+By default, Databricks creates a VPC in your AWS account for each workspace. Databricks uses it for running clusters in the workspace. Optionally, you can use your VPC for the workspace, using the feature customer-managed VPC. Databricks recommends that you provide your VPC with [databricks_mws_networks](mws_networks.md) so that you can configure it according to your organization's enterprise cloud standards while still conforming to Databricks requirements. You cannot migrate an existing workspace to your VPC. Please see the difference described through IAM policy actions [on this page](https://docs.databricks.com/administration-guide/account-api/iam-role.html).
 
 ```hcl
 variable "databricks_account_id" {
@@ -209,7 +224,7 @@ output "databricks_token" {
 
 In order to create a [Databricks Workspace that leverages AWS PrivateLink](https://docs.databricks.com/administration-guide/cloud-configurations/aws/privatelink.html) please ensure that you have read and understood the [Enable Private Link](https://docs.databricks.com/administration-guide/cloud-configurations/aws/privatelink.html) documentation and then customise the example above with the relevant examples from [mws_vpc_endpoint](mws_vpc_endpoint.md), [mws_private_access_settings](mws_private_access_settings.md) and [mws_networks](mws_networks.md).
 
-### Creating a Databricks on GCP workspace
+### Creating a workspace on GCP
 
 To get workspace running, you have to configure a network object:
 
@@ -270,11 +285,11 @@ output "databricks_token" {
 
 In order to create a [Databricks Workspace that leverages GCP Private Service Connect](https://docs.gcp.databricks.com/administration-guide/cloud-configurations/gcp/private-service-connect.html) please ensure that you have read and understood the [Enable Private Service Connect](https://docs.gcp.databricks.com/administration-guide/cloud-configurations/gcp/private-service-connect.html) documentation and then customise the example above with the relevant examples from [mws_vpc_endpoint](mws_vpc_endpoint.md), [mws_private_access_settings](mws_private_access_settings.md) and [mws_networks](mws_networks.md).
 
-#### Creating a Databricks on GCP workspace with Databricks-Managed VPC
+#### Creating a workspace on GCP with Databricks-Managed VPC
 
 ![VPCs](https://docs.databricks.com/_images/customer-managed-vpc.png)
 
-By default, Databricks creates a VPC in your GCP project for each workspace. Databricks uses it for running clusters in the workspace. Optionally, you can use your VPC for the workspace, using the feature customer-managed VPC. Databricks recommends that you provide your VPC with [databricks_mws_networks](mws_networks.md) so that you can configure it according to your organization’s enterprise cloud standards while still conforming to Databricks requirements. You cannot migrate an existing workspace to your VPC.
+By default, Databricks creates a VPC in your GCP project for each workspace. Databricks uses it for running clusters in the workspace. Optionally, you can use your VPC for the workspace, using the feature customer-managed VPC. Databricks recommends that you provide your VPC with [databricks_mws_networks](mws_networks.md) so that you can configure it according to your organization's enterprise cloud standards while still conforming to Databricks requirements. You cannot migrate an existing workspace to your VPC.
 
 ```hcl
 variable "databricks_account_id" {
@@ -324,7 +339,8 @@ The following arguments are available:
 * `workspace_name` - name of the workspace, will appear on UI.
 * `network_id` - (Optional) `network_id` from [networks](mws_networks.md).
 * `aws_region` - (AWS only) region of VPC.
-* `storage_configuration_id` - (AWS only)`storage_configuration_id` from [storage configuration](mws_storage_configurations.md).
+* `storage_configuration_id` - (AWS only, Optional) `storage_configuration_id` from [storage configuration](mws_storage_configurations.md). This must not be specified when `compute_mode` is set to `SERVERLESS`.
+* `credentials_id` - (AWS only, Optional) `credentials_id` from [credentials](mws_credentials.md). This must not be specified when `compute_mode` is set to `SERVERLESS`.
 * `managed_services_customer_managed_key_id` - (Optional) `customer_managed_key_id` from [customer managed keys](mws_customer_managed_keys.md) with `use_cases` set to `MANAGED_SERVICES`. This is used to encrypt the workspace's notebook and secret data in the control plane.
 * `storage_customer_managed_key_id` - (Optional) `customer_managed_key_id` from [customer managed keys](mws_customer_managed_keys.md) with `use_cases` set to `STORAGE`. This is used to encrypt the DBFS Storage & Cluster Volumes.
 * `location` - (GCP only) region of the subnet.
@@ -337,6 +353,7 @@ The following arguments are available:
 * `private_access_settings_id` - (Optional) Canonical unique identifier of [databricks_mws_private_access_settings](mws_private_access_settings.md) in Databricks Account.
 * `custom_tags` - (Optional / AWS only) - The custom tags key-value pairing that is attached to this workspace. These tags will be applied to clusters automatically in addition to any `default_tags` or `custom_tags` on a cluster level. Please note it can take up to an hour for custom_tags to be set due to scheduling on Control Plane. After custom tags are applied, they can be modified however they can never be completely removed.
 * `pricing_tier` - (Optional) - The pricing tier of the workspace.
+* `compute_mode` - (Optional) - The compute mode for the workspace. When unset, a classic workspace is created, and both `credentials_id` and `storage_configuration_id` must be specified. When set to `SERVERLESS`, the resulting workspace is a serverless workspace, and `credentials_id` and `storage_configuration_id` must not be set. The only allowed value for this is `SERVERLESS`. Changing this field requires recreation of the workspace.
 
 ### token block
 
@@ -369,6 +386,7 @@ In addition to all arguments above, the following attributes are exported:
 * `workspace_url` - (String) URL of the workspace
 * `custom_tags` - (Map) Custom Tags (if present) added to workspace
 * `gcp_workspace_sa` - (String, GCP only) identifier of a service account created for the workspace in form of `db-<workspace-id>@prod-gcp-<region>.iam.gserviceaccount.com`
+* `effective_compute_mode` - (String) The effective compute mode for the workspace. This is either `SERVERLESS` for serverless workspaces or `HYBRID` for classic workspaces.
 
 ## Timeouts
 

mws/mws_workspaces_test.go

@@ -76,6 +76,22 @@ func TestMwsAccWorkspaces(t *testing.T) {
 	})
 }
 
+func TestMwsAccWorkspaces_Serverless(t *testing.T) {
+	acceptance.LoadAccountEnv(t)
+	if !acceptance.IsAws(t) {
+		acceptance.Skipf(t)("TestMwsAccWorkspaces_Serverless is currently only supported on AWS")
+	}
+	acceptance.AccountLevel(t, acceptance.Step{
+		Template: `
+		resource "databricks_mws_workspaces" "this" {
+			account_id      = "{env.DATABRICKS_ACCOUNT_ID}"
+			workspace_name  = "terra-{var.RANDOM}"
+			aws_region      = "{env.AWS_REGION}"
+			compute_mode    = "SERVERLESS"
+		}`,
+	})
+}
+
 func TestMwsAccWorkspacesTokenUpdate(t *testing.T) {
 	acceptance.AccountLevel(t, acceptance.Step{
 		Template: `

mws/resource_mws_workspaces.go

@@ -21,6 +21,7 @@ import (
 	"github.com/hashicorp/go-cty/cty"
 	"github.com/hashicorp/terraform-plugin-log/tflog"
 	"github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema"
+	"github.com/hashicorp/terraform-plugin-sdk/v2/helper/validation"
 	"github.com/hashicorp/terraform-plugin-testing/helper/resource"
 )
 
@@ -104,6 +105,8 @@ type Workspace struct {
 	Cloud                               string                   `json:"cloud,omitempty" tf:"computed"`
 	Location                            string                   `json:"location,omitempty"`
 	CustomTags                          map[string]string        `json:"custom_tags,omitempty"` // Optional for AWS, not allowed for GCP
+	ComputeMode                         string                   `json:"compute_mode,omitempty" tf:"force_new"`
+	EffectiveComputeMode                string                   `json:"effective_compute_mode" tf:"computed"`
 }
 
 // this type alias hack is required for Marshaller to work without an infinite loop
@@ -141,6 +144,9 @@ func (w *Workspace) MarshalJSON() ([]byte, error) {
 	if w.StorageCustomerManagedKeyID != "" {
 		workspaceCreationRequest["storage_customer_managed_key_id"] = w.StorageCustomerManagedKeyID
 	}
+	if w.ComputeMode != "" {
+		workspaceCreationRequest["compute_mode"] = w.ComputeMode
+	}
 	return json.Marshal(workspaceCreationRequest)
 }
 
@@ -314,6 +320,10 @@ func (a WorkspacesAPI) Read(mwsAcctID, workspaceID string) (Workspace, error) {
 		host := generateWorkspaceHostname(a.client, mwsWorkspace)
 		mwsWorkspace.WorkspaceURL = fmt.Sprintf("https://%s", host)
 	}
+	// Set the effective compute mode to the compute mode
+	mwsWorkspace.EffectiveComputeMode = mwsWorkspace.ComputeMode
+	mwsWorkspace.ComputeMode = ""
+
 	return mwsWorkspace, err
 }
 
@@ -546,23 +556,13 @@ func ResourceMwsWorkspaces() common.Resource {
 			common.CustomizeSchemaPath(s, "gke_config").SetDeprecated(getGkeDeprecationMessage("gke_config", docOptions))
 			common.CustomizeSchemaPath(s, "gcp_managed_network_config", "gke_cluster_pod_ip_range").SetDeprecated(getGkeDeprecationMessage("gcp_managed_network_config.gke_cluster_pod_ip_range", docOptions))
 			common.CustomizeSchemaPath(s, "gcp_managed_network_config", "gke_cluster_service_ip_range").SetDeprecated(getGkeDeprecationMessage("gcp_managed_network_config.gke_cluster_service_ip_range", docOptions))
+			common.CustomizeSchemaPath(s, "compute_mode").SetValidateDiagFunc(validation.ToDiagFunc(validation.StringInSlice([]string{"SERVERLESS"}, false)))
 			return s
 		})
 	p := common.NewPairSeparatedID("account_id", "workspace_id", "/").Schema(
 		func(_ map[string]*schema.Schema) map[string]*schema.Schema {
 			return workspaceSchema
 		})
-	requireFields := func(onThisCloud bool, d *schema.ResourceData, fields ...string) error {
-		if !onThisCloud {
-			return nil
-		}
-		for _, fieldName := range fields {
-			if d.Get(fieldName) == workspaceSchema[fieldName].ZeroValue() {
-				return fmt.Errorf("%s is required", fieldName)
-			}
-		}
-		return nil
-	}
 	return common.Resource{
 		Schema:        workspaceSchema,
 		SchemaVersion: 3,
@@ -577,11 +577,22 @@ func ResourceMwsWorkspaces() common.Resource {
 			var workspace Workspace
 			workspacesAPI := NewWorkspacesAPI(ctx, c)
 			common.DataToStructPointer(d, workspaceSchema, &workspace)
-			if err := requireFields(c.IsAws(), d, "aws_region", "credentials_id", "storage_configuration_id"); err != nil {
-				return err
-			}
-			if err := requireFields(c.IsGcp(), d, "location"); err != nil {
-				return err
+			if c.IsAws() {
+				if _, ok := d.GetOk("aws_region"); !ok {
+					return fmt.Errorf("aws_region is required for AWS workspaces")
+				}
+				if d.Get("compute_mode") != "SERVERLESS" {
+					if _, ok := d.GetOk("credentials_id"); !ok {
+						return fmt.Errorf("credentials_id is required for non-serverless workspaces")
+					}
+					if _, ok := d.GetOk("storage_configuration_id"); !ok {
+						return fmt.Errorf("storage_configuration_id is required for non-serverless workspaces")
+					}
+				}
+			} else if c.IsGcp() {
+				if _, ok := d.GetOk("location"); !ok {
+					return fmt.Errorf("location is required for GCP workspaces")
+				}
 			}
 			if !c.IsAws() && workspace.CustomTags != nil {
 				return fmt.Errorf("custom_tags are only allowed for AWS workspaces")