You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: CHANGELOG.md
+4Lines changed: 4 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,5 +1,9 @@
1
1
# Version changelog
2
2
3
+
## 0.5.1
4
+
5
+
* Added an extended documentation from provisioning AWS PrivateLink workspace ([#1084](https://github.com/databrickslabs/terraform-provider-databricks/pull/1084)).
6
+
3
7
## 0.5.0
4
8
5
9
* Added `workspace_url` attribute to the `databricks_current_user` data source ([#1107](https://github.com/databrickslabs/terraform-provider-databricks/pull/1107)).
page_title: "Enable Backend AWS PrivateLink for Databricks Workspace"
2
+
page_title: "Provisioning Databricks on AWS with PrivateLink"
3
3
---
4
4
5
5
# Deploying pre-requisite resources and enabling PrivateLink connections (AWS Preview)
@@ -15,16 +15,16 @@ This guide uses the following variables in configurations:
15
15
-`databricks_account_username`: The username an account-level admin uses to log in to [https://accounts.cloud.databricks.com](https://accounts.cloud.databricks.com).
16
16
-`databricks_account_password`: The password for `databricks_account_username`.
17
17
-`databricks_account_id`: The numeric ID for your Databricks account. When you are logged in, it appears in the bottom left corner of the page.
18
-
-`vpc_id` - The ID for the AWS VPC
19
-
-`region` - AWS region
20
-
-`security_group_id` - Security groups set up for the existing VPC
21
-
-`subnet_ids` - Existing subnets being used for the customer managed VPC
18
+
-`vpc_id` - The ID for the AWS VPC.
19
+
-`region` - AWS region.
20
+
-`security_group_id` - Security groups set up for the existing VPC.
21
+
-`subnet_ids` - Existing subnets being used for the customer managed VPC.
22
22
-`workspace_vpce_service` - Choose the region-specific service endpoint from this table.
23
23
-`relay_vpce_service` - Choose the region-specific service from this table.
24
-
-`vpce_subnet_cidr` - CIDR range for the subnet chosen for the VPC endpoint
25
-
-`tags` - tags for the Private Link backend setup
26
-
-`root_bucket_name` - AWS bucket name required for [storage mws resource](https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/resources/mws_storage_configurations) reference
27
-
-`cross_account_arn` - AWS EC2 role ARN required for [credentials mws resource](https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/resources/mws_credentials)
24
+
-`vpce_subnet_cidr` - CIDR range for the subnet chosen for the VPC endpoint.
25
+
-`tags` - tags for the Private Link backend setup.
26
+
-`root_bucket_name` - AWS bucket name required for [databricks_mws_storage_configurations](https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/resources/mws_storage_configurations).
27
+
-`cross_account_arn` - AWS EC2 role ARN required for [databricks_mws_credentials](https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/resources/mws_credentials).
28
28
29
29
This guide is provided as-is and you can use this guide as the basis for your custom Terraform module.
30
30
@@ -44,11 +44,10 @@ Initialize [provider with `mws` alias](https://www.terraform.io/language/provide
44
44
terraform {
45
45
required_providers {
46
46
databricks = {
47
-
source = "databrickslabs/databricks"
48
-
version = "0.5.0"
47
+
source = "databrickslabs/databricks"
49
48
}
50
49
aws = {
51
-
source = "hashicorp/aws"
50
+
source = "hashicorp/aws"
52
51
version = "3.49.0"
53
52
}
54
53
}
@@ -58,15 +57,12 @@ provider "aws" {
58
57
region = var.region
59
58
}
60
59
61
-
// initialize provider in "MWS" mode for provisioning workspace with AWS PrivateLink
62
60
provider "databricks" {
63
61
alias = "mws"
64
62
host = "https://accounts.cloud.databricks.com"
65
63
username = var.databricks_account_username
66
64
password = var.databricks_account_password
67
65
}
68
-
69
-
70
66
```
71
67
72
68
Define the required variables
@@ -75,33 +71,25 @@ Define the required variables
75
71
variable "databricks_account_id" {}
76
72
variable "databricks_account_username" {}
77
73
variable "databricks_account_password" {}
74
+
variable "root_bucket_name" {}
75
+
variable "cross_account_arn" {}
78
76
variable "vpc_id" {}
79
77
variable "region" {}
80
78
variable "security_group_id" {}
81
-
82
-
// this input variable is of array type
83
-
variable "subnet_ids" {
84
-
type = list(string)
85
-
}
86
-
79
+
variable "subnet_ids" { type = list(string) }
87
80
variable "workspace_vpce_service" {}
88
81
variable "relay_vpce_service" {}
89
82
variable "vpce_subnet_cidr" {}
90
-
91
-
variable "private_dns_enabled" { default = false}
92
-
variable "tags" { default = {}}
93
-
94
-
// these resources (bucket and IAM role) are assumed created using your AWS provider and the examples here https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/resources/mws_storage_configurations and https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/resources/mws_credentials, respectively.
The following object is used in order to reference the credential configuration ID.
102
+
## Cross-account IAM role
103
+
Create new cross-account credentials with [databricks_mws_credentials](../resources/mws_credentials.md):
116
104
```hcl
117
105
resource "databricks_mws_credentials" "this" {
118
106
provider = databricks.mws
119
107
account_id = var.databricks_account_id
120
108
role_arn = var.cross_account_arn
121
109
credentials_name = "${local.prefix}-credentials"
122
110
}
123
-
124
111
```
125
112
126
-
127
-
## Configure AWS objects
113
+
## Configure networking
128
114
The first step is to create the required AWS objects:
129
-
- A subnet dedicated to your VPC endpoints
115
+
- A subnet dedicated to your VPC endpoints.
130
116
- A security group dedicated to your VPC endpoints and satisfying required inbound/outbound TCP/HTTPS traffic rules on ports 443 and 6666, respectively.
131
-
- Lastly, creation of the private access settings and workspace.
132
117
133
118
```hcl
119
+
data "aws_vpc" "prod" {
120
+
id = var.vpc_id
121
+
}
122
+
134
123
// this subnet houses the data plane VPC endpoints
135
124
resource "aws_subnet" "dataplane_vpce" {
136
125
vpc_id = var.vpc_id
137
126
cidr_block = var.vpce_subnet_cidr
138
127
139
-
tags = merge(
140
-
data.aws_vpc.prod.tags,
141
-
{
142
-
Name = "${local.prefix}-${data.aws_vpc.prod.id}-pl-vpce"
143
-
},
144
-
)
128
+
tags = merge(data.aws_vpc.prod.tags, {
129
+
Name = "${local.prefix}-${data.aws_vpc.prod.id}-pl-vpce"
130
+
})
145
131
}
146
132
147
133
resource "aws_route_table" "this" {
148
-
vpc_id = var.vpc_id
149
-
150
-
tags = merge(
151
-
data.aws_vpc.prod.tags,
152
-
{
153
-
Name = "${local.prefix}-${data.aws_vpc.prod.id}-pl-local-route-tbl"
154
-
},
155
-
)
134
+
vpc_id = var.vpc_id
135
+
136
+
tags = merge(data.aws_vpc.prod.tags, {
137
+
Name = "${local.prefix}-${data.aws_vpc.prod.id}-pl-local-route-tbl"
Name = "${local.prefix}-${data.aws_vpc.prod.id}-pl-vpce-sg-rules"
219
-
},
220
-
)
199
+
tags = merge(data.aws_vpc.prod.tags, {
200
+
Name = "${local.prefix}-${data.aws_vpc.prod.id}-pl-vpce-sg-rules"
201
+
})
221
202
}
222
203
```
223
204
224
-
```hcl
225
-
data "aws_vpc" "prod" {
226
-
id = var.vpc_id
227
-
}
205
+
Run terraform apply twice when configuring PrivateLink: see an [outstanding issue](https://github.com/hashicorp/terraform-provider-aws/issues/7148) for more information.
206
+
* Run 1 - comment the `private_dns_enabled` lines.
207
+
* Run 2 - uncomment the `private_dns_enabled` lines.
// run terraform apply twice when configuring PrivateLink - see this outstanding issue for understanding why this is required - https://github.com/hashicorp/terraform-provider-aws/issues/7148
236
-
// Run 1 - comment the `private_dns_enabled` line
237
-
// Run 2 - uncomment the `private_dns_enabled` line
// run terraform apply twice when configuring PrivateLink - see this outstanding issue for understanding why this is required - https://github.com/hashicorp/terraform-provider-aws/issues/7148
249
-
// Run 1 - comment the `private_dns_enabled` line
250
-
// Run 2 - uncomment the `private_dns_enabled` line
Once the VPC endpoints are created, they can be supplied in the `databricks_mws_networks` resource for workspace creation with AWS PrivateLink. After the terraform apply is run once (see the comment in the aws_vpc_endpoint resource above), run the terraform apply a second time with the line for private_dns_enabled set to true uncommented to set the proper DNS settings for PrivateLink. For understanding the reason that this needs to be applied twice, see this existing [issue](hashicorp/terraform-provider-aws#7148) in the underlying AWS provider.
279
-
280
-
The credentials ID which is referenced below is one of the attributes which is created as a result of configuring the cross-account IAM role, which Databricks uses to orchestrate EC2 resources. The credentials are created via [databricks_mws_credentials](https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/resources/mws_credentials). Similarly, the storage configuration ID is obtained from the [databricks_mws_storage_configurations](https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/resources/mws_storage_configurations) resource.
249
+
Once the VPC endpoints are created, they can be supplied in the [databricks_mws_networks](../resources/mws_networks.md) resource for workspace creation with AWS PrivateLink. After the `terraform apply` is run once (see the comment in the `aws_vpc_endpoint` resource above), run the terraform apply a second time with the line for `private_dns_enabled` set to true uncommented to set the proper DNS settings for PrivateLink. For understanding the reason that this needs to be applied twice, see this existing [issue](https://github.com/hashicorp/terraform-provider-aws/issues/7148) in the underlying AWS provider.
281
250
282
251
```hcl
283
-
// Inputs are 2 subnets and one security group from existing VPC that will be used for your Databricks workspace
The credentials ID which is referenced below is one of the attributes which is created as a result of configuring the cross-account IAM role, which Databricks uses to orchestrate EC2 resources. The credentials are created via [databricks_mws_credentials](https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/resources/mws_credentials). Similarly, the storage configuration ID is obtained from the [databricks_mws_storage_configurations](https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/resources/mws_storage_configurations) resource.
Copy file name to clipboardExpand all lines: docs/resources/mws_networks.md
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -113,6 +113,7 @@ In addition to all arguments above, the following attributes are exported:
113
113
The following resources are used in the same context:
114
114
115
115
*[Provisioning Databricks on AWS](../guides/aws-workspace.md) guide.
116
+
*[Provisioning Databricks on AWS with PrivateLink](../guides/aws-private-link-workspace.md) guide.
116
117
*[Provisioning AWS Databricks E2 with a Hub & Spoke firewall for data exfiltration protection](../guides/aws-e2-firewall-hub-and-spoke.md) guide.
117
118
*[databricks_mws_vpc_endpoint](mws_vpc_endpoint.md) to register [aws_vpc_endpoint](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/vpc_endpoint) resources with Databricks such that they can be used as part of a [databricks_mws_networks](mws_networks.md) configuration.
118
119
*[databricks_mws_private_access_settings](mws_private_access_settings.md) to create a [Private Access Setting](https://docs.databricks.com/administration-guide/cloud-configurations/aws/privatelink.html#step-5-create-a-private-access-settings-configuration-using-the-databricks-account-api) that can be used as part of a [databricks_mws_workspaces](mws_workspaces.md) resource to create a [Databricks Workspace that leverages AWS PrivateLink](https://docs.databricks.com/administration-guide/cloud-configurations/aws/privatelink.html).
Copy file name to clipboardExpand all lines: docs/resources/mws_private_access_settings.md
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -65,6 +65,7 @@ In addition to all arguments above, the following attributes are exported:
65
65
The following resources are used in the same context:
66
66
67
67
*[Provisioning Databricks on AWS](../guides/aws-workspace.md) guide.
68
+
*[Provisioning Databricks on AWS with PrivateLink](../guides/aws-private-link-workspace.md) guide.
68
69
*[Provisioning AWS Databricks E2 with a Hub & Spoke firewall for data exfiltration protection](../guides/aws-e2-firewall-hub-and-spoke.md) guide.
69
70
*[databricks_mws_vpc_endpoint](mws_vpc_endpoint.md) to register [aws_vpc_endpoint](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/vpc_endpoint) resources with Databricks such that they can be used as part of a [databricks_mws_networks](mws_networks.md) configuration.
70
71
*[databricks_mws_networks](mws_networks.md) to [configure VPC](https://docs.databricks.com/administration-guide/cloud-configurations/aws/customer-managed-vpc.html) & subnets for new workspaces within AWS.
Copy file name to clipboardExpand all lines: docs/resources/mws_storage_configurations.md
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -54,6 +54,7 @@ In addition to all arguments above, the following attributes are exported:
54
54
The following resources are used in the same context:
55
55
56
56
*[Provisioning Databricks on AWS](../guides/aws-workspace.md) guide.
57
+
*[Provisioning Databricks on AWS with PrivateLink](../guides/aws-private-link-workspace.md) guide.
57
58
*[databricks_mws_credentials](mws_credentials.md) to configure the cross-account role for creation of new workspaces within AWS.
58
59
*[databricks_mws_customer_managed_keys](mws_customer_managed_keys.md) to configure KMS keys for new workspaces within AWS.
59
60
*[databricks_mws_log_delivery](mws_log_delivery.md) to configure delivery of [billable usage logs](https://docs.databricks.com/administration-guide/account-settings/billable-usage-delivery.html) and [audit logs](https://docs.databricks.com/administration-guide/account-settings/audit-logs.html).
Copy file name to clipboardExpand all lines: docs/resources/mws_vpc_endpoint.md
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -152,6 +152,7 @@ In addition to all arguments above, the following attributes are exported:
152
152
The following resources are used in the same context:
153
153
154
154
*[Provisioning Databricks on AWS](../guides/aws-workspace.md) guide.
155
+
*[Provisioning Databricks on AWS with PrivateLink](../guides/aws-private-link-workspace.md) guide.
155
156
*[Provisioning AWS Databricks E2 with a Hub & Spoke firewall for data exfiltration protection](../guides/aws-e2-firewall-hub-and-spoke.md) guide.
156
157
*[databricks_mws_networks](mws_networks.md) to [configure VPC](https://docs.databricks.com/administration-guide/cloud-configurations/aws/customer-managed-vpc.html) & subnets for new workspaces within AWS.
157
158
*[databricks_mws_private_access_settings](mws_private_access_settings.md) to create a [Private Access Setting](https://docs.databricks.com/administration-guide/cloud-configurations/aws/privatelink.html#step-5-create-a-private-access-settings-configuration-using-the-databricks-account-api) that can be used as part of a [databricks_mws_workspaces](mws_workspaces.md) resource to create a [Databricks Workspace that leverages AWS PrivateLink](https://docs.databricks.com/administration-guide/cloud-configurations/aws/privatelink.html).
0 commit comments