Skip to content

Commit d5de730

Browse files
jessiedu-dbjessiedu-db
authored
Added support for vpc endpoint on GCP preview (#2080)
1 parent 9ca86a2 commit d5de730

File tree

5 files changed

+199
-19
lines changed

5 files changed

+199
-19
lines changed

‎docs/resources/mws_vpc_endpoint.md‎

Lines changed: 83 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -7,12 +7,14 @@ subcategory: "Deployment"
77

88
-> **Note** This resource has an evolving API, which will change in the upcoming versions of the provider in order to simplify user experience.
99

10-
Enables you to register [aws_vpc_endpoint](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/vpc_endpoint) resources with Databricks such that they can be used as part of a [databricks_mws_networks](mws_networks.md) configuration.
10+
Enables you to register [aws_vpc_endpoint](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/vpc_endpoint) resources or gcp vpc_endpoint resources with Databricks such that they can be used as part of a [databricks_mws_networks](mws_networks.md) configuration.
1111

12-
It is strongly recommended that customers read the [Enable Private Link](https://docs.databricks.com/administration-guide/cloud-configurations/aws/privatelink.html) documentation before trying to leverage this resource.
12+
It is strongly recommended that customers read the [Enable AWS Private Link](https://docs.databricks.com/administration-guide/cloud-configurations/aws/privatelink.html) or the [Enable GCP Private Service Connect](https://docs.gcp.databricks.com/administration-guide/cloud-configurations/gcp/private-service-connect.html) documentation before trying to leverage this resource.
1313

1414
## Example Usage
1515

16+
### Databricks on AWS usage
17+
1618
Before using this resource, you will need to create the necessary VPC Endpoints as per your [VPC endpoint requirements](https://docs.databricks.com/administration-guide/cloud-configurations/aws/privatelink.html#vpc-endpoint-requirements). You can use the [aws_vpc_endpoint](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/vpc_endpoint) resource for this, for example:
1719

1820
```hcl
@@ -107,23 +109,96 @@ resource "databricks_mws_workspaces" "this" {
107109
}
108110
```
109111

112+
### Databricks on GCP usage
113+
114+
Before using this resource, you will need to create the necessary Private Service Connect (PSC) connections on your Google Cloud VPC networks. You can see [Enable Private Service Connect for your workspace](https://docs.gcp.databricks.com/administration-guide/cloud-configurations/gcp/private-service-connect.html) for more details.
115+
116+
Once you have created the necessary PSC connections, you need to register each of them via *this* Terraform resource, which calls out to the Databricks Account API.
117+
118+
```hcl
119+
variable "databricks_account_id" {
120+
description = "Account Id that could be found in https://accounts.gcp.databricks.com/"
121+
}
122+
variable "databricks_google_service_account" {}
123+
variable "google_project" {}
124+
variable "subnet_region" {}
125+
126+
provider "databricks" {
127+
alias = "mws"
128+
host = "https://accounts.gcp.databricks.com"
129+
}
130+
131+
resource "databricks_mws_vpc_endpoint" "workspace" {
132+
provider = databricks.mws
133+
account_id = var.databricks_account_id
134+
vpc_endpoint_name = "PSC Rest API endpoint"
135+
gcp_vpc_endpoint_info {
136+
project_id = var.google_project
137+
psc_endpoint_name = "PSC Rest API endpoint"
138+
endpoint_region = var.subnet_region
139+
}
140+
}
141+
142+
resource "databricks_mws_vpc_endpoint" "relay" {
143+
provider = databricks.mws
144+
account_id = var.databricks_account_id
145+
vpc_endpoint_name = "PSC Relay endpoint"
146+
gcp_vpc_endpoint_info {
147+
project_id = var.google_project
148+
psc_endpoint_name = "PSC Relay endpoint"
149+
endpoint_region = var.subnet_region
150+
}
151+
}
152+
```
153+
154+
Typically the next steps after this would be to create a [databricks_mws_private_access_settings](mws_private_access_settings.md) and [databricks_mws_networks](mws_networks.md) configuration, before passing the `databricks_mws_private_access_settings.pas.private_access_settings_id` and `databricks_mws_networks.this.network_id` into a [databricks_mws_workspaces](mws_workspaces.md) resource:
155+
156+
```hcl
157+
resource "databricks_mws_workspaces" "this" {
158+
provider = databricks.mws
159+
account_id = var.databricks_account_id
160+
workspace_name = "gcp workspace"
161+
location = var.subnet_region
162+
cloud_resource_container {
163+
gcp {
164+
project_id = var.google_project
165+
}
166+
}
167+
gke_config {
168+
connectivity_type = "PRIVATE_NODE_PUBLIC_MASTER"
169+
master_ip_range = "10.3.0.0/28"
170+
}
171+
network_id = databricks_mws_networks.this.network_id
172+
private_access_settings_id = databricks_mws_private_access_settings.pas.private_access_settings_id
173+
pricing_tier = "PREMIUM"
174+
depends_on = [databricks_mws_networks.this]
175+
}
176+
```
177+
110178
## Argument Reference
111179

112180
The following arguments are required:
113181

114-
* `account_id` - Account Id that could be found in the bottom left corner of [Accounts Console](https://accounts.cloud.databricks.com/)
115-
* `aws_vpc_endpoint_id` - ID of configured [aws_vpc_endpoint](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/vpc_endpoint)
182+
* `account_id` - Account Id that could be found in the Accounts Console for [AWS](https://accounts.cloud.databricks.com/) or [GCP](https://accounts.gcp.databricks.com/)
183+
* `aws_vpc_endpoint_id` - (AWS only) ID of configured [aws_vpc_endpoint](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/vpc_endpoint)
116184
* `vpc_endpoint_name` - Name of VPC Endpoint in Databricks Account
117-
* `region` - Region of AWS VPC
185+
* `region` - (AWS only) Region of AWS VPC
186+
* `gcp_vpc_endpoint_info` - (GCP only) a block consists of Google Cloud specific information for this PSC endpoint. It has the following fields:
187+
* `project_id` - The Google Cloud project ID of the VPC network where the PSC connection resides.
188+
* `psc_endpoint_name` - The name of the PSC endpoint in the Google Cloud project.
189+
* `endpoint_region` - Region of the PSC endpoint.
118190

119191
## Attribute Reference
120192

121193
In addition to all arguments above, the following attributes are exported:
122194

123195
* `vpc_endpoint_id` - Canonical unique identifier of VPC Endpoint in Databricks Account
124-
* `aws_endpoint_service_id` - The ID of the Databricks endpoint service that this VPC endpoint is connected to. Please find the list of endpoint service IDs for each supported region in the [Databricks PrivateLink documentation](https://docs.databricks.com/administration-guide/cloud-configurations/aws/privatelink.html)
125-
* `state` - State of VPC Endpoint
126-
196+
* `aws_endpoint_service_id` - (AWS Only) The ID of the Databricks endpoint service that this VPC endpoint is connected to. Please find the list of endpoint service IDs for each supported region in the [Databricks PrivateLink documentation](https://docs.databricks.com/administration-guide/cloud-configurations/aws/privatelink.html)
197+
* `state` - (AWS Only) State of VPC Endpoint
198+
* `gcp_vpc_endpoint_info`- (GCP only) a block consists of Google Cloud specific information for this PSC endpoint. It has the following fields exported:
199+
* `psc_connection_id` - The unique ID of this PSC connection.
200+
* `service_attachment_id` - The service attachment this PSC connection connects to.
201+
127202
## Import
128203

129204
-> **Note** Importing this resource is not currently supported.

‎internal/acceptance/mws_vpc_endpoint_test.go‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,3 +17,20 @@ func TestMwsAccVpcEndpoint(t *testing.T) {
1717
}`,
1818
})
1919
}
20+
21+
func TestMwsAccVpcEndpoint_GCP(t *testing.T) {
22+
accountLevel(t, step{
23+
Template: `
24+
resource "databricks_mws_vpc_endpoint" "this" {
25+
account_id = "{env.DATABRICKS_ACCOUNT_ID}"
26+
vpc_endpoint_name = "vpce-{var.RANDOM}"
27+
28+
gcp_vpc_endpoint_info {
29+
project_id = "{env.GOOGLE_PROJECT}"
30+
psc_endpoint_name = "{env.TEST_RELAY_PSC_ENDPOINT}"
31+
endpoint_region = "{env.GOOGLE_REGION}"
32+
}
33+
34+
}`,
35+
})
36+
}

‎mws/mws.go‎

Lines changed: 19 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -72,17 +72,27 @@ type Network struct {
7272
GcpNetworkInfo *GcpNetworkInfo `json:"gcp_network_info,omitempty"`
7373
}
7474

75+
// GcpVpcEndpointInfo is the objecy that configures GCP Private Service Connect endpoints.
76+
type GcpVpcEndpointInfo struct {
77+
PscConnectionId string `json:"psc_connection_id,omitempty" tf:"computed"`
78+
ProjectId string `json:"project_id"`
79+
PscEndpointName string `json:"psc_endpoint_name"`
80+
EndpointRegion string `json:"endpoint_region"`
81+
ServiceAttachmentId string `json:"service_attachment_id,omitempty" tf:"computed"`
82+
}
83+
7584
// VPCEndpoint is the object that contains all the information for registering an VPC endpoint
7685
type VPCEndpoint struct {
77-
VPCEndpointID string `json:"vpc_endpoint_id,omitempty" tf:"computed"`
78-
AwsVPCEndpointID string `json:"aws_vpc_endpoint_id"`
79-
AccountID string `json:"account_id,omitempty"`
80-
VPCEndpointName string `json:"vpc_endpoint_name"`
81-
AwsVPCEndpointServiceID string `json:"aws_endpoint_service_id,omitempty" tf:"computed"`
82-
AWSAccountID string `json:"aws_account_id,omitempty" tf:"computed"`
83-
UseCase string `json:"use_case,omitempty" tf:"computed"`
84-
Region string `json:"region"`
85-
State string `json:"state,omitempty" tf:"computed"`
86+
VPCEndpointID string `json:"vpc_endpoint_id,omitempty" tf:"computed"`
87+
AwsVPCEndpointID string `json:"aws_vpc_endpoint_id,omitempty"`
88+
AccountID string `json:"account_id,omitempty"`
89+
VPCEndpointName string `json:"vpc_endpoint_name"`
90+
AwsVPCEndpointServiceID string `json:"aws_endpoint_service_id,omitempty" tf:"computed"`
91+
AWSAccountID string `json:"aws_account_id,omitempty" tf:"computed"`
92+
UseCase string `json:"use_case,omitempty" tf:"computed"`
93+
Region string `json:"region,omitempty"`
94+
State string `json:"state,omitempty" tf:"computed"`
95+
GcpVpcEndpointInfo *GcpVpcEndpointInfo `json:"gcp_vpc_endpoint_info,omitempty"`
8696
}
8797

8898
// PrivateAccessSettings (PAS) is the object that contains all the information for creating an PrivateAccessSettings (PAS)

‎mws/resource_mws_vpc_endpoint.go‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,6 @@ import (
1010

1111
"github.com/hashicorp/terraform-plugin-sdk/v2/helper/resource"
1212
"github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema"
13-
"github.com/hashicorp/terraform-plugin-sdk/v2/helper/validation"
1413
)
1514

1615
// NewVPCEndpointAPI creates VPCEndpointAPI instance from provider meta
@@ -36,6 +35,9 @@ func (a VPCEndpointAPI) Create(vpcEndpoint *VPCEndpoint) error {
3635
if err != nil {
3736
return resource.NonRetryableError(err)
3837
}
38+
if ve.GcpVpcEndpointInfo != nil {
39+
return nil
40+
}
3941
switch state := strings.ToLower(ve.State); state {
4042
case "available":
4143
return nil
@@ -75,7 +77,11 @@ func (a VPCEndpointAPI) List(mwsAcctID string) ([]VPCEndpoint, error) {
7577
func ResourceMwsVpcEndpoint() *schema.Resource {
7678
s := common.StructToSchema(VPCEndpoint{}, func(s map[string]*schema.Schema) map[string]*schema.Schema {
7779
// nolint
78-
s["vpc_endpoint_name"].ValidateFunc = validation.StringLenBetween(4, 256)
80+
s["aws_vpc_endpoint_id"].ExactlyOneOf = []string{"aws_vpc_endpoint_id", "gcp_vpc_endpoint_info"}
81+
s["region"].ExactlyOneOf = []string{"region", "gcp_vpc_endpoint_info"}
82+
s["aws_endpoint_service_id"].ConflictsWith = []string{"gcp_vpc_endpoint_info"}
83+
s["aws_account_id"].ConflictsWith = []string{"gcp_vpc_endpoint_info"}
84+
s["gcp_vpc_endpoint_info"].ConflictsWith = []string{"aws_vpc_endpoint_id", "region", "aws_endpoint_service_id", "aws_account_id"}
7985
return s
8086
})
8187
p := common.NewPairSeparatedID("account_id", "vpc_endpoint_id", "/")

‎mws/resource_mws_vpc_endpoint_test.go‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,78 @@ func TestResourceVPCEndpointCreate(t *testing.T) {
5454
assert.Equal(t, "abc/ve_id", d.Id())
5555
}
5656

57+
func TestResourceVPCEndpointCreate_GCP(t *testing.T) {
58+
qa.ResourceFixture{
59+
Fixtures: []qa.HTTPFixture{
60+
{
61+
Method: "POST",
62+
Resource: "/api/2.0/accounts/abc/vpc-endpoints",
63+
ExpectedRequest: VPCEndpoint{
64+
AccountID: "abc",
65+
VPCEndpointName: "ve_name",
66+
GcpVpcEndpointInfo: &GcpVpcEndpointInfo{
67+
ProjectId: "project_a",
68+
PscEndpointName: "psc_endpoint_a",
69+
EndpointRegion: "region_a",
70+
},
71+
},
72+
Response: VPCEndpoint{
73+
VPCEndpointID: "ve_id",
74+
},
75+
},
76+
{
77+
Method: "GET",
78+
Resource: "/api/2.0/accounts/abc/vpc-endpoints/ve_id",
79+
ReuseRequest: true,
80+
Response: VPCEndpoint{
81+
AccountID: "abc",
82+
VPCEndpointName: "ve_name",
83+
GcpVpcEndpointInfo: &GcpVpcEndpointInfo{
84+
ProjectId: "project_a",
85+
PscEndpointName: "psc_endpoint_a",
86+
EndpointRegion: "region_a",
87+
PscConnectionId: "120938102938209",
88+
ServiceAttachmentId: "service_attachment_a",
89+
},
90+
},
91+
},
92+
},
93+
Resource: ResourceMwsVpcEndpoint(),
94+
HCL: `
95+
account_id = "abc"
96+
vpc_endpoint_name = "ve_name"
97+
gcp_vpc_endpoint_info {
98+
project_id = "project_a"
99+
psc_endpoint_name = "psc_endpoint_a"
100+
endpoint_region = "region_a"
101+
}
102+
`,
103+
Create: true,
104+
}.ApplyNoError(t)
105+
}
106+
107+
func TestResourceVPCEndpointCreate_ConflictErrors(t *testing.T) {
108+
_, err := qa.ResourceFixture{
109+
Fixtures: []qa.HTTPFixture{},
110+
Resource: ResourceMwsVpcEndpoint(),
111+
HCL: `
112+
account_id = "abc"
113+
vpc_endpoint_name = "ve_name"
114+
region = "ar"
115+
aws_vpc_endpoint_id = "ave_id"
116+
gcp_vpc_endpoint_info {
117+
project_id = "project_a"
118+
psc_endpoint_name = "psc_endpoint_a"
119+
endpoint_region = "region_a"
120+
}
121+
`,
122+
Create: true,
123+
}.Apply(t)
124+
assert.ErrorContains(t, err, "[gcp_vpc_endpoint_info] Conflicting configuration arguments")
125+
assert.ErrorContains(t, err, "[region] Invalid combination of arguments")
126+
assert.ErrorContains(t, err, "[aws_vpc_endpoint_id] Invalid combination of arguments")
127+
}
128+
57129
func TestResourceVPCEndpointCreate_Error(t *testing.T) {
58130
d, err := qa.ResourceFixture{
59131
Fixtures: []qa.HTTPFixture{

0 commit comments

Comments
 (0)