Skip to content

Commit e7b20cd

Browse files
GorginZapeabody
GorginZ
and
apeabody
authored
fix: firewall rules for autopilot clusters are ineffective. add cluster_network_tag to autopilot cluster network_tags if firewalls are toggled on (#1817)
Co-authored-by: Andrew Peabody <[email protected]>
1 parent 654868e commit e7b20cd

File tree

15 files changed

+1125
-6
lines changed

15 files changed

+1125
-6
lines changed

‎autogen/main/cluster.tf.tmpl‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -258,10 +258,10 @@ resource "google_container_cluster" "primary" {
258258
}
259259
{% if autopilot_cluster %}
260260
dynamic "node_pool_auto_config" {
261-
for_each = length(var.network_tags) > 0 ? [1] : []
261+
for_each = length(var.network_tags) > 0 || var.add_cluster_firewall_rules || var.add_master_webhook_firewall_rules || var.add_shadow_firewall_rules ? [1] : []
262262
content {
263263
network_tags {
264-
tags = var.network_tags
264+
tags = var.add_cluster_firewall_rules || var.add_master_webhook_firewall_rules || var.add_shadow_firewall_rules ? concat(var.network_tags, [local.cluster_network_tag]) : var.network_tags
265265
}
266266
}
267267
}

‎build/int.cloudbuild.yaml‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -454,6 +454,27 @@ steps:
454454
- verify simple-autopilot-private-local
455455
name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
456456
args: ['/bin/bash', '-c', 'cft test run TestSimpleAutopilotPrivate --stage teardown --verbose']
457+
- id: init autopilot-private-firewalls
458+
waitFor:
459+
- create-all
460+
- destroy simple-autopilot-private-local
461+
name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
462+
args: ['/bin/bash', '-c', 'cft test run TestAutopilotPrivateFirewalls --stage init --verbose']
463+
- id: apply autopilot-private-firewalls
464+
waitFor:
465+
- init autopilot-private-firewalls
466+
name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
467+
args: ['/bin/bash', '-c', 'cft test run TestAutopilotPrivateFirewalls --stage apply --verbose']
468+
- id: verify autopilot-private-firewalls
469+
waitFor:
470+
- apply autopilot-private-firewalls
471+
name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
472+
args: ['/bin/bash', '-c', 'cft test run TestAutopilotPrivateFirewalls --stage verify --verbose']
473+
- id: destroy autopilot-private-firewalls
474+
waitFor:
475+
- verify autopilot-private-firewalls
476+
name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
477+
args: ['/bin/bash', '-c', 'cft test run TestAutopilotPrivateFirewalls --stage destroy --verbose']
457478
- id: init simple-autopilot-public-local
458479
waitFor:
459480
- create-all

‎examples/autopilot_private_firewalls/README.md‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# Private Regional Autopilot Cluster With Firewall Rules
2+
3+
This example creates a regional autopilot cluster with beta features
4+
5+
It will:
6+
- Create a private autopilot cluster
7+
- Variables ```add_cluster_firewall_rules```, ```add_master_webhook_firewall_rules``` and ```add_shadow_firewal_rules``` are toggled on. In a environment with stringent firewall rules, these cluster firewall rules may be required to allow intra cluster communication
8+
- Adds an example network tag. This example network tag aligns with a firewall rule target tag from the [terraform-example-foundation](https://github.com/terraform-google-modules/terraform-example-foundation) configuration to allow google api access
9+
10+
>note: this example does **not** create a network with stringent firewall rules. This example shows what cluster configuration may be required for a networking configuration comparable to the [terraform-example-foundation](https://github.com/terraform-google-modules/terraform-example-foundation)'s base shared VPCs, or any VPC with firewall rules to deny 0.0.0.0/0 egress and an allow rule for private google api egress access which targets tags "allow-google-apis". By toggling on the firewall rules variables and adding network_tags to allow private google api access, a private cluster can come up healthy with no internet egress. This configuration shows how to ensure those firewalls that explicitly allow intra cluster ingress and egress are created and appropriate network tags are attached to the cluster.
11+
12+
<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
13+
## Inputs
14+
15+
| Name | Description | Type | Default | Required |
16+
|------|-------------|------|---------|:--------:|
17+
| project\_id | The project ID to host the cluster in | `any` | n/a | yes |
18+
| region | The region the cluster in | `string` | `"us-central1"` | no |
19+
20+
## Outputs
21+
22+
| Name | Description |
23+
|------|-------------|
24+
| ca\_certificate | The cluster ca certificate (base64 encoded) |
25+
| cluster\_name | Cluster name |
26+
| kubernetes\_endpoint | The cluster endpoint |
27+
| location | n/a |
28+
| master\_kubernetes\_version | Kubernetes version of the master |
29+
| network\_name | The name of the VPC being created |
30+
| project\_id | The project ID the cluster is in |
31+
| region | The region in which the cluster resides |
32+
| service\_account | The service account to default running nodes as if not overridden in `node_pools`. |
33+
| subnet\_names | The names of the subnet being created |
34+
| zones | List of zones in which the cluster resides |
35+
36+
<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
37+
38+
To provision this example, run the following from within this directory:
39+
- `terraform init` to get the plugins
40+
- `terraform plan` to see the infrastructure plan
41+
- `terraform apply` to apply the infrastructure build
42+
- `terraform destroy` to destroy the built infrastructure

‎examples/autopilot_private_firewalls/main.tf‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
/**
2+
* Copyright 2024 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
locals {
18+
cluster_type = "autopilot-private-firewalls"
19+
network_name = "${local.cluster_type}-network"
20+
subnet_name = "${local.cluster_type}-subnet"
21+
master_auth_subnetwork = "${local.cluster_type}-master-subnet"
22+
pods_range_name = "ip-range-pods-${local.cluster_type}"
23+
svc_range_name = "ip-range-svc-${local.cluster_type}"
24+
subnet_names = [for subnet_self_link in module.gcp-network.subnets_self_links : split("/", subnet_self_link)[length(split("/", subnet_self_link)) - 1]]
25+
}
26+
27+
data "google_client_config" "default" {}
28+
29+
provider "kubernetes" {
30+
host = "https://${module.gke.endpoint}"
31+
token = data.google_client_config.default.access_token
32+
cluster_ca_certificate = base64decode(module.gke.ca_certificate)
33+
}
34+
35+
module "gke" {
36+
source = "../../modules/beta-autopilot-private-cluster/"
37+
project_id = var.project_id
38+
name = "${local.cluster_type}-cluster"
39+
regional = true
40+
region = var.region
41+
network = module.gcp-network.network_name
42+
subnetwork = local.subnet_names[index(module.gcp-network.subnets_names, local.subnet_name)]
43+
ip_range_pods = local.pods_range_name
44+
ip_range_services = local.svc_range_name
45+
release_channel = "REGULAR"
46+
enable_vertical_pod_autoscaling = true
47+
enable_private_endpoint = true
48+
enable_private_nodes = true
49+
master_ipv4_cidr_block = "172.16.0.0/28"
50+
add_cluster_firewall_rules = true
51+
add_master_webhook_firewall_rules = true
52+
add_shadow_firewall_rules = true
53+
network_tags = ["allow-google-apis"]
54+
deletion_protection = false
55+
56+
master_authorized_networks = [
57+
{
58+
cidr_block = "10.60.0.0/17"
59+
display_name = "VPC"
60+
},
61+
]
62+
}
63+

‎examples/autopilot_private_firewalls/network.tf‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
/**
2+
* Copyright 2024 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
module "gcp-network" {
18+
source = "terraform-google-modules/network/google"
19+
version = ">= 9.0.0"
20+
21+
project_id = var.project_id
22+
network_name = local.network_name
23+
24+
subnets = [
25+
{
26+
subnet_name = local.subnet_name
27+
subnet_ip = "10.0.0.0/17"
28+
subnet_region = var.region
29+
},
30+
{
31+
subnet_name = local.master_auth_subnetwork
32+
subnet_ip = "10.60.0.0/17"
33+
subnet_region = var.region
34+
},
35+
]
36+
37+
secondary_ranges = {
38+
(local.subnet_name) = [
39+
{
40+
range_name = local.pods_range_name
41+
ip_cidr_range = "192.168.0.0/18"
42+
},
43+
{
44+
range_name = local.svc_range_name
45+
ip_cidr_range = "192.168.64.0/18"
46+
},
47+
]
48+
}
49+
}

‎examples/autopilot_private_firewalls/outputs.tf‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
/**
2+
* Copyright 2024 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
output "kubernetes_endpoint" {
18+
description = "The cluster endpoint"
19+
sensitive = true
20+
value = module.gke.endpoint
21+
}
22+
23+
output "cluster_name" {
24+
description = "Cluster name"
25+
value = module.gke.name
26+
}
27+
28+
output "location" {
29+
value = module.gke.location
30+
}
31+
32+
output "master_kubernetes_version" {
33+
description = "Kubernetes version of the master"
34+
value = module.gke.master_version
35+
}
36+
37+
output "ca_certificate" {
38+
sensitive = true
39+
description = "The cluster ca certificate (base64 encoded)"
40+
value = module.gke.ca_certificate
41+
}
42+
43+
output "service_account" {
44+
description = "The service account to default running nodes as if not overridden in `node_pools`."
45+
value = module.gke.service_account
46+
}
47+
48+
output "network_name" {
49+
description = "The name of the VPC being created"
50+
value = module.gcp-network.network_name
51+
}
52+
53+
output "subnet_names" {
54+
description = "The names of the subnet being created"
55+
value = module.gcp-network.subnets_names
56+
}
57+
58+
output "region" {
59+
description = "The region in which the cluster resides"
60+
value = module.gke.region
61+
}
62+
63+
output "zones" {
64+
description = "List of zones in which the cluster resides"
65+
value = module.gke.zones
66+
}
67+
68+
output "project_id" {
69+
description = "The project ID the cluster is in"
70+
value = var.project_id
71+
}

‎examples/autopilot_private_firewalls/variables.tf‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
/**
2+
* Copyright 2024 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
variable "project_id" {
18+
description = "The project ID to host the cluster in"
19+
}
20+
21+
variable "region" {
22+
description = "The region the cluster in"
23+
default = "us-central1"
24+
}

‎examples/autopilot_private_firewalls/versions.tf‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
/**
2+
* Copyright 2024 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
terraform {
18+
required_providers {
19+
google = {
20+
source = "hashicorp/google"
21+
}
22+
kubernetes = {
23+
source = "hashicorp/kubernetes"
24+
}
25+
}
26+
required_version = ">= 0.13"
27+
}

‎modules/beta-autopilot-private-cluster/cluster.tf‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -99,10 +99,10 @@ resource "google_container_cluster" "primary" {
9999
}
100100
}
101101
dynamic "node_pool_auto_config" {
102-
for_each = length(var.network_tags) > 0 ? [1] : []
102+
for_each = length(var.network_tags) > 0 || var.add_cluster_firewall_rules || var.add_master_webhook_firewall_rules || var.add_shadow_firewall_rules ? [1] : []
103103
content {
104104
network_tags {
105-
tags = var.network_tags
105+
tags = var.add_cluster_firewall_rules || var.add_master_webhook_firewall_rules || var.add_shadow_firewall_rules ? concat(var.network_tags, [local.cluster_network_tag]) : var.network_tags
106106
}
107107
}
108108
}

‎modules/beta-autopilot-public-cluster/cluster.tf‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -99,10 +99,10 @@ resource "google_container_cluster" "primary" {
9999
}
100100
}
101101
dynamic "node_pool_auto_config" {
102-
for_each = length(var.network_tags) > 0 ? [1] : []
102+
for_each = length(var.network_tags) > 0 || var.add_cluster_firewall_rules || var.add_master_webhook_firewall_rules || var.add_shadow_firewall_rules ? [1] : []
103103
content {
104104
network_tags {
105-
tags = var.network_tags
105+
tags = var.add_cluster_firewall_rules || var.add_master_webhook_firewall_rules || var.add_shadow_firewall_rules ? concat(var.network_tags, [local.cluster_network_tag]) : var.network_tags
106106
}
107107
}
108108
}

0 commit comments

Comments
 (0)