Skip to content

Commit 259dbfb

Browse files
ericyzericyz
authored
feat: Add support for creating "shadow" firewall rules for logging purposes (#741)
1 parent f32dea7 commit 259dbfb

File tree

23 files changed

+798
-0
lines changed

23 files changed

+798
-0
lines changed

‎README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -124,6 +124,7 @@ Then perform the following commands on the root folder:
124124
| Name | Description | Type | Default | Required |
125125
|------|-------------|------|---------|:--------:|
126126
| add\_cluster\_firewall\_rules | Create additional firewall rules | `bool` | `false` | no |
127+
| add\_shadow\_firewall\_rules | Create GKE shadow firewall (the same as default firewall rules with firewall logs enabled). | `bool` | `false` | no |
127128
| basic\_auth\_password | The password to be used with Basic Authentication. | `string` | `""` | no |
128129
| basic\_auth\_username | The username to be used with Basic Authentication. An empty value will disable Basic Authentication, which is the recommended configuration. | `string` | `""` | no |
129130
| cluster\_autoscaling | Cluster autoscaling configuration. See [more details](https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1beta1/projects.locations.clusters#clusterautoscaling) | <pre>object({<br> enabled = bool<br> min_cpu_cores = number<br> max_cpu_cores = number<br> min_memory_gb = number<br> max_memory_gb = number<br> })</pre> | <pre>{<br> "enabled": false,<br> "max_cpu_cores": 0,<br> "max_memory_gb": 0,<br> "min_cpu_cores": 0,<br> "min_memory_gb": 0<br>}</pre> | no |
@@ -181,6 +182,7 @@ Then perform the following commands on the root folder:
181182
| remove\_default\_node\_pool | Remove default node pool while setting up the cluster | `bool` | `false` | no |
182183
| resource\_usage\_export\_dataset\_id | The ID of a BigQuery Dataset for using BigQuery as the destination of resource usage export. | `string` | `""` | no |
183184
| service\_account | The service account to run nodes as if not overridden in `node_pools`. The create\_service\_account variable default value (true) will cause a cluster-specific service account to be created. | `string` | `""` | no |
185+
| shadow\_firewall\_rules\_priority | The firewall priority of GKE shadow firewall rules. The priority should be less than default firewall, which is 1000. | `number` | `999` | no |
184186
| skip\_provisioners | Flag to skip all local-exec provisioners. It breaks `stub_domains` and `upstream_nameservers` variables functionality. | `bool` | `false` | no |
185187
| stub\_domains | Map of stub domains and their resolvers to forward DNS queries for a certain domain to an external DNS server | `map(list(string))` | `{}` | no |
186188
| subnetwork | The subnetwork to host the cluster in (required) | `string` | n/a | yes |

‎autogen/main/firewall.tf.tmpl‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -95,3 +95,89 @@ resource "google_compute_firewall" "master_webhooks" {
9595
{% endif %}
9696

9797
}
98+
99+
100+
/******************************************
101+
Create shadow firewall rules to capture the
102+
traffic flow between the managed firewall rules
103+
*****************************************/
104+
resource "google_compute_firewall" "shadow_allow_pods" {
105+
count = var.add_shadow_firewall_rules ? 1 : 0
106+
107+
name = "gke-shadow-${substr(var.name, 0, min(25, length(var.name)))}-all"
108+
description = "Managed by terraform gke module: A shadow firewall rule to match the default rule allowing pod communication."
109+
project = local.network_project_id
110+
network = var.network
111+
priority = var.shadow_firewall_rules_priority
112+
direction = "INGRESS"
113+
114+
source_ranges = [local.cluster_alias_ranges_cidr[var.ip_range_pods]]
115+
target_tags = [local.cluster_network_tag]
116+
117+
# Allow all possible protocols
118+
allow { protocol = "tcp" }
119+
allow { protocol = "udp" }
120+
allow { protocol = "icmp" }
121+
allow { protocol = "sctp" }
122+
allow { protocol = "esp" }
123+
allow { protocol = "ah" }
124+
125+
log_config {
126+
metadata = "INCLUDE_ALL_METADATA"
127+
}
128+
}
129+
130+
resource "google_compute_firewall" "shadow_allow_master" {
131+
count = var.add_shadow_firewall_rules ? 1 : 0
132+
133+
name = "gke-shadow-${substr(var.name, 0, min(25, length(var.name)))}-master"
134+
description = "Managed by terraform GKE module: A shadow firewall rule to match the default rule allowing worker nodes communication."
135+
project = local.network_project_id
136+
network = var.network
137+
priority = var.shadow_firewall_rules_priority
138+
direction = "INGRESS"
139+
140+
source_ranges = [local.cluster_endpoint_for_nodes]
141+
target_tags = [local.cluster_network_tag]
142+
143+
allow {
144+
protocol = "tcp"
145+
ports = ["10250", "443"]
146+
}
147+
148+
log_config {
149+
metadata = "INCLUDE_ALL_METADATA"
150+
}
151+
}
152+
153+
resource "google_compute_firewall" "shadow_allow_nodes" {
154+
count = var.add_shadow_firewall_rules ? 1 : 0
155+
156+
name = "gke-shadow-${substr(var.name, 0, min(25, length(var.name)))}-vms"
157+
description = "Managed by Terraform GKE module: A shadow firewall rule to match the default rule allowing worker nodes communication."
158+
project = local.network_project_id
159+
network = var.network
160+
priority = var.shadow_firewall_rules_priority
161+
direction = "INGRESS"
162+
163+
source_ranges = [local.cluster_subnet_cidr]
164+
target_tags = [local.cluster_network_tag]
165+
166+
allow {
167+
protocol = "icmp"
168+
}
169+
170+
allow {
171+
protocol = "udp"
172+
ports = ["1-65535"]
173+
}
174+
175+
allow {
176+
protocol = "tcp"
177+
ports = ["1-65535"]
178+
}
179+
180+
log_config {
181+
metadata = "INCLUDE_ALL_METADATA"
182+
}
183+
}

‎autogen/main/variables.tf.tmpl‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -580,6 +580,18 @@ variable "gcloud_upgrade" {
580580
default = false
581581
}
582582

583+
variable "add_shadow_firewall_rules" {
584+
type = bool
585+
description = "Create GKE shadow firewall (the same as default firewall rules with firewall logs enabled)."
586+
default = false
587+
}
588+
589+
variable "shadow_firewall_rules_priority" {
590+
type = number
591+
description = "The firewall priority of GKE shadow firewall rules. The priority should be less than default firewall, which is 1000."
592+
default = 999
593+
}
594+
583595
{% if beta_cluster %}
584596
variable "disable_default_snat" {
585597
type = bool

‎firewall.tf‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,3 +82,89 @@ resource "google_compute_firewall" "master_webhooks" {
8282
]
8383

8484
}
85+
86+
87+
/******************************************
88+
Create shadow firewall rules to capture the
89+
traffic flow between the managed firewall rules
90+
*****************************************/
91+
resource "google_compute_firewall" "shadow_allow_pods" {
92+
count = var.add_shadow_firewall_rules ? 1 : 0
93+
94+
name = "gke-shadow-${substr(var.name, 0, min(25, length(var.name)))}-all"
95+
description = "Managed by terraform gke module: A shadow firewall rule to match the default rule allowing pod communication."
96+
project = local.network_project_id
97+
network = var.network
98+
priority = var.shadow_firewall_rules_priority
99+
direction = "INGRESS"
100+
101+
source_ranges = [local.cluster_alias_ranges_cidr[var.ip_range_pods]]
102+
target_tags = [local.cluster_network_tag]
103+
104+
# Allow all possible protocols
105+
allow { protocol = "tcp" }
106+
allow { protocol = "udp" }
107+
allow { protocol = "icmp" }
108+
allow { protocol = "sctp" }
109+
allow { protocol = "esp" }
110+
allow { protocol = "ah" }
111+
112+
log_config {
113+
metadata = "INCLUDE_ALL_METADATA"
114+
}
115+
}
116+
117+
resource "google_compute_firewall" "shadow_allow_master" {
118+
count = var.add_shadow_firewall_rules ? 1 : 0
119+
120+
name = "gke-shadow-${substr(var.name, 0, min(25, length(var.name)))}-master"
121+
description = "Managed by terraform GKE module: A shadow firewall rule to match the default rule allowing worker nodes communication."
122+
project = local.network_project_id
123+
network = var.network
124+
priority = var.shadow_firewall_rules_priority
125+
direction = "INGRESS"
126+
127+
source_ranges = [local.cluster_endpoint_for_nodes]
128+
target_tags = [local.cluster_network_tag]
129+
130+
allow {
131+
protocol = "tcp"
132+
ports = ["10250", "443"]
133+
}
134+
135+
log_config {
136+
metadata = "INCLUDE_ALL_METADATA"
137+
}
138+
}
139+
140+
resource "google_compute_firewall" "shadow_allow_nodes" {
141+
count = var.add_shadow_firewall_rules ? 1 : 0
142+
143+
name = "gke-shadow-${substr(var.name, 0, min(25, length(var.name)))}-vms"
144+
description = "Managed by Terraform GKE module: A shadow firewall rule to match the default rule allowing worker nodes communication."
145+
project = local.network_project_id
146+
network = var.network
147+
priority = var.shadow_firewall_rules_priority
148+
direction = "INGRESS"
149+
150+
source_ranges = [local.cluster_subnet_cidr]
151+
target_tags = [local.cluster_network_tag]
152+
153+
allow {
154+
protocol = "icmp"
155+
}
156+
157+
allow {
158+
protocol = "udp"
159+
ports = ["1-65535"]
160+
}
161+
162+
allow {
163+
protocol = "tcp"
164+
ports = ["1-65535"]
165+
}
166+
167+
log_config {
168+
metadata = "INCLUDE_ALL_METADATA"
169+
}
170+
}

‎modules/beta-private-cluster-update-variant/README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -155,6 +155,7 @@ Then perform the following commands on the root folder:
155155
| Name | Description | Type | Default | Required |
156156
|------|-------------|------|---------|:--------:|
157157
| add\_cluster\_firewall\_rules | Create additional firewall rules | `bool` | `false` | no |
158+
| add\_shadow\_firewall\_rules | Create GKE shadow firewall (the same as default firewall rules with firewall logs enabled). | `bool` | `false` | no |
158159
| authenticator\_security\_group | The name of the RBAC security group for use with Google security groups in Kubernetes RBAC. Group name must be in format [email protected] | `string` | `null` | no |
159160
| basic\_auth\_password | The password to be used with Basic Authentication. | `string` | `""` | no |
160161
| basic\_auth\_username | The username to be used with Basic Authentication. An empty value will disable Basic Authentication, which is the recommended configuration. | `string` | `""` | no |
@@ -237,6 +238,7 @@ Then perform the following commands on the root folder:
237238
| resource\_usage\_export\_dataset\_id | The ID of a BigQuery Dataset for using BigQuery as the destination of resource usage export. | `string` | `""` | no |
238239
| sandbox\_enabled | (Beta) Enable GKE Sandbox (Do not forget to set `image_type` = `COS_CONTAINERD` to use it). | `bool` | `false` | no |
239240
| service\_account | The service account to run nodes as if not overridden in `node_pools`. The create\_service\_account variable default value (true) will cause a cluster-specific service account to be created. | `string` | `""` | no |
241+
| shadow\_firewall\_rules\_priority | The firewall priority of GKE shadow firewall rules. The priority should be less than default firewall, which is 1000. | `number` | `999` | no |
240242
| skip\_provisioners | Flag to skip all local-exec provisioners. It breaks `stub_domains` and `upstream_nameservers` variables functionality. | `bool` | `false` | no |
241243
| stub\_domains | Map of stub domains and their resolvers to forward DNS queries for a certain domain to an external DNS server | `map(list(string))` | `{}` | no |
242244
| subnetwork | The subnetwork to host the cluster in (required) | `string` | n/a | yes |

‎modules/beta-private-cluster-update-variant/firewall.tf‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,3 +77,89 @@ resource "google_compute_firewall" "master_webhooks" {
7777

7878

7979
}
80+
81+
82+
/******************************************
83+
Create shadow firewall rules to capture the
84+
traffic flow between the managed firewall rules
85+
*****************************************/
86+
resource "google_compute_firewall" "shadow_allow_pods" {
87+
count = var.add_shadow_firewall_rules ? 1 : 0
88+
89+
name = "gke-shadow-${substr(var.name, 0, min(25, length(var.name)))}-all"
90+
description = "Managed by terraform gke module: A shadow firewall rule to match the default rule allowing pod communication."
91+
project = local.network_project_id
92+
network = var.network
93+
priority = var.shadow_firewall_rules_priority
94+
direction = "INGRESS"
95+
96+
source_ranges = [local.cluster_alias_ranges_cidr[var.ip_range_pods]]
97+
target_tags = [local.cluster_network_tag]
98+
99+
# Allow all possible protocols
100+
allow { protocol = "tcp" }
101+
allow { protocol = "udp" }
102+
allow { protocol = "icmp" }
103+
allow { protocol = "sctp" }
104+
allow { protocol = "esp" }
105+
allow { protocol = "ah" }
106+
107+
log_config {
108+
metadata = "INCLUDE_ALL_METADATA"
109+
}
110+
}
111+
112+
resource "google_compute_firewall" "shadow_allow_master" {
113+
count = var.add_shadow_firewall_rules ? 1 : 0
114+
115+
name = "gke-shadow-${substr(var.name, 0, min(25, length(var.name)))}-master"
116+
description = "Managed by terraform GKE module: A shadow firewall rule to match the default rule allowing worker nodes communication."
117+
project = local.network_project_id
118+
network = var.network
119+
priority = var.shadow_firewall_rules_priority
120+
direction = "INGRESS"
121+
122+
source_ranges = [local.cluster_endpoint_for_nodes]
123+
target_tags = [local.cluster_network_tag]
124+
125+
allow {
126+
protocol = "tcp"
127+
ports = ["10250", "443"]
128+
}
129+
130+
log_config {
131+
metadata = "INCLUDE_ALL_METADATA"
132+
}
133+
}
134+
135+
resource "google_compute_firewall" "shadow_allow_nodes" {
136+
count = var.add_shadow_firewall_rules ? 1 : 0
137+
138+
name = "gke-shadow-${substr(var.name, 0, min(25, length(var.name)))}-vms"
139+
description = "Managed by Terraform GKE module: A shadow firewall rule to match the default rule allowing worker nodes communication."
140+
project = local.network_project_id
141+
network = var.network
142+
priority = var.shadow_firewall_rules_priority
143+
direction = "INGRESS"
144+
145+
source_ranges = [local.cluster_subnet_cidr]
146+
target_tags = [local.cluster_network_tag]
147+
148+
allow {
149+
protocol = "icmp"
150+
}
151+
152+
allow {
153+
protocol = "udp"
154+
ports = ["1-65535"]
155+
}
156+
157+
allow {
158+
protocol = "tcp"
159+
ports = ["1-65535"]
160+
}
161+
162+
log_config {
163+
metadata = "INCLUDE_ALL_METADATA"
164+
}
165+
}

‎modules/beta-private-cluster-update-variant/variables.tf‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -562,6 +562,18 @@ variable "gcloud_upgrade" {
562562
default = false
563563
}
564564

565+
variable "add_shadow_firewall_rules" {
566+
type = bool
567+
description = "Create GKE shadow firewall (the same as default firewall rules with firewall logs enabled)."
568+
default = false
569+
}
570+
571+
variable "shadow_firewall_rules_priority" {
572+
type = number
573+
description = "The firewall priority of GKE shadow firewall rules. The priority should be less than default firewall, which is 1000."
574+
default = 999
575+
}
576+
565577
variable "disable_default_snat" {
566578
type = bool
567579
description = "Whether to disable the default SNAT to support the private use of public IP addresses"

‎modules/beta-private-cluster/README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -133,6 +133,7 @@ Then perform the following commands on the root folder:
133133
| Name | Description | Type | Default | Required |
134134
|------|-------------|------|---------|:--------:|
135135
| add\_cluster\_firewall\_rules | Create additional firewall rules | `bool` | `false` | no |
136+
| add\_shadow\_firewall\_rules | Create GKE shadow firewall (the same as default firewall rules with firewall logs enabled). | `bool` | `false` | no |
136137
| authenticator\_security\_group | The name of the RBAC security group for use with Google security groups in Kubernetes RBAC. Group name must be in format [email protected] | `string` | `null` | no |
137138
| basic\_auth\_password | The password to be used with Basic Authentication. | `string` | `""` | no |
138139
| basic\_auth\_username | The username to be used with Basic Authentication. An empty value will disable Basic Authentication, which is the recommended configuration. | `string` | `""` | no |
@@ -215,6 +216,7 @@ Then perform the following commands on the root folder:
215216
| resource\_usage\_export\_dataset\_id | The ID of a BigQuery Dataset for using BigQuery as the destination of resource usage export. | `string` | `""` | no |
216217
| sandbox\_enabled | (Beta) Enable GKE Sandbox (Do not forget to set `image_type` = `COS_CONTAINERD` to use it). | `bool` | `false` | no |
217218
| service\_account | The service account to run nodes as if not overridden in `node_pools`. The create\_service\_account variable default value (true) will cause a cluster-specific service account to be created. | `string` | `""` | no |
219+
| shadow\_firewall\_rules\_priority | The firewall priority of GKE shadow firewall rules. The priority should be less than default firewall, which is 1000. | `number` | `999` | no |
218220
| skip\_provisioners | Flag to skip all local-exec provisioners. It breaks `stub_domains` and `upstream_nameservers` variables functionality. | `bool` | `false` | no |
219221
| stub\_domains | Map of stub domains and their resolvers to forward DNS queries for a certain domain to an external DNS server | `map(list(string))` | `{}` | no |
220222
| subnetwork | The subnetwork to host the cluster in (required) | `string` | n/a | yes |

0 commit comments

Comments
 (0)