chore: Add Safer Cluster with IAP bastion host access pattern example (#544)

* add safer cluster access pattern

* Apply suggestions from code review

address comments

Co-authored-by: Morgante Pell <[email protected]>

* remove redundant var

* add e2e test

* fix e2e

* fix sandbox-enabled tests

Co-authored-by: Morgante Pell <[email protected]>

Author: bharathkkb

.kitchen.yml

@@ -221,3 +221,10 @@ suites:
       systems:
         - name: sandbox_enabled
           backend: local
+  - name: "safer_cluster_iap_bastion"
+    driver:
+      root_module_directory: test/fixtures/safer_cluster_iap_bastion
+    verifier:
+      systems:
+        - name: safer_cluster_iap_bastion
+          backend: local

build/int.cloudbuild.yaml

@@ -384,6 +384,26 @@ steps:
     - verify workload-identity-local
   name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
   args: ['/bin/bash', '-c', 'source /usr/local/bin/task_helper_functions.sh && kitchen_do destroy workload-identity-local']
+- id: create safer-cluster-iap-bastion-local
+  waitFor:
+    - prepare
+  name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
+  args: ['/bin/bash', '-c', 'source /usr/local/bin/task_helper_functions.sh && kitchen_do create safer-cluster-iap-bastion-local']
+- id: converge safer-cluster-iap-bastion-local
+  waitFor:
+    - create safer-cluster-iap-bastion-local
+  name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
+  args: ['/bin/bash', '-c', 'source /usr/local/bin/task_helper_functions.sh && kitchen_do converge safer-cluster-iap-bastion-local']
+- id: verify safer-cluster-iap-bastion-local
+  waitFor:
+    - converge safer-cluster-iap-bastion-local
+  name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
+  args: ['/bin/bash', '-c', 'source /usr/local/bin/task_helper_functions.sh && kitchen_do verify safer-cluster-iap-bastion-local']
+- id: destroy safer-cluster-iap-bastion-local
+  waitFor:
+    - verify safer-cluster-iap-bastion-local
+  name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
+  args: ['/bin/bash', '-c', 'source /usr/local/bin/task_helper_functions.sh && kitchen_do destroy safer-cluster-iap-bastion-local']
 tags:
 - 'ci'
 - 'integration'

examples/safer_cluster_iap_bastion/README.md

@@ -0,0 +1,70 @@
+# Safer Cluster Access with IAP Bastion Host
+
+This end to end example aims to showcase access patterns to a [Safer Cluster](../../modules/safer-cluster/README.md), which is a hardened GKE Private Cluster, through a bastion host utilizing [Identity Awareness Proxy](https://cloud.google.com/iap/) without an external ip address. Access to this cluster's control plane is restricted to the bastion host's internal IP using [authorized networks](https://cloud.google.com/kubernetes-engine/docs/how-to/authorized-networks#overview).
+
+Additionally we deploy a [tinyproxy](https://tinyproxy.github.io/) daemon which allows `kubectl` commands to be piped through the bastion host allowing ease of development from a local machine with the security of GKE Private Clusters.
+
+## Setup
+
+To deploy this example:
+
+1. Run `terraform init`.
+
+2. Create a `terraform.tfvars` to provide values for `project_id`, `bastion_members`. Optionally override any variables if necessary.
+
+3. Run `terraform apply`.
+
+4. After apply is complete, generate kubeconfig for the private cluster. _The command with the right parameters will displayed as the Terraform output `get_credentials_command`._
+
+   ```sh
+   gcloud container clusters get-credentials --project $PROJECT_ID --zone $ZONE --internal-ip $CLUSTER_NAME
+   ```
+
+5. SSH to the Bastion Host while port forwarding to the bastion host through an IAP tunnel. _The command with the right parameters will displayed by running `terraform output bastion_ssh_command`._
+
+   ```sh
+   gcloud beta compute ssh $BASTION_VM_NAME --tunnel-through-iap --project $PROJECT_ID --zone $ZONE -- -L8888:127.0.0.1:8888
+   ```
+
+6. You can now run `kubectl` commands though the proxy. _An example command will displayed as the Terraform output `bastion_kubectl_command`._
+
+   ```sh
+   HTTPS_PROXY=localhost:8888 kubectl get pods --all-namespaces
+   ```
+
+ <!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
+## Inputs
+
+| Name | Description | Type | Default | Required |
+|------|-------------|:----:|:-----:|:-----:|
+| bastion\_members | List of users, groups, SAs who need access to the bastion host | list(string) | `<list>` | no |
+| cluster\_name | The name of the cluster | string | `"safer-cluster-iap-bastion"` | no |
+| ip\_range\_pods\_name | The secondary ip range to use for pods | string | `"ip-range-pods"` | no |
+| ip\_range\_services\_name | The secondary ip range to use for pods | string | `"ip-range-svc"` | no |
+| ip\_source\_ranges\_ssh | Additional source ranges to allow for ssh to bastion host. 35.235.240.0/20 allowed by default for IAP tunnel. | list(string) | `<list>` | no |
+| network\_name | The name of the network being created to host the cluster in | string | `"safer-cluster-network"` | no |
+| project\_id | The project ID to host the cluster in | string | n/a | yes |
+| region | The region to host the cluster in | string | `"us-central1"` | no |
+| subnet\_ip | The cidr range of the subnet | string | `"10.10.10.0/24"` | no |
+| subnet\_name | The name of the subnet being created to host the cluster in | string | `"safer-cluster-subnet"` | no |
+
+## Outputs
+
+| Name | Description |
+|------|-------------|
+| bastion\_kubectl\_command | kubectl command using the local proxy once the bastion_ssh command is running |
+| bastion\_name | Name of the bastion host |
+| bastion\_ssh\_command | gcloud command to ssh and port forward to the bastion host command |
+| bastion\_zone | Location of bastion host |
+| ca\_certificate | Cluster ca certificate (base64 encoded) |
+| cluster\_name | Cluster name |
+| endpoint | Cluster endpoint |
+| get\_credentials\_command | gcloud get-credentials command to generate kubeconfig for the private cluster |
+| location | Cluster location (region if regional cluster, zone if zonal cluster) |
+| master\_authorized\_networks\_config | Networks from which access to master is permitted |
+| network\_name | The name of the VPC being created |
+| region | Subnet/Router/Bastion Host region |
+| router\_name | Name of the router that was created |
+| subnet\_name | The name of the VPC subnet being created |
+
+ <!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

examples/safer_cluster_iap_bastion/apis.tf

@@ -0,0 +1,35 @@
+/**
+ * Copyright 2020 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+module "enabled_google_apis" {
+  source  = "terraform-google-modules/project-factory/google//modules/project_services"
+  version = "~> 8.0"
+
+  project_id                  = var.project_id
+  disable_services_on_destroy = false
+
+  activate_apis = [
+    "iam.googleapis.com",
+    "compute.googleapis.com",
+    "logging.googleapis.com",
+    "monitoring.googleapis.com",
+    "containerregistry.googleapis.com",
+    "container.googleapis.com",
+    "binaryauthorization.googleapis.com",
+    "stackdriver.googleapis.com",
+    "iap.googleapis.com",
+  ]
+}

examples/safer_cluster_iap_bastion/bastion.tf

@@ -0,0 +1,44 @@
+/**
+ * Copyright 2018 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+locals {
+  bastion_name = format("%s-bastion", var.cluster_name)
+  bastion_zone = format("%s-a", var.region)
+}
+
+data "template_file" "startup_script" {
+  template = <<-EOF
+  sudo apt-get update -y
+  sudo apt-get install -y tinyproxy
+  EOF
+}
+
+module "bastion" {
+  source         = "terraform-google-modules/bastion-host/google"
+  version        = "~> 2.0"
+  network        = module.vpc.network_self_link
+  subnet         = module.vpc.subnets_self_links[0]
+  project        = module.enabled_google_apis.project_id
+  host_project   = module.enabled_google_apis.project_id
+  name           = local.bastion_name
+  zone           = local.bastion_zone
+  image_project  = "debian-cloud"
+  image_family   = "debian-9"
+  machine_type   = "g1-small"
+  startup_script = data.template_file.startup_script.rendered
+  members        = var.bastion_members
+  shielded_vm    = "false"
+}

examples/safer_cluster_iap_bastion/cluster.tf

@@ -0,0 +1,43 @@
+/**
+ * Copyright 2018 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+module "gke" {
+  source                  = "terraform-google-modules/kubernetes-engine/google//modules/safer-cluster"
+  version                 = "~> 9.0"
+  project_id              = module.enabled_google_apis.project_id
+  name                    = var.cluster_name
+  region                  = var.region
+  network                 = module.vpc.network_name
+  subnetwork              = module.vpc.subnets_names[0]
+  ip_range_pods           = module.vpc.subnets_secondary_ranges[0].*.range_name[0]
+  ip_range_services       = module.vpc.subnets_secondary_ranges[0].*.range_name[1]
+  enable_private_endpoint = false
+  master_authorized_networks = [{
+    cidr_block   = "${module.bastion.ip_address}/32"
+    display_name = "Bastion Host"
+  }]
+  grant_registry_access = true
+  node_pools = [
+    {
+      name          = "safer-pool"
+      min_count     = 1
+      max_count     = 4
+      machine_type  = "n1-standard-2"
+      auto_upgrade  = true
+      node_metadata = "GKE_METADATA_SERVER"
+    }
+  ]
+}

examples/safer_cluster_iap_bastion/network.tf

@@ -0,0 +1,58 @@
+/**
+ * Copyright 2020 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+module "vpc" {
+  source  = "terraform-google-modules/network/google"
+  version = "~> 2.3"
+
+  project_id   = module.enabled_google_apis.project_id
+  network_name = var.network_name
+  routing_mode = "GLOBAL"
+
+  subnets = [
+    {
+      subnet_name           = var.subnet_name
+      subnet_ip             = var.subnet_ip
+      subnet_region         = var.region
+      subnet_private_access = true
+      description           = "This subnet is managed by Terraform"
+    }
+  ]
+  secondary_ranges = {
+    "${var.subnet_name}" = [
+      {
+        range_name    = var.ip_range_pods_name
+        ip_cidr_range = "192.168.0.0/18"
+      },
+      {
+        range_name    = var.ip_range_services_name
+        ip_cidr_range = "192.168.64.0/18"
+      },
+    ]
+  }
+}
+
+
+module "cloud-nat" {
+  source        = "terraform-google-modules/cloud-nat/google"
+  version       = "~> 1.2"
+  project_id    = module.enabled_google_apis.project_id
+  region        = var.region
+  router        = "safer-router"
+  network       = module.vpc.network_self_link
+  create_router = true
+}

examples/safer_cluster_iap_bastion/outputs.tf

@@ -0,0 +1,87 @@
+/**
+ * Copyright 2018 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+output "cluster_name" {
+  description = "Cluster name"
+  value       = module.gke.name
+}
+
+output "location" {
+  description = "Cluster location (region if regional cluster, zone if zonal cluster)"
+  value       = module.gke.location
+}
+
+output "region" {
+  description = "Subnet/Router/Bastion Host region"
+  value       = var.region
+}
+
+output "endpoint" {
+  sensitive   = true
+  description = "Cluster endpoint"
+  value       = module.gke.endpoint
+}
+
+output "master_authorized_networks_config" {
+  description = "Networks from which access to master is permitted"
+  value       = module.gke.master_authorized_networks_config
+}
+
+output "router_name" {
+  description = "Name of the router that was created"
+  value       = module.cloud-nat.router_name
+}
+
+output "ca_certificate" {
+  sensitive   = true
+  description = "Cluster ca certificate (base64 encoded)"
+  value       = module.gke.ca_certificate
+}
+
+output "network_name" {
+  value       = module.vpc.network_name
+  description = "The name of the VPC being created"
+}
+
+output "subnet_name" {
+  value       = module.vpc.subnets_names[0]
+  description = "The name of the VPC subnet being created"
+}
+
+output "get_credentials_command" {
+  description = "gcloud get-credentials command to generate kubeconfig for the private cluster"
+  value       = format("gcloud container clusters get-credentials --project %s --zone %s --internal-ip %s", var.project_id, module.gke.location, module.gke.name)
+}
+
+output "bastion_name" {
+  description = "Name of the bastion host"
+  value       = module.bastion.hostname
+}
+
+output "bastion_zone" {
+  description = "Location of bastion host"
+  value       = local.bastion_zone
+}
+
+output "bastion_ssh_command" {
+  description = "gcloud command to ssh and port forward to the bastion host command"
+  value       = format("gcloud beta compute ssh %s --tunnel-through-iap --project %s --zone %s -- -L8888:127.0.0.1:8888", module.bastion.hostname, var.project_id, local.bastion_zone)
+}
+
+output "bastion_kubectl_command" {
+  description = "kubectl command using the local proxy once the bastion_ssh command is running"
+  value       = "HTTPS_PROXY=localhost:8888 kubectl get pods --all-namespaces"
+}