Skip to content

Commit b44bb6f

Browse files
sanmaymsanmaym
authored
feat: adding a module for networking/iam requirements and example for composer-v2 (#60)
* initial * test * test * integration test * changed for each * broken by module * added outputs * test * added fixtures * linting * added outputs * added headers * fixed linting * fixing lint * fixing lint * fixing lint * adding README files * incorporating feedback from bharath * fixed service account iam * renamed files * moved out of fixtures * moved out of fixtures * changed package name * removed invalid code from example * testing * resolving errors * adding variables * adding variables * Revert "adding variables" This reverts commit 6ab5156. * fixed variables.tf * formatted properly * added documentation * resolved fmt errors * fixed bharath's comments * fixed bharath's comments * fixed bharath's comments * fixed bharath's comments * fixed bharath's comments * fixed bharath's comments * added network * added network * added iam * added dns * added dns * added dns * added dns * added shared vpc iam * added shared vpc iam * shared vpc stuff * fix lint * fixing fmt errors * fixing fmt errors * using custom service account * cleaned up shared vpc iam * cleaned up shared vpc iam * fixing based on bharath's inputs * fixed some linting * fixing based on bharath's inputs * fixed some linting * fixed some linting * added container googleapis.com * added container googleapis.com * only trying shared vpc module * adding new vpc name * ran generate docs * added back other integration tests * resolved bharaths final comments * bharath's inputs
1 parent 8ac171f commit b44bb6f

File tree

18 files changed

+1097
-1
lines changed

18 files changed

+1097
-1
lines changed

‎build/int.cloudbuild.yaml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,28 @@ steps:
111111
# name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
112112
# args: ['/bin/bash', '-c', 'source /usr/local/bin/task_helper_functions.sh && kitchen_do destroy airflow-pool-local']
113113

114+
# ----- SUITE composer-v2-sharedvpc-prereq-local
115+
116+
- id: init-composer-v2-sharedvpc-prereq
117+
waitFor:
118+
- prepare
119+
name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
120+
args: ['/bin/bash', '-c', 'cft test run TestSimpleComposerEnvV2SharedVpcModule --stage init --verbose']
121+
- id: apply-composer-v2-sharedvpc-prereq
122+
waitFor:
123+
- init-composer-v2-sharedvpc-prereq
124+
name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
125+
args: ['/bin/bash', '-c', 'cft test run TestSimpleComposerEnvV2SharedVpcModule --stage apply --verbose']
126+
- id: verify-composer-v2-sharedvpc-prereq
127+
waitFor:
128+
- apply-composer-v2-sharedvpc-prereq
129+
name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
130+
args: ['/bin/bash', '-c', 'cft test run TestSimpleComposerEnvV2SharedVpcModule --stage verify --verbose']
131+
- id: destroy-composer-v2-sharedvpc-prereq
132+
waitFor:
133+
- verify-composer-v2-sharedvpc-prereq
134+
name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
135+
args: ['/bin/bash', '-c', 'cft test run TestSimpleComposerEnvV2SharedVpcModule --stage destroy --verbose']
114136
tags:
115137
- 'ci'
116138
- 'integration'

‎examples/composer_v2_sharedvpc_prereq/README.md‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
# Cloud Composer Example within VPC SC and Shared VPC
2+
3+
This guide provides the infrastructure as code as a reference point for users/ customers
4+
who want to perform all the steps documented in these links through Terraform
5+
- Shared VPC : https://cloud.google.com/composer/docs/composer-2/configure-shared-vpc
6+
- Firewall rules : https://cloud.google.com/composer/docs/composer-2/configure-private-ip#step_3_configure_firewall_rules
7+
- Cloud DNS rules: https://cloud.google.com/composer/docs/composer-2/configure-vpc-sc#connectivity_to_the_restrictedgoogleapiscom_endpoint
8+
9+
## Compatibility
10+
11+
This module is meant for use with Terraform 0.13+ and tested using Terraform 1.0+.
12+
13+
## Usage
14+
15+
16+
```hcl
17+
18+
19+
module "composer_env" {
20+
source = "terraform-google-modules/composer/google//modules/create_environment_v2"
21+
project_id = var.service_project_id
22+
network_project_id = var.network_project_id
23+
composer_env_name = var.composer_env_name
24+
region = var.region
25+
composer_service_account = google_service_account.composer_sa.email
26+
network = var.network
27+
subnetwork = var.subnetwork
28+
pod_ip_allocation_range_name = var.pod_ip_allocation_range_name
29+
service_ip_allocation_range_name = var.service_ip_allocation_range_name
30+
grant_sa_agent_permission = true
31+
use_private_environment = true
32+
enable_private_endpoint = true
33+
environment_size = "ENVIRONMENT_SIZE_SMALL"
34+
scheduler = {
35+
cpu = 1
36+
memory_gb = 1.875
37+
storage_gb = 1
38+
count = 1
39+
}
40+
web_server = {
41+
cpu = 1
42+
memory_gb = 2
43+
storage_gb = 10
44+
}
45+
worker = {
46+
cpu =1
47+
memory_gb = 2
48+
storage_gb = 1
49+
min_count = 1
50+
max_count = 6
51+
}
52+
}
53+
54+
```
55+
56+
57+
## Requirements
58+
59+
Before this module can be used on a project, you must ensure that the following pre-requisites are fulfilled:
60+
61+
1. The projects (host and service) are added within a VPC Service Control Perimeter and have the Cloud DNS/ routes configured appropriately for for Restricted VIP.
62+
2. The Service Account you execute the module with has the right [permissions](#configure-a-service-account).
63+
3. The Compute Engine and Kubernetes Engine APIs are [active](#enable-apis) on the project you will launch the cluster in.
64+
4. If you are using a Shared VPC, the APIs must also be activated on the Shared VPC host project and your service account needs the proper permissions there.
65+
5. As Composer uses GKE under the hood, certain networking configurations such as creation of network, subnets, GKE IP ranges are done before hand. This code only configures composer specific firewall rules and DNS entry.
66+
67+
68+
69+
### Configure a Service Account
70+
In order to execute this module you must have a Service Account with the
71+
following project roles in the Service Project:
72+
- roles/viewer - to get the service project
73+
- roles/composer.admin - to create the composer environment
74+
- roles/iam.serviceAccountCreator- to create the Service account attached to the composer environment
75+
- roles/resourcemanager.projectIamAdmin - to make iam bindings in service project for composer worker
76+
- roles/serviceaccountuser - If you do not assign this, you get the User not authorized to act as service account 'xxxx. The user must be granted iam.serviceAccounts.actAs permission
77+
78+
Roles needed in the Network project (where host vpc resides)
79+
- roles/dns.admin - Create DNS Zone that contains records to access Airflow UI
80+
- roles/compute.securityAdmin - You can explore custom roles as well to compute.firewalls.create in network project (Compute Security Admin)
81+
- roles/resourcemanager.projectIamAdmin - to make iam bindings in host project for Compute Network user
82+
83+
As the project's are within VPC SC, you may need to allow list the orchestrator service account that has these permissions
84+
### Destroy
85+
When destroying, destroy the composer resource first through targeted operation
86+
87+
terraform destroy -target=module.composer_env
88+
89+
Then proceed to destroying everything else
90+
### Enable APIs
91+
In order to operate with the Service Account you must activate the following APIs on the project where the Service Account was created:
92+
93+
- Cloud Composer - composer.googleapis.com
94+
- Kubernetes Engine API - container.googleapis.com
95+
- Service Usage API - serviceusage.googleapis.com
96+
97+
<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
98+
## Inputs
99+
100+
| Name | Description | Type | Default | Required |
101+
|------|-------------|------|---------|:--------:|
102+
| network | The name of the network being created | `string` | n/a | yes |
103+
| project\_id | The project ID of the shared VPC's host (for shared vpc support) | `string` | n/a | yes |
104+
| service\_project\_id | Project ID where Cloud Composer Environment is created. | `string` | n/a | yes |
105+
106+
## Outputs
107+
108+
| Name | Description |
109+
|------|-------------|
110+
| airflow\_uri | URI of the Apache Airflow Web UI hosted within the Cloud Composer Environment. |
111+
| composer\_env\_id | ID of Cloud Composer Environment. |
112+
| composer\_env\_name | Name of the Cloud Composer Environment. |
113+
| gcs\_bucket | Google Cloud Storage bucket which hosts DAGs for the Cloud Composer Environment. |
114+
| gke\_cluster | Google Kubernetes Engine cluster used to run the Cloud Composer Environment. |
115+
| service\_project\_id | The id of the project where cloud composer was created. |
116+
117+
<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

‎examples/composer_v2_sharedvpc_prereq/composer.tf‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
# Copyright 2022 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
module "composer_net" {
16+
source = "../../modules/composer_net"
17+
service_project_id = var.service_project_id
18+
network_project_id = var.project_id
19+
composer_env_name = "san-composer-2"
20+
region = "us-central1"
21+
network = var.network
22+
subnetwork = "composer-subnet"
23+
cloud_composer_network_ipv4_cidr_block = "192.168.192.0/24"
24+
master_ipv4_cidr = "192.168.193.0/28"
25+
cloud_sql_ipv4_cidr = "192.168.0.0/17"
26+
gke_subnet_ip_range = ["10.100.232.0/27"]
27+
gke_pods_services_ip_ranges = ["10.1.0.0/16", "10.4.0.0/16", "10.10.10.0/24", "10.10.14.0/24"]
28+
}
29+
module "composer_env" {
30+
depends_on = [
31+
module.composer_net
32+
]
33+
source = "../../modules/create_environment_v2"
34+
project_id = var.service_project_id
35+
network_project_id = var.project_id
36+
composer_env_name = "san-composer-2"
37+
composer_service_account = module.composer_net.composer_sa_email
38+
region = "us-central1"
39+
network = var.network
40+
subnetwork = "composer-subnet"
41+
cloud_composer_network_ipv4_cidr_block = "192.168.192.0/24"
42+
master_ipv4_cidr = "192.168.193.0/28"
43+
cloud_sql_ipv4_cidr = "192.168.0.0/17"
44+
pod_ip_allocation_range_name = "composer-pods-1"
45+
service_ip_allocation_range_name = "composer-services-1"
46+
grant_sa_agent_permission = true
47+
use_private_environment = true
48+
enable_private_endpoint = true
49+
environment_size = "ENVIRONMENT_SIZE_SMALL"
50+
scheduler = {
51+
cpu = 1
52+
memory_gb = 1.875
53+
storage_gb = 1
54+
count = 1
55+
}
56+
web_server = {
57+
cpu = 1
58+
memory_gb = 2
59+
storage_gb = 10
60+
}
61+
worker = {
62+
cpu = 1
63+
memory_gb = 2
64+
storage_gb = 1
65+
min_count = 1
66+
max_count = 6
67+
}
68+
}

‎examples/composer_v2_sharedvpc_prereq/outputs.tf‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
/**
2+
* Copyright 2022 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
output "service_project_id" {
18+
description = "The id of the project where cloud composer was created."
19+
value = var.service_project_id
20+
}
21+
22+
output "composer_env_name" {
23+
description = "Name of the Cloud Composer Environment."
24+
value = module.composer_env.composer_env_name
25+
}
26+
27+
output "composer_env_id" {
28+
description = "ID of Cloud Composer Environment."
29+
value = module.composer_env.composer_env_id
30+
}
31+
32+
output "gke_cluster" {
33+
description = "Google Kubernetes Engine cluster used to run the Cloud Composer Environment."
34+
value = module.composer_env.gke_cluster
35+
}
36+
37+
output "gcs_bucket" {
38+
description = "Google Cloud Storage bucket which hosts DAGs for the Cloud Composer Environment."
39+
value = module.composer_env.gcs_bucket
40+
}
41+
42+
output "airflow_uri" {
43+
description = "URI of the Apache Airflow Web UI hosted within the Cloud Composer Environment."
44+
value = module.composer_env.airflow_uri
45+
}

‎examples/composer_v2_sharedvpc_prereq/variables.tf‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
/**
2+
* Copyright 2020 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
variable "service_project_id" {
18+
description = "Project ID where Cloud Composer Environment is created."
19+
type = string
20+
}
21+
22+
variable "project_id" {
23+
type = string
24+
description = "The project ID of the shared VPC's host (for shared vpc support)"
25+
}
26+
27+
variable "network" {
28+
type = string
29+
description = "The name of the network being created"
30+
}
31+

‎modules/composer_net/README.md‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# Composer Network Module Example
2+
3+
This example illustrates how to use the `composer-net` module. Please see examples directory (Composer_v2_shared_vpc_prereq) on how this can be used.
4+
5+
<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
6+
## Inputs
7+
8+
| Name | Description | Type | Default | Required |
9+
|------|-------------|------|---------|:--------:|
10+
| cloud\_composer\_network\_ipv4\_cidr\_block | The CIDR block from which IP range in tenant project will be reserved. | `string` | `null` | no |
11+
| cloud\_sql\_ipv4\_cidr | The CIDR block from which IP range in tenant project will be reserved for Cloud SQL. | `string` | `null` | no |
12+
| composer\_env\_name | Name of Cloud Composer Environment | `string` | n/a | yes |
13+
| gke\_pods\_services\_ip\_ranges | The secondary IP ranges for the GKE Pods and Services IP ranges | `list(string)` | n/a | yes |
14+
| gke\_subnet\_ip\_range | The GKE subnet IP range | `list(string)` | n/a | yes |
15+
| master\_ipv4\_cidr | The CIDR block from which IP range in tenant project will be reserved for the master. | `string` | `null` | no |
16+
| network | The VPC network to host the composer cluster. | `string` | n/a | yes |
17+
| network\_project\_id | The project ID of the shared VPC's host (for shared vpc support) | `string` | n/a | yes |
18+
| region | Region where the Cloud Composer Environment is created. | `string` | `"us-central1"` | no |
19+
| service\_project\_id | Project ID where Cloud Composer Environment is created. | `string` | n/a | yes |
20+
| subnetwork | The subnetwork to host the composer cluster. | `string` | n/a | yes |
21+
22+
## Outputs
23+
24+
| Name | Description |
25+
|------|-------------|
26+
| composer\_sa\_email | n/a |
27+
28+
<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
29+
30+
To provision this example, run the following from within this directory:
31+
- `terraform init` to get the plugins
32+
- `terraform plan` to see the infrastructure plan
33+
- `terraform apply` to apply the infrastructure build
34+
- `terraform destroy` to destroy the built infrastructure

‎modules/composer_net/dns.tf‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
# Copyright 2022 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
/***************************************
16+
composer.cloud.google.com
17+
***************************************/
18+
19+
resource "google_dns_managed_zone" "composer_cloud_zone" {
20+
name = "composer-google-cloud-dns"
21+
project = var.network_project_id
22+
dns_name = "composer.cloud.google.com."
23+
description = "composer.cloud.google.com zone"
24+
25+
visibility = "private"
26+
27+
private_visibility_config {
28+
networks {
29+
network_url = "https://www.googleapis.com/compute/v1/projects/${var.network_project_id}/global/networks/${var.network}"
30+
}
31+
}
32+
}
33+
34+
resource "google_dns_record_set" "composer_cloud_zone-dev-A-record" {
35+
name = "composer.cloud.google.com."
36+
project = var.network_project_id
37+
type = "A"
38+
ttl = 300
39+
40+
managed_zone = google_dns_managed_zone.composer_cloud_zone.name
41+
42+
rrdatas = ["199.36.153.4", "199.36.153.5", "199.36.153.6", "199.36.153.7"]
43+
}
44+
45+
resource "google_dns_record_set" "composer_cloud_zone-CNAME" {
46+
name = "*.composer.cloud.google.com."
47+
project = var.network_project_id
48+
type = "CNAME"
49+
ttl = 300
50+
51+
managed_zone = google_dns_managed_zone.composer_cloud_zone.name
52+
53+
rrdatas = ["composer.cloud.google.com."]
54+
}

0 commit comments

Comments
 (0)