Skip to content

Commit 73a0f94

Browse files
jond3kjond3k
authored
feat: add submodules for managing connections and pools (#25)
1 parent 4dbb09d commit 73a0f94

File tree

42 files changed

+1637
-8
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

42 files changed

+1637
-8
lines changed

‎build/int.cloudbuild.yaml‎

Lines changed: 43 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -22,16 +22,17 @@ steps:
2222
- 'TF_VAR_folder_id=$_FOLDER_ID'
2323
- 'TF_VAR_billing_account=$_BILLING_ACCOUNT'
2424

25-
# ----- SUITE simple-composer-env-local
26-
27-
- id: create simple-composer-env-local
25+
- id: create all
2826
waitFor:
2927
- prepare
3028
name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
31-
args: ['/bin/bash', '-c', 'source /usr/local/bin/task_helper_functions.sh && kitchen_do create simple-composer-env-local']
29+
args: ['/bin/bash', '-c', 'source /usr/local/bin/task_helper_functions.sh && kitchen_do create']
30+
31+
# ----- SUITE simple-composer-env-local
32+
3233
- id: converge simple-composer-env-local
3334
waitFor:
34-
- create simple-composer-env-local
35+
- create all
3536
name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
3637
args: ['/bin/bash', '-c', 'source /usr/local/bin/task_helper_functions.sh && kitchen_do converge simple-composer-env-local']
3738
- id: verify simple-composer-env-local
@@ -45,6 +46,43 @@ steps:
4546
name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
4647
args: ['/bin/bash', '-c', 'source /usr/local/bin/task_helper_functions.sh && kitchen_do destroy simple-composer-env-local']
4748

49+
# ----- SUITE airflow-connection-local
50+
51+
- id: converge airflow-connection-local
52+
waitFor:
53+
- create all
54+
name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
55+
args: ['/bin/bash', '-c', 'source /usr/local/bin/task_helper_functions.sh && kitchen_do converge airflow-connection-local']
56+
- id: verify airflow-connection-local
57+
waitFor:
58+
- converge airflow-connection-local
59+
name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
60+
args: ['/bin/bash', '-c', 'source /usr/local/bin/task_helper_functions.sh && kitchen_do verify airflow-connection-local']
61+
# - id: destroy airflow-connection-local
62+
# waitFor:
63+
# - verify airflow-connection-local
64+
# name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
65+
# args: ['/bin/bash', '-c', 'source /usr/local/bin/task_helper_functions.sh && kitchen_do destroy airflow-connection-local']
66+
67+
68+
# ----- SUITE airflow-pool-local
69+
70+
- id: converge airflow-pool-local
71+
waitFor:
72+
- create all
73+
name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
74+
args: ['/bin/bash', '-c', 'source /usr/local/bin/task_helper_functions.sh && kitchen_do converge airflow-pool-local']
75+
- id: verify airflow-pool-local
76+
waitFor:
77+
- converge airflow-pool-local
78+
name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
79+
args: ['/bin/bash', '-c', 'source /usr/local/bin/task_helper_functions.sh && kitchen_do verify airflow-pool-local']
80+
# - id: destroy airflow-pool-local
81+
# waitFor:
82+
# - verify airflow-pool-local
83+
# name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
84+
# args: ['/bin/bash', '-c', 'source /usr/local/bin/task_helper_functions.sh && kitchen_do destroy airflow-pool-local']
85+
4886
tags:
4987
- 'ci'
5088
- 'integration'

‎examples/airflow_connection/README.md‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# Demonstrates Airflow Connections
2+
3+
This example illustrates how to use the `airflow_connection` module.
4+
5+
<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
6+
## Inputs
7+
8+
| Name | Description | Type | Default | Required |
9+
|------|-------------|------|---------|:--------:|
10+
| composer\_env\_name | Name of Cloud Composer Environment. | `string` | `"ci-composer"` | no |
11+
| composer\_service\_account | Service Account to be used for running Cloud Composer Environment. | `string` | n/a | yes |
12+
| network | Network where Cloud Composer is created. | `string` | n/a | yes |
13+
| pod\_ip\_allocation\_range\_name | The name of the cluster's secondary range used to allocate IP addresses to pods. | `string` | n/a | yes |
14+
| project\_id | Project ID where Cloud Composer Environment is created. | `string` | n/a | yes |
15+
| region | Region where Cloud Composer Environment is created. | `string` | n/a | yes |
16+
| service\_ip\_allocation\_range\_name | The name of the services' secondary range used to allocate IP addresses to the cluster. | `string` | n/a | yes |
17+
| subnetwork | Subetwork where Cloud Composer is created. | `string` | n/a | yes |
18+
| zone | Zone where the Cloud Composer nodes are created. | `string` | n/a | yes |
19+
20+
## Outputs
21+
22+
| Name | Description |
23+
|------|-------------|
24+
| airflow\_uri | URI of the Apache Airflow Web UI hosted within the Cloud Composer Environment. |
25+
| composer\_env\_id | ID of Cloud Composer Environment. |
26+
| composer\_env\_name | Name of the Cloud Composer Environment. |
27+
| gcs\_bucket | Google Cloud Storage bucket which hosts DAGs for the Cloud Composer Environment. |
28+
| gke\_cluster | Google Kubernetes Engine cluster used to run the Cloud Composer Environment. |
29+
30+
<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
31+
32+
To provision this example, run the following from within this directory:
33+
- `terraform init` to get the plugins
34+
- `terraform plan` to see the infrastructure plan
35+
- `terraform apply` to apply the infrastructure build
36+
- `terraform destroy` to destroy the built infrastructure

‎examples/airflow_connection/main.tf‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
/**
2+
* Copyright 2020 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
/******************************************
18+
Provider configuration
19+
*****************************************/
20+
provider "google" {
21+
version = "~> 3.3"
22+
}
23+
24+
provider "google-beta" {
25+
version = "~> 3.3"
26+
}
27+
28+
provider "external" {
29+
version = "2.1.0"
30+
}
31+
32+
module "simple-composer-environment" {
33+
source = "../../modules/create_environment"
34+
project_id = var.project_id
35+
composer_env_name = var.composer_env_name
36+
region = var.region
37+
composer_service_account = var.composer_service_account
38+
network = var.network
39+
subnetwork = var.subnetwork
40+
use_ip_aliases = true
41+
pod_ip_allocation_range_name = var.pod_ip_allocation_range_name
42+
service_ip_allocation_range_name = var.service_ip_allocation_range_name
43+
}
44+
45+
# Making the k8s master globally available is only to make the integration testing portable and should be removed
46+
module "master-authorized-networks" {
47+
source = "../../modules/master_authorized_networks"
48+
project_id = var.project_id
49+
zone = var.zone
50+
gke_cluster = module.simple-composer-environment.gke_cluster
51+
master_authorized_networks = [
52+
{ cidr_block = "0.0.0.0/0", display_name = "Everybody" }
53+
]
54+
}
55+
56+
module "example-1" {
57+
source = "../../modules/airflow_connection"
58+
project_id = var.project_id
59+
composer_env_name = module.simple-composer-environment.composer_env_name
60+
region = var.region
61+
id = "example-1"
62+
type = "postgres"
63+
host = "host-1"
64+
port = "5432"
65+
login = "login1"
66+
password = "p4ssw0rd"
67+
}
68+
69+
module "example-2" {
70+
source = "../../modules/airflow_connection"
71+
project_id = var.project_id
72+
composer_env_name = module.simple-composer-environment.composer_env_name
73+
region = var.region
74+
id = "example-2"
75+
uri = "gcpcloudsql://login1:p4ssw0rd@host-2:5433/db-1"
76+
extra = {
77+
database_type = "postgres"
78+
project_id = var.project_id
79+
location = var.region
80+
instance = "instance-1"
81+
use_proxy = true
82+
sql_proxy_use_tcp = true
83+
}
84+
}
85+
86+
# Connections can be defined externally if you wish
87+
module "example-3" {
88+
source = "../../modules/airflow_connection"
89+
project_id = var.project_id
90+
composer_env_name = module.simple-composer-environment.composer_env_name
91+
region = var.region
92+
id = "example-3"
93+
uri = "https://login1:p4ssw0rd@host-3/"
94+
}

‎examples/airflow_connection/outputs.tf‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
/**
2+
* Copyright 2020 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
output "composer_env_name" {
18+
description = "Name of the Cloud Composer Environment."
19+
value = module.simple-composer-environment.composer_env_name
20+
}
21+
22+
output "composer_env_id" {
23+
description = "ID of Cloud Composer Environment."
24+
value = module.simple-composer-environment.composer_env_id
25+
}
26+
27+
output "gke_cluster" {
28+
description = "Google Kubernetes Engine cluster used to run the Cloud Composer Environment."
29+
value = module.simple-composer-environment.gke_cluster
30+
}
31+
32+
output "gcs_bucket" {
33+
description = "Google Cloud Storage bucket which hosts DAGs for the Cloud Composer Environment."
34+
value = module.simple-composer-environment.gcs_bucket
35+
}
36+
37+
output "airflow_uri" {
38+
description = "URI of the Apache Airflow Web UI hosted within the Cloud Composer Environment."
39+
value = module.simple-composer-environment.airflow_uri
40+
}

‎examples/airflow_connection/variables.tf‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
/**
2+
* Copyright 2020 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
variable "project_id" {
18+
description = "Project ID where Cloud Composer Environment is created."
19+
type = string
20+
}
21+
22+
variable "composer_env_name" {
23+
description = "Name of Cloud Composer Environment."
24+
default = "ci-composer"
25+
type = string
26+
}
27+
28+
variable "region" {
29+
description = "Region where Cloud Composer Environment is created."
30+
type = string
31+
}
32+
33+
variable "zone" {
34+
description = "Zone where the Cloud Composer nodes are created."
35+
type = string
36+
}
37+
38+
variable "composer_service_account" {
39+
description = "Service Account to be used for running Cloud Composer Environment."
40+
type = string
41+
}
42+
43+
variable "network" {
44+
description = "Network where Cloud Composer is created."
45+
type = string
46+
}
47+
48+
variable "subnetwork" {
49+
description = "Subetwork where Cloud Composer is created."
50+
type = string
51+
}
52+
53+
variable "pod_ip_allocation_range_name" {
54+
description = "The name of the cluster's secondary range used to allocate IP addresses to pods."
55+
type = string
56+
}
57+
58+
variable "service_ip_allocation_range_name" {
59+
type = string
60+
description = "The name of the services' secondary range used to allocate IP addresses to the cluster."
61+
}

‎examples/airflow_connection/versions.tf‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
/**
2+
* Copyright 2020 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
terraform {
18+
required_version = ">= 0.12"
19+
}

‎examples/airflow_pool/README.md‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# Demonstrates Airflow Pools
2+
3+
This example illustrates how to use the `airflow_pool` module.
4+
5+
<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
6+
## Inputs
7+
8+
| Name | Description | Type | Default | Required |
9+
|------|-------------|------|---------|:--------:|
10+
| composer\_env\_name | Name of Cloud Composer Environment. | `string` | `"ci-composer"` | no |
11+
| composer\_service\_account | Service Account to be used for running Cloud Composer Environment. | `string` | n/a | yes |
12+
| network | Network where Cloud Composer is created. | `string` | n/a | yes |
13+
| pod\_ip\_allocation\_range\_name | The name of the cluster's secondary range used to allocate IP addresses to pods. | `string` | n/a | yes |
14+
| project\_id | Project ID where Cloud Composer Environment is created. | `string` | n/a | yes |
15+
| region | Region where Cloud Composer Environment is created. | `string` | n/a | yes |
16+
| service\_ip\_allocation\_range\_name | The name of the services' secondary range used to allocate IP addresses to the cluster. | `string` | n/a | yes |
17+
| subnetwork | Subetwork where Cloud Composer is created. | `string` | n/a | yes |
18+
| zone | Zone where the Cloud Composer nodes are created. | `string` | n/a | yes |
19+
20+
## Outputs
21+
22+
| Name | Description |
23+
|------|-------------|
24+
| airflow\_uri | URI of the Apache Airflow Web UI hosted within the Cloud Composer Environment. |
25+
| composer\_env\_id | ID of Cloud Composer Environment. |
26+
| composer\_env\_name | Name of the Cloud Composer Environment. |
27+
| gcs\_bucket | Google Cloud Storage bucket which hosts DAGs for the Cloud Composer Environment. |
28+
| gke\_cluster | Google Kubernetes Engine cluster used to run the Cloud Composer Environment. |
29+
30+
<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
31+
32+
To provision this example, run the following from within this directory:
33+
- `terraform init` to get the plugins
34+
- `terraform plan` to see the infrastructure plan
35+
- `terraform apply` to apply the infrastructure build
36+
- `terraform destroy` to destroy the built infrastructure

0 commit comments

Comments
 (0)