Skip to content

Commit 56f43ef

Browse files
k8s-ci-robotk8s-ci-robot
authored
Merge pull request #8085 from chrischdi/pr-infra-vsphere
infra: add k8s-infra-broadcom-admins and terraform for vSphere based CI
2 parents 7511c4c + c01d084 commit 56f43ef

File tree

28 files changed

+1226
-1
lines changed

28 files changed

+1226
-1
lines changed

‎groups/restrictions.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -290,6 +290,7 @@ restrictions:
290290
- "^[email protected]$"
291291
- "^[email protected]$"
292292
- "^[email protected]$"
293+
- "^[email protected]$"
293294
- "^[email protected]$"
294295
- "^[email protected]$"
295296
- "^[email protected]$"

‎groups/sig-k8s-infra/groups.yaml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -551,3 +551,17 @@ groups:
551551
- [email protected]
552552
- [email protected]
553553
- [email protected]
554+
555+
- email-id: [email protected]
556+
name: k8s-infra-gcp-gcve-admins
557+
description: |-
558+
ACL for Broadcom/vSphere project on GCP
559+
settings:
560+
WhoCanPostMessage: "ANYONE_CAN_POST"
561+
ReconcileMembers: "true"
562+
members:
563+
- [email protected]
564+
- [email protected]
565+
- [email protected]
566+
- [email protected]
567+
- [email protected]

‎infra/gcp/terraform/k8s-infra-gcp-gcve/OWNERS‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# See the OWNERS docs at https://go.k8s.io/owners
2+
3+
approvers:
4+
- chrischdi
5+
- sbueringer
6+
- fabriziopandini

‎infra/gcp/terraform/k8s-infra-gcp-gcve/README.md‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
# Setup
2+
3+
## Creation of GCVE
4+
5+
```sh
6+
gcloud auth application-default login
7+
terraform init
8+
terraform apply
9+
```
10+
11+
## Setup jumphost/vpn for further configuration
12+
13+
See [maintenance-jumphost/README.md](./maintenance-jumphost/README.md).
14+
15+
## Manual creation of a user and other IAM configuration in vSphere
16+
17+
> **Note:**
18+
> The configuration described here cannot be done via terraform due to non-existing functionality.
19+
20+
First we generate a password for the user which will be used in prow and set it as environment variable:
21+
22+
```sh
23+
export GCVE_PROW_CI_PASSWORD="SomePassword"
24+
```
25+
26+
And set credentials for `govc`:
27+
28+
```sh
29+
export GOVC_URL="$(gcloud vmware private-clouds describe k8s-gcp-gcve-pc --location us-central1-a --format='get(vcenter.fqdn)')"
30+
export GOVC_USERNAME='[email protected]'
31+
export GOVC_PASSWORD="$(gcloud vmware private-clouds vcenter credentials describe --private-cloud=k8s-gcp-gcve-pc [email protected] --location=us-central1-a --format='get(password)')"
32+
```
33+
34+
Run the script to setup the user, groups and IAM in vSphere.
35+
36+
```
37+
./vsphere/scripts/ensure-users-groups.sh
38+
```
39+
40+
Create relevant secrets in Secrets Manager
41+
42+
```sh
43+
gcloud secrets describe k8s-gcp-gcve-ci-url 2>/dev/null || echo "$GOVC_URL" | gcloud secrets create k8s-gcp-gcve-ci-url --data-file=-
44+
gcloud secrets describe k8s-gcp-gcve-ci-username 2>/dev/null || echo "[email protected]" | gcloud secrets create k8s-gcp-gcve-ci-username --data-file=-
45+
gcloud secrets describe k8s-gcp-gcve-ci-password 2>/dev/null || echo "${GCVE_PROW_CI_PASSWORD}" | gcloud secrets create k8s-gcp-gcve-ci-password --data-file=-
46+
gcloud secrets describe k8s-gcp-gcve-ci-thumbprint 2>/dev/null || echo "$(govc about.cert -json | jq -r '.thumbprintSHA256')" | gcloud secrets create k8s-gcp-gcve-ci-thumbprint --data-file=-
47+
```
48+
49+
* `k8s-gcp-gcve-ci-username` with value `[email protected]`
50+
* `k8s-gcp-gcve-ci-password` with value set above for `GCVE_PROW_CI_PASSWORD`
51+
* `k8s-gcp-gcve-ci-url` with value set above for `GOVC_URL`
52+
53+
> **Note:** Changing the GCVE CI user's password
54+
>
55+
> 1. Set GOVC credentials as above.
56+
> 2. Run govc command to update password: `govc sso.user.update -p "${GCVE_PROW_CI_PASSWORD}" prow-ci-user`
57+
> 3. Update secret `k8s-gcp-gcve-ci-password` in secrets-manager: `echo "${GCVE_PROW_CI_PASSWORD}" | gcloud secrets versions add k8s-gcp-gcve-ci-password --data-file=-`
58+
59+
## Configuration of GCVE
60+
61+
```sh
62+
export [email protected]
63+
export TF_VAR_vsphere_password="$(gcloud vmware private-clouds vcenter credentials describe --private-cloud=k8s-gcp-gcve-pc [email protected] --location=us-central1-a --format='get(password)')" # gcloud command
64+
export TF_VAR_vsphere_server="$(gcloud vmware private-clouds describe k8s-gcp-gcve-pc --location us-central1-a --format='get(vcenter.fqdn)')"
65+
export TF_VAR_nsxt_user=admin
66+
export TF_VAR_nsxt_password="$(gcloud vmware private-clouds nsx credentials describe --private-cloud k8s-gcp-gcve-pc --location us-central1-a --format='get(password)')"
67+
export TF_VAR_nsxt_server="$(gcloud vmware private-clouds describe k8s-gcp-gcve-pc --location us-central1-a --format='get(nsx.fqdn)')"
68+
```
69+
70+
```sh
71+
cd vsphere
72+
terraform init
73+
terraform apply
74+
```

‎infra/gcp/terraform/k8s-infra-gcp-gcve/main.tf‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
/*
2+
Copyright 2025 The Kubernetes Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
locals {
18+
project_id = "broadcom-451918"
19+
}
20+
21+
data "google_project" "project" {
22+
project_id = local.project_id
23+
}
24+
25+
resource "google_project_service" "project" {
26+
project = data.google_project.project.id
27+
28+
for_each = toset([
29+
"compute.googleapis.com",
30+
"secretmanager.googleapis.com",
31+
"vmwareengine.googleapis.com"
32+
])
33+
34+
service = each.key
35+
}
36+
37+
// Ensure [email protected] has admin access to this project
38+
resource "google_project_iam_member" "k8s_infra_leads" {
39+
project = data.google_project.project.id
40+
role = "roles/admin"
41+
member = "group:[email protected]"
42+
}
43+
44+
# TODO(chrischdi): we first need the group
45+
# // Ensure [email protected] has owner access to this project
46+
# resource "google_project_iam_member" "k8s_infra_vsphere" {
47+
# project = data.google_project.project.id
48+
# role = "roles/owner"
49+
# member = "group:[email protected]"
50+
# }

‎infra/gcp/terraform/k8s-infra-gcp-gcve/maintenance-jumphost/README.md‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
# Maintenance VM
2+
3+
## Reprovisioning
4+
5+
```sh
6+
terraform taint google_compute_instance.jumphost
7+
terraform apply
8+
```
9+
10+
## Configuration
11+
12+
### Configure a connection
13+
14+
First we need to generate a wireguard keypair using `wg` (install on ubuntu via `apt-get install wireguard-tools`)
15+
16+
```sh
17+
export CLIENT_PRIVATE_KEY="$(wg genkey)"
18+
export CLIENT_PUBLIC_KEY="$(echo $CLIENT_PRIVATE_KEY | wg pubkey)"
19+
```
20+
21+
Next we have to pick a free IP address from the `192.168.29.0/24` subnet, which is not already in the server config file (see gcloud secret `maintenance-vm-wireguard-config`):
22+
23+
```sh
24+
export CLIENT_IP_ADDRESS="192.168.29.X"
25+
```
26+
27+
After that we generate a peer entry for the server configuration by using the output of the following script:
28+
29+
```sh
30+
cat << EOF
31+
[Peer]
32+
PublicKey = ${CLIENT_PUBLIC_KEY}
33+
AllowedIPs = ${CLIENT_IP_ADDRESS}/32
34+
35+
EOF
36+
```
37+
38+
Then we add it to the secret `maintenance-vm-wireguard-config` [here](https://console.cloud.google.com/security/secret-manager/secret/maintenance-vm-wireguard-config/versions).
39+
40+
To add it to a running VM:
41+
42+
```sh
43+
gcloud compute ssh maintenance-jumphost
44+
sudo systemctl stop wg-quick@wg0
45+
sudo vim /etc/wireguard/wg0.conf
46+
sudo systemctl start wg-quick@wg0
47+
```
48+
49+
Last we generate a wireguard client configuration:
50+
51+
```sh
52+
cat << EOF
53+
[Interface]
54+
PrivateKey = ${CLIENT_PRIVATE_KEY}
55+
Address = ${CLIENT_IP_ADDRESS}/24
56+
MTU = 1360
57+
58+
[Peer]
59+
PublicKey = $(gcloud secrets versions access --secret maintenance-vm-wireguard-pubkey latest)
60+
AllowedIPs = 192.168.30.0/24, 192.168.32.0/21
61+
Endpoint = $(gcloud compute instances list --format='get(networkInterfaces[0].accessConfigs[0].natIP)' --filter='name=maintenance-jumphost'):51820
62+
PersistentKeepalive = 25
63+
EOF
64+
```

‎infra/gcp/terraform/k8s-infra-gcp-gcve/maintenance-jumphost/cloud-config.yaml.tftpl‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
#cloud-config
2+
3+
write_files:
4+
- path: /etc/wireguard/wg0.conf
5+
content: "${wg0}"
6+
encoding: b64
7+
permissions: "0600"
8+
9+
- path: /etc/sysctl.d/10-wireguard.conf
10+
content: |
11+
net.ipv4.ip_forward = 1
12+
13+
runcmd:
14+
- apt-get update
15+
- apt install wireguard -q -y
16+
- sysctl -p /etc/sysctl.d/10-wireguard.conf
17+
- systemctl enable wg-quick@wg0
18+
- systemctl start wg-quick@wg0

‎infra/gcp/terraform/k8s-infra-gcp-gcve/maintenance-jumphost/jumphost.tf‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
/*
2+
Copyright 2025 The Kubernetes Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
locals {
18+
project_id = "broadcom-451918"
19+
}
20+
21+
resource "google_compute_instance" "jumphost" {
22+
project = local.project_id
23+
name = "maintenance-jumphost"
24+
machine_type = "f1-micro"
25+
zone = "us-central1-f"
26+
27+
boot_disk {
28+
initialize_params {
29+
image = "ubuntu-os-cloud/ubuntu-2404-lts-amd64"
30+
}
31+
}
32+
33+
network_interface {
34+
network = "maintenance-vpc-network"
35+
subnetwork = "maintenance-subnet"
36+
subnetwork_project = local.project_id
37+
access_config {
38+
network_tier = "STANDARD"
39+
}
40+
}
41+
42+
metadata = {
43+
user-data = templatefile("${path.module}/cloud-config.yaml.tftpl", { wg0 = base64encode(data.google_secret_manager_secret_version_access.wireguard-config.secret_data) })
44+
}
45+
}
46+
47+
data "google_secret_manager_secret_version_access" "wireguard-config" {
48+
project = local.project_id
49+
secret = "maintenance-vm-wireguard-config"
50+
}

‎infra/gcp/terraform/k8s-infra-gcp-gcve/maintenance-jumphost/provider.tf‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
/*
2+
Copyright 2025 The Kubernetes Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
/*
18+
This file defines:
19+
- Required provider versions
20+
- Storage backend details
21+
*/
22+
23+
terraform {
24+
25+
backend "gcs" {
26+
bucket = "k8s-infra-tf-gcp-gcve"
27+
prefix = "k8s-infra-gcp-maintenance-jumphost"
28+
}
29+
30+
31+
required_providers {
32+
google = {
33+
source = "hashicorp/google"
34+
version = "~> 6.34.1"
35+
}
36+
}
37+
}

‎infra/gcp/terraform/k8s-infra-gcp-gcve/provider.tf‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
/*
2+
Copyright 2025 The Kubernetes Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
/*
18+
This file defines:
19+
- Required provider versions
20+
- Storage backend details
21+
*/
22+
23+
terraform {
24+
25+
backend "gcs" {
26+
bucket = "k8s-infra-tf-gcp-gcve"
27+
prefix = "k8s-infra-gcp-gcve"
28+
}
29+
30+
31+
required_providers {
32+
google = {
33+
source = "hashicorp/google"
34+
version = "~> 6.34.1"
35+
}
36+
}
37+
}

0 commit comments

Comments
 (0)