Skip to content

Commit 87918cb

Browse files
sbueringersbueringer
committed
Run e2e tests in AWS/GCP
1 parent 3a37397 commit 87918cb

File tree

8 files changed

+1248
-0
lines changed

8 files changed

+1248
-0
lines changed

‎hack/boskos.py‎

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
#!/usr/bin/env python3
2+
3+
# Copyright 2021 The Kubernetes Authors.
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
import argparse
18+
import json
19+
import os
20+
21+
import requests
22+
import time
23+
24+
BOSKOS_HOST = os.environ.get("BOSKOS_HOST", "boskos")
25+
BOSKOS_RESOURCE_NAME = os.environ.get('BOSKOS_RESOURCE_NAME')
26+
27+
28+
def checkout_account_request(resource_type, user, input_state):
29+
url = f'http://{BOSKOS_HOST}/acquire?type={resource_type}&state={input_state}&dest=busy&owner={user}'
30+
r = requests.post(url)
31+
status = r.status_code
32+
reason = r.reason
33+
result = ""
34+
35+
if status == 200:
36+
content = r.content.decode()
37+
result = json.loads(content)
38+
39+
return status, reason, result
40+
41+
42+
def checkout_account(resource_type, user):
43+
status, reason, result = checkout_account_request(resource_type, user, "clean")
44+
# TODO(sbueringer): find out if we still need this
45+
# replicated the implementation of cluster-api-provider-gcp
46+
# we're working around an issue with the data in boskos.
47+
# We'll remove the code that tries both free and clean once all the data is good.
48+
# Afterwards we should just check for free
49+
if status == 404:
50+
status, reason, result = checkout_account_request(resource_type, user, "free")
51+
52+
if status != 200:
53+
raise Exception(f"Got invalid response {status}: {reason}")
54+
55+
print(f"export BOSKOS_RESOURCE_NAME={result['name']}")
56+
print(f"export GCP_PROJECT={result['name']}")
57+
58+
59+
def release_account(user):
60+
url = f'http://{BOSKOS_HOST}/release?name={BOSKOS_RESOURCE_NAME}&dest=dirty&owner={user}'
61+
62+
r = requests.post(url)
63+
64+
if r.status_code != 200:
65+
raise Exception(f"Got invalid response {r.status_code}: {r.reason}")
66+
67+
68+
def send_heartbeat(user):
69+
url = f'http://{BOSKOS_HOST}/update?name={BOSKOS_RESOURCE_NAME}&state=busy&owner={user}'
70+
71+
while True:
72+
print(f"POST-ing heartbeat for resource {BOSKOS_RESOURCE_NAME} to {BOSKOS_HOST}")
73+
r = requests.post(url)
74+
75+
if r.status_code == 200:
76+
print(f"response status: {r.status_code}")
77+
else:
78+
print(f"Got invalid response {r.status_code}: {r.reason}")
79+
80+
time.sleep(60)
81+
82+
83+
def main():
84+
parser = argparse.ArgumentParser(description='Boskos GCP Account Management')
85+
86+
parser.add_argument(
87+
'--get', dest='checkout_account', action="store_true",
88+
help='Checkout a Boskos GCP Account'
89+
)
90+
91+
parser.add_argument(
92+
'--release', dest='release_account', action="store_true",
93+
help='Release a Boskos GCP Account'
94+
)
95+
96+
parser.add_argument(
97+
'--heartbeat', dest='send_heartbeat', action="store_true",
98+
help='Send heartbeat for the checked out a Boskos GCP Account'
99+
)
100+
101+
parser.add_argument(
102+
'--resource-type', dest="resource_type", type=str,
103+
default="gce-project",
104+
help="Type of Boskos resource to manage"
105+
)
106+
107+
parser.add_argument(
108+
'--user', dest="user", type=str,
109+
default="cluster-api",
110+
help="username"
111+
)
112+
113+
args = parser.parse_args()
114+
115+
if args.checkout_account:
116+
checkout_account(args.resource_type, args.user)
117+
118+
elif args.release_account:
119+
release_account(args.user)
120+
121+
elif args.send_heartbeat:
122+
send_heartbeat(args.user)
123+
124+
125+
if __name__ == "__main__":
126+
main()

‎hack/remote/README.md‎

Lines changed: 168 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,168 @@
1+
2+
# TODOs:
3+
4+
* Test on MacOS
5+
6+
* We should consider also pushing a test-extension image
7+
8+
* go over all files in the diff, finalize + FIXME /TODOs
9+
10+
* Test & fixup GCP script
11+
12+
* Test on a PR, special labels for cloud access:
13+
* preset-service-account (GCP) => Workload identity?
14+
* preset-aws-ssh: "true" (AWS)
15+
* preset-aws-credential: "true" (AWS)
16+
17+
Backlog:
18+
* Optimize scripting / automation
19+
* Implement in Go?
20+
* MacOS: Debug why it crashes the local Docker Desktop
21+
* try without IPv6: docker network create -d=bridge -o com.docker.network.bridge.enable_ip_masquerade=true -o com.docker.network.driver.mtu=1500 --subnet=172.24.4.0/24 --gateway=172.24.4.1 kind
22+
* => re-add IPv6
23+
24+
# Setting up a Docker engine on AWS:
25+
26+
Prerequisites:
27+
* AWS CLI must be installed & configured with credentials
28+
29+
Setup server on AWS with Docker engine:
30+
```bash
31+
./hack/remote/setup-docker-on-aws-account.sh
32+
```
33+
34+
Note: The script can also be run repeatedly, e.g. to create the ssh tunnel when the server already exists.
35+
36+
# Use remote Docker engine
37+
38+
## Docker CLI
39+
40+
```bash
41+
export DOCKER_HOST=tcp://10.0.3.15:2375
42+
docker version
43+
docker info
44+
```
45+
46+
## Local management cluster
47+
48+
### e2e tests via IDE
49+
50+
Prerequisites:
51+
```bash
52+
make generate-e2e-templates
53+
make docker-build-e2e
54+
```
55+
56+
Run configuration:
57+
* Add to environment: `CAPD_DOCKER_HOST=tcp://10.0.3.15:2375`
58+
59+
### Tilt
60+
61+
tilt-settings.yaml:
62+
```yaml
63+
kustomize_substitutions:
64+
# Use remote Docker host in CAPD.
65+
CAPD_DOCKER_HOST: "tcp://10.0.3.15:2375"
66+
```
67+
68+
```bash
69+
tilt up
70+
```
71+
72+
### Quickstart
73+
74+
```bash
75+
export CAPD_DOCKER_HOST="tcp://10.0.3.15:2375"
76+
```
77+
78+
## Remote management cluster
79+
80+
Create remote kind cluster:
81+
```bash
82+
# SSH to server
83+
ssh-add ~/.ssh/aws-capi-docker
84+
ssh cloud@${SERVER_PUBLIC_IP}
85+
sudo su
86+
87+
# Note: this has to be run on the server.
88+
# Running it locally will fails because 10.0.3.15 is not a valid IP there.
89+
kind create cluster --name=capi-test --config=${HOME}/kind.yaml
90+
```
91+
92+
### e2e tests via IDE
93+
94+
Prerequisites:
95+
```bash
96+
make generate-e2e-templates
97+
98+
# If local images are required (e.g. because code has changed)
99+
export DOCKER_HOST=tcp://10.0.3.15:2375
100+
make docker-build-e2e
101+
kind load docker-image --name=capi-test gcr.io/k8s-staging-cluster-api/cluster-api-controller-amd64:dev
102+
kind load docker-image --name=capi-test gcr.io/k8s-staging-cluster-api/kubeadm-bootstrap-controller-amd64:dev
103+
kind load docker-image --name=capi-test gcr.io/k8s-staging-cluster-api/kubeadm-control-plane-controller-amd64:dev
104+
kind load docker-image --name=capi-test gcr.io/k8s-staging-cluster-api/capd-manager-amd64:dev
105+
kind load docker-image --name=capi-test gcr.io/k8s-staging-cluster-api/test-extension-amd64:dev
106+
```
107+
108+
Run configuration:
109+
* Add to environment: `DOCKER_HOST=tcp://10.0.3.15:2375;CAPD_DOCKER_HOST=tcp://10.0.3.15:2375`
110+
* Add to program arguments: `-e2e.use-existing-cluster=true`
111+
112+
### Tilt
113+
114+
tilt-settings.yaml:
115+
```yaml
116+
kustomize_substitutions:
117+
# Use remote Docker host in CAPD.
118+
CAPD_DOCKER_HOST: "tcp://10.0.3.15:2375"
119+
```
120+
121+
```bash
122+
export DOCKER_HOST=tcp://10.0.3.15:2375
123+
tilt up
124+
```
125+
126+
FIXME(sbueringer): enable local registry
127+
* let's check if it is faster (as redeploy also just copies the binary over)
128+
* copy&paste kind-install-for-capd.sh script over(?) (already done => just test it)
129+
* ensure registry is reachable from local machine
130+
131+
## Getting access to workload clusters
132+
133+
Retrieve kubeconfig for workload clusters via:
134+
```bash
135+
clusterctl get kubeconfig capi-quickstart > /tmp/kubeconfig
136+
kubectl --kubeconfig /tmp/kubeconfig get no,po -A -A
137+
```
138+
Note: The kubeconfigs returned by `kind get kubeconfig` don't work.
139+
140+
# Troubleshooting
141+
142+
Verify connectivity:
143+
144+
```bash
145+
# SSH to server
146+
ssh-add ~/.ssh/aws-capi-docker
147+
ssh cloud@${SERVER_PUBLIC_IP}
148+
149+
# On the server:
150+
nc -l 10.0.3.15 8005
151+
152+
# Locally:
153+
nc 10.0.3.15 8005
154+
```
155+
156+
# Tested scenarios
157+
158+
* Local mgmt cluster:
159+
* Tilt:
160+
* works well
161+
* e2e tests (via Intellij):
162+
* works well
163+
* Remote mgmt cluster:
164+
* Tilt:
165+
* loading images via kind load is slow
166+
* e2e tests (via Intellij):
167+
* building e2e images with make is quick
168+
* loading images with kind load is slow

0 commit comments

Comments
 (0)