Skip to content

Commit f8dd697

Browse files
authored
Merge pull request #1114 from artemvmin/patch-1
Document edge cases for local e2e tests
2 parents 6f3e28b + 7def5df commit f8dd697

File tree

1 file changed

+121
-48
lines changed

1 file changed

+121
-48
lines changed

docs/kubernetes/development.md

Lines changed: 121 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -1,61 +1,126 @@
11
# Kubernetes Development
22

3-
## Manual
3+
## Manual deployment
44

55
To build and install a development version of the driver:
66

7-
```
8-
$ GCE_PD_CSI_STAGING_IMAGE=gcr.io/path/to/driver/image:dev # Location to push dev image to
9-
$ make push-container
7+
```sh
8+
GCE_PD_CSI_STAGING_IMAGE=gcr.io/path/to/driver/image:dev # Location to push dev image to
9+
make push-container
1010

1111
# Modify controller.yaml and node.yaml in ./deploy/kubernetes/dev to use dev image
12-
$ GCE_PD_DRIVER_VERSION=dev
13-
$ ./deploy/kubernetes/deploy-driver.sh
12+
GCE_PD_DRIVER_VERSION=dev
13+
./deploy/kubernetes/deploy-driver.sh
1414
```
1515

1616
To bring down driver:
1717

18+
```sh
19+
./deploy/kubernetes/delete-driver.sh
1820
```
19-
$ ./deploy/kubernetes/delete-driver.sh
20-
```
21+
2122
## Testing
2223

23-
### E2E Tests:
24+
### E2E tests:
25+
26+
The following environment variables will be required for setup and all
27+
subsequent test runs.
28+
29+
```sh
30+
export PROJECT=$USER-e2e # GCP project used for testing
31+
export GCE_PD_SA_NAME=$PROJECT-pd-sa # Service account name
32+
export IAM_NAME=$GCE_PD_SA_NAME@$PROJECT.iam.gserviceaccount.com # Service account IAM name
33+
export GCE_PD_SA_DIR=~/credentials # Service account directory
34+
export GOOGLE_APPLICATION_CREDENTIALS=$GCE_PD_SA_DIR/cloud-sa.json # Service account key
35+
```
2436

2537
#### One time setup
2638

27-
```console
28-
$ export PROJECT=my-project # GCP Project to run tests in
29-
$ export GCE_PD_SA_NAME=$PROJECT-pd-sa # Name of the service account to create
30-
$ export GCE_PD_SA_DIR=/my/safe/credentials/directory # Directory to save the service account key
31-
$ export ENABLE_KMS=false
32-
$ ./deploy/setup-project.sh
39+
The following steps will need to be completed once per project.
40+
41+
##### Create service account
42+
43+
To create a service account with correct permissions, your test project must:
44+
45+
1. Be assigned
46+
[Project IAM Admin](https://cloud.google.com/iam/docs/understanding-roles#resourcemanager.projectIamAdmin)
47+
and
48+
[Role Administrator](https://cloud.google.com/iam/docs/understanding-roles#iam.roleAdmin)
49+
IAM roles.
50+
2. Be part of an organization that allows creation of service account keys.
51+
- Google developer projects restrict SA keys and will not work for E2E
52+
testing.
53+
54+
```sh
55+
export ENABLE_KMS=false
56+
./deploy/setup-project.sh
3357
```
34-
#### Ongoing runs
35-
```console
36-
$ export PROJECT=my-project # GCP Project to run tests in
37-
$ export GCE_PD_SA_NAME=$PROJECT-pd-sa
38-
$ export IAM_NAME=$GCE_PD_SA_NAME@$PROJECT.iam.gserviceaccount.com # IAM SA that was set up in "one time setup"
3958

40-
$ ./test/run-e2e-local.sh
59+
##### Create SSH key
60+
61+
1. Follow instructions to
62+
[generate an SSH key](https://cloud.google.com/compute/docs/connect/create-ssh-keys#create_an_ssh_key_pair)
63+
with the following parameters:
64+
- `KEY_FILENAME: "google_compute_engine"`
65+
- `USERNAME: "$USER"`
66+
2. Add your public key (`cat ~/.ssh/google_compute_engine.pub`) to your
67+
[GCP project metatdata](https://cloud.google.com/compute/docs/connect/add-ssh-keys#add_ssh_keys_to_project_metadata).
68+
69+
#### Running tests
70+
71+
```sh
72+
./test/run-e2e-local.sh
4173
```
4274

43-
### Sanity Tests
44-
> **_NOTE:_** Sanity tests are currently failing, tracked by https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/issues/990.
75+
##### Debugging
76+
77+
Local stderr will show you all GCE calls and SSH commands that were performed by
78+
the test.
79+
80+
###### GCE failures
81+
82+
You may see errors during instance creation. It is likely your service account
83+
is missing permissions.
84+
85+
1. Make sure the resources are showing up in pantheon under [Compute Engine >
86+
VM Instances].
87+
2. Look for errors in pantheon under [Logging > Logs Explorer].
88+
3. Double check and re-run the
89+
[service account creation](#create-service-account) step.
90+
91+
###### SSH failures
92+
93+
You may see errors during driver deployment.
94+
95+
1. Double check that you have an [SSH key file](#create-ssh-key) in the correct
96+
directory.
97+
2. `gcloud compute ssh $VM_NAME --zone $ZONE` into the VM instance and attempt
98+
to execute the last few commands and look for errors.
99+
- E.g.
100+
[vm and driver have a dependency mismatch](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/pull/1113)).
101+
102+
### Sanity tests
103+
104+
> ***NOTE:*** Sanity tests are currently failing, tracked by
105+
> https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/issues/990.
45106
46-
Sanity tests can be run from VS Code, etc directly or via the cmd line:
47107
```
48-
$ ./test/run-sanity.sh
108+
./test/run-sanity.sh
49109
```
50110

51-
### Unit Tests
52-
Unit tests can be run from VS Code, etc directly or via the cmd line:
53-
```
54-
$ ./test/run-unit.sh
111+
### Unit tests:
112+
113+
```sh
114+
./test/run-unit.sh
55115
```
56116

57-
### Performance Tests
58-
Performance tests are run in automated testing frameworks, but you may wish to run them locally to benchmark changes. For a given test configuration, you need to modify the `test-config.yaml` file automatically generated from running one of the `run-k8s-integration...` or `run-windows-k8s-integration.sh` scripts to enable performance tests.
117+
### Performance tests
118+
119+
Performance tests are run in automated testing frameworks, but you may wish to
120+
run them locally to benchmark changes. For a given test configuration, you need
121+
to modify the `test-config.yaml` file automatically generated from running one
122+
of the `run-k8s-integration...` or `run-windows-k8s-integration.sh` scripts to
123+
enable performance tests.
59124

60125
```yaml
61126
DriverInfo:
@@ -71,47 +136,54 @@ DriverInfo:
71136
72137
You may modify the parameter values to customize the test.
73138
74-
You also need to modify the `StorageClass` file pointed to in `test-config.yaml` to set `volumeBindingMode: Immediate`, as the performance tests only support this mode.
139+
You also need to modify the `StorageClass` file pointed to in `test-config.yaml`
140+
to set `volumeBindingMode: Immediate`, as the performance tests only support
141+
this mode.
75142

76-
We will be running the Kubernetes integration tests directly from its repository. Install `kubetest` and set up a test cluster with the driver installed. Now, cd into `$GOPATH/src/k8s.io/kubernetes` and run
143+
We will be running the Kubernetes integration tests directly from its
144+
repository. Install `kubetest` and set up a test cluster with the driver
145+
installed. Now, cd into `$GOPATH/src/k8s.io/kubernetes` and run
77146

78-
```bash
147+
```sh
79148
# pwd=k/k
80149
kubetest <custom flags based on your provider> \
81150
--check-version-skew=false \
82151
--test \
83152
--test_args="--ginkgo.focus=External.Storage.*volume-lifecycle-performance --allowed-not-ready-nodes=10 --node-os-distro=<linux or windows> --storage.testdriver=<path-to-test-config>"
84153
```
85154

86-
## Dependency Management
155+
## Dependency management
87156

88157
Use [dep](https://github.com/golang/dep)
89-
```
90-
$ dep ensure
158+
159+
```sh
160+
dep ensure
91161
```
92162

93163
To modify dependencies or versions change `./Gopkg.toml`
94164

95165
## Debugging
96166

97-
We use https://github.com/go-delve/delve and its feature for remote debugging. This feature
98-
is only available in the PD CSI Controller (which runs in a linux node).
167+
We use https://github.com/go-delve/delve and its feature for remote debugging.
168+
This feature is only available in the PD CSI Controller (which runs in a linux
169+
node).
99170

100171
Requirements:
101172

102-
- https://github.com/go-delve/delve
173+
- https://github.com/go-delve/delve
103174

104175
Steps:
105176

106-
- Build the PD CSI driver with additional compiler flags.
177+
- Build the PD CSI driver with additional compiler flags.
107178

108-
```
179+
```sh
109180
export GCE_PD_CSI_STAGING_VERSION=latest
110181
export GCE_PD_CSI_STAGING_IMAGE=image/repo/gcp-compute-persistent-disk-csi-driver
111182
make build-and-push-multi-arch-debug
112183
```
113184

114-
- Update `deploy/kubernetes/overlays/noauth-debug/kustomization.yaml` to match the repo you wrote above e.g.
185+
- Update `deploy/kubernetes/overlays/noauth-debug/kustomization.yaml` to match
186+
the repo you wrote above e.g.
115187

116188
```yaml
117189
images:
@@ -120,7 +192,7 @@ images:
120192
newTag: latest
121193
```
122194

123-
- Delete and deploy the driver with this overlay
195+
- Delete and deploy the driver with this overlay
124196

125197
```sh
126198
./deploy/kubernetes/delete-driver.sh && \
@@ -135,13 +207,13 @@ API server listening at: [::]:2345
135207
2021-04-15T18:28:53Z debug layer=debugger continuing
136208
```
137209

138-
- Enable port forwading of the PD CSI controller of port 2345
210+
- Enable port forwading of the PD CSI controller of port 2345
139211

140212
```sh
141213
kubectl -n gce-pd-csi-driver get pods | grep controller | awk '{print $1}' | xargs -I % kubectl -n gce-pd-csi-driver port-forward % 2345:2345
142214
```
143215

144-
- Connect to the headless server and issue commands
216+
- Connect to the headless server and issue commands
145217

146218
```sh
147219
dlv connect localhost:2345
@@ -175,5 +247,6 @@ Command failed: command not available
175247
(dlv)
176248
```
177249

178-
See https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/pull/742 for the implementation details
179-
250+
See
251+
https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/pull/742
252+
for the implementation details

0 commit comments

Comments
 (0)