|
| 1 | +# Testing |
| 2 | + |
| 3 | +## Table of Contents |
| 4 | + |
| 5 | +- [Bootstrapping a Kubeadm Test Environment](#bootstrapping-a-kubeadm-test-environment) |
| 6 | + - [Prerequisites](#prerequisites) |
| 7 | + - [Script Reference](#script-reference) |
| 8 | + - [Creating a Cluster](#creating-a-cluster) |
| 9 | + - [Accessing the Cluster](#accessing-the-cluster) |
| 10 | + - [Destroying the Cluster](#destroying-the-cluster) |
| 11 | + - [Testing Custom Branches or Images](#testing-custom-branches-or-images) |
| 12 | +- [Running End-to-End (E2E) Tests for the CSI Driver](#running-end-to-end-e2e-tests-for-the-csi-driver) |
| 13 | + - [Parallel E2E Test Suite](#parallel-e2e-test-suite) |
| 14 | + - [Sequential E2E Test Suite (Snapshots & Backups)](#sequential-e2e-test-suite-snapshots--backups) |
| 15 | + - [Customizing Test Execution](#customizing-test-execution) |
| 16 | + - [Full Example](#full-example) |
| 17 | + |
| 18 | +## Bootstrapping a Kubeadm Test Environment |
| 19 | + |
| 20 | +To run the E2E tests, you first need a Kubernetes cluster. The script located at `test/e2e/e2e-test-script.sh` (this path is assumed, please adjust if it's located elsewhere) automates the creation and destruction of a single-node Kubeadm cluster on a STACKIT VM. |
| 21 | + |
| 22 | +This script handles provisioning all necessary STACKIT resources, including: |
| 23 | + |
| 24 | +- VM (Server) |
| 25 | +- Network |
| 26 | +- Security Group (with rules for SSH and K8s API) |
| 27 | +- Public IP |
| 28 | +- Service Account and Key |
| 29 | +- SSH Key (in STACKIT) |
| 30 | + |
| 31 | +It then configures the VM with `kubeadm`, installs Calico CNI, and deploys the STACKIT CCM and CSI driver. |
| 32 | + |
| 33 | +### Prerequisites |
| 34 | + |
| 35 | +Before running the script, you must have the following installed and configured on your local machine: |
| 36 | + |
| 37 | +1. **STACKIT CLI:** The `stackit` command-line tool. |
| 38 | + - You must be authenticated. Run `stackit auth login` if you haven't already. |
| 39 | +2. **jq:** The `jq` command-line JSON processor. |
| 40 | +3. **SSH Key Pair:** The script needs an SSH key pair to access the VM. |
| 41 | + |
| 42 | + - By default, it looks for `$HOME/.ssh/stackit-ccm-test.pub` and `$HOME/.ssh/stackit-ccm-test`. |
| 43 | + - You can generate a new key pair with: |
| 44 | + |
| 45 | + ```bash |
| 46 | + ssh-keygen -f $HOME/.ssh/stackit-ccm-test -t rsa -b 4096 |
| 47 | + ``` |
| 48 | + |
| 49 | + - If you want to use a different key, you can set the `E2E_SSH_KEY_NAME` environment variable (e.g., `E2E_SSH_KEY_NAME="my-other-key"`). |
| 50 | + |
| 51 | +### Script Reference |
| 52 | + |
| 53 | +You can control other parameters, such as the VM name or machine type, via environment variables. For a full list of commands and environment variables, run the script with the `--help` flag. |
| 54 | + |
| 55 | +```bash |
| 56 | +$ ./test/e2e/e2e.sh --help |
| 57 | +Usage: ./test/e2e/e2e.sh <create|destroy> [options] |
| 58 | +
|
| 59 | +Actions: |
| 60 | + create Create a new Kubernetes test environment. |
| 61 | + destroy Destroy an existing Kubernetes test environment. |
| 62 | +
|
| 63 | +Options: |
| 64 | + --project-id <ID> STACKIT Project ID. (Required for create & destroy) |
| 65 | + --kubernetes-version <VERSION> Kubernetes version (e.g., 1.34.1). (Required for create) |
| 66 | + --help Show this help message. |
| 67 | +
|
| 68 | +Environment Variables (Optional Overrides): |
| 69 | + E2E_MACHINE_NAME: Name for the VM, network, SA, and security group. |
| 70 | + (Default: "stackit-ccm-test") |
| 71 | + E2E_MACHINE_TYPE: STACKIT machine type for the VM. |
| 72 | + (Default: "c2i.4") |
| 73 | + E2E_SSH_KEY_NAME: Name of the SSH key pair to use (must exist at $HOME/.ssh/<name>). |
| 74 | + (Default: value of E2E_MACHINE_NAME) |
| 75 | + E2E_NETWORK_NAME: Name of the STACKIT network to create or use. |
| 76 | + (Default: value of E2E_MACHINE_NAME) |
| 77 | + E2E_DEPLOY_BRANCH: Specify a git branch for the CCM/CSI manifests. |
| 78 | + (Default: auto-detects 'release-vX.Y' or 'main') |
| 79 | + E2E_DEPLOY_CCM_IMAGE: Specify a full container image ref to override the CCM deployment. |
| 80 | + (Default: uses the image from the kustomize base) |
| 81 | + E2E_DEPLOY_CSI_IMAGE: Specify a full container image ref to override the CSI plugin. |
| 82 | + (Default: uses the image from the kustomize base) |
| 83 | +``` |
| 84 | + |
| 85 | +### Creating a Cluster |
| 86 | + |
| 87 | +To create and provision the entire test environment, run the `create` command: |
| 88 | + |
| 89 | +```bash |
| 90 | +
|
| 91 | +# Assumes the script is at test/e2e/e2e-test-script.sh |
| 92 | +
|
| 93 | +./test/e2e/e2e-test-script.sh create \ |
| 94 | + --project-id <STACKIT_PROJECT_ID> \ |
| 95 | + --kubernetes-version <K8S_VERSION> |
| 96 | +``` |
| 97 | + |
| 98 | +- **`<STACKIT_PROJECT_ID>`:** Your STACKIT Project UUID. |
| 99 | +- **`<K8S_VERSION>`:** The specific Kubernetes version to install (e.g., `1.29.2`). |
| 100 | + |
| 101 | +This process will take several minutes. The script will create all resources, wait for the VM to be ready, install Kubernetes, and deploy the STACKIT components. |
| 102 | + |
| 103 | +### Accessing the Cluster |
| 104 | + |
| 105 | +Upon successful creation, the script generates a `kubeconfig` file locally. |
| 106 | + |
| 107 | +- **Kubeconfig Path:** `test/e2e/kubeconfig-<PROJECT_ID>-<MACHINE_NAME>.yaml` |
| 108 | +- **SA Key Path:** `test/e2e/sa-key-<PROJECT_ID>-<MACHINE_NAME>.json` |
| 109 | +- **Inventory Path:** `test/e2e/inventory-<PROJECT_ID>-<MACHINE_NAME>.json` (Used for deletion) |
| 110 | + |
| 111 | +You can export the `KUBECONFIG` variable to interact with your new cluster: |
| 112 | + |
| 113 | +```bash |
| 114 | +export KUBECONFIG=$PWD/test/e2e/kubeconfig-<PROJECT_ID>-<MACHINE_NAME>.yaml |
| 115 | +kubectl get nodes |
| 116 | +``` |
| 117 | + |
| 118 | +### Destroying the Cluster |
| 119 | + |
| 120 | +To tear down all resources created by the script (including the VM, network, SA, etc.), use the `destroy` command. |
| 121 | + |
| 122 | +```bash |
| 123 | +./test/e2e/e2e-test-script.sh destroy --project-id <STACKIT_PROJECT_ID> |
| 124 | +``` |
| 125 | + |
| 126 | +The script uses the `inventory-....json` file to identify which resources to delete. |
| 127 | + |
| 128 | +### Testing Custom Branches or Images |
| 129 | + |
| 130 | +The bootstrap script is ideal for testing changes in a feature branch or custom-built images. You can use environment variables to override the deployment defaults. |
| 131 | + |
| 132 | +#### Testing a Specific Git Branch |
| 133 | + |
| 134 | +Use **`E2E_DEPLOY_BRANCH`** to instruct the provisioner to pull the CCM/CSI manifests from a specific git branch (e.g., a feature branch or a release branch) instead of the default. |
| 135 | + |
| 136 | +```bash |
| 137 | +
|
| 138 | +# This will create a cluster using manifests from the 'release-v1.31' branch |
| 139 | +
|
| 140 | +E2E_DEPLOY_BRANCH="release-v1.31" \ |
| 141 | +./test/e2e/e2e-test-script.sh create \ |
| 142 | + --project-id <STACKIT_PROJECT_ID> \ |
| 143 | + --kubernetes-version <K8S_VERSION> |
| 144 | +``` |
| 145 | + |
| 146 | +#### Testing a Custom CSI Driver Image |
| 147 | + |
| 148 | +Use **`E2E_DEPLOY_CSI_IMAGE`** to override the CSI driver image. |
| 149 | + |
| 150 | +```bash |
| 151 | +
|
| 152 | +# This will create a cluster and deploy your custom-built CSI image |
| 153 | +
|
| 154 | +E2E_DEPLOY_CSI_IMAGE="ghcr.io/stackitcloud/cloud-provider-stackit/stackit-csi-plugin:v1.31.6-4-gf2d85f1" \ |
| 155 | +./test/e2e/e2e-test-script.sh create \ |
| 156 | + --project-id <STACKIT_PROJECT_ID> \ |
| 157 | + --kubernetes-version <K8S_VERSION> |
| 158 | +``` |
| 159 | + |
| 160 | +#### Testing a Custom CCM Image |
| 161 | + |
| 162 | +Use **`E2E_DEPLOY_CCM_IMAGE`** to override the Cloud Controller Manager image. |
| 163 | + |
| 164 | +```bash |
| 165 | +
|
| 166 | +# This will create a cluster and deploy your custom-built CCM image |
| 167 | +
|
| 168 | +E2E_DEPLOY_CCM_IMAGE="ghcr.io/stackitcloud/cloud-provider-stackit/cloud-controller-manager:v1.31.6-4-gf2d85f1" \ |
| 169 | +./test/e2e/e2e-test-script.sh create \ |
| 170 | + --project-id <STACKIT_PROJECT_ID> \ |
| 171 | + --kubernetes-version <K8S_VERSION> |
| 172 | +``` |
| 173 | + |
| 174 | +## Running End-to-End (E2E) Tests for the CSI Driver |
| 175 | + |
| 176 | +The CSI E2E test suite validates the full functionality of the CSI driver. Tests are divided into parallel and sequential execution sets to accommodate different operational requirements, particularly for stateful operations like snapshots. |
| 177 | + |
| 178 | +> :warning: Make sure that the kubernetes version of the e2e test (hack/tools.mk@KUBERNETES_TEST_VERSION) matches your kubernetes version! |
| 179 | + |
| 180 | +### Parallel E2E Test Suite |
| 181 | + |
| 182 | +To execute the main CSI E2E test suite, which covers most driver features, use the following command. This suite runs tests in parallel to maximize efficiency. |
| 183 | + |
| 184 | +- **Command:** `make verify-e2e-csi-parallel` |
| 185 | +- **Scope:** Tests a broad spectrum of the CSI driver's core features (e.g., volume provisioning, mounting, unmounting, deletion), except `VolumeSnapshot`. |
| 186 | +
|
| 187 | +### Sequential E2E Test Suite (Snapshots & Backups) |
| 188 | +
|
| 189 | +Specific tests related to `VolumeSnapshot` must be run sequentially to ensure proper state management and ordering of operations. |
| 190 | +
|
| 191 | +### Prerequisite: VolumeSnapshotClass |
| 192 | +
|
| 193 | +Before running the sequential tests, the necessary `VolumeSnapshotClass` resource must be applied to the Kubernetes cluster. This configuration ensures the CSI driver is correctly configured to handle snapshot operations. |
| 194 | +
|
| 195 | +Example: |
| 196 | +
|
| 197 | +```yaml |
| 198 | +apiVersion: snapshot.storage.k8s.io/v1 |
| 199 | +kind: VolumeSnapshotClass |
| 200 | +metadata: |
| 201 | + name: stackit |
| 202 | +driver: block-storage.csi.stackit.cloud |
| 203 | +deletionPolicy: Delete |
| 204 | +parameters: |
| 205 | + type: "snapshot" |
| 206 | +``` |
| 207 | +
|
| 208 | +> You can also run the same test by using the `type: "backup"` parameter instead. This will run the same test against STACKIT Backups instead of Snapshots. |
| 209 | +
|
| 210 | +You can apply this manifest using `kubectl apply -f <filename.yaml>` or ensuring it is present in the cluster. |
| 211 | +
|
| 212 | +### Running the Sequential Tests |
| 213 | +
|
| 214 | +- **Command:** `make verify-e2e-csi-sequential` |
| 215 | +- **Scope:** Specifically targets CSI operations related to volume state, including Snapshots and Restores (Backups). |
| 216 | +- **Requirement:** The `VolumeSnapshotClass` (as defined above) must be present in the cluster. |
| 217 | +
|
| 218 | +### Customizing Test Execution |
| 219 | +
|
| 220 | +The test configuration file allows for granular control over which specific tests are executed. This is useful for debugging or targeting a subset of features. |
| 221 | +
|
| 222 | +- **Configuration File:** `test/e2e/csi/block-storage.yaml` |
| 223 | +- **Action:** Modify this YAML file to include or exclude specific test cases, adjust parameters, or change timeout settings for the E2E runs. This allows for focused testing without running the entire suite. |
| 224 | +
|
| 225 | +### Full Example |
| 226 | +
|
| 227 | +The full example is a walk-trough to run the E2E CSI Driver tests for a specific Kubernetes Version v1.32.9. |
| 228 | +
|
| 229 | +```bash |
| 230 | +# Create a single-node test cluster for Kubernetes v1.32.9 |
| 231 | +# |
| 232 | +# E2E_DEPLOY_BRANCH will auto-detect the Kubernetes version from --kubernetes-version flag |
| 233 | +# and use the corresponding release-vX.Y branch. This can be customized by using E2E_DEPLOY_BRANCH. |
| 234 | +$ E2E_MACHINE_NAME=stackit-ccm-test-1-32 ./test/e2e/e2e.sh create \ |
| 235 | + --project-id <ID> \ |
| 236 | + --kubernetes-version 1.32.9 |
| 237 | +
|
| 238 | +# Check node and pod status |
| 239 | +$ kubectl get nodes |
| 240 | +NAME STATUS ROLES AGE VERSION |
| 241 | +stackit-ccm-test-1-32 Ready control-plane 4d1h v1.32.9 |
| 242 | +
|
| 243 | +# Ensure csi-stackit-* and stackit-cloud-controller-manager-* pods are running and healthy |
| 244 | +$ kubectl get pods -n kube-system |
| 245 | +NAME READY STATUS RESTARTS AGE |
| 246 | +coredns-66bc5c9577-dxvk4 1/1 Running 0 4d1h |
| 247 | +coredns-66bc5c9577-gzkt5 1/1 Running 0 4d1h |
| 248 | +csi-stackit-controllerplugin-5844c9df74-9s4x6 6/6 Running 0 21h |
| 249 | +csi-stackit-nodeplugin-rrffk 3/3 Running 0 21h |
| 250 | +etcd-stackit-ccm-test-1-32 1/1 Running 0 4d1h |
| 251 | +kube-apiserver-stackit-ccm-test-1-32 1/1 Running 0 4d1h |
| 252 | +kube-controller-manager-stackit-ccm-test-1-32 1/1 Running 0 4d1h |
| 253 | +kube-proxy-4pg89 1/1 Running 0 4d1h |
| 254 | +kube-scheduler-stackit-ccm-test-1-32 1/1 Running 0 4d1h |
| 255 | +snapshot-controller-5b7776766f-cjjgv 1/1 Running 0 4d1h |
| 256 | +snapshot-controller-5b7776766f-mp8dp 1/1 Running 0 4d1h |
| 257 | +stackit-cloud-controller-manager-9cbc5fb6-cvj5f 1/1 Running 0 21h |
| 258 | +
|
| 259 | +# Run parallel test suite, ensure KUBERNETES_TEST_VERSION matches the cluster Kubernetes version |
| 260 | +§ KUBERNETES_TEST_VERSION=1.32.9 make verify-e2e-csi-parallel |
| 261 | +
|
| 262 | +# The test result should look like this |
| 263 | +[...] |
| 264 | +Ran 61 of 7450 Specs in 426.519 seconds |
| 265 | +SUCCESS! -- 61 Passed | 0 Failed | 0 Pending | 7389 Skipped |
| 266 | +``` |
0 commit comments