Skip to content

Commit e50e0a6

Browse files
jwttyjwtty
committed
test: use clusterloader2 framework for large scale testing
Signed-off-by: Wantong Jiang <[email protected]>
1 parent 8a5bb61 commit e50e0a6

File tree

5 files changed

+182
-0
lines changed

5 files changed

+182
-0
lines changed

‎hack/cl2/README.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
This folder contains a list of (clusterloader2)[https://github.com/kubernetes/perf-tests/blob/master/clusterloader2/docs/GETTING_STARTED.md] configuration manifests for large scale testing.
2+
3+
## Prerequisites
4+
5+
Follow (clusterloader2 GETTING STARTED)[https://github.com/kubernetes/perf-tests/blob/master/clusterloader2/docs/GETTING_STARTED.md] to clone the `perf-tests` repository.
6+
7+
## Execute Tests
8+
9+
Run `clusterloader2` under `perf-tests/clusterloader2` directory:
10+
```bash
11+
go run cmd/clusterloader.go --testconfig=<path to the test yaml> --provider=local --kubeconfig=<path to the hub cluster kubeconfig> --v=2 --enable-exec-service=false
12+
```
13+
We need to set `--enable-exec-service=false` to prevent creating agnostic deployment on the hub cluster as hub
14+
cluster does not allow pod creation.
15+
16+
## Cleanup Resources After Tests
17+
18+
By default, clusterloader2 automatically deletes all the generated namespaces after the tests.
19+
In addition, we also want to delete all the CRPs created. To facilitate cleanup process, run the simple script
20+
provided in this folder:
21+
```
22+
export KUBECONFIG=<your path to the kubeconfig file>
23+
./cleanup.sh
24+
```

‎hack/cl2/busy_cluster.yaml‎

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
name: busy_cluster_test
2+
3+
{{$duration := "10m"}}
4+
{{$count := 1000}}
5+
6+
namespace:
7+
number: 2
8+
prefix: busy-cluster-test-ns
9+
10+
tuningSets:
11+
- name: Uniform100qps
12+
qpsLoad:
13+
qps: 100
14+
15+
steps:
16+
- name: Create CRPs
17+
phases:
18+
- replicasPerNamespace: 2
19+
tuningSet: Uniform100qps
20+
objectBundle:
21+
- basename: test-crp
22+
objectTemplatePath: "manifests/test-crp.yaml"
23+
- name: Create ConfigMaps
24+
phases:
25+
- namespaceRange:
26+
min: 1
27+
max: 2
28+
replicasPerNamespace: {{$count}}
29+
tuningSet: Uniform100qps
30+
objectBundle:
31+
- basename: test-configmap
32+
objectTemplatePath: "manifests/test-configmap.yaml"
33+
- name: Wait for CRPs to be Ready
34+
measurements:
35+
- Identifier: WaitForGenericK8sObjects
36+
Method: WaitForGenericK8sObjects
37+
Params:
38+
objectGroup: placement.kubernetes-fleet.io
39+
objectVersion: v1beta1
40+
objectResource: clusterresourceplacements
41+
timeout: {{$duration}}
42+
successfulConditions:
43+
- ClusterResourcePlacementAvailable=True
44+
failedConditions:
45+
- ClusterResourcePlacementAvailable=False
46+
minDesiredObjectCount: 2
47+
maxFailedObjectCount: 0
48+
- name: Delete ConfigMaps
49+
phases:
50+
- namespaceRange:
51+
min: 1
52+
max: 2
53+
replicasPerNamespace: 0
54+
tuningSet: Uniform100qps
55+
objectBundle:
56+
- basename: test-configmap
57+
objectTemplatePath: "manifests/test-configmap.yaml"
58+
- name: Wait for CRPs to be Ready
59+
measurements:
60+
- Identifier: WaitForGenericK8sObjects
61+
Method: WaitForGenericK8sObjects
62+
Params:
63+
objectGroup: placement.kubernetes-fleet.io
64+
objectVersion: v1beta1
65+
objectResource: clusterresourceplacements
66+
timeout: {{$duration}}
67+
successfulConditions:
68+
- ClusterResourcePlacementAvailable=True
69+
failedConditions:
70+
- ClusterResourcePlacementAvailable=False
71+
minDesiredObjectCount: 2
72+
maxFailedObjectCount: 0
73+
- name: Create ConfigMaps
74+
phases:
75+
- namespaceRange:
76+
min: 1
77+
max: 2
78+
replicasPerNamespace: {{$count}}
79+
tuningSet: Uniform100qps
80+
objectBundle:
81+
- basename: test-configmap
82+
objectTemplatePath: "manifests/test-configmap.yaml"
83+
- name: Wait for CRPs to be Ready
84+
measurements:
85+
- Identifier: WaitForGenericK8sObjects
86+
Method: WaitForGenericK8sObjects
87+
Params:
88+
objectGroup: placement.kubernetes-fleet.io
89+
objectVersion: v1beta1
90+
objectResource: clusterresourceplacements
91+
timeout: {{$duration}}
92+
successfulConditions:
93+
- ClusterResourcePlacementAvailable=True
94+
failedConditions:
95+
- ClusterResourcePlacementAvailable=False
96+
minDesiredObjectCount: 2
97+
maxFailedObjectCount: 0
98+
- name: Delete ConfigMaps
99+
phases:
100+
- namespaceRange:
101+
min: 1
102+
max: 2
103+
replicasPerNamespace: 0
104+
tuningSet: Uniform100qps
105+
objectBundle:
106+
- basename: test-configmap
107+
objectTemplatePath: "manifests/test-configmap.yaml"
108+
- name: Wait for CRPs to be Ready
109+
measurements:
110+
- Identifier: WaitForGenericK8sObjects
111+
Method: WaitForGenericK8sObjects
112+
Params:
113+
objectGroup: placement.kubernetes-fleet.io
114+
objectVersion: v1beta1
115+
objectResource: clusterresourceplacements
116+
timeout: {{$duration}}
117+
successfulConditions:
118+
- ClusterResourcePlacementAvailable=True
119+
failedConditions:
120+
- ClusterResourcePlacementAvailable=False
121+
minDesiredObjectCount: 2
122+
maxFailedObjectCount: 0

‎hack/cl2/cleanup.sh‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
#!/usr/bin/env bash
2+
3+
set -o errexit
4+
set -o nounset
5+
set -o pipefail
6+
7+
: "${KUBECONFIG:?Environment variable KUBECONFIG must be set}"
8+
9+
echo "Deleting all CRPs generated during the load test from the hub cluster..."
10+
kubectl delete crp -l test=cl2-test

‎hack/cl2/manifests/test-configmap.yaml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# Test configmap with 1k payload
2+
apiVersion: v1
3+
kind: ConfigMap
4+
metadata:
5+
name: {{.Name}}
6+
data:
7+
random: Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec, pellentesque eu, pretium quis, sem. Nulla consequat massa quis enim. Donec pede justo, fringilla vel, aliquet nec, vulputate eget, arcu. In enim justo, rhoncus ut, imperdiet a, venenatis vitae, justo. Nullam dictum felis eu pede mollis pretium. Integer tincidunt. Cras dapibus. Vivamus elementum semper nisi. Aenean vulputate eleifend tellus. Aenean leo ligula, porttitor eu, consequat vitae, eleifend ac, enim. Aliquam lorem ante, dapibus in, viverra quis, feugiat a, tellus. Phasellus viverra nulla ut metus varius laoreet. Quisque rutrum. Aenean imperdiet. Etiam ultricies nisi vel augue. Curabitur ullamcorper ultricies nisi. Nam eget dui. Etiam rhoncus. Maecenas tempus, tellus eget condimentum rhoncus, sem quam semper libero, sit amet adipiscing sem neque sed ipsum. Nam quam nunc, blandit ve

‎hack/cl2/manifests/test-crp.yaml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
apiVersion: placement.kubernetes-fleet.io/v1beta1
2+
kind: ClusterResourcePlacement
3+
metadata:
4+
name: {{.Name}}
5+
labels:
6+
test: cl2-test
7+
spec:
8+
resourceSelectors:
9+
- group: ""
10+
kind: Namespace
11+
name: busy-cluster-test-ns-{{AddInt .Index 1}}
12+
version: v1
13+
strategy:
14+
type: RollingUpdate
15+
rollingUpdate:
16+
maxUnavailable: 25%
17+
maxSurge: 25%
18+
unavailablePeriodSeconds: 60
19+
revisionHistoryLimit: 15

0 commit comments

Comments
 (0)