test: use clusterloader2 framework for large scale testing

Signed-off-by: Wantong Jiang <[email protected]>

Author: jwtty

hack/cl2/README.md

@@ -0,0 +1,24 @@
+This folder contains a list of (clusterloader2)[https://github.com/kubernetes/perf-tests/blob/master/clusterloader2/docs/GETTING_STARTED.md] configuration manifests for large scale testing.
+
+## Prerequisites
+
+Follow (clusterloader2 GETTING STARTED)[https://github.com/kubernetes/perf-tests/blob/master/clusterloader2/docs/GETTING_STARTED.md] to clone the `perf-tests` repository.
+
+## Execute Tests
+
+Run `clusterloader2` under `perf-tests/clusterloader2` directory:
+```bash
+go run cmd/clusterloader.go --testconfig=<path to the test yaml> --provider=local --kubeconfig=<path to the hub cluster kubeconfig> --v=2 --enable-exec-service=false
+```
+We need to set `--enable-exec-service=false` to prevent creating agnostic deployment on the hub cluster as hub
+cluster does not allow pod creation.
+
+## Cleanup Resources After Tests
+
+By default, clusterloader2 automatically deletes all the generated namespaces after the tests.
+In addition, we also want to delete all the CRPs created. To facilitate cleanup process, run the simple script
+provided in this folder:
+```
+export KUBECONFIG=<your path to the kubeconfig file>
+./cleanup.sh
+```

hack/cl2/busy_cluster.yaml

@@ -0,0 +1,91 @@
+name: busy_cluster_test
+
+{{$duration := "10m"}}
+{{$count := 1000}}
+
+namespace:
+  number: 2
+  prefix: busy-cluster-test-ns
+
+tuningSets:
+- name: Uniform100qps
+  qpsLoad:
+    qps: 100
+
+steps:
+- name: Create CRPs
+  phases:
+  - replicasPerNamespace: 2
+    tuningSet: Uniform100qps
+    objectBundle:
+    - basename: test-crp 
+      objectTemplatePath: "manifests/test-crp.yaml"
+- name: Create ConfigMaps
+  phases:
+  - namespaceRange:
+      min: 1
+      max: 2
+    replicasPerNamespace: {{$count}}
+    tuningSet: Uniform100qps
+    objectBundle:
+    - basename: test-configmap 
+      objectTemplatePath: "manifests/test-configmap.yaml"
+- name: Sleep
+  measurements:
+  - Identifier: sleep
+    Method: Sleep
+    Params:
+      duration: {{$duration}}
+# - name: Wait for CRPs to be Ready
+#   measurements:
+#   - Identifier: WaitForGenericK8sObjects
+#     Method: WaitForGenericK8sObjects
+#     Params:
+#       objectGroup: placement.kubernetes-fleet.io
+#       objectVersion: v1beta1
+#       objectResource: clusterresourceplacements
+#       timeout: {{$duration}}
+#       successfulConditions: ResourceAvailable=True
+#       minDesiredObjectCount: 2
+- name: Delete ConfigMaps
+  phases:
+  - namespaceRange:
+      min: 1
+      max: 2
+    replicasPerNamespace: 0
+    tuningSet: Uniform100qps
+    objectBundle:
+    - basename: test-configmap 
+      objectTemplatePath: "manifests/test-configmap.yaml"
+- name: Sleep
+  measurements:
+  - Identifier: sleep
+    Method: Sleep
+    Params:
+      duration: {{$duration}}
+- name: Create ConfigMaps
+  phases:
+  - namespaceRange:
+      min: 1
+      max: 2
+    replicasPerNamespace: {{$count}}
+    tuningSet: Uniform100qps
+    objectBundle:
+    - basename: test-configmap 
+      objectTemplatePath: "manifests/test-configmap.yaml"
+- name: Sleep
+  measurements:
+  - Identifier: sleep
+    Method: Sleep
+    Params:
+      duration: {{$duration}}
+- name: Delete ConfigMaps
+  phases:
+  - namespaceRange:
+      min: 1
+      max: 2
+    replicasPerNamespace: 0
+    tuningSet: Uniform100qps
+    objectBundle:
+    - basename: test-configmap 
+      objectTemplatePath: "manifests/test-configmap.yaml"

hack/cl2/cleanup.sh

@@ -0,0 +1,10 @@
+#!/usr/bin/env bash
+
+set -o errexit
+set -o nounset
+set -o pipefail
+
+: "${KUBECONFIG:?Environment variable KUBECONFIG must be set}"
+
+echo "Deleting all CRPs generated during the load test from the hub cluster..."
+kubectl delete crp -l test=cl2-test

hack/cl2/manifests/test-configmap.yaml

@@ -0,0 +1,7 @@
+# Test configmap with 1k payload
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: {{.Name}}
+data:
+  random: Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec, pellentesque eu, pretium quis, sem. Nulla consequat massa quis enim. Donec pede justo, fringilla vel, aliquet nec, vulputate eget, arcu. In enim justo, rhoncus ut, imperdiet a, venenatis vitae, justo. Nullam dictum felis eu pede mollis pretium. Integer tincidunt. Cras dapibus. Vivamus elementum semper nisi. Aenean vulputate eleifend tellus. Aenean leo ligula, porttitor eu, consequat vitae, eleifend ac, enim. Aliquam lorem ante, dapibus in, viverra quis, feugiat a, tellus. Phasellus viverra nulla ut metus varius laoreet. Quisque rutrum. Aenean imperdiet. Etiam ultricies nisi vel augue. Curabitur ullamcorper ultricies nisi. Nam eget dui. Etiam rhoncus. Maecenas tempus, tellus eget condimentum rhoncus, sem quam semper libero, sit amet adipiscing sem neque sed ipsum. Nam quam nunc, blandit ve

hack/cl2/manifests/test-crp.yaml

@@ -0,0 +1,19 @@
+apiVersion: placement.kubernetes-fleet.io/v1beta1
+kind: ClusterResourcePlacement
+metadata:
+  name: {{.Name}}
+  labels:
+    test: cl2-test
+spec:
+  resourceSelectors:
+    - group: ""
+      kind: Namespace
+      name: busy-cluster-test-ns-{{AddInt .Index 1}}
+      version: v1
+  strategy:
+    type: RollingUpdate
+    rollingUpdate:
+      maxUnavailable: 25%
+      maxSurge: 25%
+      unavailablePeriodSeconds: 60
+  revisionHistoryLimit: 15