|
| 1 | +--- |
| 2 | +title: "Nginx A/B Testing" |
| 3 | +linkTitle: "Nginx A/B Testing" |
| 4 | +weight: 20 |
| 5 | +description: > |
| 6 | + A comprehensive guide on Kurator's A/B Testing uses Nginx as ingress, providing an overview and quick start guide. |
| 7 | +--- |
| 8 | + |
| 9 | +## Prerequisites |
| 10 | + |
| 11 | +In the subsequent sections, we'll guide you through a hands-on demonstration. |
| 12 | + |
| 13 | +These are some of the prerequisites needed to use Kurator Rollout: |
| 14 | + |
| 15 | +### Kubernetes Clusters |
| 16 | + |
| 17 | +Kubernetes v1.27.3 or higher is supported. |
| 18 | + |
| 19 | +You can use [Kind](https://kind.sigs.k8s.io/) to create clusters as needed. |
| 20 | +It is recommended to use [Kurator's scripts](https://kurator.dev/docs/setup/install-cluster-operator/#setup-kubernetes-clusters-with-kind) to create multi-clusters environment. |
| 21 | + |
| 22 | +Notes: You can find the mapping between Kind node image versions and Kubernetes versions on [Kind Release](https://github.com/kubernetes-sigs/kind/releases). Additionally, the website provides a lookup table showing compatible Kind and node image versions. |
| 23 | + |
| 24 | + |
| 25 | +### Nginx |
| 26 | + |
| 27 | +When Nginx is specified in fleet's `rollout.trafficRoutingProvider` , Kurator will install Nginx and its supporting Prometheus via helm in the fleet-managed clusters. |
| 28 | + |
| 29 | +You can review the results a few minutes after applying fleet: |
| 30 | + |
| 31 | +```console |
| 32 | +kubectl get po -n ingress-nginx --kubeconfig=/root/.kube/kurator-member1.config |
| 33 | +NAME READY STATUS RESTARTS AGE |
| 34 | +ingress-nginx-flagger-kurator-member-7fbdfb7f7-hphc2 1/1 Running 0 5m44s |
| 35 | +ingress-nginx-flagger-kurator-member-prometheus-56bdbf4855l4jkx 1/1 Running 0 5m44s |
| 36 | +ingress-nginx-nginx-kurator-member-controller-6566b7886-b7g8f 1/1 Running 0 5m33s |
| 37 | +ingress-nginx-testloader-kurator-member-loadtester-7ff7d75l2dwj 1/1 Running 0 5m51s |
| 38 | + |
| 39 | +``` |
| 40 | + |
| 41 | +### Kurator Rollout Plugin |
| 42 | + |
| 43 | +Before delving into the how to Perform a Unified Rollout, ensure you have successfully installed the Rollout plugin as outlined in the [Rollout plugin installation guide](/docs/fleet-manager/rollout/rollout-plugin/). |
| 44 | + |
| 45 | +## How to Perform a Unified Rollout |
| 46 | + |
| 47 | +### Configuring the Rollout Policy |
| 48 | + |
| 49 | +You can deploy a abtest application demo using Nginx by the following command: |
| 50 | + |
| 51 | +```console |
| 52 | +kubectl apply -f examples/rollout/ab-testingNginx.yaml |
| 53 | +``` |
| 54 | + |
| 55 | +Here is the configuration: |
| 56 | + |
| 57 | +```yaml |
| 58 | +apiVersion: apps.kurator.dev/v1alpha1 |
| 59 | +kind: Application |
| 60 | +metadata: |
| 61 | + name: abtesting-nginx-demo |
| 62 | + namespace: default |
| 63 | +spec: |
| 64 | + source: |
| 65 | + gitRepository: |
| 66 | + interval: 3m0s |
| 67 | + ref: |
| 68 | + branch: master |
| 69 | + timeout: 1m0s |
| 70 | + url: https://github.com/stefanprodan/podinfo |
| 71 | + syncPolicies: |
| 72 | + - destination: |
| 73 | + fleet: quickstart |
| 74 | + kustomization: |
| 75 | + interval: 0s |
| 76 | + path: ./deploy/webapp |
| 77 | + prune: true |
| 78 | + timeout: 2m0s |
| 79 | + rollout: |
| 80 | + testLoader: true |
| 81 | + trafficRoutingProvider: nginx |
| 82 | + workload: |
| 83 | + apiVersion: apps/v1 |
| 84 | + name: backend |
| 85 | + kind: Deployment |
| 86 | + namespace: webapp |
| 87 | + serviceName: backend |
| 88 | + port: 9898 |
| 89 | + rolloutPolicy: |
| 90 | + trafficRouting: |
| 91 | + analysisTimes: 3 |
| 92 | + timeoutSeconds: 60 |
| 93 | + host: "app.example.com" |
| 94 | + match: |
| 95 | + - headers: |
| 96 | + x-canary: |
| 97 | + exact: "insider" |
| 98 | + - headers: |
| 99 | + cookie: |
| 100 | + exact: "canary" |
| 101 | + trafficAnalysis: |
| 102 | + checkIntervalSeconds: 90 |
| 103 | + checkFailedTimes: 2 |
| 104 | + metrics: |
| 105 | + - name: nginx-request-success-rate |
| 106 | + intervalSeconds: 90 |
| 107 | + thresholdRange: |
| 108 | + min: 99 |
| 109 | + customMetric: |
| 110 | + provider: |
| 111 | + type: prometheus |
| 112 | + address: http://ingress-nginx-flagger-kurator-member-prometheus.ingress-nginx:9090 |
| 113 | + query: | |
| 114 | + sum( |
| 115 | + rate( |
| 116 | + http_requests_total{ |
| 117 | + status!~"5.*" |
| 118 | + }[{{ interval }}] |
| 119 | + ) |
| 120 | + ) |
| 121 | + / |
| 122 | + sum( |
| 123 | + rate( |
| 124 | + http_requests_total[{{ interval }}] |
| 125 | + ) |
| 126 | + ) * 100 |
| 127 | + webhooks: |
| 128 | + timeoutSeconds: 60 |
| 129 | + command: |
| 130 | + - "hey -z 1m -q 10 -c 2 http://app.example.com/" |
| 131 | + rolloutTimeoutSeconds: 600 |
| 132 | + - destination: |
| 133 | + fleet: quickstart |
| 134 | + kustomization: |
| 135 | + targetNamespace: default |
| 136 | + interval: 5m0s |
| 137 | + path: ./kustomize |
| 138 | + prune: true |
| 139 | + timeout: 2m0s |
| 140 | +``` |
| 141 | +
|
| 142 | +**Notes:** There is a problem with the metric provided by the current flagger, so `customMetric` is used.Here is the detailed [API](/docs/references/app-api/#apps.kurator.dev/v1alpha1.Metric). |
| 143 | + |
| 144 | +To use Nginx, you need to provide the `host` it uses. Kurator will generate an ingress resource based on this field. Here is the [specific configuration generated](/docs/references/app-api/#apps.kurator.dev/v1alpha1.TrafficRoutingConfig). Other configurations are as follow. |
| 145 | + |
| 146 | +Given the output provided, let's dive deeper to understand the various elements and their implications: |
| 147 | + |
| 148 | +- Kurator allows customizing Rollout strategies under the `Spec.syncPolicies.rollout` section for services deployed via kustomization. It will establish and implement A/B Testing for these services according to the configuration defined here. |
| 149 | +- The `workload` defines the target resource for the A/B Testing. The `kind` specifies the resource type, which can be either deployment or daemonset. |
| 150 | +- The `serviceName` and `port` specify the name of the service for the workload as well as the exposed port number. |
| 151 | +- The `trafficAnalysis` section defines the configuration for evaluating a new release version's health and readiness during a rollout process. |
| 152 | + - The `checkFailedTimes` parameter specifies the maximum number of failed check results allowed throughout the A/B Testing lifecycle. |
| 153 | + - `checkIntervalSeconds` denotes the time interval between consecutive health evaluation checks. |
| 154 | + - The `metrics` identify the metrics that will be monitored to determine the deployment's health status. You can choose between the two built-in metric types `request-success-rate` and `request-duration` or write your own metric |
| 155 | + - The `webhooks` provide an extensibility mechanism for the analysis procedures. In this configuration, webhooks communicate with the testloader to generate test traffic for the healthchecks. |
| 156 | +- The `trafficRouting` configuration specifies how traffic will be shifted to the A/B Testing during the rollout process. |
| 157 | + - The `analysisTimes` signifies the number of testing iterations that will be conducted. |
| 158 | + - The `match` defines the criteria that an incoming request must satisfy in order to be routed to the new version. This includes header match definitions which specify rules for request headers. Other match dimensions like port, URL path etc. can also be configured. It's important to note that HTTP matching only takes effect during code analysis, and does not apply to normal usage afterwards. Please refer to [Application API Reference](https://kurator.dev/docs/references/app-api/#apps.kurator.dev/v1alpha1.TrafficRoutingConfig) for more details on directly setting the release and test traffic distributions. |
| 159 | +- The `rolloutStatus` section displays the actual processing status of rollout within the fleet. |
| 160 | + |
| 161 | +About a minute after submitting this configuration, you can check the rollout status by running the following command: |
| 162 | + |
| 163 | +```console |
| 164 | +kubectl get canary -n webapp --kubeconfig=/root/.kube/kurator-member1.config |
| 165 | +
|
| 166 | +NAME STATUS WEIGHT LASTTRANSITIONTIME |
| 167 | +backend Initialized 0 2024-01-12T08:53:40Z |
| 168 | +``` |
| 169 | + |
| 170 | +If the status shows as `Initialized`, it means the initialization of rollout process has completed successfully. |
| 171 | + |
| 172 | +**Notes**: In the above configuration, we set the `kustomization.interval` to 0s. This disables Fluxcd's periodic synchronization of configurations between the local mirror and cluster. The reason is that Flagger needs to modify the replica counts in Deployments to complete its initialization process. If you are uncertain whether the replicas for all applications in your deployments are set to zero, it is recommended to also set `kustomization.interval` to 0s. |
| 173 | + |
| 174 | +### Trigger Rollout |
| 175 | + |
| 176 | +An A/B Testing can be triggered by either updating the container image referenced in the git repository configuration, or directly updating the image of the deployment resource locally in the Kubernetes cluster. |
| 177 | + |
| 178 | +Review the results: |
| 179 | + |
| 180 | +```console |
| 181 | +kubectl get canary -n webapp -w --kubeconfig=/root/.kube/kurator-member1.config |
| 182 | +
|
| 183 | +NAME STATUS WEIGHT LASTTRANSITIONTIME |
| 184 | +backend Initialized 0 2024-01-12T08:53:40Z |
| 185 | +backend Progressing 0 2024-01-12T08:55:10Z |
| 186 | +backend Progressing 0 2024-01-12T08:56:40Z |
| 187 | +backend Progressing 0 2024-01-12T08:58:10Z |
| 188 | +backend Progressing 0 2024-01-12T08:59:40Z |
| 189 | +backend Progressing 0 2024-01-12T09:01:10Z |
| 190 | +backend Promoting 0 2024-01-12T09:02:40Z |
| 191 | +backend Finalising 0 2024-01-12T09:04:10Z |
| 192 | +backend Succeeded 0 2024-01-12T09:05:40Z |
| 193 | +``` |
| 194 | + |
| 195 | +{{< image width="100%" |
| 196 | +link="./image/abtesting.svg" |
| 197 | +>}} |
| 198 | + |
| 199 | +- As shown in the diagram, after triggering an A/B Testing, the Kurator Rollout Plugin will first create pod(s) for the new version. |
| 200 | +- The new version will then undergo multiple test iterations. During this testing period, incoming requests matching the defined criteria will be routed to the new version. Various testing metrics will be evaluated to determine the health and stability of the new release. |
| 201 | +- Upon validating the new version through testing and confirming it is ready for release, Kurator will proceed to replace the old version with the new version across the entire cluster. |
| 202 | +- It will then remove the canary pod, completing the rollout process. |
| 203 | + |
| 204 | +```console |
| 205 | +kubectl get application abtesting-nginx-demo -oyaml |
| 206 | +
|
| 207 | +rolloutStatus: |
| 208 | + backupNameInCluster: backend |
| 209 | + backupStatusInCluster: |
| 210 | + canaryWeight: 0 |
| 211 | + conditions: |
| 212 | + - lastTransitionTime: "2024-01-12T09:05:40Z" |
| 213 | + lastUpdateTime: "2024-01-12T09:05:40Z" |
| 214 | + message: Canary analysis completed successfully, promotion finished. |
| 215 | + reason: Succeeded |
| 216 | + status: "True" |
| 217 | + type: Promoted |
| 218 | + failedChecks: 1 |
| 219 | + iterations: 0 |
| 220 | + lastAppliedSpec: 7b779dcc48 |
| 221 | + lastPromotedSpec: 7b779dcc48 |
| 222 | + lastTransitionTime: "2024-01-12T09:05:40Z" |
| 223 | + phase: Succeeded |
| 224 | + trackedConfigs: {} |
| 225 | + clusterName: kurator-member1 |
| 226 | +``` |
| 227 | + |
| 228 | +An A/B Testing is triggered by changes in any of the following objects: |
| 229 | + |
| 230 | +- Deployment PodSpec (container image, command, ports, env, resources, etc) |
| 231 | +- ConfigMaps mounted as volumes or mapped to environment variables |
| 232 | +- Secrets mounted as volumes or mapped to environment variables |
| 233 | + |
| 234 | +**Notes:** If you apply new changes to the deployment during the analysis, Kurator Rollout will restart the analysis. |
| 235 | + |
| 236 | +## Cleanup |
| 237 | + |
| 238 | +### 1.Cleanup the Rollout Policy |
| 239 | + |
| 240 | +If you only need to remove the Rollout Policy, simply edit the current application and remove the corresponding description: |
| 241 | + |
| 242 | +```console |
| 243 | +kubectl edit application abtesting-nginx-demo |
| 244 | +``` |
| 245 | + |
| 246 | +To check the results of the deletion, you can observe that the rollout-related pods have been removed: |
| 247 | + |
| 248 | +```console |
| 249 | +kubectl get po -A --kubeconfig=/root/.kube/kurator-member1.config |
| 250 | +kubectl get po -A --kubeconfig=/root/.kube/kurator-member2.config |
| 251 | +``` |
| 252 | + |
| 253 | +If you want to configure an A/B Testing for it again, you can simply edit the application and add the necessary configurations. |
| 254 | + |
| 255 | +### 2.Cleanup the Application |
| 256 | + |
| 257 | +When the application is delete, all associated resources will also be removed: |
| 258 | + |
| 259 | +```console |
| 260 | +kubectl delete application abtesting-nginx-demo |
| 261 | +``` |
0 commit comments