@@ -17,8 +17,9 @@ How do we bring a new application online? How do we upgrade an application? How
17
17
platform, ops, and development teams?
18
18
19
19
In this post, we're going to use Gloo to design a two-phased canary rollout workflow for application upgrades:
20
- * In the first phase, we'll do canary testing by shifting a small subset of traffic to the new version. This allows you to safely perform smoke and correctness tests.
21
- * In the second phase, we'll progressively shift traffic to the new version, allowing us to monitor the new version under load, and eventually, decommission the old version.
20
+
21
+ - In the first phase, we'll do canary testing by shifting a small subset of traffic to the new version. This allows you to safely perform smoke and correctness tests.
22
+ - In the second phase, we'll progressively shift traffic to the new version, allowing us to monitor the new version under load, and eventually, decommission the old version.
22
23
23
24
To keep it simple, we're going to focus on designing the workflow using [ open source Gloo] ( https://github.com/solo-io/gloo ) , and we're going to deploy the gateway and
24
25
application to Kubernetes. At the end, we'll talk about a few extensions and advanced topics that could be interesting to explore in a follow up.
@@ -30,10 +31,10 @@ features, and can be run against a local test cluster such as [minikube](https:/
30
31
This post assumes a basic understanding of Kubernetes and how to interact with it using ` kubectl ` .
31
32
32
33
We'll install the latest [ open source Gloo] ( https://github.com/solo-io/gloo ) to the ` gloo-system ` namespace and deploy
33
- version ` v1 ` of an example application to the ` echo ` namespace. We'll expose this application outside the cluster
34
- by creating a route in Gloo, to end up with a picture like this:
34
+ version ` v1 ` of an example application to the ` echo ` namespace. We'll expose this application outside the cluster
35
+ by creating a route in Gloo, to end up with a picture like this:
35
36
36
- ![ Setup] ( /static/ images/blog/2020-04-17 -two-phased-canary-rollout-with-gloo/setup.png )
37
+ ![ Setup] ( /images/blog/2020-04-22 -two-phased-canary-rollout-with-gloo/setup.png )
37
38
38
39
### Deploying Gloo
39
40
@@ -88,11 +89,12 @@ shifting traffic to a `v2` version of the application.
88
89
89
90
Kubernetes gives us a lot of flexibility in terms of modeling this application. We'll adopt the following
90
91
conventions:
91
- * We'll include the version in the deployment name so we can run two versions of the application
92
- side-by-side and manage their lifecycle differently.
93
- * We'll label pods with an app label (` app: echo ` ) and a version label (` version: v1 ` ) to help with our canary rollout.
94
- * We'll deploy a single Kubernetes ` Service ` for the application to set up networking. Instead of updating
95
- this or using multiple services to manage routing to different versions, we'll manage the rollout with Gloo configuration.
92
+
93
+ - We'll include the version in the deployment name so we can run two versions of the application
94
+ side-by-side and manage their lifecycle differently.
95
+ - We'll label pods with an app label (` app: echo ` ) and a version label (` version: v1 ` ) to help with our canary rollout.
96
+ - We'll deploy a single Kubernetes ` Service ` for the application to set up networking. Instead of updating
97
+ this or using multiple services to manage routing to different versions, we'll manage the rollout with Gloo configuration.
96
98
97
99
The following is our ` v1 ` echo application:
98
100
@@ -148,13 +150,15 @@ kubectl apply -f https://raw.githubusercontent.com/solo-io/gloo-ref-arch/blog-30
148
150
` ` `
149
151
150
152
We should see the following output :
153
+
151
154
` ` `
152
155
namespace/echo created
153
156
deployment.apps/echo-v1 created
154
157
service/echo created
155
158
` ` `
156
159
157
160
And we should be able to see all the resources healthy in the `echo` namespace :
161
+
158
162
` ` `
159
163
➜ kubectl get all -n echo
160
164
NAME READY STATUS RESTARTS AGE
@@ -210,7 +214,7 @@ metadata:
210
214
spec:
211
215
virtualHost:
212
216
domains:
213
- - '*'
217
+ - "*"
214
218
routes:
215
219
- matchers:
216
220
- prefix: /
@@ -237,14 +241,14 @@ version:v1
237
241
238
242
Our setup is complete, and our cluster now looks like this :
239
243
240
- 
244
+ 
241
245
242
246
# # Two-Phased Rollout Strategy
243
247
244
248
Now we have a new version `v2` of the echo application that we wish to roll out. We know that when the
245
249
rollout is complete, we are going to end up with this picture :
246
250
247
- 
251
+ 
248
252
249
253
However, to get there, we may want to perform a few rounds of testing to ensure the new version of the application
250
254
meets certain correctness and/or performance acceptance criteria. In this post, we'll introduce a two-phased approach to
@@ -268,7 +272,7 @@ In this phase, we'll deploy `v2`, and then use a header `stage: canary` to start
268
272
traffic to the new version. We'll use this header to perform some basic smoke testing and make sure `v2` is working the
269
273
way we'd expect :
270
274
271
- 
275
+ 
272
276
273
277
# ## Setting up subset routing
274
278
@@ -284,7 +288,7 @@ metadata:
284
288
spec:
285
289
virtualHost:
286
290
domains:
287
- - '*'
291
+ - "*"
288
292
routes:
289
293
- matchers:
290
294
- prefix: /
@@ -382,7 +386,7 @@ metadata:
382
386
spec:
383
387
virtualHost:
384
388
domains:
385
- - '*'
389
+ - "*"
386
390
routes:
387
391
- matchers:
388
392
- headers:
@@ -464,7 +468,7 @@ metadata:
464
468
spec:
465
469
virtualHost:
466
470
domains:
467
- - '*'
471
+ - "*"
468
472
routes:
469
473
# We'll keep our route from before if we want to continue testing with this header
470
474
- matchers:
@@ -502,17 +506,17 @@ spec:
502
506
values:
503
507
version: v2
504
508
weight: 0
505
-
506
509
` ` `
507
510
508
511
We can apply this virtual service update to the cluster with the following commands :
512
+
509
513
` ` `
510
514
kubectl apply -f https://raw.githubusercontent.com/solo-io/gloo-ref-arch/blog-30-mar-20/platform/prog-delivery/two-phased-with-os-gloo/3-progressive-traffic-shift-to-v2/vs-1.yaml
511
515
` ` `
512
516
513
517
Now the cluster looks like this, for any request that doesn't have the `stage : canary` header:
514
518
515
- 
519
+ 
516
520
517
521
With the initial weights, we should see the gateway continue to serve `v1` for all traffic.
518
522
@@ -525,7 +529,7 @@ version:v1
525
529
526
530
To simulate a load test, let's shift half the traffic to `v2` :
527
531
528
- 
532
+ 
529
533
530
534
This can be expressed on our virtual service by adjusting the weights :
531
535
@@ -538,7 +542,7 @@ metadata:
538
542
spec:
539
543
virtualHost:
540
544
domains:
541
- - '*'
545
+ - "*"
542
546
routes:
543
547
- matchers:
544
548
- headers:
@@ -607,7 +611,7 @@ We will save these topics for a future post on advanced canary testing use cases
607
611
608
612
We will continue adjusting weights until eventually, all of the traffic is now being routed to `v2`:
609
613
610
- 
614
+ 
611
615
612
616
Our virtual service will look like this:
613
617
@@ -620,7 +624,7 @@ metadata:
620
624
spec:
621
625
virtualHost:
622
626
domains:
623
- - '*'
627
+ - "*"
624
628
routes:
625
629
- matchers:
626
630
- headers:
@@ -661,6 +665,7 @@ spec:
661
665
```
662
666
663
667
We can apply that to the cluster with the following command:
668
+
664
669
```
665
670
kubectl apply -f https://raw.githubusercontent.com/solo-io/gloo-ref-arch/blog-30-mar-20/platform/prog-delivery/two-phased-with-os-gloo/3-progressive-traffic-shift-to-v2/vs-3.yaml
666
671
```
@@ -693,7 +698,7 @@ metadata:
693
698
spec :
694
699
virtualHost :
695
700
domains :
696
- - ' * '
701
+ - " * "
697
702
routes :
698
703
- matchers :
699
704
- prefix : /
@@ -721,7 +726,7 @@ kubectl delete deploy -n echo echo-v1
721
726
722
727
Now our cluster looks like this:
723
728
724
- ![ End State] ( /static/ images/blog/2020-04-17 -two-phased-canary-rollout-with-gloo/end-state.png )
729
+ ![ End State] ( /images/blog/2020-04-22 -two-phased-canary-rollout-with-gloo/end-state.png )
725
730
726
731
And requests to the gateway return this:
727
732
@@ -736,26 +741,26 @@ We have now completed our two-phased canary rollout of an application update usi
736
741
737
742
Over the course of this post, we collected a few topics that could be a good starting point for advanced exploration:
738
743
739
- * Using the ** JWT** filter to verify JWTs, extract claims onto headers, and route to canary versions depending on a claim value.
740
- * Looking at ** Prometheus metrics** and ** Grafana dashboards** created by Gloo to monitor the health of the rollout.
741
- * Automating the rollout by integrating ** Flagger** with ** Gloo** .
744
+ - Using the ** JWT** filter to verify JWTs, extract claims onto headers, and route to canary versions depending on a claim value.
745
+ - Looking at ** Prometheus metrics** and ** Grafana dashboards** created by Gloo to monitor the health of the rollout.
746
+ - Automating the rollout by integrating ** Flagger** with ** Gloo** .
742
747
743
748
A few other topics that warrant further exploration:
744
749
745
- * Supporting ** self-service** upgrades by giving teams ownership over their upstream and route configuration
746
- * Utilizing Gloo's ** delegation** feature and Kubernetes ** RBAC** to decentralize the configuration management safely
747
- * Fully automating the continuous delivery process by applying ** GitOps** principles and using tools like ** Flux** to push config to the cluster
748
- * Supporting ** hybrid** or ** non-Kubernetes** application use-cases by setting up Gloo with a different deployment pattern
749
- * Utilizing ** traffic shadowing** to begin testing the new version with realistic data before shifting production traffic to it
750
+ - Supporting ** self-service** upgrades by giving teams ownership over their upstream and route configuration
751
+ - Utilizing Gloo's ** delegation** feature and Kubernetes ** RBAC** to decentralize the configuration management safely
752
+ - Fully automating the continuous delivery process by applying ** GitOps** principles and using tools like ** Flux** to push config to the cluster
753
+ - Supporting ** hybrid** or ** non-Kubernetes** application use-cases by setting up Gloo with a different deployment pattern
754
+ - Utilizing ** traffic shadowing** to begin testing the new version with realistic data before shifting production traffic to it
750
755
751
756
## Get Involved in the Gloo Community
752
757
753
758
Gloo has a large and growing community of open source users, in addition to an enterprise customer base. To learn more about
754
759
Gloo:
755
760
756
- * Check out the [ repo] ( https://github.com/solo-io/gloo ) , where you can see the code and file issues
757
- * Check out the [ docs] ( https://docs.solo.io/gloo/latest ) , which have an extensive collection of guides and examples
758
- * Join the [ slack channel] ( http://slack.solo.io/ ) and start chatting with the Solo engineering team and user community
761
+ - Check out the [ repo] ( https://github.com/solo-io/gloo ) , where you can see the code and file issues
762
+ - Check out the [ docs] ( https://docs.solo.io/gloo/latest ) , which have an extensive collection of guides and examples
763
+ - Join the [ slack channel] ( http://slack.solo.io/ ) and start chatting with the Solo engineering team and user community
759
764
760
765
If you'd like to get in touch with me (feedback is always appreciated!), you can find me on the
761
766
[ Solo slack
] ( http://slack.solo.io/ ) or email me at
** [email protected] ** .
0 commit comments