Skip to content

Commit fa575ad

Browse files
rborbo
committed
Added cross cluster live migration
1 parent 71bd017 commit fa575ad

14 files changed

+622
-0
lines changed

‎content/kubevirt/livemigration/cclm.md‎

Lines changed: 283 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,283 @@
1+
---
2+
title: Cross Cluster Live Migration
3+
linktitle: Cross Cluster Live Migration
4+
description: Cross Cluster Live Migration
5+
tags: ['cnv','kubevirt','ocp-v','v4.12']
6+
---
7+
# Cross cluster live migration
8+
9+
Official documentation: [12.5. Configuring a cross-cluster live migration network](https://docs.redhat.com/en/documentation/openshift_container_platform/4.20/html/virtualization/live-migration#virt-configuring-cross-cluster-live-migration-network)
10+
11+
Tested with:
12+
13+
|Component|Version|
14+
|---|---|
15+
|OpenShift|v4.20.4|
16+
|OpenShift Virt|v4.20.1|
17+
|MTV|v2.10.0|
18+
19+
Without ACM, just a pure cross cluster live migration with two OpenShift clusters.
20+
21+
## Cluster overview
22+
23+
We have two identicial clusters in terms of
24+
25+
* OpenShift Version
26+
* CPU Type and Model
27+
28+
Cluster one called OCP1 is the target cluster with mtv.
29+
Cluster two called OCP7 is the source cluster.
30+
31+
This cluster are running on bare OpenShift Cluster called ISAR.
32+
33+
![](cclm/overview.drawio)
34+
35+
### Details about the OCP1 & OCP7 adjustments at ISAR
36+
37+
OCP1 and OCP7 are provided via our [stormshift automation](https://github.com/stormshift/automation)
38+
39+
??? quote "OCP1 & OCP7 Infrastructure details"
40+
41+
#### Patch the cpu model
42+
43+
=== "Command"
44+
45+
```shell
46+
oc get vm -o name | xargs oc patch --type=merge -p '{"spec":{"template":{"spec":{"domain":{"cpu":{"model":"Haswell-v4"}}}}}}'
47+
```
48+
49+
#### Enable VT-X/vmx feature
50+
51+
=== "Command"
52+
53+
```shell
54+
oc get vm -o name | xargs oc patch --type=merge -p '{"spec":{"template":{"spec":{"domain":{"cpu":{"features":[{"name":"vmx","policy":"require"}]}}}}}}'
55+
```
56+
57+
#### Restart all VM's
58+
59+
=== "Command"
60+
61+
```shell
62+
oc get vm --no-headers -o custom-columns="NAME:.metadata.name" | xargs -n1 virtctl restart
63+
```
64+
65+
#### Check setttings as ISAR
66+
67+
=== "Command"
68+
69+
```shell
70+
oc get vm -o custom-columns=NAME:.metadata.name,CPU:.spec.template.spec.domain.cpu
71+
```
72+
73+
=== "Example output"
74+
75+
```shell
76+
oc get vm -o custom-columns=NAME:.metadata.name,CPU:.spec.template.spec.domain.cpu
77+
NAME CPU
78+
ocp1-cp-0 map[cores:8 features:[map[name:vmx policy:require]] model:Haswell-v4 sockets:1 threads:1]
79+
ocp1-cp-1 map[cores:8 features:[map[name:vmx policy:require]] model:Haswell-v4 sockets:1 threads:1]
80+
ocp1-cp-2 map[cores:8 features:[map[name:vmx policy:require]] model:Haswell-v4 sockets:1 threads:1]
81+
ocp1-worker-0 map[cores:8 features:[map[name:vmx policy:require]] model:Haswell-v4 sockets:1 threads:1]
82+
ocp1-worker-1 map[cores:8 features:[map[name:vmx policy:require]] model:Haswell-v4 sockets:1 threads:1]
83+
ocp1-worker-2 map[cores:8 features:[map[name:vmx policy:require]] model:Haswell-v4 sockets:1 threads:1]
84+
```
85+
86+
#### Add second interface into vlan 2001 for the VM's/nodes
87+
88+
=== "oc apply -f ...."
89+
90+
```bash
91+
oc apply -n stormshift-ocp1-infra -f {{ page.canonical_url }}cclm/isar-2001-net-attach-def.yaml
92+
oc apply -n stormshift-ocp7-infra -f {{ page.canonical_url }}cclm/isar-2001-net-attach-def.yaml
93+
94+
```
95+
96+
=== "isar-2001-net-attach-def.yaml"
97+
98+
```yaml
99+
--8<-- "content/kubevirt/livemigration/cclm/isar-2001-net-attach-def.yaml"
100+
```
101+
102+
#### Adjust VM's to add second interface to worker nodes:
103+
104+
![](cclm/isar-second-interface.png)
105+
106+
## OCP1 and OCP7 cluster preperation
107+
108+
### Install following operators
109+
110+
* Nmstate Operator (instantiate the operator now)
111+
* OpenShift Virtualization (instantiate the operator **LATER!**)
112+
* Migration Toolkit for Virtualization (instantiate the operator **LATER!**)
113+
114+
### Prepare required live migration network
115+
116+
Both clusters have to be connected via an L2 network.
117+
In my case it's vlan 2001 with `192.168.201.0/24 subnet
118+
119+
Here an high level overview:
120+
121+
![](cclm/live-migration-network.drawio)
122+
123+
???+ bug "There is an documetion bug in the offical docs"
124+
125+
<https://issues.redhat.com/browse/CNV-74609>
126+
127+
??? example "NodeNetworkConfigurationPolicy for linux bridge into VLAN 2001"
128+
129+
Apply this to OCP1 and OCP7
130+
131+
=== "coe-bridge-via-enp2s0.yaml"
132+
133+
```yaml
134+
--8<-- "content/kubevirt/livemigration/cclm/coe-bridge-via-enp2s0.yaml"
135+
```
136+
137+
=== "oc apply -f ...."
138+
139+
```bash
140+
oc apply -f {{ page.canonical_url }}cclm/coe-bridge-via-enp2s0.yaml
141+
```
142+
143+
??? example "NetworkAttachmentDefinition for OCP1 and OCP7"
144+
145+
Little helper for find out the interfaces on the nodes:
146+
147+
```shell
148+
oc get nodes -l node-role.kubernetes.io/worker -o name | while read line ; do echo "# $line";oc debug -q $line -- ip -br l | grep enp ; done
149+
```
150+
151+
Apply this to OCP1
152+
153+
=== "ocp1.net-attach-def.yaml"
154+
155+
```yaml
156+
--8<-- "content/kubevirt/livemigration/cclm/ocp1.net-attach-def.yaml"
157+
```
158+
159+
=== "oc apply -f ...."
160+
161+
```bash
162+
oc apply -f {{ page.canonical_url }}cclm/ocp1.net-attach-def.yaml
163+
```
164+
165+
Apply this to OCP7
166+
167+
=== "ocp7.net-attach-def.yaml"
168+
169+
```yaml
170+
--8<-- "content/kubevirt/livemigration/cclm/ocp7.net-attach-def.yaml"
171+
```
172+
173+
=== "oc apply -f ...."
174+
175+
```bash
176+
oc apply -f {{ page.canonical_url }}cclm/ocp7.net-attach-def.yaml
177+
```
178+
179+
### Instantiate the operator
180+
181+
#### OpenShift Virtualization on OCP1 and OCP7
182+
183+
Instantiate with following changes:
184+
185+
```yaml
186+
spec:
187+
liveMigrationConfig:
188+
network: livemigration-network
189+
featureGates:
190+
decentralizedLiveMigration: true
191+
```
192+
193+
Wait until `virt-synchronization-controller-xxx` pods are running:
194+
195+
```shell
196+
oc get pods -n openshift-cnv -l kubevirt.io=virt-synchronization-controller
197+
```
198+
199+
```shell
200+
% oc get pods -n openshift-cnv -l kubevirt.io=virt-synchronization-controller
201+
NAME READY STATUS RESTARTS AGE
202+
virt-synchronization-controller-5b58bd4478-5l25n 1/1 Running 0 3d21h
203+
virt-synchronization-controller-5b58bd4478-zwpn4 1/1 Running 0 3d21h
204+
```
205+
206+
Optional: Check the virt-handler migration network configuration:
207+
208+
```shell
209+
% oc project openshift-cnv
210+
% oc get pods -l kubevirt.io=virt-handler -o name | while read line ; do oc exec -q $line -- /bin/sh -c 'echo -n "$HOSTNAME $NODE_NAME "; ip -4 -br a show dev migration0' ; done
211+
virt-handler-dm5mh ocp1-worker-2 migration0@if9 UP 192.168.201.129/24
212+
virt-handler-h6bn9 ocp1-worker-1 migration0@if9 UP 192.168.201.131/24
213+
virt-handler-nndkq ocp1-worker-0 migration0@if9 UP 192.168.201.130/24
214+
```
215+
216+
```shell
217+
% oc project openshift-cnv
218+
% oc get pods -l kubevirt.io=virt-handler -o name | while read line ; do oc exec -q $line -- /bin/sh -c 'echo -n "$HOSTNAME $NODE_NAME "; ip -4 -br a show dev migration0' ; done
219+
virt-handler-bjwvt ocp7-worker-1 migration0@if9 UP 192.168.201.3/24
220+
virt-handler-gl8gs ocp7-worker-0 migration0@if9 UP 192.168.201.1/24
221+
virt-handler-kxg7k ocp7-worker-2 migration0@if8 UP 192.168.201.4/24
222+
```
223+
224+
#### Migration toolkit for Virtualization on OCP1
225+
226+
Instantiate with following change:
227+
228+
```yaml
229+
spec:
230+
feature_ocp_live_migration: 'true'
231+
```
232+
233+
## Migration toolkit for Virtualization
234+
235+
### Create provide at OCP1
236+
237+
#### Create service account and token at OCP7
238+
239+
Create clusterrole `live-migration-role`
240+
241+
```shell
242+
oc apply -f {{ page.canonical_url }}cclm/clusterrole.yaml
243+
```
244+
245+
```shell
246+
oc create namespace openshift-mtv
247+
oc create serviceaccount cclm -n openshift-mtv
248+
oc create clusterrolebinding cclm --clusterrole=live-migration-role --serviceaccount=openshift-mtv:cclm
249+
oc apply -f - <<EOF
250+
apiVersion: v1
251+
kind: Secret
252+
metadata:
253+
name: cclm
254+
namespace: openshift-mtv
255+
annotations:
256+
kubernetes.io/service-account.name: cclm
257+
type: kubernetes.io/service-account-token
258+
EOF
259+
```
260+
261+
Get the token:
262+
263+
```shell
264+
{% raw %}
265+
oc get secret "cclm" -n "openshift-mtv" -o go-template='{{ .data.token | base64decode}}{{"\n"}}'
266+
{% endraw %}
267+
```
268+
269+
#### Add provider `ocp7` to OCP1
270+
271+
![](cclm/create-mtv-provider.png)
272+
273+
## Run a cross cluster live migration
274+
275+
<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/ctx9qYyQlms" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
276+
277+
## Resources
278+
279+
* <https://access.redhat.com/solutions/7130438>
280+
* <https://docs.redhat.com/en/documentation/openshift_container_platform/4.20/html/virtualization/networking#virt-dedicated-network-live-migration>
281+
* <https://github.com/k8snetworkplumbingwg/whereabouts>
282+
* <https://docs.google.com/document/d/1x4r9gJVXVGe8ef6lMcciOqNbywJ5HKtR7E9kHVzIjAQ/edit?tab=t.0>
283+
* <https://github.com/openshift/runbooks/pull/362>

‎content/kubevirt/livemigration/cclm/clusterrole.yaml‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
apiVersion: rbac.authorization.k8s.io/v1
2+
kind: ClusterRole
3+
metadata:
4+
name: live-migration-role
5+
rules:
6+
- apiGroups:
7+
- forklift.konveyor.io
8+
resources:
9+
- '*'
10+
verbs:
11+
- get
12+
- list
13+
- watch
14+
- apiGroups:
15+
- ""
16+
resources:
17+
- secrets
18+
- namespaces
19+
- configmaps
20+
- persistentvolumes
21+
- persistentvolumeclaims
22+
verbs:
23+
- get
24+
- list
25+
- watch
26+
- create
27+
- update
28+
- patch
29+
- delete
30+
- apiGroups:
31+
- k8s.cni.cncf.io
32+
resources:
33+
- network-attachment-definitions
34+
verbs:
35+
- get
36+
- list
37+
- watch
38+
- apiGroups:
39+
- storage.k8s.io
40+
resources:
41+
- storageclasses
42+
verbs:
43+
- get
44+
- list
45+
- watch
46+
- apiGroups:
47+
- kubevirt.io
48+
resources:
49+
- virtualmachines
50+
- virtualmachines/finalizers
51+
- virtualmachineinstancemigrations
52+
verbs:
53+
- get
54+
- list
55+
- watch
56+
- create
57+
- update
58+
- patch
59+
- delete
60+
- apiGroups:
61+
- kubevirt.io
62+
resources:
63+
- kubevirts
64+
- virtualmachineinstances
65+
verbs:
66+
- get
67+
- list
68+
- watch
69+
- apiGroups:
70+
- cdi.kubevirt.io
71+
resources:
72+
- datavolumes
73+
- datavolumes/finalizers
74+
verbs:
75+
- get
76+
- list
77+
- watch
78+
- create
79+
- update
80+
- patch
81+
- delete
82+
- apiGroups:
83+
- apps
84+
resources:
85+
- deployments
86+
verbs:
87+
- get
88+
- list
89+
- watch
90+
- create
91+
- update
92+
- patch
93+
- delete
94+
- apiGroups:
95+
- instancetype.kubevirt.io
96+
resources:
97+
- virtualmachineclusterpreferences
98+
- virtualmachineclusterinstancetypes
99+
verbs:
100+
- get
101+
- list
102+
- watch
103+
- apiGroups:
104+
- instancetype.kubevirt.io
105+
resources:
106+
- virtualmachinepreferences
107+
- virtualmachineinstancetypes
108+
verbs:
109+
- get
110+
- list
111+
- watch
112+
- create
113+
- update
114+
- patch
115+
- delete

0 commit comments

Comments
 (0)