Skip to content

Commit 5da611b

Browse files
gryfmaxwelldb
gryf
and
maxwelldb
authored
OSASINFRA-3066: Added migration procedure for Kuryr SDN. (#54822)
Co-authored-by: Max Bridges <[email protected]>
1 parent da59e62 commit 5da611b

File tree

5 files changed

+727
-0
lines changed

5 files changed

+727
-0
lines changed

‎_topic_maps/_topic_map.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1292,6 +1292,8 @@ Topics:
12921292
File: migrate-from-openshift-sdn
12931293
- Name: Rolling back to the OpenShift SDN network plugin
12941294
File: rollback-to-openshift-sdn
1295+
- Name: Migrating from Kuryr
1296+
File: migrate-from-kuryr-sdn
12951297
- Name: Converting to IPv4/IPv6 dual stack networking
12961298
File: converting-to-dual-stack
12971299
- Name: Logging for egress firewall and network policy rules

‎modules/nw-kuryr-cleanup.adoc

Lines changed: 289 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,289 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * networking/ovn_kubernetes_network_provider/migrate-from-kuryr-sdn.adoc
4+
5+
:_content-type: PROCEDURE
6+
[id="nw-kuryr-cleanup_{context}"]
7+
= Cleaning up resources after migration
8+
9+
After migration from the Kuryr network plugin to the OVN-Kubernetes network
10+
plugin, you must clean up the resources that Kuryr created previously.
11+
12+
[NOTE]
13+
====
14+
The clean up process relies on a Python virtual environment to ensure that the package versions that you use support tags for Octavia objects. You do not need a virtual environment if you are certain that your environment uses at minimum:
15+
* `openstacksdk` version 0.54.0
16+
* `python-openstackclient` version 5.5.0
17+
* `python-octaviaclient` version 2.3.0
18+
====
19+
20+
.Prerequisites
21+
22+
* You installed the {product-title} CLI (`oc`).
23+
* You installed a Python interpreter.
24+
* You installed the `openstacksdk` Python package.
25+
* You installed the `openstack` CLI.
26+
* You have access to the underlying {rh-openstack} cloud.
27+
* You can access the cluster as a user with the `cluster-admin` role.
28+
29+
.Procedure
30+
. Create a clean-up Python virtual environment:
31+
.. Create a temporary directory for your environment. For example:
32+
+
33+
[source,terminal]
34+
----
35+
$ python3 -m venv /tmp/venv
36+
----
37+
+
38+
The virtual environment located in `/tmp/venv` directory is used in all clean up examples.
39+
.. Enter the virtual environment. For example:
40+
+
41+
[source,terminal]
42+
----
43+
$ source /tmp/venv/bin/activate
44+
----
45+
.. Upgrade the `pip` command in the virtual environment by running the following command:
46+
+
47+
[source,terminal]
48+
----
49+
(venv) $ pip install pip --upgrade
50+
----
51+
.. Install the required Python packages by running the following command:
52+
+
53+
[source,terminal]
54+
----
55+
(venv) $ pip install openstacksdk==0.54.0 python-openstackclient==5.5.0 python-octaviaclient==2.3.0
56+
----
57+
58+
. In your terminal, set variables to cluster and Kuryr identifiers by running the following commands:
59+
60+
.. Set the cluster ID:
61+
+
62+
[source,terminal]
63+
----
64+
(venv) $ CLUSTERID=$(oc get infrastructure.config.openshift.io cluster -o=jsonpath='{.status.infrastructureName}')
65+
----
66+
67+
.. Set the cluster tag:
68+
+
69+
[source,terminal]
70+
----
71+
(venv) $ CLUSTERTAG="openshiftClusterID=${CLUSTERID}"
72+
----
73+
.. Set the router ID:
74+
+
75+
[source,terminal]
76+
----
77+
(venv) $ ROUTERID=$(oc get kuryrnetwork -A --no-headers -o custom-columns=":status.routerId"|head -n 1)
78+
----
79+
80+
. Create a Bash function that removes finalizers from specified resources by running the following command:
81+
+
82+
[source,terminal]
83+
----
84+
(venv) $ function REMFIN {
85+
local resource=$1
86+
local finalizer=$2
87+
for res in $(oc get $resource -A --template='{{range $i,$p := .items}}{{ $p.metadata.name }}|{{ $p.metadata.namespace }}{{"\n"}}{{end}}'); do
88+
name=${res%%|*}
89+
ns=${res##*|}
90+
yaml=$(oc get -n $ns $resource $name -o yaml)
91+
if echo "${yaml}" | grep -q "${finalizer}"; then
92+
echo "${yaml}" | grep -v "${finalizer}" | oc replace -n $ns $resource $name -f -
93+
fi
94+
done
95+
}
96+
----
97+
+
98+
The function takes two parameters: the first parameter is name of the resource, and the second parameter is the finalizer to remove.
99+
The named resource is removed from the cluster and its definition is replaced with copied data, excluding the specified finalizer.
100+
101+
. To remove Kuryr finalizers from services, enter the following command:
102+
+
103+
[source,terminal]
104+
----
105+
(venv) $ REMFIN services kuryr.openstack.org/service-finalizer
106+
----
107+
108+
. To remove the Kuryr `service-subnet-gateway-ip` service, enter the following command:
109+
+
110+
[source,terminal]
111+
----
112+
(venv) $ if $(oc get -n openshift-kuryr service service-subnet-gateway-ip &>/dev/null); then
113+
oc -n openshift-kuryr delete service service-subnet-gateway-ip
114+
fi
115+
----
116+
117+
. To remove all tagged {rh-openstack} load balancers from Octavia, enter the following command:
118+
+
119+
[source,terminal]
120+
----
121+
(venv) $ for lb in $(openstack loadbalancer list --tags $CLUSTERTAG -f value -c id); do
122+
openstack loadbalancer delete --cascade $lb
123+
done
124+
----
125+
126+
. To remove Kuryr finalizers from all `KuryrLoadBalancer` CRs, enter the following command:
127+
+
128+
[source,terminal]
129+
----
130+
(venv) $ REMFIN kuryrloadbalancers.openstack.org kuryr.openstack.org/kuryrloadbalancer-finalizers
131+
----
132+
133+
. To remove the `openshift-kuryr` namespace, enter the following command:
134+
+
135+
[source,terminal]
136+
----
137+
(venv) $ oc delete namespace openshift-kuryr
138+
----
139+
140+
. To remove the Kuryr service subnet from the router, enter the following command:
141+
+
142+
[source,terminal]
143+
----
144+
(venv) $ openstack router remove subnet $ROUTERID ${CLUSTERID}-kuryr-service-subnet
145+
----
146+
147+
. To remove the Kuryr service network, enter the following command:
148+
+
149+
[source,terminal]
150+
----
151+
(venv) $ openstack network delete ${CLUSTERID}-kuryr-service-network
152+
----
153+
154+
. To remove Kuryr finalizers from all pods, enter the following command:
155+
+
156+
[source,terminal]
157+
----
158+
(venv) $ REMFIN pods kuryr.openstack.org/pod-finalizer
159+
----
160+
161+
. To remove Kuryr finalizers from all `KuryrPort` CRs, enter the following command:
162+
+
163+
[source,terminal]
164+
----
165+
(venv) $ REMFIN kuryrports.openstack.org kuryr.openstack.org/kuryrport-finalizer
166+
----
167+
This command deletes the `KuryrPort` CRs.
168+
169+
. To remove Kuryr finalizers from network policies, enter the following command:
170+
+
171+
[source,terminal]
172+
----
173+
(venv) $ REMFIN networkpolicy kuryr.openstack.org/networkpolicy-finalizer
174+
----
175+
176+
. To remove Kuryr finalizers from remaining network policies, enter the following command:
177+
+
178+
[source,terminal]
179+
----
180+
(venv) $ REMFIN kuryrnetworkpolicies.openstack.org kuryr.openstack.org/networkpolicy-finalizer
181+
----
182+
183+
. To remove subports that Kuryr created from trunks, enter the following command:
184+
+
185+
[source,terminal]
186+
----
187+
(venv) $ read -ra trunks <<< $(python -c "import openstack; n = openstack.connect().network; print(' '.join([x.id for x in n.trunks(any_tags='$CLUSTERTAG')]))") && \
188+
i=0 && \
189+
for trunk in "${trunks[@]}"; do
190+
i=$((i+1))
191+
echo "Processing trunk $trunk, ${i}/${#trunks[@]}."
192+
subports=()
193+
for subport in $(python -c "import openstack; n = openstack.connect().network; print(' '.join([x['port_id'] for x in n.get_trunk('$trunk').sub_ports if '$CLUSTERTAG' in n.get_port(x['port_id']).tags]))"); do
194+
subports+=("$subport");
195+
done
196+
args=()
197+
for sub in "${subports[@]}" ; do
198+
args+=("--subport $sub")
199+
done
200+
if [ ${#args[@]} -gt 0 ]; then
201+
openstack network trunk unset ${args[*]} $trunk
202+
fi
203+
done
204+
----
205+
206+
. To retrieve all networks and subnets from `KuryrNetwork` CRs and remove ports, router interfaces and the network itself, enter the following command:
207+
+
208+
[source,terminal]
209+
----
210+
(venv) $ mapfile -t kuryrnetworks < <(oc get kuryrnetwork -A --template='{{range $i,$p := .items}}{{ $p.status.netId }}|{{ $p.status.subnetId }}{{"\n"}}{{end}}') && \
211+
i=0 && \
212+
for kn in "${kuryrnetworks[@]}"; do
213+
i=$((i+1))
214+
netID=${kn%%|*}
215+
subnetID=${kn##*|}
216+
echo "Processing network $netID, ${i}/${#kuryrnetworks[@]}"
217+
# Remove all ports from the network.
218+
for port in $(python -c "import openstack; n = openstack.connect().network; print(' '.join([x.id for x in n.ports(network_id='$netID') if x.device_owner != 'network:router_interface']))"); do
219+
( openstack port delete $port ) &
220+
221+
# Only allow 20 jobs in parallel.
222+
if [[ $(jobs -r -p | wc -l) -ge 20 ]]; then
223+
wait -n
224+
fi
225+
done
226+
wait
227+
228+
# Remove the subnet from the router.
229+
openstack router remove subnet $ROUTERID $subnetID
230+
231+
# Remove the network.
232+
openstack network delete $netID
233+
done
234+
----
235+
236+
. To remove the Kuryr security group, enter the following command:
237+
+
238+
[source,terminal]
239+
----
240+
(venv) $ openstack security group delete ${CLUSTERID}-kuryr-pods-security-group
241+
----
242+
243+
. To remove all tagged subnet pools, enter the following command:
244+
+
245+
[source,terminal]
246+
----
247+
(venv) $ for subnetpool in $(openstack subnet pool list --tags $CLUSTERTAG -f value -c ID); do
248+
openstack subnet pool delete $subnetpool
249+
done
250+
----
251+
252+
. To check that all of the networks based on `KuryrNetwork` CRs were removed, enter the following command:
253+
+
254+
[source,terminal]
255+
----
256+
(venv) $ networks=$(oc get kuryrnetwork -A --no-headers -o custom-columns=":status.netId") && \
257+
for existingNet in $(openstack network list --tags $CLUSTERTAG -f value -c ID); do
258+
if [[ $networks =~ $existingNet ]]; then
259+
echo "Network still exists: $existingNet"
260+
fi
261+
done
262+
----
263+
+
264+
If the command returns any existing networks, intestigate and remove them before you continue.
265+
266+
. To remove security groups that are related to network policy, enter the following command:
267+
+
268+
[source,terminal]
269+
----
270+
(venv) $ for sgid in $(openstack security group list -f value -c ID -c Description | grep 'Kuryr-Kubernetes Network Policy' | cut -f 1 -d ' '); do
271+
openstack security group delete $sgid
272+
done
273+
----
274+
275+
. To remove finalizers from `KuryrNetwork` CRs, enter the following command:
276+
+
277+
[source,terminal]
278+
----
279+
(venv) $ REMFIN kuryrnetworks.openstack.org kuryrnetwork.finalizers.kuryr.openstack.org
280+
----
281+
282+
. To remove the Kuryr router, enter the following command:
283+
+
284+
[source,terminal]
285+
----
286+
(venv) $ if $(python3 -c "import sys; import openstack; n = openstack.connect().network; r = n.get_router('$ROUTERID'); sys.exit(0) if r.description != 'Created By OpenShift Installer' else sys.exit(1)"); then
287+
openstack router delete $ROUTERID
288+
fi
289+
----

‎modules/nw-kuryr-migration-about.adoc

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * networking/ovn_kubernetes_network_provider/migrate-from-openshift-sdn.adoc
4+
5+
:_content-type: CONCEPT
6+
[id="nw-kuryr-ovn-kubernetes-migration-about_{context}"]
7+
= Migration to the OVN-Kubernetes network provider
8+
9+
You can manually migrate a cluster that runs on {rh-openstack-first} to the OVN-Kubernetes network provider.
10+
11+
[IMPORTANT]
12+
====
13+
Migration to OVN-Kubernetes is a one-way process.
14+
During migration, your cluster will be unreachable for a brief time.
15+
====
16+
17+
[id="considerations-kuryr-migrating-network-provider_{context}"]
18+
== Considerations when migrating to the OVN-Kubernetes network provider
19+
20+
Kubernetes namespaces are kept by Kuryr in separate {rh-openstack} networking service (Neutron) subnets. Those subnets and the IP addresses that are assigned to individual pods are not preserved during the migration.
21+
22+
[id="how-the-kuryr-migration-process-works_{context}"]
23+
== How the migration process works
24+
25+
The following table summarizes the migration process by relating the steps that you perform with the actions that your cluster and Operators take.
26+
27+
.The Kuryr to OVN-Kubernetes migration process
28+
[cols="1,1a",options="header"]
29+
|===
30+
31+
|User-initiated steps|Migration activity
32+
33+
|
34+
Set the `migration` field of the `Network.operator.openshift.io` custom resource (CR) named `cluster` to `OVNKubernetes`. Verify that the value of the `migration` field prints the `null` value before setting it to another value.
35+
|
36+
Cluster Network Operator (CNO):: Updates the status of the `Network.config.openshift.io` CR named `cluster` accordingly.
37+
Machine Config Operator (MCO):: Deploys an update to the systemd configuration that is required by OVN-Kubernetes. By default, the MCO updates a single machine per pool at a time. As a result, large clusters have longer migration times.
38+
39+
|Update the `networkType` field of the `Network.config.openshift.io` CR.
40+
|
41+
CNO:: Performs the following actions:
42+
+
43+
--
44+
* Destroys the Kuryr control plane pods: Kuryr CNIs and the Kuryr controller.
45+
* Deploys the OVN-Kubernetes control plane pods.
46+
* Updates the Multus objects to reflect the new network plugin.
47+
--
48+
49+
|
50+
Reboot each node in the cluster.
51+
|
52+
Cluster:: As nodes reboot, the cluster assigns IP addresses to pods on the OVN-Kubernetes cluster network.
53+
54+
|
55+
Clean up remaining resources Kuryr controlled.
56+
|
57+
Cluster:: Holds {rh-openstack} resources that need to be freed, as well as {product-title} resources to configure.
58+
|===

0 commit comments

Comments
 (0)