Skip to content

Commit 529cc83

Browse files
lmiccinifultonj
lmiccini
authored and
fultonj
committed
Add bgp-l3-xl dt
This commit adds a "bgp-l3-xl" dt where workers/computes/networkers are deployed across three racks over a virtualized spine/leaf fabric using bgp. The overall footprint consists of 3 masters, 9 workers (3x rack), 6 computes (2x rack), 3 networkers (1x rack) plus 1 router, 2 spines and 6 leaf switches (2x rack). ~~~ $ virsh list Id Name State ---------------------------------------- 1292 cifmw-controller-0 running 1293 cifmw-r0-compute-0 running 1294 cifmw-r0-compute-1 running 1295 cifmw-r1-compute-0 running 1296 cifmw-r1-compute-1 running 1297 cifmw-r2-compute-0 running 1298 cifmw-r2-compute-1 running 1299 cifmw-r0-networker-0 running 1300 cifmw-r1-networker-0 running 1301 cifmw-r2-networker-0 running 1302 cifmw-ocp-master-0 running 1303 cifmw-ocp-master-1 running 1304 cifmw-ocp-master-2 running 1305 cifmw-ocp-worker-0 running 1306 cifmw-ocp-worker-1 running 1307 cifmw-ocp-worker-2 running 1308 cifmw-ocp-worker-3 running 1309 cifmw-ocp-worker-4 running 1310 cifmw-ocp-worker-5 running 1311 cifmw-ocp-worker-6 running 1312 cifmw-ocp-worker-7 running 1313 cifmw-ocp-worker-8 running 1314 cifmw-ocp-worker-9 running 1315 cifmw-router-0 running 1316 cifmw-spine-0 running 1317 cifmw-spine-1 running 1318 cifmw-leaf-0 running 1319 cifmw-leaf-1 running 1320 cifmw-leaf-2 running 1321 cifmw-leaf-3 running 1322 cifmw-leaf-4 running 1323 cifmw-leaf-5 running ~~~
1 parent 408c010 commit 529cc83

38 files changed

+6471
-15
lines changed

‎automation/mocks/bgp-l3-xl.yaml‎

Lines changed: 1440 additions & 0 deletions
Large diffs are not rendered by default.

‎automation/vars/bgp-l3-xl.yaml‎

Lines changed: 197 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,197 @@
1+
---
2+
vas:
3+
bgp-l3-xl:
4+
stages:
5+
- pre_stage_run:
6+
- name: Apply taint on worker-9
7+
type: cr
8+
definition:
9+
spec:
10+
taints:
11+
- effect: NoSchedule
12+
key: testOperator
13+
value: 'true'
14+
- effect: NoExecute
15+
key: testOperator
16+
value: 'true'
17+
kind: Node
18+
resource_name: worker-9
19+
state: patched
20+
- name: Disable rp_filters on OCP nodes
21+
type: cr
22+
definition:
23+
spec:
24+
profile:
25+
- data: |
26+
[main]
27+
summary=Optimize systems running OpenShift (provider specific parent profile)
28+
include=-provider-${f:exec:cat:/var/lib/ocp-tuned/provider},openshift
29+
30+
[sysctl]
31+
net.ipv4.conf.enp7s0.rp_filter=0
32+
net.ipv4.conf.enp8s0.rp_filter=0
33+
name: openshift-no-reapply-sysctl
34+
recommend:
35+
- match:
36+
# applied to all nodes except worker-9, because worker-9 has no enp8s0
37+
- label: kubernetes.io/hostname
38+
value: worker-0
39+
- label: kubernetes.io/hostname
40+
value: worker-1
41+
- label: kubernetes.io/hostname
42+
value: worker-2
43+
- label: kubernetes.io/hostname
44+
value: worker-3
45+
- label: kubernetes.io/hostname
46+
value: worker-4
47+
- label: kubernetes.io/hostname
48+
value: worker-5
49+
- label: kubernetes.io/hostname
50+
value: worker-6
51+
- label: kubernetes.io/hostname
52+
value: worker-7
53+
- label: kubernetes.io/hostname
54+
value: worker-8
55+
- label: node-role.kubernetes.io/master
56+
operand:
57+
tunedConfig:
58+
reapply_sysctl: false
59+
priority: 15
60+
profile: openshift-no-reapply-sysctl
61+
api_version: tuned.openshift.io/v1
62+
kind: Tuned
63+
resource_name: openshift-no-reapply-sysctl
64+
namespace: openshift-cluster-node-tuning-operator
65+
state: present
66+
path: examples/dt/bgp-l3-xl/control-plane/nncp
67+
wait_conditions:
68+
- >-
69+
oc -n openstack wait nncp
70+
-l osp/nncm-config-type=standard
71+
--for jsonpath='{.status.conditions[0].reason}'=SuccessfullyConfigured
72+
--timeout=300s
73+
values:
74+
- name: network-values
75+
src_file: values.yaml
76+
build_output: nncp.yaml
77+
78+
- path: examples/dt/bgp-l3-xl/control-plane
79+
wait_conditions:
80+
- >-
81+
oc -n openstack wait openstackcontrolplane
82+
controlplane
83+
--for condition=Ready
84+
--timeout=30m
85+
values:
86+
- name: network-values
87+
src_file: nncp/values.yaml
88+
- name: service-values
89+
src_file: service-values.yaml
90+
build_output: control-plane.yaml
91+
post_stage_run:
92+
- name: Create BGPConfiguration after controlplane is deployed
93+
type: cr
94+
definition:
95+
spec: {}
96+
api_version: network.openstack.org/v1beta1
97+
kind: BGPConfiguration
98+
resource_name: bgpconfiguration
99+
namespace: openstack
100+
state: present
101+
102+
- # stage_2
103+
path: examples/dt/bgp-l3-xl/edpm/computes/r0
104+
wait_conditions:
105+
- >-
106+
oc -n openstack wait openstackdataplanenodeset
107+
r0-compute-nodes
108+
--for condition=SetupReady
109+
--timeout=600s
110+
values:
111+
- name: edpm-r0-compute-nodeset-values
112+
src_file: values.yaml
113+
build_output: edpm-r0-compute-nodeset.yaml
114+
115+
- # stage_3
116+
path: examples/dt/bgp-l3-xl/edpm/computes/r1
117+
wait_conditions:
118+
- >-
119+
oc -n openstack wait openstackdataplanenodeset
120+
r1-compute-nodes
121+
--for condition=SetupReady
122+
--timeout=600s
123+
values:
124+
- name: edpm-r1-compute-nodeset-values
125+
src_file: values.yaml
126+
build_output: edpm-r1-compute-nodeset.yaml
127+
128+
- # stage_4
129+
path: examples/dt/bgp-l3-xl/edpm/computes/r2
130+
wait_conditions:
131+
- >-
132+
oc -n openstack wait openstackdataplanenodeset
133+
r2-compute-nodes
134+
--for condition=SetupReady
135+
--timeout=600s
136+
values:
137+
- name: edpm-r2-compute-nodeset-values
138+
src_file: values.yaml
139+
build_output: edpm-r2-compute-nodeset.yaml
140+
141+
- # stage_5
142+
path: examples/dt/bgp-l3-xl/edpm/networkers/r0
143+
wait_conditions:
144+
- >-
145+
oc -n openstack wait openstackdataplanenodeset
146+
r0-networker-nodes
147+
--for condition=SetupReady
148+
--timeout=600s
149+
values:
150+
- name: edpm-r0-networker-nodeset-values
151+
src_file: values.yaml
152+
build_output: edpm-r0-networker-nodeset.yaml
153+
154+
- # stage_6
155+
path: examples/dt/bgp-l3-xl/edpm/networkers/r1
156+
wait_conditions:
157+
- >-
158+
oc -n openstack wait openstackdataplanenodeset
159+
r1-networker-nodes
160+
--for condition=SetupReady
161+
--timeout=600s
162+
values:
163+
- name: edpm-r1-networker-nodeset-values
164+
src_file: values.yaml
165+
build_output: edpm-r1-networker-nodeset.yaml
166+
167+
- # stage_7
168+
path: examples/dt/bgp-l3-xl/edpm/networkers/r2
169+
wait_conditions:
170+
- >-
171+
oc -n openstack wait openstackdataplanenodeset
172+
r2-networker-nodes
173+
--for condition=SetupReady
174+
--timeout=600s
175+
values:
176+
- name: edpm-r2-networker-nodeset-values
177+
src_file: values.yaml
178+
build_output: edpm-r2-networker-nodeset.yaml
179+
180+
- # stage_8
181+
path: examples/dt/bgp-l3-xl/edpm/deployment
182+
wait_conditions:
183+
- >-
184+
oc -n openstack wait openstackdataplanedeployment
185+
edpm-deployment
186+
--for condition=Ready
187+
--timeout=120m
188+
values:
189+
- name: edpm-deployment-values
190+
src_file: values.yaml
191+
build_output: edpm-deployment.yaml
192+
post_stage_run:
193+
- name: Wait until computes are ready
194+
type: playbook
195+
source: "../../playbooks/bgp-l3-computes-ready.yml"
196+
extra_vars:
197+
num_computes: 6

‎dt/bgp/kustomization.yaml‎

Lines changed: 0 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -305,18 +305,6 @@ replacements:
305305
options:
306306
create: true
307307

308-
- source:
309-
kind: ConfigMap
310-
name: service-values
311-
fieldPath: data.ovn.ovnController.nicMappings
312-
targets:
313-
- select:
314-
kind: OpenStackControlPlane
315-
fieldPaths:
316-
- spec.ovn.template.ovnController.nicMappings
317-
options:
318-
create: true
319-
320308
- source:
321309
kind: ConfigMap
322310
name: network-values

‎examples/dt/bgp-l3-xl/README.md‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
# RHOSO Deployed Topology - BGP-L3-XL DT - Kernel routing and OVN NB DB driver
2+
3+
This document describes a scaled out BGP Deployed Topology (DT), used to test the
4+
BGP Dynamic Routing functionality on Red Hat OpenStack Services on OpenShift
5+
(RHOSO).
6+
7+
The CRs included within this DT should be applied on an environment where EDPM
8+
and OCP nodes are connected through a spine/leaf network. The BGP protocol
9+
should be enabled on those spine and leaf routers.
10+
11+
## Purpose
12+
13+
This BGP DT (l3-xl) tests default BGP configuration:
14+
* Kernel routing (instead of OVN routing)
15+
* OVN NB DB driver (instead of OVN SB DB driver)
16+
17+
The OCP cluster consists on the following nodes:
18+
* 3 OCP master nodes
19+
* 9 OCP worker nodes
20+
* 1 OCP worker node with special configuration (OCP tester node)
21+
22+
This DT creates an OCP cluster that includes both master and worker nodes,
23+
instead of the usual master/worker combo nodes. The reason for this is to run
24+
disruptive tests only on the OCP workers, which host the Openstack Control
25+
Plane services, avoiding potential issues when OCP master nodes are disrupted
26+
that would not be relevant when testing RHOSO high availability scenarios.
27+
28+
The extra OCP worker (OCP tester) is needed to run tests from it because:
29+
* disruptive tests can be run from this worker on the other workers without
30+
affecting the test exection
31+
* this worker is connected to the spine/leaf routers with a special routing
32+
configuration, so that it can reach the Openstack provider network
33+
The OCP tester is configured so that only test pods (created by the
34+
Openstack test-operator) run on it.
35+
36+
This DT configures both compute and networker EDPM nodes. So far, networker
37+
nodes are needed when BGP is used on RHOSO, in order to expose routes to SNAT
38+
traffic (OVN Gateway IPs). In other words, when RHOSO is used with BGP, the OCP
39+
workers cannot be configured as OVN Gateways.
40+
41+
The OCP and EDPM nodes deployed with this DT are distributed into three
42+
different racks. Each rack is connected to two leaves.
43+
Hence, the distribution of the nodes in the racks is the following one:
44+
* rack0: r0-compute-0, r0-compute-1, r0-networker-0, ocp-master-0, ocp-worker-0, leaf-0, leaf-1
45+
* rack1: r1-compute-0, r1-compute-1, r1-networker-0, ocp-master-1, ocp-worker-1, leaf-2, leaf-3
46+
* rack2: r2-compute-0, r2-compute-1, r2-networker-0, ocp-master-2, ocp-worker-2, leaf-4, leaf-5
47+
48+
The OCP tester (ocp-worker-3) is not included into any rack. It is not
49+
connected to any leaves, but to a router connected to the spines, due to the
50+
reasons described before (it needs special connectivity to reach the provider
51+
network).
52+
53+
## Node topology
54+
| Node role | bm/vm | amount |
55+
| ----------------------- | ----- | ------ |
56+
| Openshift master nodes | vm | 3 |
57+
| Openshift worker nodes | vm | 10 |
58+
| Openstack Computes | vm | 6 |
59+
| Openstack Networker | vm | 3 |
60+
| Leaf routers | vm | 6 |
61+
| Spine routers | vm | 2 |
62+
| External routers | vm | 1 |
63+
| Ansible Controller | vm | 1 |
64+
65+
### Networks
66+
67+
| Name | Type | CIDR |
68+
| ------------------------ | -------- | ---------------- |
69+
| Controlplane rack0 | untagged | 192.168.122.0/24 |
70+
| Controlplane rack1 | untagged | 192.168.123.0/24 |
71+
| Controlplane rack2 | untagged | 192.168.124.0/24 |
72+
| Provider network | untagged | 192.168.133.0/24 |
73+
| RH OSP | untagged | 192.168.111.0/24 |
74+
| edpm/ocp to left leaves | untagged | 100.64.x.y/30 |
75+
| edpm/ocp to right leaves | untagged | 100.65.x.y/30 |
76+
77+
## Services, enabled features and configurations
78+
79+
| Service | configuration | Lock-in coverage? |
80+
| ---------------- | ---------------- | ------------------ |
81+
| Glance | Swift | Must have |
82+
| Swift | (default) | Must have |
83+
| Octavia | (default) | Must have |
84+
| Heat | (default) | Must have |
85+
| frr | dataplane | Must have |
86+
| ovn-bgp-agent | dataplane | Must have |
87+
88+
## Considerations/Constraints
89+
90+
1. Virtual networks should be created to connect the nodes to the routers.
91+
2. All the VMs that are neither Openstack nor Openshift nodes, i.e. those that
92+
act as routers, need to be properly configured in order to support the BGP
93+
protocol.
94+
3. The spine/leaf topology separates the nodes into different L2
95+
network segments, called racks. Each rack includes two leaves, some OCP
96+
nodes and some EDPM nodes.
97+
4. The Openstack services running on the EDPM nodes are installed using the BGP
98+
network, i.e. the Openstack services running on OCP nodes connect to the
99+
Openstack services running on EDPM nodes using BGP. There is no direct L2
100+
network connectivity between them. OCP version 4.18 or higher is required
101+
because the Openstack Operators use the frr-k8s feature for this and frr-k8s
102+
is not available in OCP 4.16.
103+
5. Once Openstack is installed on them, both dataplane and controlplane
104+
connections are achieved using the BGP protocol.
105+
6. Tests are executed from the OCP worker to verify external connectivity.
106+
107+
## Stages
108+
109+
All stages must be executed in the order listed below. Everything is required unless otherwise indicated.
110+
111+
1. [Configure taints on the OCP worker](configure-taints.md)
112+
2. [Disable RP filters on OCP nodes](disable-rp-filters.md)
113+
3. [Install the OpenStack K8S operators and their dependencies](../../common/)
114+
4. [Apply metallb customization required to run a speaker pod on the OCP tester node](metallb/)
115+
5. [Configuring networking and deploy the OpenStack control plane](control-plane.md)
116+
6. [Create BGPConfiguration after controplane is deployed](bgp-configuration.md)
117+
7. [Configure and deploy the dataplane - networker and compute nodes](data-plane.md)

‎examples/dt/bgp-l3-xl/bgp-configuration.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Create BGPConfiguration after controplane is deployed
2+
3+
An empty BGPConfiguration Openshift resource needs to be created.
4+
The infra-operator will detect this resource is created and will automatically
5+
apply the required Openshift BGP configuration.
6+
OCP 4.18 release is necessary for this.
7+
8+
The following CR needs to be applied:
9+
```
10+
apiVersion: network.openstack.org/v1beta1
11+
kind: BGPConfiguration
12+
metadata:
13+
name: bgpconfiguration
14+
namespace: openstack
15+
spec: {}
16+
```

‎examples/dt/bgp-l3-xl/configure-taints.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# Apply taints on OCP tester node
2+
3+
This OCP worker node should not run any Openstack service apart from those
4+
created by the test-operator.
5+
It should also run a metallb's speaker pod, in order to obtain the proper
6+
network configuration.
7+
Due to this, taints should be configured on this worker.
8+
9+
Execute the following command:
10+
```
11+
oc patch node/worker-9 --type merge --patch '
12+
spec:
13+
taints:
14+
- effect: NoSchedule
15+
key: testOperator
16+
value: "true"
17+
- effect: NoExecute
18+
key: testOperator
19+
value: "true"
20+
'
21+
```

0 commit comments

Comments
 (0)