Skip to content

Commit de1d812

Browse files
jab-rhjab-rh
authored
Merge pull request #38448 from jboxman-rh/OSDOCS-2682
OSDOCS-2682: Add public cloud support for Egress IPs
2 parents f6f2a3a + c32892d commit de1d812

File tree

4 files changed

+138
-51
lines changed

4 files changed

+138
-51
lines changed

‎modules/nw-egress-ips-about.adoc

Lines changed: 125 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
// Module included in the following assemblies:
22
//
33
// * networking/openshift_sdn/assigning-egress-ips.adoc
4-
// * networking/ovn_kubernetes_network_provider/assigning-egress-ips-ovn.adoc
4+
// * networking/ovn_kubernetes_network_provider/configuring-egress-ips-ovn.adoc
55

66
ifeval::["{context}" == "egress-ips"]
77
:openshift-sdn:
@@ -10,7 +10,6 @@ ifeval::["{context}" == "configuring-egress-ips-ovn"]
1010
:ovn:
1111
endif::[]
1212

13-
ifdef::ovn[]
1413
[id="nw-egress-ips-about_{context}"]
1514
= Egress IP address architectural design and implementation
1615

@@ -19,30 +18,136 @@ The {product-title} egress IP address functionality allows you to ensure that th
1918
For example, you might have a pod that periodically queries a database that is hosted on a server outside of your cluster. To enforce access requirements for the server, a packet filtering device is configured to allow traffic only from specific IP addresses.
2019
To ensure that you can reliably allow access to the server from only that specific pod, you can configure a specific egress IP address for the pod that makes the requests to the server.
2120

21+
An egress IP address assigned to a namespace is different from an egress router, which is used to send traffic to specific destinations.
22+
23+
ifdef::openshift-sdn[]
2224
An egress IP address is implemented as an additional IP address on the primary network interface of a node and must be in the same subnet as the primary IP address of the node. The additional IP address must not be assigned to any other node in the cluster.
25+
endif::openshift-sdn[]
26+
27+
[IMPORTANT]
28+
====
29+
Egress IP addresses must not be configured in any Linux network configuration files, such as `ifcfg-eth0`.
30+
====
2331

2432
[id="nw-egress-ips-platform-support_{context}"]
2533
== Platform support
2634

2735
Support for the egress IP address functionality on various platforms is summarized in the following table:
2836

29-
[IMPORTANT]
30-
====
31-
The egress IP address implementation is not compatible with Amazon Web Services (AWS), Azure Cloud, or any other public cloud platform incompatible with the automatic layer 2 network manipulation required by the egress IP feature.
32-
====
33-
3437
[cols="1,1",options="header"]
3538
|===
3639

3740
| Platform | Supported
3841

3942
| Bare metal | Yes
40-
| vSphere | Yes
43+
| VMWare vSphere | Yes
4144
| {rh-openstack-first} | No
42-
| Public cloud | No
45+
| Amazon Web Services (AWS) | Yes
46+
| Google Cloud Platform (GCP) | Yes
47+
| Microsoft Azure | Yes
4348

4449
|===
4550

51+
[IMPORTANT]
52+
====
53+
The assignment of egress IP addresses to control plane nodes with the EgressIP feature is not supported on a cluster provisioned on Amazon Web Services (AWS). (link:https://bugzilla.redhat.com/show_bug.cgi?id=2039656[*BZ#2039656*])
54+
====
55+
56+
[id="nw-egress-ips-public-cloud-platform-considerations_{context}"]
57+
== Public cloud platform considerations
58+
59+
For clusters provisioned on public cloud infrastructure, there is a constraint on the absolute number of assignable IP addresses per node. The maximum number of assignable IP addresses per node, or the _IP capacity_, can be described in the following formula:
60+
61+
[source,text]
62+
----
63+
IP capacity = public cloud default capacity - sum(current IP assignments)
64+
----
65+
66+
While the Egress IPs capability manages the IP address capacity per node, it is important to plan for this constraint in your deployments. For example, for a cluster installed on bare-metal infrastructure with 8 nodes you can configure 150 egress IP addresses. However, if a public cloud provider limits IP address capacity to 10 IP addresses per node, the total number of assignable IP addresses is only 80. To achieve the same IP address capacity in this example cloud provider, you would need to allocate 7 additional nodes.
67+
68+
To confirm the IP capacity and subnets for any node in your public cloud environment, you can enter the `oc get node <node_name> -o yaml` command. The `cloud.network.openshift.io/egress-ipconfig` annotation includes capacity and subnet information for the node.
69+
70+
The annotation value is an array with a single object with fields that provide the following information for the primary network interface:
71+
72+
* `interface`: Specifies the interface ID on AWS and Azure and the interface name on GCP.
73+
* `ifaddr`: Specifies the subnet mask for one or both IP address families.
74+
* `capacity`: Specifies the IP address capacity for the node. On AWS, the IP address capacity is provided per IP address family. On Azure and GCP, the IP address capacity includes both IPv4 and IPv6 addresses.
75+
76+
The following examples illustrate the annotation from nodes on several public cloud providers. The annotations are indented for readability.
77+
78+
.Example `cloud.network.openshift.io/egress-ipconfig` annotation on AWS
79+
[source,yaml]
80+
----
81+
cloud.network.openshift.io/egress-ipconfig: [
82+
{
83+
"interface":"eni-078d267045138e436",
84+
"ifaddr":{"ipv4":"10.0.128.0/18"},
85+
"capacity":{"ipv4":14,"ipv6":15}
86+
}
87+
]
88+
----
89+
90+
.Example `cloud.network.openshift.io/egress-ipconfig` annotation on GCP
91+
[source,yaml]
92+
----
93+
cloud.network.openshift.io/egress-ipconfig: [
94+
{
95+
"interface":"nic0",
96+
"ifaddr":{"ipv4":"10.0.128.0/18"},
97+
"capacity":{"ip":14}
98+
}
99+
]
100+
----
101+
102+
The following sections describe the IP address capacity for supported public cloud environments for use in your capacity calculation.
103+
104+
[id="nw-egress-ips-capacity-aws_{context}"]
105+
=== Amazon Web Services (AWS) IP address capacity limits
106+
107+
On AWS, constraints on IP address assignments depend on the instance type configured. For more information, see link:https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI[IP addresses per network interface per instance type]
108+
109+
[id="nw-egress-ips-capacity-gcp_{context}"]
110+
=== Google Cloud Platform (GCP) IP address capacity limits
111+
112+
On GCP, the networking model implements additional node IP addresses through IP address aliasing, rather than IP address assignments. However, IP address capacity maps directly to IP aliasing capacity.
113+
114+
The following capacity limits exist for IP aliasing assignment:
115+
116+
- Per node, the maximum number of IP aliases, both IPv4 and IPv6, is 10.
117+
- Per VPC, the maximum number of IP aliases is unspecified, but {product-title} scalability testing reveals the maximum to be approximately 15,000.
118+
119+
For more information, see link:https://cloud.google.com/vpc/docs/quota#per_instance[Per instance] quotas and link:https://cloud.google.com/vpc/docs/alias-ip[Alias IP ranges overview].
120+
121+
[id="nw-egress-ips-capacity-azure_{context}"]
122+
=== Microsoft Azure IP address capacity limits
123+
124+
On Azure, the following capacity limits exist for IP address assignment:
125+
126+
- Per NIC, the maximum number of assignable IP addresses, for both IPv4 and IPv6, is 256.
127+
- Per virtual network, the maximum number of assigned IP addresses cannot exceed 65,536.
128+
129+
For more information, see link:https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits?toc=/azure/virtual-network/toc.json#networking-limits[Networking limits].
130+
131+
ifdef::openshift-sdn[]
132+
[id="nw-egress-ips-limitations_{context}"]
133+
== Limitations
134+
135+
The following limitations apply when using egress IP addresses with the OpenShift SDN cluster network provider:
136+
137+
- You cannot use manually assigned and automatically assigned egress IP addresses on the same nodes.
138+
- If you manually assign egress IP addresses from an IP address range, you must not make that range available for automatic IP assignment.
139+
- You cannot share egress IP addresses across multiple namespaces using the OpenShift SDN egress IP address implementation.
140+
141+
If you need to share IP addresses across namespaces, the OVN-Kubernetes cluster network provider egress IP address implementation allows you to span IP addresses across multiple namespaces.
142+
143+
[NOTE]
144+
====
145+
If you use OpenShift SDN in multitenant mode, you cannot use egress IP addresses with any namespace that is joined to another namespace by the projects that are associated with them.
146+
For example, if `project1` and `project2` are joined by running the `oc adm pod-network join-projects --to=project1 project2` command, neither project can use an egress IP address. For more information, see link:https://bugzilla.redhat.com/show_bug.cgi?id=1645577[BZ#1645577].
147+
====
148+
endif::openshift-sdn[]
149+
150+
ifdef::ovn[]
46151
[id="nw-egress-ips-considerations_{context}"]
47152
== Assignment of egress IPs to pods
48153

@@ -142,23 +247,10 @@ For the configuration in the previous example, {product-title} assigns both egre
142247
endif::ovn[]
143248

144249
ifdef::openshift-sdn[]
145-
[id="nw-egress-ips-about_{context}"]
146-
= Egress IP address assignment for project egress traffic
250+
[id="automatic-manual-assignment-approaches"]
251+
== IP address assignment approaches
147252

148-
By configuring an egress IP address for a project, all outgoing external connections from the specified project will share the same, fixed source IP address.
149-
External resources can recognize traffic from a particular project based on the egress IP address.
150-
An egress IP address assigned to a project is different from the egress router, which is used to send traffic to specific destinations.
151-
152-
Egress IP addresses are implemented as additional IP addresses on the primary network interface of the node and must be in the same subnet as the node's primary IP address.
153-
154-
[IMPORTANT]
155-
====
156-
Egress IP addresses must not be configured in any Linux network configuration files, such as `ifcfg-eth0`.
157-
158-
Allowing additional IP addresses on the primary network interface might require extra configuration when using some cloud or VM solutions.
159-
====
160-
161-
You can assign egress IP addresses to namespaces by setting the `egressIPs` parameter of the `NetNamespace` object. After an egress IP is associated with a project, OpenShift SDN allows you to assign egress IPs to hosts in two ways:
253+
You can assign egress IP addresses to namespaces by setting the `egressIPs` parameter of the `NetNamespace` object. After an egress IP address is associated with a project, OpenShift SDN allows you to assign egress IP addresses to hosts in two ways:
162254

163255
* In the _automatically assigned_ approach, an egress IP address range is assigned to a node.
164256
* In the _manually assigned_ approach, a list of one or more egress IP address is assigned to a node.
@@ -170,26 +262,8 @@ High availability of nodes is automatic.
170262
If a node that hosts an egress IP address is unreachable and there are nodes that are able to host that egress IP address, then the egress IP address will move to a new node.
171263
When the unreachable node comes back online, the egress IP address automatically moves to balance egress IP addresses across nodes.
172264

173-
[IMPORTANT]
174-
====
175-
The following limitations apply when using egress IP addresses with the OpenShift SDN cluster network provider:
176-
177-
- You cannot use manually assigned and automatically assigned egress IP addresses on the same nodes.
178-
- If you manually assign egress IP addresses from an IP address range, you must not make that range available for automatic IP assignment.
179-
- You cannot share egress IP addresses across multiple namespaces using the OpenShift SDN egress IP address implementation.
180-
ifeval::["{product-version}" >= "4.6"]
181-
If you need to share IP addresses across namespaces, the OVN-Kubernetes cluster network provider egress IP address implementation allows you to span IP addresses across multiple namespaces.
182-
endif::[]
183-
====
184-
185-
[NOTE]
186-
====
187-
If you use OpenShift SDN in multitenant mode, you cannot use egress IP addresses with any namespace that is joined to another namespace by the projects that are associated with them.
188-
For example, if `project1` and `project2` are joined by running the `oc adm pod-network join-projects --to=project1 project2` command, neither project can use an egress IP address. For more information, see link:https://bugzilla.redhat.com/show_bug.cgi?id=1645577[BZ#1645577].
189-
====
190-
191265
[id="considerations-automatic-egress-ips"]
192-
== Considerations when using automatically assigned egress IP addresses
266+
=== Considerations when using automatically assigned egress IP addresses
193267

194268
When using the automatic assignment approach for egress IP addresses the following considerations apply:
195269

@@ -203,9 +277,14 @@ If the node hosting the namespace's egress IP address is unreachable, {product-t
203277
The automatic assignment approach works best for clusters installed in environments with flexibility in associating additional IP addresses with nodes.
204278

205279
[id="considerations-manual-egress-ips"]
206-
== Considerations when using manually assigned egress IP addresses
280+
=== Considerations when using manually assigned egress IP addresses
281+
282+
This approach allows you to control which nodes can host an egress IP address.
207283

208-
This approach is recommended for clusters installed in public cloud environments, where there can be limitations on associating additional IP addresses with nodes.
284+
[NOTE]
285+
====
286+
If your cluster is installed on public cloud infrastructure, you must ensure that each node that you assign egress IP addresses to has sufficient spare capacity to host the IP addresses. For more information, see "Platform considerations" in a previous section.
287+
====
209288

210289
When using the manual assignment approach for egress IP addresses the following considerations apply:
211290

‎modules/nw-egress-ips-static.adoc

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -50,9 +50,11 @@ To provide high availability, set the `egressIPs` value to two or more IP addres
5050
Because OpenShift SDN manages the `NetNamespace` object, you can make changes only by modifying the existing `NetNamespace` object. Do not create a new `NetNamespace` object.
5151
====
5252
53-
. Manually assign the egress IP to the node hosts. Set the `egressIPs` parameter
54-
on the `HostSubnet` object on the node host. Using the following JSON, include
55-
as many IP addresses as you want to assign to that node host:
53+
. Manually assign the egress IP address to the node hosts.
54+
+
55+
If your cluster is installed on public cloud infrastructure, you must confirm that the node has available IP address capacity.
56+
+
57+
Set the `egressIPs` parameter on the `HostSubnet` object on the node host. Using the following JSON, include as many IP addresses as you want to assign to that node host:
5658
+
5759
[source,terminal]
5860
----

‎networking/openshift_sdn/assigning-egress-ips.adoc

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,10 +6,16 @@ include::modules/common-attributes.adoc[]
66

77
toc::[]
88

9-
As a cluster administrator, you can configure the OpenShift SDN default Container Network Interface (CNI) network provider to assign one or more egress IP addresses to a project.
9+
[role="_abstract"]
10+
As a cluster administrator, you can configure the OpenShift SDN Container Network Interface (CNI) cluster network provider to assign one or more egress IP addresses to a project.
1011

1112
include::modules/nw-egress-ips-about.adoc[leveloffset=+1]
1213

1314
include::modules/nw-egress-ips-automatic.adoc[leveloffset=+1]
1415

1516
include::modules/nw-egress-ips-static.adoc[leveloffset=+1]
17+
18+
[id="{context}-additional-resources"]
19+
== Additional resources
20+
21+
* If you are configuring manual egress IP address assignment, see xref:../../networking/openshift_sdn/assigning-egress-ips.adoc#nw-egress-ips-public-cloud-platform-considerations_egress-ips[Platform considerations] for information about IP capacity planning.

‎networking/ovn_kubernetes_network_provider/configuring-egress-ips-ovn.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ include::modules/common-attributes.adoc[]
66

77
toc::[]
88

9-
As a cluster administrator, you can configure the OVN-Kubernetes default Container Network Interface (CNI) network provider to assign one or more egress IP addresses to a namespace, or to specific pods in a namespace.
9+
As a cluster administrator, you can configure the OVN-Kubernetes Container Network Interface (CNI) cluster network provider to assign one or more egress IP addresses to a namespace, or to specific pods in a namespace.
1010

1111
include::modules/nw-egress-ips-about.adoc[leveloffset=+1]
1212

0 commit comments

Comments
 (0)