CORENET-6133:Enhancement for no-overlay mode #1859
Conversation
Signed-off-by: Riccardo Ravaioli <[email protected]>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@ricky-rav: The following test failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
ricky-rav
changed the title
openshift-ci-robot
added
the
jira/valid-reference
|
@ricky-rav: This pull request references CORENET-6133 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
|
||
| E2E tests and QE tests should ensure that the no-overlay mode works as expected in all supported configurations and that we fully support: | ||
| - conformance kubernetes tests on the default network in no-overlay mode | ||
| - ovn-kubernetes E2E tests, excluding features not supported in no-overlay mode (e.g. NetworkPolicy, EgressIP), for the default network and for CUDNs in no-overlay mode |
There was a problem hiding this comment.
networkpolicy should be supported.
EgressService, EIP, multicast, ipsec are not supported AFAIK
| <!-- and https://github.com/openshift/enhancements/blob/master/enhancements/agent-installer/automated-workflow-for-agent-based-installer.md for more detailed examples. --> | ||
|
|
||
|
|
||
| At cluster installation time, the Openshift API will allow a cluster administrator to: |
There was a problem hiding this comment.
one things for D/S is, if no-overlay mode is enabled at installation, means the pod MTU will be 1500 same with physical interface, However RA and frrconfigration are not applied yet in this time because it's day2 configuration. if so, I guess pods traffic for different nodes cannot be working before applying RA and frrconfiguration, this will caused operators cannot be ready and installed failed.
There was a problem hiding this comment.
I think just reading it how it is expressed here makes it look like it is day-2. But it should be day-0 and we need more detail here on how that happens.
That detail is: by providing the manifests upfront to the openshift installer (day-0), the installer will deploy those manifests along with the cluster install (day-1). From that point on, everything should converge.
The bare minimum that will need to be provided is the CNO configuration manifest to enable FRR and the no overlay mode. Additionally for unmanaged configurations, RA and FRR config will need to be provided as well.
The hard requirement is that ovn-k is started with the appropriate parameters.
The soft requirement is that the bootstrap and other operators tolerate the potentially added latency to convergence. If not, we probably need to take additional measures (for example, delaying when we set the node as ready or deploying in a specific order).
I wouldn't be shocked if we have surprises here so everything that we can test early in this regard might help. For example, we should be able to use the same mechanism to deploy a cluster configured in local gateway mode.
| <!-- and https://github.com/openshift/enhancements/blob/master/enhancements/agent-installer/automated-workflow-for-agent-based-installer.md for more detailed examples. --> | ||
|
|
||
|
|
||
| At cluster installation time, the Openshift API will allow a cluster administrator to: |
There was a problem hiding this comment.
I think just reading it how it is expressed here makes it look like it is day-2. But it should be day-0 and we need more detail here on how that happens.
That detail is: by providing the manifests upfront to the openshift installer (day-0), the installer will deploy those manifests along with the cluster install (day-1). From that point on, everything should converge.
The bare minimum that will need to be provided is the CNO configuration manifest to enable FRR and the no overlay mode. Additionally for unmanaged configurations, RA and FRR config will need to be provided as well.
The hard requirement is that ovn-k is started with the appropriate parameters.
The soft requirement is that the bootstrap and other operators tolerate the potentially added latency to convergence. If not, we probably need to take additional measures (for example, delaying when we set the node as ready or deploying in a specific order).
I wouldn't be shocked if we have surprises here so everything that we can test early in this regard might help. For example, we should be able to use the same mechanism to deploy a cluster configured in local gateway mode.
| - "managed": delegate to OCP the configuration of the BGP routers or reflectors necessary to advertise the network pod subnet to all cluster nodes; | ||
| - "unmanaged": use the FRRConfig and RouteAdvertisements provided by the cluster administrator to implement the no-overlay mode | ||
|
|
||
| For CUDNs these parameters will be added to the CUDN CRD and can be configured by the cluster administrator when creating a CUDN. For the default network, these parameters must be input by the cluster administrator at installation time and passed over to ovn-kubernetes by the Cluster Network Operator. |
There was a problem hiding this comment.
It's fine to overview the support for CUDNs here (enough to understand testing requirements) here but I wouldn't focus on it too much on this section (Proposal) of document.
In general, impacts for CNO are:
- configuration of the CDN
- configuration of the internal BGP fabric, which might be employed for CDN and CUDNs.
Or maybe we can have different chapters describing the functionality vs CNO impacts. It's kinda mixed now.
| + // +kubebuilder:validation:Enum=NoOverlay;Geneve | ||
| + // +kubebuilder:default=Geneve | ||
| + // +optional | ||
| + Transport TransportOption `json:"transport,omitempty"` |
There was a problem hiding this comment.
I am missing a CEL rule that checks this field is not updated.
| + // NoOverlayManagedConfig configures the BGP properties for the default network when | ||
| + // NoOverlayOptions.Routing is set to "Managed". | ||
| + // +optional | ||
| + NoOverlayManagedConfig *NoOverlayManagedConfig `json:"NoOverlayManagedConfig,omitempty"` |
There was a problem hiding this comment.
Already mentioned this upstream: we probably need to find a name that does not mention "NoOverlay" as the internal bgp fabric could have more uses cases in the future unrelated to no-overlay.
| // ovnKubernetesConfig contains the configuration parameters for networks | ||
| // using the ovn-kubernetes network project | ||
| +// +kubebuilder:validation:XValidation:rule='self.transport == "NoOverlay" || !has(self.noOverlayOptions)',message="noOverlayOptions is only supported for no-overlay networks" | ||
| +// +kubebuilder:validation:XValidation:rule='!has(self.NoOverlayManagedConfig) || (self.transport == "NoOverlay" && has(self.noOverlayOptions) && self.noOverlayOptions.routing == "Managed")',message="NoOverlayManagedConfig can only be specified when Transport is NoOverlay, NoOverlayOptions is set and NoOverlayOptions.Routing=\"Managed\"" |
There was a problem hiding this comment.
I would probably make the check the other way around: if you have NoOverlay transport and Routing Managed, then NoOverlayManagedConfig is required.
You may want to specify NoOverlayManagedConfig for use with CUDNs and not the CDN.
| There are also two global parameters specific to the way that the no-overlay mode is to be implemented when NoOverlayOptions.routing="Managed", affecting the generated BGP configuration: | ||
| - asNumber: the Autonomous System (AS) number to be used in the generated FRRConfig | ||
| - bgpTopology: | ||
| - "routeReflector": every master node will run a BGP route reflector, in order to reduce the number of BGP connections in the cluster; this is particularly useful for large clusters. |
There was a problem hiding this comment.
I don't think the routeReflector option will be supported. SO other than mentioning it as a future possibility, I wouldn't consider it.
| For the default network, we need a separate CI lane where we can enable the no-overlay mode at installation time. This lane will cover only the unmanaged scenario (with, say, outboundSNAT=enable) and will run upstream conformance tests on the default network: | ||
| - e2e-metal-ipi-ovn-dualstack-no-overlay-unmanaged-techpreview | ||
|
|
||
| For the managed scenario, we need yet two CI lanes, where we can test no-overlay mode with the route reflector topology and with the full mesh topology, for both the default network and CUDNs. In order to keep the number of lanes manageable, we can have one lane run ovn-kubernetes in shared gateway mode and the other in local gateway mode. In both lanes the default network is in no-overlay mode and we can create CUDNs in no-overlay mode on the fly, with outboundSNAT=enable and then outboundSNAT=disable, thus covering both cases in the same lane. | ||
| - e2e-metal-ipi-ovn-dualstack-no-overlay-managed-route-reflector-techpreview | ||
| - e2e-metal-ipi-ovn-dualstack-local-gw-no-overlay-managed-full-mesh-techpreview |
There was a problem hiding this comment.
What if we only test managed mode with CDN? It should be a subset of unmanaged mode, it might be a good compromise.
| <!-- CVO does not currently delete resources that no longer exist in --> | ||
| <!-- the target version. --> | ||
|
|
||
| There are no special concerns about upgrades, since the feature can only be turned on for the default network at installation time and for CUDNs at the time of creation of the affected CUDNs. Configuration changes should not be allowed by CNO during upgrades. |
There was a problem hiding this comment.
It might be interesting to test an upgrade for a cluster already running in no overlay mode. QA coverage for this would be enough I think.
|
|
||
|
|
||
| ### QE Testing | ||
| When testing a managed full mesh topology, we should pay some special attention to resource consumption as we scale up the cluster. The number of links in a full mesh topology grows as N*(N-1)/2, where N is the number of nodes in the cluster. We need to ensure that ovn-kubernetes and FRR can handle the number of BGP connections in a large cluster without excessive resource consumption. |
There was a problem hiding this comment.
I think we need to explicitly mention two separate things:
- testing cluster node scale up functionally
- testing cluster scale performance wise
| The no-overlay feature largely leverages the existing BGP functionality in OVN-Kubernetes and only needs few API changes. | ||
| The feature can be applied to: | ||
| - the default network, at cluster installation time | ||
| - Cluster User Defined Networks (CUDNs) with non-overlapping subnets at CUDN creation time. |
There was a problem hiding this comment.
Overlap concern does not apply for CUDNs in VRF-Lite configuration for which no-overlay is also supported.
| There are also two global parameters specific to the way that the no-overlay mode is to be implemented when NoOverlayOptions.routing="Managed", affecting the generated BGP configuration: | ||
| - asNumber: the Autonomous System (AS) number to be used in the generated FRRConfig | ||
| - bgpTopology: | ||
| - "routeReflector": every master node will run a BGP route reflector, in order to reduce the number of BGP connections in the cluster; this is particularly useful for large clusters. |
There was a problem hiding this comment.
We may not necessarily use master nodes as BGP route reflector. Please find more details in the updated u/s enhancement.
| + // +optional | ||
| + NoOverlayOptions *NoOverlayOptions `json:"noOverlayOptions,omitempty"` | ||
| + | ||
| + // NoOverlayManagedConfig configures the BGP properties for the default network when |
There was a problem hiding this comment.
This field is shared by the CDN and UDNs.
| + | ||
| + // BGPTopology defines the BGP topology to be used. Allowed values | ||
| + // are "routeReflector" and "fullMesh". | ||
| + // - "routeReflector": Every master node will run a BGP route reflector. |
There was a problem hiding this comment.
We shall also mention that other nodes will only peer with the route reflectors.
| + // Defaults to "fullMesh". | ||
| + // +kubebuilder:validation:Enum=routeReflector;fullMesh | ||
| + // +kubebuilder:default=fullMesh | ||
| + // +optional |
There was a problem hiding this comment.
I removed the default from the ovn-k API. Maybe we shall do the same here also. Since we may want to have routeReflector as the default in the future.
| + OutboundSNAT SnatOption `json:"outboundSNAT,omitempty"` | ||
| + // Routing specifies whether the pod network routing is managed by OVN-Kubernetes or users. | ||
| + // +kubebuilder:validation:Enum=Managed;Unmanaged | ||
| + // +kubebuilder:default=Unmanaged |
There was a problem hiding this comment.
Also remove the default value here.
|
|
||
| ### Goals | ||
| https://github.com/ovn-kubernetes/ovn-kubernetes/blob/bf3ec6cc1288971a95d95ca22646249e5f19fb6b/docs/okeps/okep-5259-no-overlay.md#goals | ||
|
|
There was a problem hiding this comment.
For d/s, I think we shall call out that this is only for BM for now.
5 participants
Enhancement for no-overlay mode,
Upstream enhancement: ovn-kubernetes/ovn-kubernetes#5289