From 36102275030223b4e2fa6c45b91f9ade15103e22 Mon Sep 17 00:00:00 2001 From: Flynn Date: Sun, 29 Jun 2025 18:48:15 -0400 Subject: [PATCH 01/13] Add API to GEP-3793. Signed-off-by: Flynn --- geps/gep-3793/index.md | 307 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 298 insertions(+), 9 deletions(-) diff --git a/geps/gep-3793/index.md b/geps/gep-3793/index.md index 9f15ead87d..8d6bb7b7ff 100644 --- a/geps/gep-3793/index.md +++ b/geps/gep-3793/index.md @@ -73,6 +73,11 @@ she doesn't have to explicitly name, and can simply trust to exist. not in scope for this GEP, in order to have a fighting chance of getting functionality into Gateway API 1.4. + Additionally, note that providing support for Chihiro to swap the default + Gateway without downtime may very well require supporting multiple default + Gateways at the same time, since Kubernetes does not support atomic swaps of + resources. + - Allow Ana to override Chihiro's choice for the default Gateway for a given Route without explicitly specifying the Gateway. @@ -161,10 +166,10 @@ Gateways. ## API -Most of the API work for this GEP is TBD at this point. The challenge is to -find a way to allow Ana to use Routes without requiring her to specify the -Gateway explicitly, while still allowing Chihiro and Ian to retain control -over the Gateway and its configuration. +The main challenge in the API design is to find a way to allow Ana to use +Routes without requiring her to specify the Gateway explicitly, while still +allowing Chihiro and Ian to retain control over the Gateway and its +configuration. An additional concern is CD tools and GitOps workflows. In very broad terms, these tools function by applying manifests from a Git repository to a @@ -189,10 +194,274 @@ will need resolution before this GEP can graduate. [discussion]: https://github.com/kubernetes-sigs/gateway-api/pull/3852#discussion_r2140117567 +Finally, although support for multiple default Gateways is a non-goal for this +GEP, it's worth noting that allowing Chihiro full control over the default +Gateway is very much a goal, which includes giving Chihiro a clean way to swap +one default Gateway for another. This is important because a zero-downtime +swap implies having two default Gateways running at the same time, since +Kubernetes does not support any sort of atomic swap operation. + ### Gateway for Ingress (North/South) +There are two main aspects to the API design for default Gateways: + +1. Giving Ana a way to bind Routes to the default Gateway. + +2. Giving Chihiro a way to control which Gateway is the default, and to + enumerate which Routes are bound to it. + +#### 1. Binding a Route to the Default Gateway + +For Ana to indicate that a Route should use the default Gateway, she MUST +leave `parentRefs` empty in the `spec` of the Route, for example: + +```yaml +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: my-route +spec: + rules: + - backendRefs: + - name: my-service + port: 80 +``` + +would route _all_ HTTP traffic arriving at the default Gateway to `my-service` +on port 80. + +Note that Ana MUST omit `parentRefs` entirely: specifying an empty array for +`parentRefs` MUST fail validation. If a Route with an empty array for +`parentRefs` somehow exists in the cluster, all Gateways in the cluster MUST +refuse to accept it. (Omitting `parentRefs` entirely will work much more +cleanly with GitOps tools than specifying an empty array.) + +Note also that if Ana specifies _any_ `parentRefs`, the default Gateway MUST +NOT claim the Route unless of the `parentRefs` explicitly names the default +Gateway. To do otherwise makes it impossible for Ana to define mesh-only +Routes, or to specify a Route that is meant to use only a specific Gateway +that is not the default. This implies that for Ana to specify a Route intended +to serve both north/south and east/west roles, she MUST explicitly specify the +Gateway in `parentRefs`, even if that Gateway happens to be the default +Gateway. + +All other characteristics of a Route using the default Gateway MUST behave the +same as if the default Gateway were explicitly specified in `parentRefs`. + +The default Gateway MUST use `status.parents` to announce that it has bound +the Route, for example: + +```yaml +status: + parents: + - name: my-default-gateway + namespace: default + controllerName: gateway.networking.k8s.io/some-gateway-controller + conditions: + - type: Accepted + status: "True" + lastTransitionTime: "2025-10-01T12:00:00Z" + message: "Route is bound to default Gateway" +``` + +The default Gateway MUST NOT rewrite the `parentRefs` of a Route using the +default Gateway; it MUST leave `parentRefs` empty. This becomes important if +the default Gateway changes, or (in some situations) if GitOps tools are in +play. + +##### Enumerating Routes Bound to the Default Gateway + +To enumerate Routes bound to the default Gateway, Ana can look for Routes with +no `parentRefs` specified, and then check the `status.parents` of those Routes +to see if the Route has been claimed. This will also tell Ana which Gateway is +the default, even if she doesn't have RBAC to query Gateway resources +directly. + +While this is possible with `kubectl get -o yaml`, it's not exactly a friendly +user experience, so adding this functionality to a tool like `gwctl` would be +a dramatic improvement. In fact, looking at the `status` of a Route is very +much something that we should expect Ana to do often, whether or not default +Gateways are in play; `gwctl` or something similar SHOULD be able to show her +which Routes are bound to which Gateways in every case, not just with default +Gateways. + +**Open Questions:** + +Should the Gateway also add a `condition` explicitly expressing that the Route +has been claimed by the default Gateway, perhaps with `type: DefaultGateway`? +This could help tooling like `gwctl` more easily enumerate Routes bound to the +default Gateway. + +#### 2. Controlling which Gateway is the Default + +Since Chihiro must be able to control which Gateway is the default, selecting +the default Gateway must be an active configuration step taken by Chihiro, +rather than any kind of implicit behavior. To that end, the Gateway resource +will gain a new field, `spec.isDefault`: + +```go +type GatewaySpec struct { + // ... other fields ... + IsDefault *bool `json:"isDefault,omitempty"` +} +``` + +If `spec.isDefault` is set to `true`, the Gateway MUST claim Routes that have +specified no `parentRefs` (subject to the usual Gateway API rules about which +Routes may be bound to a Gateway), and it MUST update its own `status` to with +a `condition` of type `DefaultGateway` and `status` true to indicate that it +is the default Gateway, for example: + +```yaml +status: + conditions: + - type: DefaultGateway + status: "True" + lastTransitionTime: "2025-10-01T12:00:00Z" + message: "Gateway is the default Gateway" +``` + +If `spec.isDefault` is not present or is set to `false`, the Gateway MUST NOT +claim those Routes and MUST NOT set the `DefaultGateway` condition in its +`status`. + +##### Access to the Default Gateway + +The rules for which Routes may bind to a Gateway do not change for the default +Gateway. In particular, if a default Gateway should accept Routes from other +namespaces, then it MUST include the appropriate `AllowedRoutes` definition, +and without such an `AllowedRoutes`, a default Gateway MUST accept only Routes +from its own namespace. + +##### Behavior with No Default Gateway + +If no Gateway has `spec.isDefault` set to `true`, then the behavior is exactly +the same as for Gateway API 1.3: all Routes MUST specify `parentRefs` in order +to function, and no Gateway will claim Routes that do not specify +`parentRefs`. + +##### Deleting the Default Gateway + +Deleting the default Gateway MUST behave the same as deleting any other +Gateway: all Routes that were bound to the default Gateway MUST be unbound, +and the `Accepted` conditions in the `status` of those Routes SHOULD be +removed. + +##### Multiple Default Gateways + +Support for multiple default Gateways in a cluster is not one of the original +goals of this GEP. However, allowing Chihiro to control which Gateway is the +default - including being able to switch which Gateway is the default at +runtime, without requiring downtime - is a goal. + +Kubernetes itself will not prevent setting `spec.isDefault` to `true` on +multiple Gateways in a cluster, and it also doesn't support any atomic swap +mechanisms. If we want to enforce only a single default Gateway, the Gateway +controllers will have to implement that enforcement logic. There are three +possible options here. + +1. Don't bother with any enforcement logic. + + In this case, a Route with no `parentRefs` specified will be bound to _all_ + Gateways that have `spec.isDefault` set to `true`. Since Gateway API + already allows a Route to be bound to multiple Gateways, and the Route + `status` is already designed for it, this should function without + difficulty. + +2. Treat multiple Gateways with `spec.isDefault` set to `true` as if no + Gateway has `spec.isDefault` set to `true`. + + If we assume that all Gateway controllers in a cluster can see all the + Gateways in the cluster, then detecting that multiple Gateways have + `spec.isDefault` set to `true` is relatively straightforward. + + For option 2, every Gateway with `spec.isDefault` set to `true` can simply + refuse to accept Routes with no `parentRefs` specified, behaving as if no + Gateway has been chosen as the default. Each Gateway would also update its + `status` with a `condition` of type `DefaultGateway` and `status` false to + indicate that it is not the default Gateway, for example: + + ```yaml + status: + conditions: + - type: DefaultGateway + status: "False" + lastTransitionTime: "2025-10-01T12:00:00Z" + message: "Multiple Gateways are marked as default" + ``` + +3. Perform conflict resolution as with Routes. + + In this case, the oldest Gateway with `spec.isDefault` set to `true` will + be considered the only default Gateway. That oldest Gateway will accept all + Routes with no `parentRefs` specified, while all other Gateways with + `spec.isDefault` set to `true` will ignore those Routes. + + The oldest default Gateway will update its `status` to reflect that it the + default Gateway; all other Gateways with `spec.isDefault` set to `true` + will update their `status` as in Option 2. + +Unfortunately, option 2 will almost certainly cause downtime in any case where +Chihiro wants to change the default Gateway: + +- If Chihiro deletes the default Gateway before creating the new one, then all + routes using the default Gateway will be unbound during the time that + there's no default Gateway, resulting in errors for any requests using those + Routes. + +- If Chihiro creates the new default Gateway before deleting the old one, then + all Routes using the default Gateway are still unbound during the time that + both Gateways exist. + +Option 3 gives Chihiro a way to change the default Gateway without downtime: +when they create the new default Gateway, it will not take effect until the +old default Gateway is deleted. However, it doesn't give Chihiro any way to +test the Routes through the new default Gateway before deleting the old +Gateway. + +Reluctantly, we must therefore conclude that option 1 is the only viable +choice. Therefore: Gateways MUST NOT attempt to enforce a single default +Gateway, and MUST allow Routes with no `parentRefs` to bind to _all_ Gateways +that have `spec.isDefault` set to `true`. This is simplest to implement, it +permits zero-downtime changes to the default Gateway, and it allows for +testing of the new default Gateway before the old one is deleted. + +##### Changes in Functionality + +If Chihiro changes the default Gateway to a different implementation that does +not support all the functionality of the previous default Gateway, then the +Routes that were bound to the previous default Gateway will no longer function +as expected. This is not a new problem: it already exists when Ana changes a +Route's `parentRefs`, or when Chihiro changes the implementation of a Gateway +that is explicitly specified in a Route's `parentRefs`. + +At present, we do not propose any solution to this problem, other than to note +that `gwctl` or similar tools SHOULD be able to show Ana not just the Gateways +to which a Route is bound, but also the features supported by those Gateways, +to at least help Ana understand if she is trying to use Gateways that don't +support a feature that she needs. This is a definitely an area for future +work, and it is complicated by the fact that Ana may not have access to read +Gateway resources in the cluster at all. + +##### Listeners, ListenerSets, and Merging + +Setting `spec.isDefault` on a Gateway affects which Routes will bind to the +Gateway, not where the Gateway listens for traffic. As such, setting +`spec.isDefault` MUST NOT alter a Gateway's behavior with respect to +Listeners, ListenerSets, or merging. + +In the future, we may want to consider allowing a default ListenerSet rather +than only a default Gateway, but that is not in scope for this GEP. Even if it +is considered later, the guiding principle SHOULD be that `spec.isDefault` +SHOULD NOT affect where a Gateway listens for traffic or whether it can be +merged with other Gateways. + ### Gateway For Mesh (East/West) +Mesh traffic is defined by using a Service as a `parentRef` rather than a +Gateway. As such, there is no case where a default Gateway would be used for +mesh traffic. + ## Conformance Details #### Feature Names @@ -204,14 +473,34 @@ not seem like a good choice. ### Conformance tests +TBD. + ## Alternatives -A possible alternative API design is to modify the behavior of Listeners or -ListenerSets; rather than having a "default Gateway", perhaps we would have -"[default Listeners]". One challenge here is that the Route `status` doesn't -currently expose information about which Listener is being used, though it -does show which Gateway is being used. +- A possible alternative API design is to modify the behavior of Listeners or + ListenerSets; rather than having a "default Gateway", perhaps we would have + "[default Listeners]". One challenge here is that the Route `status` doesn't + currently expose information about which Listener is being used, though it + does show which Gateway is being used. [default Listeners]: https://github.com/kubernetes-sigs/gateway-api/pull/3852#discussion_r2149056246 +- We could define the default Gateway as a Gateway with a magic name, e.g. + "default". This doesn't actually make things that much simpler for Ana + (she'd still have to specify `parentRefs`), and it raises questions about + Chihiro's ability to control which Routes can bind to the default Gateway, + as well as how namespacing would work -- it's especially unhelpful for Ana + if she has to know the namespace of the default Gateway in order to use it. + +- A default Gateway could overwrite a Route's empty `parentRefs` with a + non-empty `parentRefs` pointing to the default Gateway. The main challenge + with this approach is that once the `parentRefs` are overwritten, it's no + longer possible to know that the Route was originally intended to use the + default Gateway. Using the `status` to indicate that the Route is bound to + the default Gateway instead both preserves Ana's original intent and also + makes it possible to change the default Gateway without requiring Ana to + recreate all her Routes. + ## References + +TBD. From 0b43d321e66d0338abfbcda9a7f007e94be3dde4 Mon Sep 17 00:00:00 2001 From: Flynn Date: Sun, 29 Jun 2025 19:04:55 -0400 Subject: [PATCH 02/13] Minor fixes in text, more important fixes in list formatting. Signed-off-by: Flynn --- geps/gep-3793/index.md | 68 +++++++++++++++++++++--------------------- 1 file changed, 34 insertions(+), 34 deletions(-) diff --git a/geps/gep-3793/index.md b/geps/gep-3793/index.md index 8d6bb7b7ff..937d312278 100644 --- a/geps/gep-3793/index.md +++ b/geps/gep-3793/index.md @@ -285,7 +285,7 @@ Gateways are in play; `gwctl` or something similar SHOULD be able to show her which Routes are bound to which Gateways in every case, not just with default Gateways. -**Open Questions:** +##### Open Questions Should the Gateway also add a `condition` explicitly expressing that the Route has been claimed by the default Gateway, perhaps with `type: DefaultGateway`? @@ -308,9 +308,9 @@ type GatewaySpec struct { If `spec.isDefault` is set to `true`, the Gateway MUST claim Routes that have specified no `parentRefs` (subject to the usual Gateway API rules about which -Routes may be bound to a Gateway), and it MUST update its own `status` to with -a `condition` of type `DefaultGateway` and `status` true to indicate that it -is the default Gateway, for example: +Routes may be bound to a Gateway), and it MUST update its own `status` with a +`condition` of type `DefaultGateway` and `status` true to indicate that it is +the default Gateway, for example: ```yaml status: @@ -362,44 +362,44 @@ possible options here. 1. Don't bother with any enforcement logic. - In this case, a Route with no `parentRefs` specified will be bound to _all_ - Gateways that have `spec.isDefault` set to `true`. Since Gateway API - already allows a Route to be bound to multiple Gateways, and the Route - `status` is already designed for it, this should function without - difficulty. + In this case, a Route with no `parentRefs` specified will be bound to _all_ + Gateways that have `spec.isDefault` set to `true`. Since Gateway API + already allows a Route to be bound to multiple Gateways, and the Route + `status` is already designed for it, this should function without + difficulty. 2. Treat multiple Gateways with `spec.isDefault` set to `true` as if no Gateway has `spec.isDefault` set to `true`. - If we assume that all Gateway controllers in a cluster can see all the - Gateways in the cluster, then detecting that multiple Gateways have - `spec.isDefault` set to `true` is relatively straightforward. - - For option 2, every Gateway with `spec.isDefault` set to `true` can simply - refuse to accept Routes with no `parentRefs` specified, behaving as if no - Gateway has been chosen as the default. Each Gateway would also update its - `status` with a `condition` of type `DefaultGateway` and `status` false to - indicate that it is not the default Gateway, for example: - - ```yaml - status: - conditions: - - type: DefaultGateway - status: "False" - lastTransitionTime: "2025-10-01T12:00:00Z" - message: "Multiple Gateways are marked as default" - ``` + If we assume that all Gateway controllers in a cluster can see all the + Gateways in the cluster, then detecting that multiple Gateways have + `spec.isDefault` set to `true` is relatively straightforward. + + For option 2, every Gateway with `spec.isDefault` set to `true` can simply + refuse to accept Routes with no `parentRefs` specified, behaving as if no + Gateway has been chosen as the default. Each Gateway would also update its + `status` with a `condition` of type `DefaultGateway` and `status` false to + indicate that it is not the default Gateway, for example: + + ```yaml + status: + conditions: + - type: DefaultGateway + status: "False" + lastTransitionTime: "2025-10-01T12:00:00Z" + message: "Multiple Gateways are marked as default" + ``` 3. Perform conflict resolution as with Routes. - In this case, the oldest Gateway with `spec.isDefault` set to `true` will - be considered the only default Gateway. That oldest Gateway will accept all - Routes with no `parentRefs` specified, while all other Gateways with - `spec.isDefault` set to `true` will ignore those Routes. + In this case, the oldest Gateway with `spec.isDefault` set to `true` will + be considered the only default Gateway. That oldest Gateway will accept all + Routes with no `parentRefs` specified, while all other Gateways with + `spec.isDefault` set to `true` will ignore those Routes. - The oldest default Gateway will update its `status` to reflect that it the - default Gateway; all other Gateways with `spec.isDefault` set to `true` - will update their `status` as in Option 2. + The oldest default Gateway will update its `status` to reflect that it the + default Gateway; all other Gateways with `spec.isDefault` set to `true` + will update their `status` as in Option 2. Unfortunately, option 2 will almost certainly cause downtime in any case where Chihiro wants to change the default Gateway: From 5eba650dc5802969f177e7d7980cb20f685a2c07 Mon Sep 17 00:00:00 2001 From: Flynn Date: Tue, 22 Jul 2025 13:23:30 -0400 Subject: [PATCH 03/13] Major update for multiple default Gateways, including clarification of goals and non-goals as needed and acknowledgment that a defaulted Route gives up some control. Signed-off-by: Flynn --- geps/gep-3793/index.md | 221 +++++++++++++++++++++-------------------- 1 file changed, 115 insertions(+), 106 deletions(-) diff --git a/geps/gep-3793/index.md b/geps/gep-3793/index.md index 937d312278..a82a2866df 100644 --- a/geps/gep-3793/index.md +++ b/geps/gep-3793/index.md @@ -17,44 +17,60 @@ just wants to create a Route that "works from the outside world" and she really doesn't care what the Gateway is called. Therefore, Ana would like a way to be able to rely on a default Gateway that -she doesn't have to explicitly name, and can simply trust to exist. +she doesn't have to explicitly name, and can simply trust to exist. Ana +recognizes that this will involve **giving up** a certain amount of control +over how requests reach her workloads. She's OK with that, and she understands +that it means that relying on a default Gateway is not always appropriate: for +example, if she needs to be sure that her Route is protected by specific +authorization policies, she should confer with Chihiro to make sure that she +explicitly specifies a Gateway that meets those requirements. -[Ana]: https://gateway-api.sigs.k8s.io/concepts/roles-and-personas/#ana +## Definitions + +- **defaulted Route**: a Route that Ana creates without explicitly specifying + a Gateway + +- **default Gateway**: a Gateway that Chihiro has configured to accept + defaulted Routes ## Goals - Give Ana a way to use Gateway API without having to explicitly specify a - Gateway for every Route, ideally without mutating Routes. + Gateway for every Route, ideally without mutating Routes. (In other words, + give Ana an easy way to create a defaulted Route.) -- Give Ana an easy way to determine which Gateway is the default, and which of - her Routes are bound to it. +- Give Ana an easy way to determine which default Gateways are present in the + cluster, if any, and which of her Routes are currently bound to these + Gateways. -- Continue supporting multiple Gateways in a cluster, while allowing exactly - one of them to be the default Gateway. +- Continue supporting multiple Gateways in a cluster, while allowing zero or + more of them to be configured as default Gateways. -- Allow [Chihiro] to retain control over which Gateway is the default, so that - they can ensure that it meets their requirements for security, performance, - and other operational concerns. +- Allow [Chihiro] to retain control over which Gateways accept defaulted + Routes, so that they can ensure that all Gateways meet their requirements + for security, performance, and other operational concerns. -- Allow Chihiro to choose not to provide a default Gateway. +- Allow Chihiro to choose not to provide any default Gateways at all. -- Allow Chihiro to rename, reconfigure, or replace the default Gateway at +- Allow Chihiro to rename, reconfigure, or replace any default Gateway at runtime. - - If Chihiro renames the default Gateway, Routes using the default Gateway - MUST remain bound to the new default Gateway. Ana shouldn't need to go - recreate all her Routes just because Chihiro is being indecisive. + - While Kubernetes does not allow renaming a resource, Chihiro MUST be able + to duplicate a default Gateway under a new name, then remove the old + default Gateway, without disrupting routing. Ana MUST NOT need to go + update all her Routes just because Chihiro is being indecisive about + naming. - - Determine how (or if) to signal changes in functionality if the default - Gateway implementation is changed. For example, suppose that Chihiro - switches the default Gateway from an implementation that supports the + - Determine how (or if) to signal changes in functionality if a default + Gateway's implementation is changed. For example, suppose that Chihiro + switches a default Gateway from an implementation that supports the `HTTPRoutePhaseOfTheMoon` filter to an implementation that does not. (Note that this problem is not unique to default Gateways; it affects explicitly-named Gateways as well.) -- Allow Chihiro to control which Routes may bind to the default Gateway, and - to enumerate which Routes are currently bound to the default Gateway. +- Allow Chihiro to control which Routes may bind to a default Gateway, and to + enumerate which Routes are currently bound to a default Gateway. - Support easy interoperation with common CI/CD and GitOps workflows. @@ -63,25 +79,11 @@ she doesn't have to explicitly name, and can simply trust to exist. ## Non-Goals -- Support multiple "default" Gateways in a single cluster. If Ana has to make - a choice about which Gateway she wants to use, she'll need to be explicit - about that. - - Loosening this restriction later is a possibility. For example, we may later - want to consider allowing a default Gateway per namespace, or a default - Gateway per implementation running in a cluster. However, these examples are - not in scope for this GEP, in order to have a fighting chance of getting - functionality into Gateway API 1.4. - - Additionally, note that providing support for Chihiro to swap the default - Gateway without downtime may very well require supporting multiple default - Gateways at the same time, since Kubernetes does not support atomic swaps of - resources. +- Allow Ana to override Chihiro's choices for default Gateways for a given + Route without explicitly specifying the Gateway: a Route can either be + defaulted, or it MUST specify a Gateway explicitly. -- Allow Ana to override Chihiro's choice for the default Gateway for a given - Route without explicitly specifying the Gateway. - -- Require that every possible routing use case be met by a Route using the +- Require that every possible routing use case be met by a Route using a default Gateway. There will be a great many situations that require Ana to explicitly choose a Gateway; the existence of a default Gateway is not a guarantee that it will be correct for any given use case. @@ -161,6 +163,7 @@ Ana has easy access to this information, and that it's clear enough for her to understand, is clearly important for many more reasons than just default Gateways. +[Ana]: https://gateway-api.sigs.k8s.io/concepts/roles-and-personas/#ana [Chihiro]: https://gateway-api.sigs.k8s.io/concepts/roles-and-personas/#chihiro [Ian]: https://gateway-api.sigs.k8s.io/concepts/roles-and-personas/#ian @@ -168,8 +171,8 @@ Gateways. The main challenge in the API design is to find a way to allow Ana to use Routes without requiring her to specify the Gateway explicitly, while still -allowing Chihiro and Ian to retain control over the Gateway and its -configuration. +allowing Chihiro and Ian to retain control over Gateways and their +configurations. An additional concern is CD tools and GitOps workflows. In very broad terms, these tools function by applying manifests from a Git repository to a @@ -189,30 +192,29 @@ Gateway controller write a new `parentRefs` stanza to the resource. There has been (much!) [discussion] about whether the ideal API for this feature will mutate the `parentRefs` of a Route using a default Gateway to reflect the Gateway chosen, or whether it should not, relying instead on the -`status` stanza to carry this information. This is obviously a key point that -will need resolution before this GEP can graduate. +`status` stanza to carry this information. Ultimately, mutating the `spec` of +a Kubernetes resource introduces complexity which we should avoid if it's not +required. Since we can gracefully provide default-Gateway functionality +without mutating `parentRefs`, we will rely on `status` instead of mutating +`parentRefs`. [discussion]: https://github.com/kubernetes-sigs/gateway-api/pull/3852#discussion_r2140117567 -Finally, although support for multiple default Gateways is a non-goal for this -GEP, it's worth noting that allowing Chihiro full control over the default -Gateway is very much a goal, which includes giving Chihiro a clean way to swap -one default Gateway for another. This is important because a zero-downtime -swap implies having two default Gateways running at the same time, since -Kubernetes does not support any sort of atomic swap operation. - ### Gateway for Ingress (North/South) There are two main aspects to the API design for default Gateways: -1. Giving Ana a way to bind Routes to the default Gateway. +1. Giving Ana a way to indicate that a Route should be defaulted. + +2. Giving Chihiro a way to control which Gateways (if any) will accept + defaulted Routes. -2. Giving Chihiro a way to control which Gateway is the default, and to - enumerate which Routes are bound to it. +3. Give anyone with read access to Routes (Ana, Chihiro, or Ian) a way to + enumerate which Routes are bound to the default Gateways. -#### 1. Binding a Route to the Default Gateway +#### 1. Binding a Route to a Default Gateway -For Ana to indicate that a Route should use the default Gateway, she MUST +For Ana to indicate that a Route should use a default Gateway, she MUST leave `parentRefs` empty in the `spec` of the Route, for example: ```yaml @@ -227,7 +229,7 @@ spec: port: 80 ``` -would route _all_ HTTP traffic arriving at the default Gateway to `my-service` +would route _all_ HTTP traffic arriving at any default Gateway to `my-service` on port 80. Note that Ana MUST omit `parentRefs` entirely: specifying an empty array for @@ -236,20 +238,19 @@ Note that Ana MUST omit `parentRefs` entirely: specifying an empty array for refuse to accept it. (Omitting `parentRefs` entirely will work much more cleanly with GitOps tools than specifying an empty array.) -Note also that if Ana specifies _any_ `parentRefs`, the default Gateway MUST -NOT claim the Route unless of the `parentRefs` explicitly names the default +Note also that if Ana specifies _any_ `parentRefs`, a default Gateway MUST NOT +claim the Route unless of the `parentRefs` explicitly names that particular Gateway. To do otherwise makes it impossible for Ana to define mesh-only Routes, or to specify a Route that is meant to use only a specific Gateway that is not the default. This implies that for Ana to specify a Route intended to serve both north/south and east/west roles, she MUST explicitly specify the -Gateway in `parentRefs`, even if that Gateway happens to be the default -Gateway. +Gateway in `parentRefs`, even if that Gateway happens to be a default Gateway. -All other characteristics of a Route using the default Gateway MUST behave the -same as if the default Gateway were explicitly specified in `parentRefs`. +All other characteristics of a Route using a default Gateway MUST behave the +same as if the default Gateway(s) were explicitly specified in `parentRefs`. -The default Gateway MUST use `status.parents` to announce that it has bound -the Route, for example: +A default Gateway MUST use `status.parents` to announce that it has bound the +Route, for example: ```yaml status: @@ -264,40 +265,18 @@ status: message: "Route is bound to default Gateway" ``` -The default Gateway MUST NOT rewrite the `parentRefs` of a Route using the +A default Gateway MUST NOT rewrite the `parentRefs` of a Route using the default Gateway; it MUST leave `parentRefs` empty. This becomes important if -the default Gateway changes, or (in some situations) if GitOps tools are in -play. - -##### Enumerating Routes Bound to the Default Gateway - -To enumerate Routes bound to the default Gateway, Ana can look for Routes with -no `parentRefs` specified, and then check the `status.parents` of those Routes -to see if the Route has been claimed. This will also tell Ana which Gateway is -the default, even if she doesn't have RBAC to query Gateway resources -directly. +the set of default Gateways changes, or (in some situations) if GitOps tools +are in play. -While this is possible with `kubectl get -o yaml`, it's not exactly a friendly -user experience, so adding this functionality to a tool like `gwctl` would be -a dramatic improvement. In fact, looking at the `status` of a Route is very -much something that we should expect Ana to do often, whether or not default -Gateways are in play; `gwctl` or something similar SHOULD be able to show her -which Routes are bound to which Gateways in every case, not just with default -Gateways. +#### 2. Controlling which Gateways accept Defaulted Routes -##### Open Questions - -Should the Gateway also add a `condition` explicitly expressing that the Route -has been claimed by the default Gateway, perhaps with `type: DefaultGateway`? -This could help tooling like `gwctl` more easily enumerate Routes bound to the -default Gateway. - -#### 2. Controlling which Gateway is the Default - -Since Chihiro must be able to control which Gateway is the default, selecting -the default Gateway must be an active configuration step taken by Chihiro, -rather than any kind of implicit behavior. To that end, the Gateway resource -will gain a new field, `spec.isDefault`: +Since Chihiro must be able to control which Gateways accept defaulted Routes, +configuring a Gateway to accept defaulted Routes must be an active +configuration step taken by Chihiro, rather than any kind of implicit +behavior. To that end, the Gateway resource will gain a new field, +`spec.isDefault`: ```go type GatewaySpec struct { @@ -325,9 +304,9 @@ If `spec.isDefault` is not present or is set to `false`, the Gateway MUST NOT claim those Routes and MUST NOT set the `DefaultGateway` condition in its `status`. -##### Access to the Default Gateway +##### Access to a Default Gateway -The rules for which Routes may bind to a Gateway do not change for the default +The rules for which Routes may bind to a Gateway do not change for a default Gateway. In particular, if a default Gateway should accept Routes from other namespaces, then it MUST include the appropriate `AllowedRoutes` definition, and without such an `AllowedRoutes`, a default Gateway MUST accept only Routes @@ -349,10 +328,11 @@ removed. ##### Multiple Default Gateways -Support for multiple default Gateways in a cluster is not one of the original -goals of this GEP. However, allowing Chihiro to control which Gateway is the -default - including being able to switch which Gateway is the default at -runtime, without requiring downtime - is a goal. +Support for multiple default Gateways in a cluster was not one of the original +goals of this GEP. However, allowing Chihiro full control over which Gateways +accept defaulted Routes - including being able to change the set of default +Gateways at runtime, without requiring downtime - has always been a goal, and +this turns out to require support for multiple default Gateways. Kubernetes itself will not prevent setting `spec.isDefault` to `true` on multiple Gateways in a cluster, and it also doesn't support any atomic swap @@ -375,11 +355,12 @@ possible options here. Gateways in the cluster, then detecting that multiple Gateways have `spec.isDefault` set to `true` is relatively straightforward. - For option 2, every Gateway with `spec.isDefault` set to `true` can simply - refuse to accept Routes with no `parentRefs` specified, behaving as if no - Gateway has been chosen as the default. Each Gateway would also update its - `status` with a `condition` of type `DefaultGateway` and `status` false to - indicate that it is not the default Gateway, for example: + In this case, every Gateway with `spec.isDefault` set to `true` would + simply refuse to accept Routes with no `parentRefs` specified, behaving as + if no Gateway has been chosen as the default. Each Gateway would also + update its `status` with a `condition` of type `DefaultGateway` and + `status` false to indicate that it is not the default Gateway, for + example: ```yaml status: @@ -428,7 +409,7 @@ testing of the new default Gateway before the old one is deleted. ##### Changes in Functionality -If Chihiro changes the default Gateway to a different implementation that does +If Chihiro changes a default Gateway to a different implementation that does not support all the functionality of the previous default Gateway, then the Routes that were bound to the previous default Gateway will no longer function as expected. This is not a new problem: it already exists when Ana changes a @@ -456,6 +437,31 @@ is considered later, the guiding principle SHOULD be that `spec.isDefault` SHOULD NOT affect where a Gateway listens for traffic or whether it can be merged with other Gateways. +#### 4. Enumerating Routes Bound to Default Gateways + +To enumerate Routes bound to the default Gateways, any of Ana, Chihiro, or Ian +can look for Routes with no `parentRefs` specified, and then check the +`status.parents` of those Routes to see if the Route has been claimed. Since +this will also show _which_ Gateways have claimed a given defaulted Route, it +neatly solves the problem of allowing Ana to determine which default +Gateway(s) her Route is using even if she doesn't have RBAC to query Gateway +resources directly. + +While this is possible with `kubectl get -o yaml`, it's not exactly a friendly +user experience, so adding this functionality to a tool like `gwctl` would be +a dramatic improvement. In fact, looking at the `status` of a Route is very +much something that we should expect any user of Gateway API to do often, +whether or not default Gateways are in play; `gwctl` or something similar +SHOULD be able to show her which Routes are bound to which Gateways in every +case, not just with default Gateways. + +##### Open Questions + +Should the Gateway also add a `condition` explicitly expressing that the Route +has been claimed by the default Gateway, perhaps with `type: DefaultGateway`? +This could help tooling like `gwctl` more easily enumerate Routes bound to the +default Gateway. + ### Gateway For Mesh (East/West) Mesh traffic is defined by using a Service as a `parentRef` rather than a @@ -492,6 +498,9 @@ TBD. as well as how namespacing would work -- it's especially unhelpful for Ana if she has to know the namespace of the default Gateway in order to use it. + (Also, this is a breaking change if Chihiro has already created a + non-default Gateway with whatever name we choose to use for the convention.) + - A default Gateway could overwrite a Route's empty `parentRefs` with a non-empty `parentRefs` pointing to the default Gateway. The main challenge with this approach is that once the `parentRefs` are overwritten, it's no From d672d9e24d4c71921256fb9d8501db1e4a945a37 Mon Sep 17 00:00:00 2001 From: Flynn Date: Tue, 22 Jul 2025 14:41:43 -0400 Subject: [PATCH 04/13] Rearrange Prior Art based on feedback; also split the explicit/implicit bit into its own paragraph. Signed-off-by: Flynn --- geps/gep-3793/index.md | 91 ++++++++++++++++++++++++++++-------------- 1 file changed, 62 insertions(+), 29 deletions(-) diff --git a/geps/gep-3793/index.md b/geps/gep-3793/index.md index a82a2866df..6e271f4380 100644 --- a/geps/gep-3793/index.md +++ b/geps/gep-3793/index.md @@ -119,37 +119,70 @@ Ian, the Gateway is a first-class thing that they think about regularly, while to Ana, it's an implementation detail that she doesn't care about. Neither point of view is wrong, but they are in tension with each other. +In practice, the trick is to find a usable balance between explicitness and +simplicity, while managing ambiguity. A good example is the humble URL, where +the port number is not always explicit, but it _is_ always unambiguous. +Requiring everyone to type `:80` or `:443` at the end of the host portion of +every URL wouldn't actually help anyone, though allowing it to be specified +explicitly when needed definitely does help people. + ### Prior Art -This is very much not a new problem: there are many other systems out there -where being unambiguous is crucial, but where being completely explicit is a -burden. One of the simplest examples is the humble URL, where the port number -is not always explicit, but it _is_ always unambiguous. Requiring everyone to -type `:80` or `:443` at the end of the host portion of every URL wouldn't -actually help anyone, though allowing it to be specified explicitly when -needed definitely does help people. - -The Ingress resource, of course, is another example of prior art: it permitted -specifying a default IngressClass, allowing users to create Ingress resources -that didn't specify the IngressClass explicitly. As with a great many things -in the Ingress API, this caused problems: - -1. Ingress never defined how conflicts between multiple Ingress resources - should be handled. Many (most?) implementations merged conflicting - resources, which is arguably the worst possible choice. - -2. Ingress also never defined a way to allow users to see which IngressClass - was being used by a given Ingress resource, which made it difficult for - users to understand what was going on if they were using the default - IngressClass. - -(Oddly enough, Ingress' general lack of attention to separation of concerns -wasn't really one of the problems here, since IngressClass was a separate -resource.) - -It's rare to find systems that are completely explicit or completely implicit: -in practice, the trick is to find a usable balance between explicitness and -simplicity, while managing ambiguity. +- **Ingress** + + The Ingress resource is the most obvious example of prior art: it permitted + specifying a default IngressClass, allowing users to create Ingress + resources that didn't specify the IngressClass explicitly. As with a great + many things in the Ingress API, this caused problems: + + 1. Ingress never defined how conflicts between multiple Ingress resources + should be handled. Many (most?) implementations merged conflicting + resources, which is arguably the worst possible choice. + + 2. Ingress also never defined a way to allow users to see which IngressClass + was being used by a given Ingress resource, which made it difficult for + users to understand what was going on if they were using the default + IngressClass. + + (Oddly enough, Ingress' general lack of attention to separation of concerns + wasn't really one of the problems here, since IngressClass was a separate + resource.) + +- **Emissary Mapping** + + Emissary-ingress turns this idea on its head: it assumes that app developers + will almost never care about which specific Emissary they're using, and will + instead only care about the hostnames and ports involved. + + In Emissary: + + - a Listener resource defines which ports and protocols are in play; + - a Host resource defines hostnames, TLS certificates, etc.; + - a Mapping resource is roughly analogous to a Route. + + The Listener resource has selectors to control which Hosts it will claim; + Mappings, though, are claimed by Hosts based on the hostname that the + Mapping specifies. In other words, Mappings are not bound to a Listener + explicitly, but rather are bound to a Listener implicitly based on the + hostname that the Mapping specifies. There is no way to _explicitly_ specify + which Listener a Mapping wants to be claimed by. + + This is obviously a very different model from Gateway API, shifting almost + all the work of controlling route binding away from the application + developer onto the cluster operator. + +- **Service** + + We could also consider a Service of `type: LoadBalancer` as a kind of prior + art: in many cases, Ana can directly create these Services and use them to + provide direct, completely unmediated access to a workload, without + worrying about the specifics of how her cluster provider implements them. + + Service's major disadvantages here are that it doesn't support Layer 7 + functionality, and that each Service of type `LoadBalancer` has direct + costs in many cases. In other words, Service allows Ana to rely on the + cluster provider to create the load balancer, while forcing Ana to shoulder + the burden of basically everything else. ### Debugging and Visibility From a5fc0c8313ff8bc9989942e46e2293944183e6a4 Mon Sep 17 00:00:00 2001 From: Flynn Date: Tue, 22 Jul 2025 17:03:29 -0400 Subject: [PATCH 05/13] :man_facepalming: I can count, really. Signed-off-by: Flynn --- geps/gep-3793/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/geps/gep-3793/index.md b/geps/gep-3793/index.md index 6e271f4380..37146101cf 100644 --- a/geps/gep-3793/index.md +++ b/geps/gep-3793/index.md @@ -235,7 +235,7 @@ without mutating `parentRefs`, we will rely on `status` instead of mutating ### Gateway for Ingress (North/South) -There are two main aspects to the API design for default Gateways: +There are three main aspects to the API design for default Gateways: 1. Giving Ana a way to indicate that a Route should be defaulted. From c4091cbf5272a3411c3a170650d8978a5c09ae1c Mon Sep 17 00:00:00 2001 From: Flynn Date: Tue, 22 Jul 2025 17:41:02 -0400 Subject: [PATCH 06/13] Massive update for `defaultOK: true` instead of "no parentRefs". Signed-off-by: Flynn --- geps/gep-3793/index.md | 222 +++++++++++++++++++++++++++-------------- 1 file changed, 145 insertions(+), 77 deletions(-) diff --git a/geps/gep-3793/index.md b/geps/gep-3793/index.md index 37146101cf..ab35186332 100644 --- a/geps/gep-3793/index.md +++ b/geps/gep-3793/index.md @@ -245,10 +245,48 @@ There are three main aspects to the API design for default Gateways: 3. Give anyone with read access to Routes (Ana, Chihiro, or Ian) a way to enumerate which Routes are bound to the default Gateways. -#### 1. Binding a Route to a Default Gateway +We will describe each of these aspects in turn, laying out changes to Gateway +API behaviors and resources that are necessary to support them. **Any behavior +not explicitly discussed in this GEP is intended to remain unchanged;** the +GEP covers **all** intended changes to Gateway API behavior. -For Ana to indicate that a Route should use a default Gateway, she MUST -leave `parentRefs` empty in the `spec` of the Route, for example: +#### 1. Creating a Defaulted Route + +Since Ana must be able to choose whether a Route is defaulted or not, marking +a Route as defaulted must be an active configuration step she takes, rather +than any kind of implicit behavior. To that end, the `CommonRouteSpec` +resource will gain a new field, `defaultOK`: + +```go +type CommonRouteSpec struct { + // ... other fields ... + DefaultOK *bool `json:"defaultOK,omitempty"` +} +``` + +For Ana to indicate that a Route should use a default Gateway, she MUST set +`defaultOK` to `true` in the `spec` of the Route: + +```yaml +... +spec: + defaultOK: true +``` + +A Route MAY include explicit `parentRefs` in addition to setting `defaultOK` +to `true`. In this case, the Route will be a candidate for being bound to +default Gateways, but it will also be bound to its explicitly-specified +Gateways. This allows Ana to create a single Route that handles N/S traffic +via the default Gateways and also handles E/W traffic via a Service, for +example. + +All other characteristics of a defaulted Route MUST behave the same as if all +default Gateways were explicitly specified in `parentRefs`. + +##### Examples + +**Simple N/S Route**: The following HTTPRoute would route _all_ HTTP traffic +arriving at any default Gateway to `my-service` on port 80: ```yaml apiVersion: gateway.networking.k8s.io/v1 @@ -256,34 +294,60 @@ kind: HTTPRoute metadata: name: my-route spec: + defaultOK: true rules: - backendRefs: - name: my-service port: 80 ``` -would route _all_ HTTP traffic arriving at any default Gateway to `my-service` -on port 80. +**N/S and E/W Route**: The following HTTPRoute would be bound to both any +default Gateways and to a Service named `face` in the `faces` namespace, +permitting a single Route to handle both N/S traffic (via the default Gateway) +and E/W traffic (via the Service): + +```yaml +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: ns-ew-route +spec: + defaultOK: true + parentRefs: + - kind: Service + name: face + namespace: faces + rules: + - backendRefs: + - name: face + port: 80 +``` -Note that Ana MUST omit `parentRefs` entirely: specifying an empty array for -`parentRefs` MUST fail validation. If a Route with an empty array for -`parentRefs` somehow exists in the cluster, all Gateways in the cluster MUST -refuse to accept it. (Omitting `parentRefs` entirely will work much more -cleanly with GitOps tools than specifying an empty array.) +**Multiple Gateways**: A defaulted Route MAY include both a `DefaultGateway` +and other specific named Gateways, although this is not expected to be common +in practice: -Note also that if Ana specifies _any_ `parentRefs`, a default Gateway MUST NOT -claim the Route unless of the `parentRefs` explicitly names that particular -Gateway. To do otherwise makes it impossible for Ana to define mesh-only -Routes, or to specify a Route that is meant to use only a specific Gateway -that is not the default. This implies that for Ana to specify a Route intended -to serve both north/south and east/west roles, she MUST explicitly specify the -Gateway in `parentRefs`, even if that Gateway happens to be a default Gateway. +```yaml +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: multi-gateway-route +spec: + defaultOK: true + parentRefs: + - kind: Gateway + name: my-gateway + namespace: default + rules: + - backendRefs: + - name: my-service + port: 80 +``` -All other characteristics of a Route using a default Gateway MUST behave the -same as if the default Gateway(s) were explicitly specified in `parentRefs`. +##### `status` for a Defaulted Route -A default Gateway MUST use `status.parents` to announce that it has bound the -Route, for example: +When a defaulted Route is claimed by a default Gateway, the Gateway MUST use +`status.parents` to announce that it has claimed the Route, for example: ```yaml status: @@ -298,10 +362,17 @@ status: message: "Route is bound to default Gateway" ``` -A default Gateway MUST NOT rewrite the `parentRefs` of a Route using the -default Gateway; it MUST leave `parentRefs` empty. This becomes important if -the set of default Gateways changes, or (in some situations) if GitOps tools -are in play. +##### Other Considerations + +A default Gateway MUST NOT modify the `parentRefs` of a defaulted Route to +indicate that the Route has been claimed by a default Gateway. This becomes +important if the set of default Gateways changes, or (in some situations) if +GitOps tools are in play. + +If there are no default Gateways in the cluster, `defaultOK` MUST be treated +as if it were set to `false` in all Routes, parallel to the situation where a +Route specifies a Gateway by name, but no Gateway of that name exists in the +cluster. #### 2. Controlling which Gateways accept Defaulted Routes @@ -319,10 +390,10 @@ type GatewaySpec struct { ``` If `spec.isDefault` is set to `true`, the Gateway MUST claim Routes that have -specified no `parentRefs` (subject to the usual Gateway API rules about which -Routes may be bound to a Gateway), and it MUST update its own `status` with a -`condition` of type `DefaultGateway` and `status` true to indicate that it is -the default Gateway, for example: +specified `defaultOK` `true` (subject to the usual Gateway API rules about +which Routes may be bound to a Gateway), and it MUST update its own `status` +with a `condition` of type `DefaultGateway` and `status` true to indicate that +it is a default Gateway, for example: ```yaml status: @@ -347,17 +418,16 @@ from its own namespace. ##### Behavior with No Default Gateway -If no Gateway has `spec.isDefault` set to `true`, then the behavior is exactly -the same as for Gateway API 1.3: all Routes MUST specify `parentRefs` in order -to function, and no Gateway will claim Routes that do not specify -`parentRefs`. +If no Gateway has `spec.isDefault` set to `true`, then all Routes MUST specify +`parentRefs` entries with specific Gateways in order to be accepted by +Gateways, and all Gateways MUST treat `defaultOK` as if it were set to `false` +in all Routes. -##### Deleting the Default Gateway +##### Deleting a Default Gateway -Deleting the default Gateway MUST behave the same as deleting any other -Gateway: all Routes that were bound to the default Gateway MUST be unbound, -and the `Accepted` conditions in the `status` of those Routes SHOULD be -removed. +Deleting a default Gateway MUST behave the same as deleting any other Gateway: +all Routes that were bound to that Gateway MUST be unbound, and the `Accepted` +conditions in the `status` of those Routes SHOULD be removed. ##### Multiple Default Gateways @@ -375,7 +445,7 @@ possible options here. 1. Don't bother with any enforcement logic. - In this case, a Route with no `parentRefs` specified will be bound to _all_ + In this case, a Route that sets `defaultOK` will be bound to _all_ Gateways that have `spec.isDefault` set to `true`. Since Gateway API already allows a Route to be bound to multiple Gateways, and the Route `status` is already designed for it, this should function without @@ -389,11 +459,10 @@ possible options here. `spec.isDefault` set to `true` is relatively straightforward. In this case, every Gateway with `spec.isDefault` set to `true` would - simply refuse to accept Routes with no `parentRefs` specified, behaving as - if no Gateway has been chosen as the default. Each Gateway would also - update its `status` with a `condition` of type `DefaultGateway` and - `status` false to indicate that it is not the default Gateway, for - example: + treat `defaultOK` as `false` in every Route, behaving as if no Gateway had + `spec.isDefault` set. Each Gateway would also update its `status` with a + `condition` of type `DefaultGateway` and `status` false to indicate that + it is not the default Gateway, for example: ```yaml status: @@ -407,38 +476,39 @@ possible options here. 3. Perform conflict resolution as with Routes. In this case, the oldest Gateway with `spec.isDefault` set to `true` will - be considered the only default Gateway. That oldest Gateway will accept all - Routes with no `parentRefs` specified, while all other Gateways with - `spec.isDefault` set to `true` will ignore those Routes. + be considered the only default Gateway. That oldest Gateway will be the + only one that honors `defaultOK` in Routes, and all other Gateways with + `spec.isDefault` set to `true` will treat `defaultOK` as `false` in every + Route. The oldest default Gateway will update its `status` to reflect that it the default Gateway; all other Gateways with `spec.isDefault` set to `true` will update their `status` as in Option 2. Unfortunately, option 2 will almost certainly cause downtime in any case where -Chihiro wants to change the default Gateway: +Chihiro wants to change the implementation behind a default Gateway: -- If Chihiro deletes the default Gateway before creating the new one, then all - routes using the default Gateway will be unbound during the time that - there's no default Gateway, resulting in errors for any requests using those +- If Chihiro deletes the old Gateway resource before creating the new one, + then all routes using that Gateway will be unbound during the time between + deletion and recreation, resulting in errors for any requests using those Routes. -- If Chihiro creates the new default Gateway before deleting the old one, then - all Routes using the default Gateway are still unbound during the time that - both Gateways exist. +- If Chihiro creates the new Gateway resource before deleting the old one, + then all Routes using the old default Gateway will still be unbound during + the time that both Gateways exist. Option 3 gives Chihiro a way to change the default Gateway without downtime: -when they create the new default Gateway, it will not take effect until the -old default Gateway is deleted. However, it doesn't give Chihiro any way to -test the Routes through the new default Gateway before deleting the old -Gateway. +when they create the new default Gateway resource, it will not take effect +until the old default Gateway resource is deleted. However, it doesn't give +Chihiro any way to test the Routes through the new default Gateway before +deleting the old Gateway. Reluctantly, we must therefore conclude that option 1 is the only viable choice. Therefore: Gateways MUST NOT attempt to enforce a single default -Gateway, and MUST allow Routes with no `parentRefs` to bind to _all_ Gateways -that have `spec.isDefault` set to `true`. This is simplest to implement, it -permits zero-downtime changes to the default Gateway, and it allows for -testing of the new default Gateway before the old one is deleted. +Gateway, and MUST allow Routes that set `defaultOK` to `true` to bind to _all_ +Gateways that have `spec.isDefault` set to `true`. This is simplest to +implement, it permits zero-downtime changes to the default Gateway, and it +allows for testing of the new default Gateway before the old one is deleted. ##### Changes in Functionality @@ -451,11 +521,11 @@ that is explicitly specified in a Route's `parentRefs`. At present, we do not propose any solution to this problem, other than to note that `gwctl` or similar tools SHOULD be able to show Ana not just the Gateways -to which a Route is bound, but also the features supported by those Gateways, -to at least help Ana understand if she is trying to use Gateways that don't -support a feature that she needs. This is a definitely an area for future -work, and it is complicated by the fact that Ana may not have access to read -Gateway resources in the cluster at all. +to which a Route is bound, but also the features supported by those Gateways. +This will at least give Ana some visibility into whether she's trying to use +Gateways that don't support a feature that she needs. This is a definitely an +area for future work, and it is complicated by the fact that Ana may not have +access to read Gateway resources in the cluster at all. ##### Listeners, ListenerSets, and Merging @@ -464,8 +534,8 @@ Gateway, not where the Gateway listens for traffic. As such, setting `spec.isDefault` MUST NOT alter a Gateway's behavior with respect to Listeners, ListenerSets, or merging. -In the future, we may want to consider allowing a default ListenerSet rather -than only a default Gateway, but that is not in scope for this GEP. Even if it +In the future, we may want to consider allowing default ListenerSets rather +than only default Gateways, but that is not in scope for this GEP. Even if it is considered later, the guiding principle SHOULD be that `spec.isDefault` SHOULD NOT affect where a Gateway listens for traffic or whether it can be merged with other Gateways. @@ -473,7 +543,7 @@ merged with other Gateways. #### 4. Enumerating Routes Bound to Default Gateways To enumerate Routes bound to the default Gateways, any of Ana, Chihiro, or Ian -can look for Routes with no `parentRefs` specified, and then check the +can look for Routes that set `defaultOK` to `true`, and then check the `status.parents` of those Routes to see if the Route has been claimed. Since this will also show _which_ Gateways have claimed a given defaulted Route, it neatly solves the problem of allowing Ana to determine which default @@ -488,19 +558,17 @@ whether or not default Gateways are in play; `gwctl` or something similar SHOULD be able to show her which Routes are bound to which Gateways in every case, not just with default Gateways. -##### Open Questions - -Should the Gateway also add a `condition` explicitly expressing that the Route -has been claimed by the default Gateway, perhaps with `type: DefaultGateway`? -This could help tooling like `gwctl` more easily enumerate Routes bound to the -default Gateway. - ### Gateway For Mesh (East/West) Mesh traffic is defined by using a Service as a `parentRef` rather than a Gateway. As such, there is no case where a default Gateway would be used for mesh traffic. +As noted above, a Route MAY both set `defaultOK` to `true` _and_ include a +`Service` `parentRef`, allowing a single Route to handle both N/S and E/W +traffic. In this case, the Route will be bound to both the default Gateway and +the mesh, and the `status` will show both parents. + ## Conformance Details #### Feature Names From cf229ee94392ec17cf2ecc5bc8b935d05c85d05c Mon Sep 17 00:00:00 2001 From: Flynn Date: Tue, 22 Jul 2025 18:26:20 -0400 Subject: [PATCH 07/13] Use `spec.defaultOK` to parallel `spec.isDefault`. Signed-off-by: Flynn --- geps/gep-3793/index.md | 58 +++++++++++++++++++++--------------------- 1 file changed, 29 insertions(+), 29 deletions(-) diff --git a/geps/gep-3793/index.md b/geps/gep-3793/index.md index ab35186332..dfbdee2b5b 100644 --- a/geps/gep-3793/index.md +++ b/geps/gep-3793/index.md @@ -265,7 +265,7 @@ type CommonRouteSpec struct { ``` For Ana to indicate that a Route should use a default Gateway, she MUST set -`defaultOK` to `true` in the `spec` of the Route: +`spec.defaultOK` to `true` in the `spec` of the Route: ```yaml ... @@ -273,12 +273,12 @@ spec: defaultOK: true ``` -A Route MAY include explicit `parentRefs` in addition to setting `defaultOK` -to `true`. In this case, the Route will be a candidate for being bound to -default Gateways, but it will also be bound to its explicitly-specified -Gateways. This allows Ana to create a single Route that handles N/S traffic -via the default Gateways and also handles E/W traffic via a Service, for -example. +A Route MAY include explicit `parentRefs` in addition to setting +`spec.defaultOK` to `true`. In this case, the Route will be a candidate for +being bound to default Gateways, but it will also be bound to its +explicitly-specified Gateways. This allows Ana to create a single Route that +handles N/S traffic via the default Gateways and also handles E/W traffic via +a Service, for example. All other characteristics of a defaulted Route MUST behave the same as if all default Gateways were explicitly specified in `parentRefs`. @@ -369,7 +369,7 @@ indicate that the Route has been claimed by a default Gateway. This becomes important if the set of default Gateways changes, or (in some situations) if GitOps tools are in play. -If there are no default Gateways in the cluster, `defaultOK` MUST be treated +If there are no default Gateways in the cluster, `spec.defaultOK` MUST be treated as if it were set to `false` in all Routes, parallel to the situation where a Route specifies a Gateway by name, but no Gateway of that name exists in the cluster. @@ -390,10 +390,10 @@ type GatewaySpec struct { ``` If `spec.isDefault` is set to `true`, the Gateway MUST claim Routes that have -specified `defaultOK` `true` (subject to the usual Gateway API rules about -which Routes may be bound to a Gateway), and it MUST update its own `status` -with a `condition` of type `DefaultGateway` and `status` true to indicate that -it is a default Gateway, for example: +specified `spec.defaultOK` `true` (subject to the usual Gateway API rules +about which Routes may be bound to a Gateway), and it MUST update its own +`status` with a `condition` of type `DefaultGateway` and `status` true to +indicate that it is a default Gateway, for example: ```yaml status: @@ -420,8 +420,8 @@ from its own namespace. If no Gateway has `spec.isDefault` set to `true`, then all Routes MUST specify `parentRefs` entries with specific Gateways in order to be accepted by -Gateways, and all Gateways MUST treat `defaultOK` as if it were set to `false` -in all Routes. +Gateways, and all Gateways MUST treat `spec.defaultOK` as if it were set to +`false` in all Routes. ##### Deleting a Default Gateway @@ -445,7 +445,7 @@ possible options here. 1. Don't bother with any enforcement logic. - In this case, a Route that sets `defaultOK` will be bound to _all_ + In this case, a Route that sets `spec.defaultOK` will be bound to _all_ Gateways that have `spec.isDefault` set to `true`. Since Gateway API already allows a Route to be bound to multiple Gateways, and the Route `status` is already designed for it, this should function without @@ -459,10 +459,10 @@ possible options here. `spec.isDefault` set to `true` is relatively straightforward. In this case, every Gateway with `spec.isDefault` set to `true` would - treat `defaultOK` as `false` in every Route, behaving as if no Gateway had - `spec.isDefault` set. Each Gateway would also update its `status` with a - `condition` of type `DefaultGateway` and `status` false to indicate that - it is not the default Gateway, for example: + treat `spec.defaultOK` as `false` in every Route, behaving as if no + Gateway had `spec.isDefault` set. Each Gateway would also update its + `status` with a `condition` of type `DefaultGateway` and `status` false to + indicate that it is not the default Gateway, for example: ```yaml status: @@ -477,9 +477,9 @@ possible options here. In this case, the oldest Gateway with `spec.isDefault` set to `true` will be considered the only default Gateway. That oldest Gateway will be the - only one that honors `defaultOK` in Routes, and all other Gateways with - `spec.isDefault` set to `true` will treat `defaultOK` as `false` in every - Route. + only one that honors `spec.defaultOK` in Routes, and all other Gateways + with `spec.isDefault` set to `true` will treat `spec.defaultOK` as `false` + in every Route. The oldest default Gateway will update its `status` to reflect that it the default Gateway; all other Gateways with `spec.isDefault` set to `true` @@ -505,8 +505,8 @@ deleting the old Gateway. Reluctantly, we must therefore conclude that option 1 is the only viable choice. Therefore: Gateways MUST NOT attempt to enforce a single default -Gateway, and MUST allow Routes that set `defaultOK` to `true` to bind to _all_ -Gateways that have `spec.isDefault` set to `true`. This is simplest to +Gateway, and MUST allow Routes that set `spec.defaultOK` to `true` to bind to +_all_ Gateways that have `spec.isDefault` set to `true`. This is simplest to implement, it permits zero-downtime changes to the default Gateway, and it allows for testing of the new default Gateway before the old one is deleted. @@ -543,7 +543,7 @@ merged with other Gateways. #### 4. Enumerating Routes Bound to Default Gateways To enumerate Routes bound to the default Gateways, any of Ana, Chihiro, or Ian -can look for Routes that set `defaultOK` to `true`, and then check the +can look for Routes that set `spec.defaultOK` to `true`, and then check the `status.parents` of those Routes to see if the Route has been claimed. Since this will also show _which_ Gateways have claimed a given defaulted Route, it neatly solves the problem of allowing Ana to determine which default @@ -564,10 +564,10 @@ Mesh traffic is defined by using a Service as a `parentRef` rather than a Gateway. As such, there is no case where a default Gateway would be used for mesh traffic. -As noted above, a Route MAY both set `defaultOK` to `true` _and_ include a -`Service` `parentRef`, allowing a single Route to handle both N/S and E/W -traffic. In this case, the Route will be bound to both the default Gateway and -the mesh, and the `status` will show both parents. +As noted above, a Route MAY both set `spec.defaultOK` to `true` _and_ include +a `Service` `parentRef` entry, allowing a single Route to handle both N/S and +E/W traffic. In this case, the Route will be bound to both the default Gateway +and the mesh, and the `status` will show both parents. ## Conformance Details From 1fd733fd87a408356b4e4c097dbe77a3945351a2 Mon Sep 17 00:00:00 2001 From: Flynn Date: Tue, 22 Jul 2025 18:26:38 -0400 Subject: [PATCH 08/13] Minor tweaks for feedback. Signed-off-by: Flynn --- geps/gep-3793/index.md | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/geps/gep-3793/index.md b/geps/gep-3793/index.md index dfbdee2b5b..46f2506431 100644 --- a/geps/gep-3793/index.md +++ b/geps/gep-3793/index.md @@ -507,8 +507,11 @@ Reluctantly, we must therefore conclude that option 1 is the only viable choice. Therefore: Gateways MUST NOT attempt to enforce a single default Gateway, and MUST allow Routes that set `spec.defaultOK` to `true` to bind to _all_ Gateways that have `spec.isDefault` set to `true`. This is simplest to -implement, it permits zero-downtime changes to the default Gateway, and it -allows for testing of the new default Gateway before the old one is deleted. +implement, it permits zero-downtime changes to the default Gateway, it allows +for testing of the new default Gateway before the old one is deleted, and it +doesn't cause trouble with respect to security posture (since Ana already +accepts that she's giving up some control over how her Routes are handled when +she's using default Gateways). ##### Changes in Functionality @@ -602,12 +605,11 @@ TBD. (Also, this is a breaking change if Chihiro has already created a non-default Gateway with whatever name we choose to use for the convention.) -- A default Gateway could overwrite a Route's empty `parentRefs` with a - non-empty `parentRefs` pointing to the default Gateway. The main challenge - with this approach is that once the `parentRefs` are overwritten, it's no - longer possible to know that the Route was originally intended to use the - default Gateway. Using the `status` to indicate that the Route is bound to - the default Gateway instead both preserves Ana's original intent and also +- A default Gateway could overwrite a defaulted Route's `parentRefs` to point + to the default Gateway. The main challenge with this approach is that once + the `parentRefs` are overwritten, it's no longer possible to know what Ana + originally intended. Using the `status` to indicate that the Route is bound + to the default Gateway instead both preserves Ana's original intent and also makes it possible to change the default Gateway without requiring Ana to recreate all her Routes. From 4f8fec12d0ecb9a80d1ee94f94dabe5551856b21 Mon Sep 17 00:00:00 2001 From: Flynn Date: Tue, 22 Jul 2025 18:32:01 -0400 Subject: [PATCH 09/13] Mark as implementable Signed-off-by: Flynn --- geps/gep-3793/index.md | 2 +- geps/gep-3793/metadata.yaml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/geps/gep-3793/index.md b/geps/gep-3793/index.md index 46f2506431..d9787c2579 100644 --- a/geps/gep-3793/index.md +++ b/geps/gep-3793/index.md @@ -1,7 +1,7 @@ # GEP-3793: Default Gateways * Issue: [#3793](https://github.com/kubernetes-sigs/gateway-api/issues/3793) -* Status: Provisional +* Status: Implementable (See [status definitions](../overview.md#gep-states).) diff --git a/geps/gep-3793/metadata.yaml b/geps/gep-3793/metadata.yaml index 060426f033..db611f5988 100644 --- a/geps/gep-3793/metadata.yaml +++ b/geps/gep-3793/metadata.yaml @@ -2,7 +2,7 @@ apiVersion: internal.gateway.networking.k8s.io/v1alpha1 kind: GEPDetails number: 3793 name: Default Gateways -status: Provisional +status: Implementable # Any authors who contribute to the GEP in any way should be listed here using # their GitHub handle. authors: From c82be4ff9aa0c8867f934b753727b2159b4ff062 Mon Sep 17 00:00:00 2001 From: Flynn Date: Wed, 23 Jul 2025 12:25:54 -0400 Subject: [PATCH 10/13] Change `defaultOK` to `useDefaultGateway`. Signed-off-by: Flynn --- geps/gep-3793/index.md | 90 +++++++++++++++++++++--------------------- 1 file changed, 45 insertions(+), 45 deletions(-) diff --git a/geps/gep-3793/index.md b/geps/gep-3793/index.md index d9787c2579..21f60d3317 100644 --- a/geps/gep-3793/index.md +++ b/geps/gep-3793/index.md @@ -255,30 +255,30 @@ GEP covers **all** intended changes to Gateway API behavior. Since Ana must be able to choose whether a Route is defaulted or not, marking a Route as defaulted must be an active configuration step she takes, rather than any kind of implicit behavior. To that end, the `CommonRouteSpec` -resource will gain a new field, `defaultOK`: +resource will gain a new field, `useDefaultGateway`: ```go type CommonRouteSpec struct { // ... other fields ... - DefaultOK *bool `json:"defaultOK,omitempty"` + useDefaultGateway *bool `json:"useDefaultGateway,omitempty"` } ``` For Ana to indicate that a Route should use a default Gateway, she MUST set -`spec.defaultOK` to `true` in the `spec` of the Route: +`spec.useDefaultGateway` to `true` in the `spec` of the Route: ```yaml ... spec: - defaultOK: true + useDefaultGateway: true ``` A Route MAY include explicit `parentRefs` in addition to setting -`spec.defaultOK` to `true`. In this case, the Route will be a candidate for -being bound to default Gateways, but it will also be bound to its -explicitly-specified Gateways. This allows Ana to create a single Route that -handles N/S traffic via the default Gateways and also handles E/W traffic via -a Service, for example. +`spec.useDefaultGateway` to `true`. In this case, the Route will be a +candidate for being bound to default Gateways, but it will also be bound to +its explicitly-specified `parentRefs`. This allows Ana to create a single +Route that handles N/S traffic via the default Gateways and also handles E/W +traffic via a Service, for example. All other characteristics of a defaulted Route MUST behave the same as if all default Gateways were explicitly specified in `parentRefs`. @@ -294,7 +294,7 @@ kind: HTTPRoute metadata: name: my-route spec: - defaultOK: true + useDefaultGateway: true rules: - backendRefs: - name: my-service @@ -312,7 +312,7 @@ kind: HTTPRoute metadata: name: ns-ew-route spec: - defaultOK: true + useDefaultGateway: true parentRefs: - kind: Service name: face @@ -333,7 +333,7 @@ kind: HTTPRoute metadata: name: multi-gateway-route spec: - defaultOK: true + useDefaultGateway: true parentRefs: - kind: Gateway name: my-gateway @@ -369,10 +369,10 @@ indicate that the Route has been claimed by a default Gateway. This becomes important if the set of default Gateways changes, or (in some situations) if GitOps tools are in play. -If there are no default Gateways in the cluster, `spec.defaultOK` MUST be treated -as if it were set to `false` in all Routes, parallel to the situation where a -Route specifies a Gateway by name, but no Gateway of that name exists in the -cluster. +If there are no default Gateways in the cluster, `spec.useDefaultGateway` MUST +be treated as if it were set to `false` in all Routes, parallel to the +situation where a Route specifies a Gateway by name, but no Gateway of that +name exists in the cluster. #### 2. Controlling which Gateways accept Defaulted Routes @@ -390,9 +390,9 @@ type GatewaySpec struct { ``` If `spec.isDefault` is set to `true`, the Gateway MUST claim Routes that have -specified `spec.defaultOK` `true` (subject to the usual Gateway API rules -about which Routes may be bound to a Gateway), and it MUST update its own -`status` with a `condition` of type `DefaultGateway` and `status` true to +specified `spec.useDefaultGateway` `true` (subject to the usual Gateway API +rules about which Routes may be bound to a Gateway), and it MUST update its +own `status` with a `condition` of type `DefaultGateway` and `status` true to indicate that it is a default Gateway, for example: ```yaml @@ -420,8 +420,8 @@ from its own namespace. If no Gateway has `spec.isDefault` set to `true`, then all Routes MUST specify `parentRefs` entries with specific Gateways in order to be accepted by -Gateways, and all Gateways MUST treat `spec.defaultOK` as if it were set to -`false` in all Routes. +Gateways, and all Gateways MUST treat `spec.useDefaultGateway` as if it were +set to `false` in all Routes. ##### Deleting a Default Gateway @@ -445,8 +445,8 @@ possible options here. 1. Don't bother with any enforcement logic. - In this case, a Route that sets `spec.defaultOK` will be bound to _all_ - Gateways that have `spec.isDefault` set to `true`. Since Gateway API + In this case, a Route that sets `spec.useDefaultGateway` will be bound to + _all_ Gateways that have `spec.isDefault` set to `true`. Since Gateway API already allows a Route to be bound to multiple Gateways, and the Route `status` is already designed for it, this should function without difficulty. @@ -459,8 +459,8 @@ possible options here. `spec.isDefault` set to `true` is relatively straightforward. In this case, every Gateway with `spec.isDefault` set to `true` would - treat `spec.defaultOK` as `false` in every Route, behaving as if no - Gateway had `spec.isDefault` set. Each Gateway would also update its + treat `spec.useDefaultGateway` as `false` in every Route, behaving as if + no Gateway had `spec.isDefault` set. Each Gateway would also update its `status` with a `condition` of type `DefaultGateway` and `status` false to indicate that it is not the default Gateway, for example: @@ -477,9 +477,9 @@ possible options here. In this case, the oldest Gateway with `spec.isDefault` set to `true` will be considered the only default Gateway. That oldest Gateway will be the - only one that honors `spec.defaultOK` in Routes, and all other Gateways - with `spec.isDefault` set to `true` will treat `spec.defaultOK` as `false` - in every Route. + only one that honors `spec.useDefaultGateway` in Routes, and all other + Gateways with `spec.isDefault` set to `true` will treat + `spec.useDefaultGateway` as `false` in every Route. The oldest default Gateway will update its `status` to reflect that it the default Gateway; all other Gateways with `spec.isDefault` set to `true` @@ -505,13 +505,13 @@ deleting the old Gateway. Reluctantly, we must therefore conclude that option 1 is the only viable choice. Therefore: Gateways MUST NOT attempt to enforce a single default -Gateway, and MUST allow Routes that set `spec.defaultOK` to `true` to bind to -_all_ Gateways that have `spec.isDefault` set to `true`. This is simplest to -implement, it permits zero-downtime changes to the default Gateway, it allows -for testing of the new default Gateway before the old one is deleted, and it -doesn't cause trouble with respect to security posture (since Ana already -accepts that she's giving up some control over how her Routes are handled when -she's using default Gateways). +Gateway, and MUST allow Routes that set `spec.useDefaultGateway` to `true` to +bind to _all_ Gateways that have `spec.isDefault` set to `true`. This is +simplest to implement, it permits zero-downtime changes to the default +Gateway, it allows for testing of the new default Gateway before the old one +is deleted, and it doesn't cause trouble with respect to security posture +(since Ana already accepts that she's giving up some control over how her +Routes are handled when she's using default Gateways). ##### Changes in Functionality @@ -546,12 +546,12 @@ merged with other Gateways. #### 4. Enumerating Routes Bound to Default Gateways To enumerate Routes bound to the default Gateways, any of Ana, Chihiro, or Ian -can look for Routes that set `spec.defaultOK` to `true`, and then check the -`status.parents` of those Routes to see if the Route has been claimed. Since -this will also show _which_ Gateways have claimed a given defaulted Route, it -neatly solves the problem of allowing Ana to determine which default -Gateway(s) her Route is using even if she doesn't have RBAC to query Gateway -resources directly. +can look for Routes that set `spec.useDefaultGateway` to `true`, and then +check the `status.parents` of those Routes to see if the Route has been +claimed. Since this will also show _which_ Gateways have claimed a given +defaulted Route, it neatly solves the problem of allowing Ana to determine +which default Gateway(s) her Route is using even if she doesn't have RBAC to +query Gateway resources directly. While this is possible with `kubectl get -o yaml`, it's not exactly a friendly user experience, so adding this functionality to a tool like `gwctl` would be @@ -567,10 +567,10 @@ Mesh traffic is defined by using a Service as a `parentRef` rather than a Gateway. As such, there is no case where a default Gateway would be used for mesh traffic. -As noted above, a Route MAY both set `spec.defaultOK` to `true` _and_ include -a `Service` `parentRef` entry, allowing a single Route to handle both N/S and -E/W traffic. In this case, the Route will be bound to both the default Gateway -and the mesh, and the `status` will show both parents. +As noted above, a Route MAY both set `spec.useDefaultGateway` to `true` _and_ +include a `Service` `parentRef` entry, allowing a single Route to handle both +N/S and E/W traffic. In this case, the Route will be bound to both the default +Gateway and the mesh, and the `status` will show both parents. ## Conformance Details From 61afff7c10e772f3b7187714c459995753a2b5e4 Mon Sep 17 00:00:00 2001 From: Flynn Date: Fri, 25 Jul 2025 16:36:57 -0400 Subject: [PATCH 11/13] Move 3793 to Implementable in mkdocs.yml Signed-off-by: Flynn --- mkdocs.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs.yml b/mkdocs.yml index 7765e320d3..d29e212746 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -132,10 +132,10 @@ nav: - geps/gep-2648/index.md - geps/gep-3779/index.md - geps/gep-3792/index.md - - geps/gep-3793/index.md - Implementable: - geps/gep-91/index.md - geps/gep-3567/index.md + - geps/gep-3793/index.md - Experimental: - geps/gep-1619/index.md - geps/gep-1713/index.md From 7ce62f90f45f882f975a2780f969d02af4bbf5e9 Mon Sep 17 00:00:00 2001 From: Flynn Date: Fri, 25 Jul 2025 16:38:31 -0400 Subject: [PATCH 12/13] Define default "scopes", with the only currently-defined scope being All. Update isDefault to defaultScope; switch both it and useDefaultGateway to scopes instead of bools. Signed-off-by: Flynn --- geps/gep-3793/index.md | 157 ++++++++++++++++++++++++----------------- 1 file changed, 94 insertions(+), 63 deletions(-) diff --git a/geps/gep-3793/index.md b/geps/gep-3793/index.md index 21f60d3317..41b505b772 100644 --- a/geps/gep-3793/index.md +++ b/geps/gep-3793/index.md @@ -25,6 +25,12 @@ example, if she needs to be sure that her Route is protected by specific authorization policies, she should confer with Chihiro to make sure that she explicitly specifies a Gateway that meets those requirements. +In the future, it may also be important to distinguish different kinds of +default Gateways -- for example, a default ingress Gateway or a default egress +Gateway. This GEP deliberately defines only a single _scope_ of default +Gateway (`All`) but recognizes the need to at least consider the possibility +of multiple scopes in the future. + ## Definitions - **defaulted Route**: a Route that Ana creates without explicitly specifying @@ -33,12 +39,17 @@ explicitly specifies a Gateway that meets those requirements. - **default Gateway**: a Gateway that Chihiro has configured to accept defaulted Routes +- **default Gateway scope**: the scope within which a default Gateway is + applicable + ## Goals - Give Ana a way to use Gateway API without having to explicitly specify a Gateway for every Route, ideally without mutating Routes. (In other words, give Ana an easy way to create a defaulted Route.) +- Give Ana an easy way to define the scope for a defaulted Route. + - Give Ana an easy way to determine which default Gateways are present in the cluster, if any, and which of her Routes are currently bound to these Gateways. @@ -52,6 +63,8 @@ explicitly specifies a Gateway that meets those requirements. - Allow Chihiro to choose not to provide any default Gateways at all. +- Allow Chihiro to define the scope of a default Gateway. + - Allow Chihiro to rename, reconfigure, or replace any default Gateway at runtime. @@ -255,30 +268,44 @@ GEP covers **all** intended changes to Gateway API behavior. Since Ana must be able to choose whether a Route is defaulted or not, marking a Route as defaulted must be an active configuration step she takes, rather than any kind of implicit behavior. To that end, the `CommonRouteSpec` -resource will gain a new field, `useDefaultGateway`: +resource will gain a new field, `useDefaultGateway`, which defines the +_scope_ for the defaulted Route: ```go +// GatewayDefaultScope defines the set of default scopes that a Gateway +// can claim. At present the only supported scope is "All". +type GatewayDefaultScope string + +const ( + // GatewayDefaultScopeAll indicates that a Gateway can claim absolutely + // any Route asking for a default Gateway. + GatewayDefaultScopeAll GatewayDefaultScope = "All" +) + type CommonRouteSpec struct { // ... other fields ... - useDefaultGateway *bool `json:"useDefaultGateway,omitempty"` + useDefaultGateway GatewayDefaultScope `json:"useDefaultGateway,omitempty"` } ``` For Ana to indicate that a Route should use a default Gateway, she MUST set -`spec.useDefaultGateway` to `true` in the `spec` of the Route: +the Route's `spec.useDefaultGateway` to the desired scope: ```yaml ... spec: - useDefaultGateway: true + useDefaultGateway: All ``` +A defaulted Route MUST be accepted only by Gateways that have been configured +with a matching `spec.useDefaultGateway` scope. + A Route MAY include explicit `parentRefs` in addition to setting -`spec.useDefaultGateway` to `true`. In this case, the Route will be a -candidate for being bound to default Gateways, but it will also be bound to -its explicitly-specified `parentRefs`. This allows Ana to create a single -Route that handles N/S traffic via the default Gateways and also handles E/W -traffic via a Service, for example. +`spec.useDefaultGateway`. In this case, the Route will be a candidate for +being bound to default Gateways, but it will also be bound to its +explicitly-specified `parentRefs`. This allows Ana to create a single Route +that handles N/S traffic via the default Gateways and also handles E/W traffic +via a Service, for example. All other characteristics of a defaulted Route MUST behave the same as if all default Gateways were explicitly specified in `parentRefs`. @@ -294,7 +321,7 @@ kind: HTTPRoute metadata: name: my-route spec: - useDefaultGateway: true + useDefaultGateway: All rules: - backendRefs: - name: my-service @@ -312,7 +339,7 @@ kind: HTTPRoute metadata: name: ns-ew-route spec: - useDefaultGateway: true + useDefaultGateway: All parentRefs: - kind: Service name: face @@ -323,9 +350,9 @@ spec: port: 80 ``` -**Multiple Gateways**: A defaulted Route MAY include both a `DefaultGateway` -and other specific named Gateways, although this is not expected to be common -in practice: +**Multiple Gateways**: A defaulted Route MAY both set `useDefaultGateway` and +name other Gateways in `parentRefs`, although this is not expected to be +common in practice: ```yaml apiVersion: gateway.networking.k8s.io/v1 @@ -333,7 +360,7 @@ kind: HTTPRoute metadata: name: multi-gateway-route spec: - useDefaultGateway: true + useDefaultGateway: All parentRefs: - kind: Gateway name: my-gateway @@ -380,20 +407,22 @@ Since Chihiro must be able to control which Gateways accept defaulted Routes, configuring a Gateway to accept defaulted Routes must be an active configuration step taken by Chihiro, rather than any kind of implicit behavior. To that end, the Gateway resource will gain a new field, -`spec.isDefault`: +`spec.defaultScope`: ```go type GatewaySpec struct { // ... other fields ... - IsDefault *bool `json:"isDefault,omitempty"` + DefaultScope GatewayDefaultScope `json:"defaultScope,omitempty"` } ``` -If `spec.isDefault` is set to `true`, the Gateway MUST claim Routes that have -specified `spec.useDefaultGateway` `true` (subject to the usual Gateway API +Again, the only currently-defined scope is `All`. + +If `spec.defaultScope` is set, the Gateway MUST claim Routes that have set +`spec.useDefaultGateway` to a matching value (subject to the usual Gateway API rules about which Routes may be bound to a Gateway), and it MUST update its own `status` with a `condition` of type `DefaultGateway` and `status` true to -indicate that it is a default Gateway, for example: +indicate that it is a default Gateway and what its scope is, for example: ```yaml status: @@ -401,12 +430,12 @@ status: - type: DefaultGateway status: "True" lastTransitionTime: "2025-10-01T12:00:00Z" - message: "Gateway is the default Gateway" + message: "Gateway has default scope All" ``` -If `spec.isDefault` is not present or is set to `false`, the Gateway MUST NOT -claim those Routes and MUST NOT set the `DefaultGateway` condition in its -`status`. +If `spec.defaultScope` is not present, the Gateway MUST NOT claim Routes that +do not name it specifically in `parentRefs`, and it MUST NOT set the +`DefaultGateway` condition in its `status`. ##### Access to a Default Gateway @@ -418,10 +447,8 @@ from its own namespace. ##### Behavior with No Default Gateway -If no Gateway has `spec.isDefault` set to `true`, then all Routes MUST specify -`parentRefs` entries with specific Gateways in order to be accepted by -Gateways, and all Gateways MUST treat `spec.useDefaultGateway` as if it were -set to `false` in all Routes. +If no Gateway has `spec.defaultScope` set, then all Gateways MUST ignore +`spec.useDefaultGateway` in all Routes. A Route will be bound to only those Gateways that it specifically names in `parentRefs` entries. ##### Deleting a Default Gateway @@ -437,32 +464,36 @@ accept defaulted Routes - including being able to change the set of default Gateways at runtime, without requiring downtime - has always been a goal, and this turns out to require support for multiple default Gateways. -Kubernetes itself will not prevent setting `spec.isDefault` to `true` on -multiple Gateways in a cluster, and it also doesn't support any atomic swap -mechanisms. If we want to enforce only a single default Gateway, the Gateway -controllers will have to implement that enforcement logic. There are three -possible options here. +Kubernetes itself will not prevent setting `spec.defaultScope` on multiple +Gateways in a cluster, and it also doesn't support any atomic swap mechanisms. +If we want to enforce only a single default Gateway, the Gateway controllers +will have to implement that enforcement logic. There are three possible +options here. 1. Don't bother with any enforcement logic. - In this case, a Route that sets `spec.useDefaultGateway` will be bound to - _all_ Gateways that have `spec.isDefault` set to `true`. Since Gateway API - already allows a Route to be bound to multiple Gateways, and the Route - `status` is already designed for it, this should function without - difficulty. + In this case, a Route that sets `spec.useDefaultGateway` would be bound to + _all_ Gateways that have `spec.defaultScope` set a matching scope. Since + Gateway API already allows a Route to be bound to multiple Gateways, and + the Route `status` is already designed for it, this should function + without difficulty. -2. Treat multiple Gateways with `spec.isDefault` set to `true` as if no - Gateway has `spec.isDefault` set to `true`. +2. Treat multiple Gateways with `spec.defaultScope` set as if no Gateway has + `spec.defaultScope` set. If we assume that all Gateway controllers in a cluster can see all the Gateways in the cluster, then detecting that multiple Gateways have - `spec.isDefault` set to `true` is relatively straightforward. + `spec.defaultScope` set is relatively straightforward. + + In this case, every Gateway with `spec.defaultScope` set would ignore it, + with the final effect being the same as if no Gateway had + `spec.defaultScope` set: all Gateways would ignore + `spec.useDefaultGateway` in all Routes, and each Gateway would only accept + Routes that explicitly named it in `parentRefs`. - In this case, every Gateway with `spec.isDefault` set to `true` would - treat `spec.useDefaultGateway` as `false` in every Route, behaving as if - no Gateway had `spec.isDefault` set. Each Gateway would also update its - `status` with a `condition` of type `DefaultGateway` and `status` false to - indicate that it is not the default Gateway, for example: + Each Gateway with `spec.defaultScope` set would also update its `status` + with a `condition` of type `DefaultGateway` and `status` false to indicate + that it is not the default Gateway, for example: ```yaml status: @@ -475,15 +506,15 @@ possible options here. 3. Perform conflict resolution as with Routes. - In this case, the oldest Gateway with `spec.isDefault` set to `true` will - be considered the only default Gateway. That oldest Gateway will be the - only one that honors `spec.useDefaultGateway` in Routes, and all other - Gateways with `spec.isDefault` set to `true` will treat - `spec.useDefaultGateway` as `false` in every Route. + In this case, the oldest Gateway with `spec.defaultScope` set would be + considered the only default Gateway. That oldest Gateway would be the only + one that honors `spec.useDefaultGateway` in Routes, and all other Gateways + with `spec.defaultScope` set would ignore `spec.useDefaultGateway` in + every Route. - The oldest default Gateway will update its `status` to reflect that it the - default Gateway; all other Gateways with `spec.isDefault` set to `true` - will update their `status` as in Option 2. + The oldest default Gateway would update its `status` to reflect that it + the default Gateway; all other Gateways with `spec.defaultScope` set to + `true` will update their `status` as in Option 2. Unfortunately, option 2 will almost certainly cause downtime in any case where Chihiro wants to change the implementation behind a default Gateway: @@ -505,8 +536,8 @@ deleting the old Gateway. Reluctantly, we must therefore conclude that option 1 is the only viable choice. Therefore: Gateways MUST NOT attempt to enforce a single default -Gateway, and MUST allow Routes that set `spec.useDefaultGateway` to `true` to -bind to _all_ Gateways that have `spec.isDefault` set to `true`. This is +Gateway, and MUST allow Routes that set `spec.useDefaultGateway` to bind to +_all_ Gateways that have `spec.defaultScope` set a matching scope. This is simplest to implement, it permits zero-downtime changes to the default Gateway, it allows for testing of the new default Gateway before the old one is deleted, and it doesn't cause trouble with respect to security posture @@ -532,14 +563,14 @@ access to read Gateway resources in the cluster at all. ##### Listeners, ListenerSets, and Merging -Setting `spec.isDefault` on a Gateway affects which Routes will bind to the +Setting `spec.defaultScope` on a Gateway affects which Routes will bind to the Gateway, not where the Gateway listens for traffic. As such, setting -`spec.isDefault` MUST NOT alter a Gateway's behavior with respect to +`spec.defaultScope` MUST NOT alter a Gateway's behavior with respect to Listeners, ListenerSets, or merging. In the future, we may want to consider allowing default ListenerSets rather than only default Gateways, but that is not in scope for this GEP. Even if it -is considered later, the guiding principle SHOULD be that `spec.isDefault` +is considered later, the guiding principle SHOULD be that `spec.defaultScope` SHOULD NOT affect where a Gateway listens for traffic or whether it can be merged with other Gateways. @@ -567,10 +598,10 @@ Mesh traffic is defined by using a Service as a `parentRef` rather than a Gateway. As such, there is no case where a default Gateway would be used for mesh traffic. -As noted above, a Route MAY both set `spec.useDefaultGateway` to `true` _and_ -include a `Service` `parentRef` entry, allowing a single Route to handle both -N/S and E/W traffic. In this case, the Route will be bound to both the default -Gateway and the mesh, and the `status` will show both parents. +As noted above, a Route MAY both set `spec.useDefaultGateway` _and_ include a +`Service` `parentRef` entry, allowing a single Route to handle both N/S and +E/W traffic. In this case, the Route will be bound to both the default Gateway +and the mesh, and the `status` will show both parents. ## Conformance Details From 04d276a30360bb6698f5ce0776ada5e782f0a9d6 Mon Sep 17 00:00:00 2001 From: Flynn Date: Fri, 25 Jul 2025 17:23:03 -0400 Subject: [PATCH 13/13] Add RFC8174 definitions Signed-off-by: Flynn --- geps/gep-3793/index.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/geps/gep-3793/index.md b/geps/gep-3793/index.md index 41b505b772..f0cad9a72d 100644 --- a/geps/gep-3793/index.md +++ b/geps/gep-3793/index.md @@ -5,6 +5,13 @@ (See [status definitions](../overview.md#gep-states).) +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", +"SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this +document are to be interpreted as described in BCP 14 ([RFC8174]) when, and +only when, they appear in all capitals, as shown here. + +[RFC8174]: https://www.rfc-editor.org/rfc/rfc8174 + ## User Story **[Ana] wants a concept of a default Gateway.**