Move GEP-3792 to Memorandum with API in other GEPs #3894

kflynn · 2025-07-01T17:44:21Z

GEP-3792 is becoming Memorandum, defining the problems that need solving for OCG support.
#3950 is the new GEP that defines the Mesh resource.
An API GEP to allow experimenting with OCG support will arrive shortly.

/kind gep

NONE

geps/gep-3792/index.md

howardjohn · 2025-07-01T18:06:20Z

geps/gep-3792/index.md

+  gatewayClassName: ocg-gateway-class
+  ...
+  mesh:   # All mesh-related configuration goes here
+    trustBundle:


I want to put in consideration adoption of ClusterTrustBundle which is now Beta in Kubernetes. It was designed exactly for this goal and is a perfect fit.

This isn't a "no", but... what would that look like? do we basically vendor the Go type for ClusterTrustBundle for codegen and hope that it doesn't change out from under us? 🤔

No, just an object ref to a ClusterTrustBundle. Or maybe not even a 'ref' needed, just a name (its cluster scoped so no namespace needed)

ClusterTrustBundle, if I understand correctly, is an alpha feature from K8s 1.27 to K8s 1.32, which is to say that in nearly every version supported by Gateway API, if you don't have the feature gate enabled, the ClusterTrustBundle resource type won't exist and there will be no way to refer to a ClusterTrustBundle resource in the Gateway resource... which is why I'm asking what using ClusterTrustBundle would look like. 🙂

(I really, really do not want Gateway API to ship definitions for resources defined by the Kubernetes core, ever.)

ClusterTrustBundle is also a cluster-scoped resource, which feels a bit less than ideal, but that's a much much smaller problem in my mind.

This is also an alpha (experimental) feature. It seems problematic that an experimental feature cannot depend on a Beta feature, and instead we are just re-inventing the exact thing ClusterTrustBundle was designed for

Yeah, this isn't a question about semantics, I kind of agree with you in principle. 🙂 I'm asking about mechanics here: how, technically, can we use ClusterTrustBundle on a cluster where the APIserver does not have any resource type defined under that name?

Saying "you have use K8s 1.33 or enable the feature gate" feels problematic.

The options here are:

we only add trustBundle inline for now, with a note about how we will add ClusterTrustBundle in the future.

we add both for now, with a note that once ClusterTrustBundle is more available in Gateway API supported versions, we will remove the inline trustBundle. We can get away with that as long as this resource is Experimental.

We only add trustBundle, and say "run 1.33 or later, or enable an alpha feature to use this". This would require us to specify the behavior for implementations; since this has a FeatureName anyway, there is a mechanism to say "requires k8s 1.33 or alpha feature enabled to work", although we've never done that before.

Because Gateway API's claimed Kubernetes support is five versions, the last would mean that we would need to wait nearly two years before bringing this API to GA in options 2 or 3. That feels like a longer timeline than people would expect for this. Unless we are okay with doing 3 and adding features that are not supported across all supported Kubernetes versions, which does not seem optimal to me.

@robscott for his thoughts on this one too.

Actually, I just checked, and ClusterTrustBundle is only beta, and not enabled by default in v1.33, so this would be even longer. (See https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/#cluster-trust-bundles for the reference)

At a minimum, if ClusterTrustBundle moves to Stable in v1.34, then the only time we could allow it with no caveats is as of Kubernetes v1.38, which is in just over two years.

In short; my thoughts here are that we can add a ClusterTrustBundle reference with a feature name and a version caveat at best. But we need something for before then as well.

We have similar (though less restrictive) versioning issues for using ClusterTrustBundle within GKE. The approach we took was to make a CustomResource ("GKEClusterTrustBundle") that had exactly the same shape as ClusterTrustBundle, and have our controllers understand both types.

Elsewhere in GW API when we've dealt with configure CA certs we've started with config maps with the idea that we'd add support for ClusterTrustBundle as it became more broadly available. That seems like a reasonable approach here, with the distinction that ClusterTrustBundle is likely going to go to GA in v1.35, so it would be good to consider that our default starting point with ConfigMap as the legacy backfill option from the outset.

howardjohn · 2025-07-01T18:14:53Z

geps/gep-3792/index.md

+- The mesh MUST use an mTLS certificate ultimately signed by a certificate in
+  the trust bundle provided to the OCG.
+
+- The OCG MUST send the `ocg.gateway.networking.k8s.io/v1` ALPN protocol


This applies to the whole proposal in general but specifically this line. There are 3 OCG implementations today. I would think to make this viable we need at least 2 to agree to this.

I know that, unless things have changed, using a custom ALPN is the hill at least one cloud will die on and will never support

I'm not comfortable making this a MUST. Custom ALPNs are certainly a way to signal mTLS, but different implementations may use tunneling protocols like HBONE instead.

@keithmattix and I discussed this on Slack a bit, since I'm very uncomfortable with making this a MAY and then having, say, the OCG expecting ALPN, the mesh expecting HBONE, but they both believe that they "support GEP-3792 OCG".

As Keith noted on Slack, though, it doesn't really make sense to require a mesh to only support one mechanism for solving the protocol problem, and it follows that it probably does make sense to add an explicit field to the Gateway and Mesh resources to say what the signalling mechanism is. That also permits warning Chihiro if the Gateway and Mesh disagree on what's offered. I'll add that.

Also, to @howardjohn's point: we explicitly say in "Graduation Criteria" section that we require at least two OCG implementations and at least two mesh implementations. Leaving aside the choice of hills upon which to die... 🙂

it follows that it probably does make sense to add an explicit field to the Gateway and Mesh resources to say what the signalling mechanism is. That also permits warning Chihiro if the Gateway and Mesh disagree on what's offered. I'll add that.

This makes me a bit nervous in that we could end up with n different protocols that OCGs would need to support here. Maybe this is unavoidable, but I'd really like to minimize the variations here if we can. The larger the number of variations here, the less likely it is that an OCG is going to be able to support them.

geps/gep-3792/index.md

keithmattix · 2025-07-01T19:00:07Z

geps/gep-3792/index.md

+- The mesh MUST use an mTLS certificate ultimately signed by a certificate in
+  the trust bundle provided to the OCG.
+
+- The OCG MUST send the `ocg.gateway.networking.k8s.io/v1` ALPN protocol


I'm not comfortable making this a MUST. Custom ALPNs are certainly a way to signal mTLS, but different implementations may use tunneling protocols like HBONE instead.

geps/gep-3792/index.md

youngnick · 2025-07-02T03:14:27Z

I won't get much of a chance to watch this one further, but the main point I wanted to add is that, because this is TLS config for a Gateway, it's important that we define the interaction (or lack thereof) with consumer BackendTLSPolicy as per #3875.

FTR, I think that saying "they are for different things, the only interaction is to end up with TLS-in-TLS, which is dumb, so you should avoid that". is fine by me.

karthikbox · 2025-07-07T17:02:49Z

geps/gep-3792/index.md

+   for [Chihiro] and [Ian], _and_ it limits us to only having entire
+   namespaces meshed.
+
+6. Add a label selector to the Gateway resource and declare that Routes with


just a thought - how about an explicit mesh MTLS policy that targetRefs a service? the meshMTLS policy would indicate to both the OCG and Mesh Clients that they need to use Mesh MTLS to talk to the service.
the explicit meshMTLS policy can be either created by the user or the mesh controller creates a default one for every service that is meshed. The user would override the default one if they wish to tweak the default settings - for e.g override the SAN checks, or SNI or disable/enable MTLS perhaps (any kind of outbound setting).

FWIW in Istio (and probably linkerd and every other mesh), the mesh TLS is a property of a workload (Pod) not service. So workloads within the same Service may not actually have the same properties

@howardjohn I don't think that we should expect Gateways to be able to differentiate TLS configuration per workload, that would be a huge increase in scope/complexity for many Gateway implementations. Instead, it seems simpler to focus on the common attributes that Gateways can understand (Service, Namespace, etc).

(You may not have been suggesting that Gateways should be able to understand per-Pod TLS config, so sorry if I'm misunderstanding here).

I am not necessarily suggesting we do, but to properly handle all cases you would need to

karthikbox · 2025-07-07T17:11:30Z

geps/gep-3792/index.md

+  as application-level mTLS that they should treat as an opaque data stream,
+  or they may simply refuse it. This alternative, therefore, both shifts the
+  entire burden of implementation to the meshes, and probably makes it
+  impossible to correctly handle application-level mTLS.


just wanted to mention that GKE Mesh is a notable exception (atleast partially). We use simply use bare mtls connection. And yes, we're subject to the downsides mentioned above. But the upside of interoperability between various GCP (off-GCP) data planes with GKE Mesh workloads using standard MTLS was/is deemed to be more important (i.e don't run into the protocol problem).

my intention with this comment is NOT to say custom protocol shouldn't be supported, but to point out that there are implementations that just use bare/standard mtls.

karthikbox · 2025-07-07T17:19:53Z

geps/gep-3792/index.md

+  gatewayClassName: ocg-gateway-class
+  ...
+  mesh:   # All mesh-related configuration goes here
+    trustBundle:


can we allow for some kind of extensibility here to indicate Platform MTLS? The Platform MTLS is a feature of the cloud platform (e.g GKE Managed Workload Certificates) or the mesh vendor (e.g Istio) where the certs and trust bundle is provisioned by the platform for each workload and OCG automatically. In such cases, User need not configure the k8s Secret explicitly reducing toil.

same comment for the counterpart in Mesh resource.

or would clusterTrustBundle (or the gkeClusterTrustBundle) be able carry the info for the platform MTLS? @ahmedtd to confirm. If that's the case, then supporting clusterTrustBundle be the path forward for indicating platform MTLS.

GKE Managed Workload Certificates uses GKEClusterTrustBundle, which is an identical copy of ClusterTrustBundle as a custom resource. We intend to migrate to ClusterTrustBundle for in-cluster distribution of trust anchors.

@ahmedtd @karthikbox just to make sure I'm following this thread correctly, it sounds like ClusterTrustBundle (or an equivalent) would work well here, even for Platform mTLS?

LiorLieberman · 2025-07-08T16:44:07Z

geps/gep-3792/index.md

+3. Add a field to the Route resource that indicates whether the Route is
+   meshed. This feels like quite a bit of an imposition on [Ana] (and, again,
+   if we do our jobs correctly, [Ana] shouldn't need to think about this).


Maybe I am missing something, cant we use parentRef: <ANY_SERVICE>?

The OCG won't be able to see a Route unless it has a parentRefs entry for the OCG itself. It feels quite strange to also have the OCG (a Gateway) look at other parentRefs entries to figure out if a mesh is involved?

robscott

Thanks @kflynn, this is really well written, appreciate the time you've put into this!

geps/gep-3792/index.md

robscott · 2025-07-10T05:47:51Z

geps/gep-3792/index.md

+- The OCG MUST use an mTLS certificate ultimately signed by a certificate in
+  the trust bundle provided to the mesh.


I thought this trust bundle was configured on the mesh resource? This may be saying that, but it's a bit confusing to me.

robscott · 2025-07-10T05:49:06Z

geps/gep-3792/index.md

+- The OCG MUST use an mTLS certificate ultimately signed by a certificate in
+  the trust bundle provided to the mesh.
+
+- The mesh MUST use an mTLS certificate ultimately signed by a certificate in


Can we call this a client cert? Is the Gateway supposed to generate this on their own, or can/should a mesh provide client cert(s) for Gateway(s) to use?

robscott · 2025-07-10T05:51:56Z

geps/gep-3792/index.md

+- The mesh MUST use an mTLS certificate ultimately signed by a certificate in
+  the trust bundle provided to the OCG.
+
+- The OCG MUST send the `ocg.gateway.networking.k8s.io/v1` ALPN protocol


it follows that it probably does make sense to add an explicit field to the Gateway and Mesh resources to say what the signalling mechanism is. That also permits warning Chihiro if the Gateway and Mesh disagree on what's offered. I'll add that.

This makes me a bit nervous in that we could end up with n different protocols that OCGs would need to support here. Maybe this is unavoidable, but I'd really like to minimize the variations here if we can. The larger the number of variations here, the less likely it is that an OCG is going to be able to support them.

robscott · 2025-07-10T05:55:27Z

geps/gep-3792/index.md

+The discovery problem is that not every workload in the cluster is required to
+be meshed, which means that the OCG needs a way to know whether a given
+connection must participate in the mesh or not. In practice, this isn't
+actually a question of _workloads_ but of _Routes_: the point of interface
+between a Gateway and a workload in the cluster is not a Pod or a Service, but
+rather a Route.


It seems like we care less about which workloads have a mesh Route, and more about which workloads have automatic mTLS provided by a Mesh. My understanding is that most mesh implementations provide automatic mTLS/mesh transport for a broader set of workloads, and that's what we want Gateways to care about when connecting to backends with mTLS following this mesh config.

robscott · 2025-07-10T05:58:19Z

geps/gep-3792/index.md

+   for [Chihiro] and [Ian], _and_ it limits us to only having entire
+   namespaces meshed.
+
+6. Add a label selector to the Gateway resource and declare that Routes with


I'm trying to understand the mechanics of this. Does this selector only apply to Routes that are attached to each relevant Gateway? Are Gateway implementations supposed to look at mesh Routes?

robscott · 2025-07-10T06:01:41Z

geps/gep-3792/index.md

+   for [Chihiro] and [Ian], _and_ it limits us to only having entire
+   namespaces meshed.
+
+6. Add a label selector to the Gateway resource and declare that Routes with


@howardjohn I don't think that we should expect Gateways to be able to differentiate TLS configuration per workload, that would be a huge increase in scope/complexity for many Gateway implementations. Instead, it seems simpler to focus on the common attributes that Gateways can understand (Service, Namespace, etc).

(You may not have been suggesting that Gateways should be able to understand per-Pod TLS config, so sorry if I'm misunderstanding here).

robscott · 2025-07-10T06:04:38Z

geps/gep-3792/index.md

+  gatewayClassName: ocg-gateway-class
+  ...
+  mesh:   # All mesh-related configuration goes here
+    trustBundle:


Elsewhere in GW API when we've dealt with configure CA certs we've started with config maps with the idea that we'd add support for ClusterTrustBundle as it became more broadly available. That seems like a reasonable approach here, with the distinction that ClusterTrustBundle is likely going to go to GA in v1.35, so it would be good to consider that our default starting point with ConfigMap as the legacy backfill option from the outset.

robscott · 2025-07-10T06:05:32Z

geps/gep-3792/index.md

+spec:
+  gatewayClassName: ocg-gateway-class
+  ...
+  mesh:   # All mesh-related configuration goes here


Why is this on the Gateway? This feels like configuration that could live on the Mesh resource?

robscott · 2025-07-10T06:11:51Z

geps/gep-3792/index.md

+    selector:
+      matchLabels:
+        linkerd.io/inject: "true"       # or whatever label is appropriate


As I understand it, this selector is intended to select namespaces and routes. I'm very nervous about a multi-resource selector here though. My understanding is that the primary reason for considering Routes here is when only a portion of a namespace is part of a mesh. Would it be reasonable to keep a label selector for namespaces, and then opt-out per Service or Service Port?

The rationale for that would be:

The selector is clearly for a single kind of resource

We still provide a way to opt-out subsets of a namespace

We're using granularities that are already well understood by a Gateway (my guess is that it will be difficult for some implementations to configure mTLS per route rule, but quite simple per backend as we're already doing with BackendTLSPolicy)

kflynn · 2025-07-23T20:23:30Z

I'm making this PR ready for review again because the API stuff is moving. However, I'm leaving many comments unresolved since they're relevant for other GEPs.

k8s-ci-robot · 2025-07-25T21:25:16Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: kflynn
Once this PR has been reviewed and has the lgtm label, please assign robscott for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: Flynn <[email protected]>

…e Mesh resource, and define that the API will be different GEPs. Signed-off-by: Flynn <[email protected]>

Signed-off-by: Flynn <[email protected]>

k8s-ci-robot · 2025-07-26T14:06:49Z

@kflynn: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-gateway-api-verify	`9c215ac`	link	true	`/test pull-gateway-api-verify`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

shaneutt · 2025-07-29T15:12:32Z

geps/gep-3792/index.md

@@ -1,10 +1,17 @@
 # GEP-3792: Out-of-Cluster Gateways

 * Issue: [#3792](https://github.com/kubernetes-sigs/gateway-api/issues/3792)
-* Status: Provisional
+* Status: Memorandum


Suggested change

* Status: Memorandum

* Status: Deferred

k8s-ci-robot requested review from gcs278 and robscott July 1, 2025 17:44

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jul 1, 2025

howardjohn reviewed Jul 1, 2025

View reviewed changes

keithmattix reviewed Jul 1, 2025

View reviewed changes

kflynn changed the title ~~Add API for Off-Cluster Gateways~~ API for Off-Cluster Gateways Jul 1, 2025

karthikbox reviewed Jul 7, 2025

View reviewed changes

LiorLieberman reviewed Jul 8, 2025

View reviewed changes

robscott reviewed Jul 10, 2025

View reviewed changes

robscott mentioned this pull request Jul 11, 2025

[GEP-2162] Updated a new field on supported features inference from boolean to enum and remove from report. #3885

Merged

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 17, 2025

kflynn force-pushed the flynn/gep-3792 branch from a1bc550 to 7a2111a Compare July 22, 2025 03:21

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 22, 2025

kflynn marked this pull request as draft July 22, 2025 03:24

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 22, 2025

kflynn mentioned this pull request Jul 22, 2025

API for Default Gateways #3887

Merged

karthikbox mentioned this pull request Jul 22, 2025

API for Off-Cluster Gateways (superseeds #3894) #3941

Open

kflynn force-pushed the flynn/gep-3792 branch 2 times, most recently from a5e438e to f5a7b01 Compare July 23, 2025 19:46

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 23, 2025

kflynn changed the title ~~API for Off-Cluster Gateways~~ Move GEP-3792 to Memorandum with API in other GEPs Jul 23, 2025

kflynn marked this pull request as ready for review July 23, 2025 20:23

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 23, 2025

k8s-ci-robot requested review from mlavacca and robscott July 23, 2025 20:23

kflynn force-pushed the flynn/gep-3792 branch from 6f0d992 to 9682774 Compare July 25, 2025 21:25

kflynn mentioned this pull request Jul 25, 2025

Defer GEP-3792 for Gateway API 1.4 #3919

Closed

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 26, 2025

kflynn added 4 commits July 26, 2025 09:51

Links for personas

72e5661

Signed-off-by: Flynn <[email protected]>

Switch to Memorandum rather than Provisional. Link to GEP-3949 for th…

9255ee4

…e Mesh resource, and define that the API will be different GEPs. Signed-off-by: Flynn <[email protected]>

Minor wordsmithing

310905f

Signed-off-by: Flynn <[email protected]>

Correctly switch to Memorandum

9c215ac

Signed-off-by: Flynn <[email protected]>

kflynn force-pushed the flynn/gep-3792 branch from 9682774 to 9c215ac Compare July 26, 2025 13:52

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 26, 2025

shaneutt reviewed Jul 29, 2025

View reviewed changes

LiorLieberman mentioned this pull request Aug 6, 2025

Extra minimal OCG API #3952

Open

		- The OCG MUST use an mTLS certificate ultimately signed by a certificate in
		the trust bundle provided to the mesh.

Move GEP-3792 to Memorandum with API in other GEPs #3894

Are you sure you want to change the base?

Move GEP-3792 to Memorandum with API in other GEPs #3894

Uh oh!

Conversation

kflynn commented Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

youngnick Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kflynn Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

youngnick commented Jul 2, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

karthikbox Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LiorLieberman Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

kflynn commented Jul 1, 2025 •

edited

Loading

youngnick Jul 2, 2025 •

edited

Loading

kflynn Jul 1, 2025 •

edited

Loading

karthikbox Jul 7, 2025 •

edited

Loading

LiorLieberman Jul 8, 2025 •

edited

Loading