EPP: Destination Endpoint Keys Should Not Be Flags

[danehans]

Here are the flags in question:

	destinationEndpointHintKey = flag.String(
		"destinationEndpointHintKey",
		runserver.DefaultDestinationEndpointHintKey,
		"Header and response metadata key used by Envoy to route to the appropriate pod. This must match Envoy configuration.")
	destinationEndpointHintMetadataNamespace = flag.String(
		"DestinationEndpointHintMetadataNamespace",
		runserver.DefaultDestinationEndpointHintMetadataNamespace,
		"The key for the outer namespace struct in the metadata field of the extproc response that is used to wrap the"+
			"target endpoint. If not set, then an outer namespace struct should not be created.")

These values must match on the Gateway. Therefore, they should not be configurable or the config should be exposed by InferencePool ExtensionConnection.

+1 on simply removing the flags until enough users demand to make them configurable.

Thoughts @ahg-g @kfswain @nirrozenbaum @robscott

[ahg-g]

sounds good to me

[nirrozenbaum]

SGTM

[danehans]

Note that #1282 added more flags that must be removed.

[kfswain]

Yeah, this sounds good.

For posterity: Since the GW implementation is not user controlled but the EPP is, having a set protocol makes interaction easier. If we made these headers configurable that requires coordination that all GW implementations would need to support. As @danehans mentions, best not to do this until there is strong demand.

[danehans]

Since InferencePool is being GA'd, we should discuss how these flags would be included in the API if they need to be configurable in the future. IMO, extending ExtensionConnection would make sense. For example:

apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferencePool
metadata:
  name: gateway-pool
  namespace: gwtest
spec:
  extensionRef:
    failureMode: FailOpen
    metadataKeys:
      destEndpoint: "foo" # Defaults to "x-gateway-destination-endpoint"
      destEndpointNamespace: "bar" # Defaults to "envoy.lb"
      destEndpointServed: "baz" # Defaults to"x-gateway-destination-endpoint-served"
)

Note: Currently, ExtensionConnection is not present in the above example since the struct is being inlined:

// Extension specifies how to configure an extension that runs the endpoint picker.
type Extension struct {
...
	// ExtensionConnection configures the connection between the gateway and the extension.
	ExtensionConnection `json:",inline"`
}

[kfswain]

That makes sense, I think the config looks clean like that too.

[nirrozenbaum]

@danehans your suggestion make sense to me.