Coordinate Gateway Implementations for InferencePool GA

[danehans]

InferencePool will be GA'd when #1173 is merged. As part of GA, we need multiple implementations to pass conformance with InferencePool GA. This issue is meant to track the progress of implementations.

• Kgateway - @danehans
• Istio - @LiorLieberman @keithmattix
• GKE - @AndresGuedez @robscott
• Envoy AI Gateway - @Xunzhuo
• Alibaba - @delavet

[kfswain]

@Xunzhuo also may be able to speak for Bytedance/AIBrix, unless we should include someone else

[keithmattix]

Confirmed that Istio 1.27 (releasing in 2 weeks) relies on Envoy 1.35, so our alpha inference extension support will be affected by Envoy regression

[danehans]

Kgateway Update:

• Requires InferencePool FailOpen Unsupported with Envoy v1.35.0 #1266 to be fixed to be fully conformant due to Envoy v1.35.0 breaking InferencePool fail-open support.
• v2.1.0 release is scheduled for 8/13/25. The plan is for both data planes (Envoy and agengateway) to support InferencePool v1 and pass all conformance tests.

[danehans]

Track kgateway-dev/kgateway#11837 for kgateway InferencePool v1 support.

[danehans]

Track istio/istio#57219 for Istio InferencePool v1 support.

[danehans]

@robscott @Xunzhuo @delavet feel free to xref your InferencePool v1 issues so we can track progress across the implementations.

[delavet]

Track AliyunContainerService/ack-gateway-api#2 for Alibaba Cloud ACK InferencePool v1 support

[nirrozenbaum]

@danehans what is the definition of done for this issue which is targeted for next release?
is an open issue in each gateway enough?

we have open issues to track progress for KGateway, Istio and Alibaba. GKE already supports v1.
@Xunzhuo do you mind opening an issue in Envoy aigw for tracking the transition to InfPool v1?

[danehans]

@danehans what is the definition of done for this issue which is targeted for next release?
is an open issue in each gateway enough?

That all implementations either support v1 or xref an associated downstream issue to track the implementation's progress towards v1.

[Xunzhuo]

Envoy AI Gateway issue tracked here: envoyproxy/ai-gateway#1032
PR is working on envoyproxy/ai-gateway#1033

[nirrozenbaum]

@danehans we now have a tracking issue for all gateways. please let me know if current issue can be closed as done.