docs: add LLMInferenceService with AgentGateway integration guide by mgonzalezo · Pull Request #697 · kserve/website

mgonzalezo · 2026-07-03T08:53:43Z

Add a procedural guide for integrating LLMInferenceService with AgentGateway
to enable LLM-aware routing, token tracking, and GenAI telemetry using the
existing LLMInferenceServiceConfig backendRef override mechanism.

Proposed Changes

Add docs/model-serving/generative-inference/llmisvc/llmisvc-agentgateway.md with step-by-step instructions for overriding HTTPRoute backendRef to use AgentgatewayBackend
Document two approaches: per-service override via spec.router.route.http and reusable LLMInferenceServiceConfig via baseRefs
Add sidebar entry under LLMInferenceService alongside the existing Envoy AI Gateway guide

netlify · 2026-07-03T08:53:47Z

✅ Deploy Preview for elastic-nobel-0aef7a ready!

Name	Link
🔨 Latest commit	`b326be3`
🔍 Latest deploy log	https://app.netlify.com/projects/elastic-nobel-0aef7a/deploys/6a48ff668153da0008883cb4
😎 Deploy Preview	https://deploy-preview-697--elastic-nobel-0aef7a.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

bartoszmajsak

Thanks for this contribution. Very happy to see that flexibility of the API makes such integration quite straightforward.

I have couple of suggestions and a gap we will address on our side.

bartoszmajsak · 2026-07-03T11:01:58Z

+| Protocol detection | `http` (generic) | `llm` (LLM-aware) |
+| Token parsing | No | Yes |
+| Model tracking | No | Yes |
+| GenAI telemetry (OTel) | No | Yes |


Is this accurate? We lately added this integration, so I am not sure this is AgentgatewayBackend specific feature.

thanks for flagging #5481. I've clarified the table: KServe's distributed tracing provides request-level spans/traces, while the GenAI semantic conventions here refer to LLM-specific OTel attributes (gen_ai.usage.input_tokens, gen_ai.request.model, etc.) that AgentGateway emits at the gateway level. Added a note explaining they're complementary.

bartoszmajsak · 2026-07-03T11:08:30Z

                "model-serving/generative-inference/llmisvc/llmisvc-config-composition",
                "model-serving/generative-inference/llmisvc/llmisvc-dependencies",
                "model-serving/generative-inference/llmisvc/llmisvc-envoy-ai-gateway",
+                "model-serving/generative-inference/llmisvc/llmisvc-agentgateway",


Maybe we could have a dedicated section for both agentgateway and envoy-ai-gateway under "Integrations"? (I am terrible at naming...)

Moved both llmisvc-envoy-ai-gateway and llmisvc-agentgateway under a new "Integrations" subcategory within LLMInferenceService

bartoszmajsak · 2026-07-03T11:55:44Z

+    host: my-model-kserve-workload-svc.kserve-test.svc.cluster.local
+    port: 8000


They should both go under .provider . https://docs.solo.io/agentgateway/2.3.x/reference/api/api/#llmprovider

Got it, host and port now sit under .provider alongside the provider type, matching the API reference. Also switched from custom to openai provider type since KServe runtimes expose OpenAI-compatible endpoints.

bartoszmajsak · 2026-07-03T11:56:33Z

+            nvidia.com/gpu: 1
+```
+
+Wait for the InferenceService to be ready:


Suggested change

Wait for the InferenceService to be ready:

Wait for the LLMInferenceService to be ready:

bartoszmajsak · 2026-07-03T12:33:44Z

+  router:
+    scheduler: {}
+    route: {}
+    gateway: {}


This is not needed b/c below you are using:

parentRefs: - group: gateway.networking.k8s.io kind: Gateway name: agentgateway namespace: kserve-test

Noted, since the route override uses explicit parentRefs, I removed it from examples.

bartoszmajsak · 2026-07-03T12:51:31Z

+    name: Qwen/Qwen2.5-0.5B-Instruct
+  replicas: 1
+  router:
+    scheduler: {}


Setting scheduler: {} is unnecessary here and actually problematic - it creates an EPP deployment and InferencePool that no HTTPRoute references. AgentGateway won't write an Accepted status for the pool since no route targets it, leaving InferencePoolReady stuck at WaitingForGateway and blocking the overall Ready condition indefinitely. On top of that, AgentGateway doesn't even watch InferencePool resources unless inferenceExtension.enabled=true is set (disabled by default).

@pierDipi, maybe we should signal misconfig in the conditions?

Thank you for the clarification, I have removed it from the examples. Happy to help with a follow-up PR once you and @pierDipi agree on the right approach here.

Signed-off-by: Marco Gonzalez <margonza@redhat.com> docs: address review feedback on AgentGateway integration guide Signed-off-by: Marco Gonzalez <margonza@redhat.com>

danehans · 2026-07-07T16:26:31Z

xref: kserve/kserve#5518

…-aware routing Proves that overriding backendRef in KServe-generated HTTPRoutes from InferencePool to AgentgatewayBackend activates AgentGateway's full LLM pipeline - GenAI telemetry, token tracking, and token-based rate limiting. Without this override, AgentGateway treats inference traffic as generic HTTP, missing model-aware observability and rate controls. The spike validates the integration described in kserve/website#697 using a tiny vLLM model on kind with no GPU required. Ref: kserve/kserve#5729 Signed-off-by: Bartosz Majsak <bartosz.majsak@gmail.com>

danehans · 2026-07-15T17:41:32Z

@bartoszmajsak when you have a moment, PTAL. I see you are testing this PR in https://github.com/bartoszmajsak/playground via bartoszmajsak/playground@f8a7a2c. Can you help me understand if this is part of the typical validation workflow.

bartoszmajsak · 2026-07-15T18:58:02Z

@bartoszmajsak when you have a moment, PTAL. I see you are testing this PR in https://github.com/bartoszmajsak/playground via bartoszmajsak/playground@f8a7a2c. Can you help me understand if this is part of the typical validation workflow.

@danehans This is related I think, PTAL kserve/kserve#5776

bartoszmajsak · 2026-07-15T20:56:10Z

Can you help me understand if this is part of the typical validation workflow.

@danehans not quite sure what the exact ask is - I was simply testing samples from this doc e2e out of curiosity. And both feedback here and PR linked previously are the results of this "spike".

mgonzalezo force-pushed the docs/llmisvc-agentgateway-integration branch from c557711 to 684e325 Compare July 3, 2026 09:06

bartoszmajsak reviewed Jul 3, 2026

View reviewed changes

docs: add LLMInferenceService with AgentGateway integration guide

b326be3

Signed-off-by: Marco Gonzalez <margonza@redhat.com> docs: address review feedback on AgentGateway integration guide Signed-off-by: Marco Gonzalez <margonza@redhat.com>

mgonzalezo force-pushed the docs/llmisvc-agentgateway-integration branch from 657fe61 to b326be3 Compare July 4, 2026 12:41

mgonzalezo mentioned this pull request Jul 4, 2026

Feature request: Make HTTPRoute backendRef kind/group configurable for InferenceService kserve/kserve#5729

Open

bartoszmajsak mentioned this pull request Jul 7, 2026

fix(llmisvc): skip scheduler for custom backendRefs kserve/kserve#5776

Open

4 tasks

danehans mentioned this pull request Jul 7, 2026

Feature request: KServe InferenceService discovery for AI backends agentgateway/agentgateway#2323

Open

		host: my-model-kserve-workload-svc.kserve-test.svc.cluster.local
		port: 8000

	Wait for the InferenceService to be ready:
	Wait for the LLMInferenceService to be ready:

Uh oh!

Conversation

mgonzalezo commented Jul 3, 2026

Proposed Changes

Uh oh!

netlify Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for elastic-nobel-0aef7a ready!

Uh oh!

bartoszmajsak left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mgonzalezo Jul 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bartoszmajsak Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danehans commented Jul 7, 2026

Uh oh!

danehans commented Jul 15, 2026

Uh oh!

bartoszmajsak commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bartoszmajsak commented Jul 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

netlify Bot commented Jul 3, 2026 •

edited

Loading

mgonzalezo Jul 4, 2026 •

edited

Loading

bartoszmajsak Jul 3, 2026 •

edited

Loading

bartoszmajsak commented Jul 15, 2026 •

edited

Loading