You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add the design for supporting the Gateway API Inference Extension. This would allow NGF to configure NGINX to route traffic to AI workloads in Kubernetes, using specialized load-balancing.
Enable NGINX Gateway Fabric to support the [Gateway API Inference Extension](https://gateway-api-inference-extension.sigs.k8s.io/), allowing dynamic routing to AI workloads. The goal for now is a basic implementation that meets the core functionality based on the API spec. There are likely many enhancements and improvements that can be made to this, but those should be considered after feedback around the usage and worth of this feature.
9
9
10
+
The [References section](#references) contains many useful links for describing the extension in depth.
11
+
10
12
## Goals
11
13
12
-
- Define and implement the basic implementation to meet the API's core specifications.
14
+
- Define and implement the basic solution to meet the API's core specifications.
13
15
- Determine which resources (e.g. InferencePool) NGF needs to watch, and what configuration should be built based upon this.
14
16
- Define the process in which NGF should integrate with the [Endpoint Picker](https://github.com/kubernetes-sigs/gateway-api-inference-extension/tree/main/pkg/epp) (EPP).
15
17
- Determine what NGINX needs to do in order to forward incoming traffic to an AI workload.
@@ -19,3 +21,216 @@ Enable NGINX Gateway Fabric to support the [Gateway API Inference Extension](htt
19
21
- Define new APIs.
20
22
- Determine how to integrate with AI Gateway.
21
23
- Any functionality beyond the core API specification.
24
+
25
+
## Terminology
26
+
27
+
The terms AI workload, LLM workload, and model workload are used interchangeably in this doc. Most of this is based on terminology provided by the Gateway API Inference Extension documentation.
28
+
29
+
## Introduction
30
+
31
+
The intention of the Gateway API Inference Extension is to optimize load-balancing for self-hosted GenAI models in Kubernetes. These workloads can serve massive amounts of data, and their performance is influenced greatly by the infrastructure that they are running on (e.g. GPUs). Routing client requests to these types of backends requires specialized decision making to ensure the best performance and responses.
32
+
33
+
To make these routing decisions, a component known as the [Endpoint Picker (EPP)](https://gateway-api-inference-extension.sigs.k8s.io/#endpoint-picker) is deployed. The EPP uses configuration and metrics to determine which AI workload in an InferencePool should receive the request. It returns this endpoint in a header to the `inference Gateway` (which would be NGINX in our case) to forward the request to that endpoint. The model name that the request should be routed to can be contained in the body of the request, or in a header.
34
+
35
+
Check out the [request flow](https://gateway-api-inference-extension.sigs.k8s.io/#request-flow) section of the Gateway API documentation to learn more.
36
+
37
+
## Use Cases
38
+
39
+
Gateway API Inference Extension provides optimized load-balancing for self-hosted Generative AI Models on Kubernetes. The project’s goal is to improve and standardize routing to inference workloads across the ecosystem.
40
+
41
+
Use cases include:
42
+
43
+
- Route traffic based on model names instead of just a path.
44
+
- Enable varying priorities of models, ensuring that more critical requests are routed to the proper model.
45
+
- Incrementally roll out new models.
46
+
- Overall reduce latency and improve accelerator (GPU) utilization for AI workloads.
47
+
48
+
## Design
49
+
50
+
This feature should be gated behind a feature flag. This is due to the API being in an alpha state and things may be subject to change. User feedback will be valuable, and telemetry data could help us understand if anyone uses it and if it's worth pursuing improvements in the future.
51
+
52
+
### Integration with EPP
53
+
54
+
The Endpoint Picker uses Envoy's [ext_proc protocol](https://github.com/kubernetes-sigs/gateway-api-inference-extension/tree/main/docs/proposals/004-endpoint-picker-protocol). NGINX needs to be able to forward the client request to the EPP and use the endpoint returned in the header response to then forward the request to that endpoint (the AI workload). Because the EPP listens for gRPC, we cannot use NGINX natively (or NJS) to make that call. For our initial implementation, we can write a simple Go application that is called by an NJS module via subrequest to get the endpoint from the EPP. NGINX would then `proxy_pass` to that endpoint.
55
+
56
+
The Go application could be built into the existing `nginx-gateway` binary, and deployed as an additional container in the NGINX Deployment when an InferencePool is created. The NJS module would then call this application over localhost. The connection from Go -> EPP should be secure if possible, as discussed in the [Security Considerations](#security-considerations).
57
+
58
+
See the [Alternatives section](#alternatives) for a future improvement to this workflow.
59
+
60
+
### Model Name extraction
61
+
62
+
When a client sends a request to an AI workload, the desired model name (e.g. gpt-4o, llama, etc.) is included in the request body.
63
+
64
+
By default, the EPP gets the model name from the request body, and then picks the proper endpoint for that model name. However, the model name could also be provided via header (`X-Gateway-Model-Name`). For example, a user could specify a desire for a traffic split or model name redirect, and therefore NGINX would need to change the model name by setting the header.
65
+
66
+
Example that redirects requests to model name `food-review` to `food-review-v1`:
67
+
68
+
```yaml
69
+
kind: HTTPRoute
70
+
apiVersion: gateway.networking.k8s.io/v1
71
+
metadata:
72
+
name: my-route
73
+
spec:
74
+
parentRefs:
75
+
- name: my-inference-gateway
76
+
rules:
77
+
- matches:
78
+
- headers:
79
+
- type: Exact
80
+
name: X-Gateway-Model-Name
81
+
value: food-review
82
+
backendRefs:
83
+
- name: vllm-llama3-8b-instruct
84
+
kind: InferencePool
85
+
group: inference.networking.x-k8s.io
86
+
- filters:
87
+
- type: RequestHeaderModifier
88
+
requestHeaderModifier:
89
+
set:
90
+
- name: X-Gateway-Model-Name
91
+
value: food-review-v1
92
+
```
93
+
94
+
Example with traffic splitting:
95
+
96
+
```yaml
97
+
kind: HTTPRoute
98
+
apiVersion: gateway.networking.k8s.io/v1
99
+
metadata:
100
+
name: my-route
101
+
spec:
102
+
parentRefs:
103
+
- name: my-inference-gateway
104
+
rules:
105
+
- matches:
106
+
- headers:
107
+
- type: Exact
108
+
name: X-Gateway-Model-Name
109
+
value: food-review
110
+
backendRefs:
111
+
- name: vllm-llama3-8b-instruct
112
+
kind: InferencePool
113
+
group: inference.networking.x-k8s.io
114
+
weight: 90
115
+
- filters:
116
+
- type: RequestHeaderModifier
117
+
requestHeaderModifier:
118
+
set:
119
+
- name: X-Gateway-Model-Name
120
+
value: food-review-v1
121
+
- name: vllm-llama3-8b-instruct
122
+
kind: InferencePool
123
+
group: inference.networking.x-k8s.io
124
+
weight: 10
125
+
- filters:
126
+
- type: RequestHeaderModifier
127
+
requestHeaderModifier:
128
+
set:
129
+
- name: X-Gateway-Model-Name
130
+
value: food-review-v2
131
+
```
132
+
133
+
In both cases, NGINX would need to extract the model name from the request body. This will probably require an NJS module. If that model name matches the condition set in the Route, then NGINX sets the header appropriately when sending the request to the EPP. For the redirect example, NGINX would set the header to `food-review-v1`. For the traffic splitting example, NGINX would set the header to either `food-review-v1` or `food-review-v2` depending on the weighted traffic decision.
134
+
135
+
### Managing InferencePools
136
+
137
+
By default, the EPP should know which endpoints are a part of an InferencePool, and then pick the correct endpoint to send to. This means that NGINX does not need to have an upstream for the AI workload servers, since it just gets the endpoint it needs to send to from the EPP.
138
+
139
+
However, there could still be a valid use case for NGF to track and configure NGINX with the endpoints that are a part of the InferencePool. For one, there is currently a `FailOpen` mode which says that if the EPP is unavailable, NGINX should pick an endpoint to send to. There's also value in verifying that the endpoint given to us by the EPP is trusted.
140
+
141
+
Because of this, NGF should watch the endpoints associated with an InferencePool, and create an upstream. One way to accomplish this is for NGF to create a Headless "shadow" Service that encompasses those endpoints. By defining this Service, NGF can use all of its existing Service/EndpointSlice logic to build the upstreams as if it was a normal Service.
142
+
143
+
**The main point of concern with this is how can we fallback to use the upstream servers if the EPP is unavailable to give us an endpoint?** This may have to be discovered during implementation.
144
+
145
+
### Flow Diagram
146
+
147
+
```mermaid
148
+
flowchart TD
149
+
A[Client Request] --> B[NGINX]
150
+
subgraph NGINX Pod
151
+
subgraph NGINX Container
152
+
B --1--> C[NJS Module: extract model name if needed]
153
+
C --2--> B
154
+
B --3--> D[NJS Module: Subrequest to Go App]
155
+
end
156
+
subgraph Go Application Container
157
+
E[Go Application]
158
+
end
159
+
D -- 4. subrequest --> E
160
+
end
161
+
E -- 5. gRPC ext_proc protocol --> F[Endpoint Picker Pod]
162
+
F -- 6. Endpoint in Header --> E
163
+
E --7--> D
164
+
D --8--> B
165
+
B --9--> G[AI Workload Endpoint]
166
+
```
167
+
168
+
## API, Customer Driven Interfaces, and User Experience
169
+
170
+
The infrastructure provider or cluster operator would first need to install the Gateway API Inference Extension CRDs, similar to how they install the Gateway API CRDs today. Two new CRDs are introduced, the `InferencePool` and `InferenceObjective`.
171
+
172
+
### InferencePool
173
+
174
+
InferencePool represents a set of Inference-focused Pods and an extension that will be used to route to them. Within the broader Gateway API resource model, this resource is considered a "backend". In practice, that means that you'd replace a Kubernetes Service with an InferencePool. This resource has some similarities to Service (a way to select Pods and specify a port), but has some unique capabilities. With InferencePool, you can configure a routing extension as well as inference-specific routing optimizations.
175
+
176
+
### InferenceObjective
177
+
178
+
InferenceObjective represents the desired state of a specific model use case. As of writing this, this API is still undergoing changes.
179
+
180
+
It is my impression that this API is purely for the EPP to handle, and does not need to be handled by NGINX Gateway Fabric.
181
+
182
+
### Personas and Processes
183
+
184
+
Two new personas are introduced, the `Inference Platform Owner/Admin` and `Inference Workload Owner`.
185
+
186
+
The Inference Platform Owner creates and manages the infrastructure necessary to run LLM workloads. They create InferencePool resources (and an associated EPP). They can do so using [this helm chart](https://github.com/kubernetes-sigs/gateway-api-inference-extension/tree/main/config/charts/inferencepool). An EPP is required per InferencePool.
187
+
188
+
The Inference Workload Owner persona is someone that trains, verifies, and leverages a large language model from a model frontend, drives the lifecycle and rollout of new versions of those models, and defines the specific performance and latency goals for the model. These workloads are expected to operate within an InferencePool sharing compute capacity with other InferenceObjectives. The Inference Workload Owner would create their InferenceObjective resources for their model workloads.
189
+
190
+
Once all resources are created, the application developers can reference the InferencePool as a BackendRef in an HTTPRoute. This will trigger NGINX Gateway Fabric to configure the specialized routing for that pathway involving the AI workloads in the InferencePool.
191
+
192
+
## Testing
193
+
194
+
The Gateway API Inference Extension repository provides [conformance tests](https://github.com/kubernetes-sigs/gateway-api-inference-extension/tree/main/conformance) that we should be able to integrate with, similar to our existing conformance suite. We'll need to evaluate if these tests are sufficient at covering all cases. If not, we'll likely need to add some functional tests as well.
195
+
196
+
At a high level, testing should ensure that traffic can be routed from a client, through NGINX, to the proper inference workload, based on the contents of the request.
197
+
198
+
For development purposes, the [Getting started guide](https://gateway-api-inference-extension.sigs.k8s.io/guides/) explains how to deploy some sample model servers.
199
+
200
+
## Security Considerations
201
+
202
+
If the Endpoint Picker (EPP) supports it, we should use a secure TLS connection. This ensures an encrypted and authenticated communication channel between the NGINX data plane and the EPP. For production environments, an integration with `cert-manager` is likely the best solution, as we recommend this for various other secure channels within the NGF ecosystem. Otherwise, our control plane may have to provision certificates in the default case (similar to NGF's startup `cert-generator` Job).
203
+
204
+
At some point, there may be opportunities for attaching Policies (like a BackendTLSPolicy) to an InferenceModel to secure the NGINX -> AI workload connection, however that is not in scope for now.
205
+
206
+
## Alternatives
207
+
208
+
Instead of building an NJS and Go connector to communicate with the EPP, we could build everything natively into NGINX using a [Rust module](https://github.com/nginx/ngx-rust). [nginx-acme](https://github.com/nginx/nginx-acme) is an example NGINX module that was built using Rust.
209
+
210
+
Advantages:
211
+
212
+
- Native integration without the need for an extra container, multiple steps, and overhead.
213
+
- Learn some Rust and writing NGINX modules!
214
+
215
+
Disadvantages:
216
+
217
+
- Learning curve.
218
+
- Would need a custom build of NGINX OSS/Plus to support Rust bindings.
219
+
220
+
If this Inference Extension feature gains traction and usage, it could be worth pursuing this as an improvement.
221
+
222
+
## Future work
223
+
224
+
- Use a Rust module, as described in the [Alternatives section](#alternatives).
225
+
- Integrate with [F5 AI Gateway](https://www.f5.com/products/ai-gateway). AI Gateway could offer a number of beneficial features that we could take advantage of for the request lifecycle of AI workloads, especially around security. Ideally, we could utilize the AI Gateway's "processors" in some way without the need for deploying the entire product. This could be dependent on architectural changes by AI Gateway itself. Policy CRDs could be defined to allow users configure various security and other processing settings for this integration. If pursued, this will require a dedicated design doc.
226
+
- Instead of `proxy_pass` directly to the endpoint returned by the EPP, we find a way to proxy to that endpoint via the upstream block that its contained in (which is the same way that we proxy to backends today). The upstream block could contain multiple endpoints for the pool, but we still only want to route to the specific endpoint that the EPP gave us. Maybe `sticky route` could be used to ensure we send to that endpoint. The benefits of routing to the upstream block versus directly to the endpoint include more robust metrics (NGINX Plus) and the ability to utilize policies (like BackendTLSPolicy) for the upstream as a whole.
227
+
228
+
## References
229
+
230
+
- [Gateway API Inference Extension](https://gateway-api-inference-extension.sigs.k8s.io/)
231
+
- [Gateway API Inference Extension Github](https://github.com/kubernetes-sigs/gateway-api-inference-extension/tree/main)
0 commit comments