Investigate Inference Extension Support

[mpstefan]

Spike to investigate the Gateway API Inference Extension and estimate the amount of complexity and effort implementation would cost based on NGF's current architecture.

[sjberman]

From a high level, this doesn't seem like too much work for us to accomplish. There are a couple points of complexity that I'm unsure about, and the most complex work might be integrating the AI Gateway processors.

Per Michael's internal PoC demo, there are a couple different possible architectures.

1. NGF and AI Gateway work in tandem, with AI Gateway performing the model lookup, endpoint selection, and forwarding to the LLM backend. This is probably the easiest for us to get working, though from a user perspective it requires them to also deploy and configure AI Gateway.
2. Model lookup and endpoint selection built in to NGF. More work on the NGF side, but it reduces the user overhead of deploying both NGF and AI Gateway. We're more self-contained.
   2a. Same as 2, but includes the AI Gateway processors for added policy/security options for the LLM traffic. This is sort of in between options 1 and 2, because NGF does most of the work, but an external component (AI Gateway processors) still needs to be deployed separately.

Without getting too into design here, it seems like what we would need to do is something along the lines of:

• Feature flag to enable this
• Controller watches for InferencePools and InferenceModels and links them together
• Support configuring an HTTPRoute with InferencePool backend
• NJS module to inspect request body for model and return list of endpoints for that model
• NJS module and Go connector to pick specific endpoint to send to (using Endpoint Picker)
  • Another potential option is update the EPP load balancer to implement something that NGINX can deal with, such as https://nginx.org/en/docs/http/ngx_http_auth_request_module.html. Then we likely wouldn't need some of these extra pieces.

There's some unknown complexity around criticality and weight that I'm not totally sure how we implement yet.

Then there's the added complexity with deploying the AI Gateway processors and calling out to those, and defining the security policy CRDs.

[sjberman]

@pleshakov Any added context here that I might have missed?