Skip to content

Adding Kubvernor to the list of implementors #1313

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions config/manifests/gateway/kubvernor/gateway.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: kubvernor-inference-gateway
spec:
gatewayClassName: kubvernor-inference-gateway
listeners:
- name: http
port: 80
protocol: HTTP
20 changes: 20 additions & 0 deletions config/manifests/gateway/kubvernor/httproute.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: llm-route
spec:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: kubvernor-inference-gateway
rules:
- backendRefs:
- group: inference.networking.x-k8s.io
kind: InferencePool
name: vllm-llama3-8b-instruct
matches:
- path:
type: PathPrefix
value: /
timeouts:
request: 300s
36 changes: 36 additions & 0 deletions site-src/guides/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -244,6 +244,42 @@ This quickstart guide is intended for engineers familiar with k8s and model serv
kubectl get httproute llm-route -o yaml
```

=== "Kubvernor Rust API Gateway"

[Kubvernor Rust API Gateway](https://github.com/kubvernor/kubvernor) is a higly experimental project so not ready for production but it supports version v0.5.1 of Inference Extension Spec.

1. Requirements
- Rust and Cargo installed

2. Run Kubvernor Rust API Gateway as documented in [README](https://github.com/kubvernor/kubvernor/blob/main/README.md)


3. Deploy the Gateway

```bash
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/kubvernor/gateway.yaml
```

Confirm that the Gateway was assigned an IP address and reports a `Programmed=True` status:
```bash
$ kubectl get gateway kubvernor-inference-gateway
NAME CLASS ADDRESS PROGRAMMED AGE
kubvernor-inference-gateway kubvernor-inference-gateway <MY_ADDRESS> True 22s
```

5. Deploy the HTTPRoute

```bash
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/kubvernor/httproute.yaml
```

6. Confirm that the HTTPRoute status conditions include `Accepted=True` and `ResolvedRefs=True`:

```bash
kubectl get httproute llm-route -o yaml
```


### Try it out

Wait until the gateway is ready.
Expand Down
11 changes: 10 additions & 1 deletion site-src/implementations/gateways.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,14 @@ This project has several implementations that are planned or in progress:
* [Google Kubernetes Engine][3]
* [Istio][4]
* [Alibaba Cloud Container Service for Kubernetes][5]
* [Kubvernor Rust API Gateway][6]

[1]:#envoy-gateway
[2]:#kgateway
[3]:#google-kubernetes-engine
[4]:#istio
[5]:#alibaba-cloud-container-service-for-kubernetes
[6]:#kubernor-api-gateway

## Envoy AI Gateway

Expand Down Expand Up @@ -85,4 +87,11 @@ by [this Issue](https://github.com/AliyunContainerService/ack-gateway-api/issues

[ack]:https://www.alibabacloud.com/help/en/ack
[ack-gie]:https://www.alibabacloud.com/help/en/ack/product-overview/ack-gateway-with-inference-extension
[ack-gie-usage]:https://www.alibabacloud.com/help/en/ack/ack-managed-and-ack-dedicated/user-guide/intelligent-routing-and-traffic-management-with-ack-gateway-inference-extension
[ack-gie-usage]:https://www.alibabacloud.com/help/en/ack/ack-managed-and-ack-dedicated/user-guide/intelligent-routing-and-traffic-management-with-ack-gateway-inference-extension

## Kubvernor Rust API Gateway
[Kubvernor Rust API Gateway][krg] is an open-source, highly experimental implementation of API controller in Rust programming language. Currently, Kubernor supports Envoy Proxy. The project aims to be as generic as possible so Kubvernor can be used to manage/deploy different gateways (Envoy, Nginx, HAProxy, etc.). Kubvernor Rust API Gateway implements Inference Extensions v0.5.1.

[krg]:https://github.com/kubvernor/kubvernor