Docs: Versions the quickstart guide

danehans · danehans · commit f9ee5315b693 · 2025-10-06T16:53:53.000-07:00
Signed-off-by: Daneyon Hansen &lt;daneyon.hansen@solo.io&gt;
diff --git a/mkdocs.yml b/mkdocs.yml
@@ -64,7 +64,8 @@ nav:
     - FAQ: faq.md
   - Guides:
     - User Guides:
-      - Getting started: guides/index.md
+      - Getting started (Released): guides/index.md
+      - Getting started (Latest/Main): guides/getting-started-latest.md
       - Use Cases:
         - Serve Multiple GenAI models: guides/serve-multiple-genai-models.md
       - Rollout:
diff --git a/site-src/_includes/bbr.md b/site-src/_includes/bbr.md
@@ -0,0 +1,3 @@
+### Deploy the Body Based Router Extension (Optional)
+
+This guide has shown how to get started with serving a single base model type per L7 URL path. If after this exercise, you wish to continue on to exercise model-aware routing such that more than 1 base model is served at the same L7 url path, that requires use of the (optional) Body Based Routing (BBR) extension which is described in a separate section of the documentation, namely the [`Serving Multiple GenAI Models`](serve-multiple-genai-models.md) section. If you wish to exercise that function, then retain the setup you have deployed so far from this guide and move on to the additional steps described in [that guide](serve-multiple-genai-models.md) or else move on to the following section to cleanup your setup. 
diff --git a/site-src/_includes/epp.md b/site-src/_includes/epp.md
@@ -0,0 +1,43 @@
+=== "GKE"
+
+      ```bash
+      export GATEWAY_PROVIDER=gke
+      helm install vllm-llama3-8b-instruct \
+      --set inferencePool.modelServers.matchLabels.app=vllm-llama3-8b-instruct \
+      --set provider.name=$GATEWAY_PROVIDER \
+      --version $IGW_CHART_VERSION \
+      oci://registry.k8s.io/gateway-api-inference-extension/charts/inferencepool
+      ```
+
+=== "Istio"
+
+      ```bash
+      export GATEWAY_PROVIDER=istio
+      helm install vllm-llama3-8b-instruct \
+      --set inferencePool.modelServers.matchLabels.app=vllm-llama3-8b-instruct \
+      --set provider.name=$GATEWAY_PROVIDER \
+      --version $IGW_CHART_VERSION \
+      oci://registry.k8s.io/gateway-api-inference-extension/charts/inferencepool
+      ```
+
+=== "Kgateway"
+
+      ```bash
+      export GATEWAY_PROVIDER=none
+      helm install vllm-llama3-8b-instruct \
+      --set inferencePool.modelServers.matchLabels.app=vllm-llama3-8b-instruct \
+      --set provider.name=$GATEWAY_PROVIDER \
+      --version $IGW_CHART_VERSION \
+      oci://registry.k8s.io/gateway-api-inference-extension/charts/inferencepool
+      ```
+
+=== "Agentgateway"
+
+      ```bash
+      export GATEWAY_PROVIDER=none
+      helm install vllm-llama3-8b-instruct \
+      --set inferencePool.modelServers.matchLabels.app=vllm-llama3-8b-instruct \
+      --set provider.name=$GATEWAY_PROVIDER \
+      --version $IGW_CHART_VERSION \
+      oci://registry.k8s.io/gateway-api-inference-extension/charts/inferencepool
+      ```
diff --git a/site-src/_includes/infobj.md b/site-src/_includes/infobj.md
@@ -0,0 +1,5 @@
+??? example "Experimental"
+
+    This project is still in an alpha state and breaking changes may occur in the future.
+
+This quickstart guide is intended for engineers familiar with k8s and model servers (vLLM in this instance). The goal of this guide is to get an Inference Gateway up and running!
diff --git a/site-src/_includes/intro.md b/site-src/_includes/intro.md
@@ -0,0 +1,5 @@
+??? example "Experimental"
+
+    This project is still in an alpha state and breaking changes may occur in the future.
+
+This quickstart guide is intended for engineers familiar with k8s and model servers (vLLM in this instance). The goal of this guide is to get an Inference Gateway up and running!
diff --git a/site-src/_includes/model-server-cpu.md b/site-src/_includes/model-server-cpu.md
@@ -0,0 +1,14 @@
+=== "CPU-Based Model Server"
+
+    ???+ warning
+
+        CPU deployment can be unreliable i.e. the pods may crash/restart because of resource contraints.
+
+    This setup is using the formal `vllm-cpu` image, which according to the documentation can run vLLM on x86 CPU platform.
+    For this setup, we use approximately 9.5GB of memory and 12 CPUs for each replica.
+
+    While it is possible to deploy the model server with less resources, this is not recommended. For example, in our tests, loading the model using 8GB of memory and 1 CPU was possible but took almost 3.5 minutes and inference requests took unreasonable time. In general, there is a tradeoff between the memory and CPU we allocate to our pods and the performance. The more memory and CPU we allocate the better performance we can get.
+
+    After running multiple configurations of these values we decided in this sample to use 9.5GB of memory and 12 CPUs for each replica, which gives reasonable response times. You can increase those numbers and potentially may even get better response times. For modifying the allocated resources, adjust the numbers in [cpu-deployment.yaml](https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/vllm/cpu-deployment.yaml) as needed.
+
+    Deploy a sample vLLM deployment with the proper protocol to work with the LLM Instance Gateway.
diff --git a/site-src/_includes/model-server-gpu.md b/site-src/_includes/model-server-gpu.md
@@ -0,0 +1,7 @@
+=== "GPU-Based Model Server"
+
+    For this setup, you will need 3 GPUs to run the sample model server. Adjust the number of replicas as needed.
+    Create a Hugging Face secret to download the model [meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct).
+    Ensure that the token grants access to this model.
+
+    Deploy a sample vLLM deployment with the proper protocol to work with the LLM Instance Gateway.
diff --git a/site-src/_includes/model-server-sim.md b/site-src/_includes/model-server-sim.md
@@ -0,0 +1,6 @@
+=== "vLLM Simulator Model Server"
+
+    This option uses the [vLLM simulator](https://github.com/llm-d/llm-d-inference-sim/tree/main) to simulate a backend model server.
+    This setup uses the least amount of compute resources, does not require GPU's, and is ideal for test/dev environments.
+
+    To deploy the vLLM simulator, run the following command.
diff --git a/site-src/_includes/model-server.md b/site-src/_includes/model-server.md
@@ -0,0 +1,19 @@
+   Three options are supported for running the model server:
+
+   1. GPU-based model server.
+      Requirements: a Hugging Face access token that grants access to the model [meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct).
+
+   1. CPU-based model server (not using GPUs).
+      The sample uses the model [Qwen/Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct).
+
+   1. [vLLM Simulator](https://github.com/llm-d/llm-d-inference-sim/tree/main) model server (not using GPUs).
+      The sample is configured to simulate the [meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) model.
+
+   Choose one of these options and follow the steps below. Please do not deploy more than one, as the deployments have the same name and will override each other.
+
+=== "GPU-Based Model Server"
+
+    For this setup, you will need 3 GPUs to run the sample model server. Adjust the number of replicas in `./config/manifests/vllm/gpu-deployment.yaml` as needed.
+    Create a Hugging Face secret to download the model [meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct). Ensure that the token grants access to this model.
+
+    Deploy a sample vLLM deployment with the proper protocol to work with the LLM Instance Gateway.
diff --git a/site-src/_includes/prereqs.md b/site-src/_includes/prereqs.md
@@ -0,0 +1,10 @@
+A cluster with:
+
+- Support for services of type `LoadBalancer`. For kind clusters, follow [this guide](https://kind.sigs.k8s.io/docs/user/loadbalancer)
+  to get services of type LoadBalancer working.
+- Support for [sidecar containers](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/) (enabled by default since Kubernetes v1.29)
+  to run the model server deployment.
+
+Tooling:
+
+- [Helm](https://helm.sh/docs/intro/install/) installed.
diff --git a/site-src/_includes/test.md b/site-src/_includes/test.md
@@ -0,0 +1,15 @@
+### Try it out
+
+   Wait until the gateway is ready.
+
+   ```bash
+   IP=$(kubectl get gateway/inference-gateway -o jsonpath='{.status.addresses[0].value}')
+   PORT=80
+
+   curl -i ${IP}:${PORT}/v1/completions -H 'Content-Type: application/json' -d '{
+   "model": "food-review-1",
+   "prompt": "Write as if you were a critic: San Francisco",
+   "max_tokens": 100,
+   "temperature": 0
+   }'
+   ```
diff --git a/site-src/guides/getting-started-latest.md b/site-src/guides/getting-started-latest.md
diff --git a/site-src/guides/index.md b/site-src/guides/index.md

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	`+### Deploy the Body Based Router Extension (Optional)`
	`2`	`+`
	`3`	+This guide has shown how to get started with serving a single base model type per L7 URL path. If after this exercise, you wish to continue on to exercise model-aware routing such that more than 1 base model is served at the same L7 url path, that requires use of the (optional) Body Based Routing (BBR) extension which is described in a separate section of the documentation, namely the [`Serving Multiple GenAI Models`](serve-multiple-genai-models.md) section. If you wish to exercise that function, then retain the setup you have deployed so far from this guide and move on to the additional steps described in [that guide](serve-multiple-genai-models.md) or else move on to the following section to cleanup your setup.