Adds vLLM Simulator Support (#898)

* Adds vLLM Simulator Support

Signed-off-by: Daneyon Hansen <[email protected]>

* Resolves Nir review feedback

Signed-off-by: Daneyon Hansen <[email protected]>

---------

Signed-off-by: Daneyon Hansen <[email protected]>

Author: danehans

Makefile

@@ -35,7 +35,9 @@ IMAGE_NAME := epp
 IMAGE_REPO ?= $(IMAGE_REGISTRY)/$(IMAGE_NAME)
 IMAGE_TAG ?= $(IMAGE_REPO):$(GIT_TAG)
 PROJECT_DIR := $(shell dirname $(abspath $(lastword $(MAKEFILE_LIST))))
-E2E_MANIFEST_PATH ?= config/manifests/vllm/gpu-deployment.yaml
+# The path to the E2E manifest file. It can be overridden by setting the
+# E2E_MANIFEST_PATH environment variable. Note that HF_TOKEN must be set when using the GPU-based manifest.
+E2E_MANIFEST_PATH ?= config/manifests/vllm/sim-deployment.yaml
 
 SYNCER_IMAGE_NAME := lora-syncer
 SYNCER_IMAGE_REPO ?= $(IMAGE_REGISTRY)/$(SYNCER_IMAGE_NAME)

config/manifests/vllm/sim-deployment.yaml

@@ -0,0 +1,43 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: vllm-llama3-8b-instruct
+spec:
+  replicas: 3
+  selector:
+    matchLabels:
+      app: vllm-llama3-8b-instruct
+  template:
+    metadata:
+      labels:
+        app: vllm-llama3-8b-instruct
+    spec:
+      containers:
+      - name: vllm-sim
+        image: ghcr.io/llm-d/llm-d-inference-sim:v0.1.0
+        imagePullPolicy: Always
+        args:
+        - --model
+        - meta-llama/Llama-3.1-8B-Instruct
+        - --port
+        - "8000"
+        - --max-loras
+        - "2"
+        - --lora
+        - food-review-1
+        env:
+        - name: POD_NAME
+          valueFrom:
+            fieldRef:
+              fieldPath: metadata.name
+        - name: NAMESPACE
+          valueFrom:
+            fieldRef:
+              fieldPath: metadata.namespace
+        ports:
+        - containerPort: 8000
+          name: http
+          protocol: TCP
+        resources:
+          requests:
+            cpu: 10m

site-src/guides/index.md

@@ -18,22 +18,26 @@ This quickstart guide is intended for engineers familiar with k8s and model serv
 
 ### Deploy Sample Model Server
 
-   Two options are supported for running the model server:
+   Three options are supported for running the model server:
 
-   1. GPU-based model server.  
+   1. GPU-based model server.
       Requirements: a Hugging Face access token that grants access to the model [meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct).
 
-   1. CPU-based model server (not using GPUs).  
-      The sample uses the model [Qwen/Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct).  
+   1. CPU-based model server (not using GPUs).
+      The sample uses the model [Qwen/Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct).
 
-   Choose one of these options and follow the steps below. Please do not deploy both, as the deployments have the same name and will override each other.
+   1. [vLLM Simulator](https://github.com/llm-d/llm-d-inference-sim/tree/main) model server (not using GPUs).
+      The sample is configured to simulate the [meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) model.
+
+   Choose one of these options and follow the steps below. Please do not deploy more than one, as the deployments have the same name and will override each other.
 
 === "GPU-Based Model Server"
 
       For this setup, you will need 3 GPUs to run the sample model server. Adjust the number of replicas in `./config/manifests/vllm/gpu-deployment.yaml` as needed.
       Create a Hugging Face secret to download the model [meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct). Ensure that the token grants access to this model.
       
       Deploy a sample vLLM deployment with the proper protocol to work with the LLM Instance Gateway.
+
       ```bash
       kubectl create secret generic hf-token --from-literal=token=$HF_TOKEN # Your Hugging Face Token with access to the set of Llama models
       kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/vllm/gpu-deployment.yaml
@@ -49,10 +53,22 @@ This quickstart guide is intended for engineers familiar with k8s and model serv
       After running multiple configurations of these values we decided in this sample to use 9.5GB of memory and 12 CPUs for each replica, which gives reasonable response times. You can increase those numbers and potentially may even get better response times. For modifying the allocated resources, adjust the numbers in [cpu-deployment.yaml](https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/vllm/cpu-deployment.yaml) as needed.  
 
       Deploy a sample vLLM deployment with the proper protocol to work with the LLM Instance Gateway.
+
       ```bash
       kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/vllm/cpu-deployment.yaml
       ```
 
+=== "vLLM Simulator Model Server"
+
+      This option uses the [vLLM simulator](https://github.com/llm-d/llm-d-inference-sim/tree/main) to simulate a backend model server.
+      This setup uses the least amount of compute resources, does not require GPU's, and is ideal for test/dev environments.
+
+      To deploy the vLLM simulator, run the following command.
+
+      ```bash
+      kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/vllm/sim-deployment.yaml
+      ```
+
 ### Install the Inference Extension CRDs
 
 === "Latest Release"

test/e2e/epp/README.md

@@ -10,7 +10,13 @@ The end-to-end tests are designed to validate end-to-end Gateway API Inference E
 
 - [Go](https://golang.org/doc/install) installed on your machine.
 - [Make](https://www.gnu.org/software/make/manual/make.html) installed to run the end-to-end test target.
-- A Hugging Face Hub token with access to the [meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) model.
+- (Optional) When using the GPU-based vLLM deployment, a Hugging Face Hub token with access to the
+  [meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) model is required.
+  After obtaining the token and being granted access to the model, set the `HF_TOKEN` environment variable:
+
+   ```sh
+   export HF_TOKEN=<MY_HF_TOKEN>
+   ```
 
 ## Running the End-to-End Tests
 
@@ -22,24 +28,28 @@ Follow these steps to run the end-to-end tests:
    git clone https://github.com/kubernetes-sigs/gateway-api-inference-extension.git && cd gateway-api-inference-extension
    ```
 
-1. **Export Your Hugging Face Hub Token**: The token is required to run the test model server:
+1. **Optional Settings**
 
-   ```sh
-   export HF_TOKEN=<MY_HF_TOKEN>
-   ```
+   - **Set the test namespace**: By default, the e2e test creates resources in the `inf-ext-e2e` namespace.
+     If you would like to change this namespace, set the following environment variable:
 
-1. **(Optional): Set the test namespace**: By default, the e2e test creates resources in the `inf-ext-e2e` namespace.
-   If you would like to change this namespace, set the following environment variable:
+     ```sh
+     export E2E_NS=<MY_NS>
+     ```
 
-   ```sh
-   export E2E_NS=<MY_NS>
-   ```
+   - **Set the model server manifest**: By default, the e2e test uses the [vLLM Simulator](https://github.com/llm-d/llm-d-inference-sim)
+     (`config/manifests/vllm/sim-deployment.yaml`) to simulate a backend model server. If you would like to change the model server
+     deployment type, set the following environment variable to one of the following:
+
+     ```sh
+     export E2E_MANIFEST_PATH=[config/manifests/vllm/gpu-deployment.yaml|config/manifests/vllm/cpu-deployment.yaml]
+     ```
 
 1. **Run the Tests**: Run the `test-e2e` target:
 
    ```sh
    make test-e2e
    ```
 
-   The test suite prints details for each step. Note that the `vllm-llama3-8b-instruct-pool` model server deployment
-   may take several minutes to report an `Available=True` status due to the time required for bootstraping.
+   The test suite prints details for each step. Note that the `vllm-llama3-8b-instruct` model server deployment
+   may take several minutes to report an `Available=True` status due to the time required for bootstrapping.