Pods not coming up for CPU deployment

[rahulgurnani]

What happened:

Pods not coming up when for CPU deployment:

$ k get pods
NAME                                       READY   STATUS     RESTARTS   AGE
vllm-llama3-8b-instruct-586778f79c-hb4vp   0/1     Init:0/1   0          6m15s
vllm-llama3-8b-instruct-586778f79c-qjk94   0/1     Init:0/1   0          6m15s
vllm-llama3-8b-instruct-586778f79c-w6pjh   0/1     Init:0/1   0          6m15s

Upon looking further, it seems the container lora is not starting:

$ kubectl logs vllm-llama3-8b-instruct-586778f79c-hb4vp
Defaulted container "lora" out of: lora, lora-adapter-syncer (init)
Error from server (BadRequest): container "lora" in pod "vllm-llama3-8b-instruct-586778f79c-7gmsb" is waiting to start: PodInitializing

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Try steps for CPU deployment from getting started guide: https://gateway-api-inference-extension.sigs.k8s.io/guides/

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):
• Inference extension version (use git describe --tags --dirty --always):
• Cloud provider or hardware configuration:
• Install tools:
• Others:

[nirrozenbaum]

which version was used for the cpu image?
can you give it a try with v0.8.5 as you did in #1526 PR?

[rahulgurnani]

Yes @nirrozenbaum , that version of vlllm is working. I created the bug so that we could track it going forward if need to make some updates on our end. I don't think pinning a version of vllm is a sustainable solution. What do you think?

[rahulgurnani]

The lora-syncer container is an init container and is part of IGW codebase , tools/dynamic-lora-sidecar does the loading of lora adapter dynamically. I tried with deleting the CPU deployment on a running cluster and then redeploying with the latest vllm image. The CPU deployment succeded and I pods did come up. So this seems to happen only on newly created cluster. I will add more investigation notes to this bug.

[nirrozenbaum]

Yes @nirrozenbaum , that version of vlllm is working. I created the bug so that we could track it going forward if need to make some updates on our end. I don't think pinning a version of vllm is a sustainable solution. What do you think?

pinning to a version is not a long term solution.
we should keep an eye and check newer versions of vllm as those are released.
we hace issue #1501 to track it from the GPU images. I assume we can track CPU images as well (and I expect it to get solved for both in the same vllm release).
maybe for now you can just add a comment with the relevant information on that issue.

[rahulgurnani]

Adding logs from the lora-adapter-syncer container:

rahulgurnani@rahulgurnani3:~/gateway-api-inference-extension$ kubectl logs vllm-llama3-8b-instruct-586778f79c-4m87j -c lora-adapter-syncer
2025-09-05 18:18:06 - INFO - sidecar.py:118 -  Settings initialized: health check timeout=300s, interval=2s, reconcile trigger=5s
2025-09-05 18:18:06 - INFO - sidecar.py:332 -  Starting metrics server on port 8080
2025-09-05 18:18:06 - INFO - sidecar.py:335 -  Running initial reconcile for config map /config/configmap.yaml
2025-09-05 18:18:06 - INFO - sidecar.py:281 -  reconciling model server localhost:8000 with config stored at /config/configmap.yaml
2025-09-05 18:23:07 - ERROR - sidecar.py:286 -  vllm server at localhost:8000 not healthy
2025-09-05 18:23:07 - INFO - sidecar.py:346 -  Starting to watch /config/configmap.yaml for changes and performing periodic reconciliation...
2025-09-05 18:23:07 - INFO - sidecar.py:350 -  Waiting 5s before next reconciliation...
2025-09-05 18:23:12 - INFO - sidecar.py:354 -  Periodic reconciliation triggered
2025-09-05 18:23:12 - INFO - sidecar.py:281 -  reconciling model server localhost:8000 with config stored at /config/configmap.yaml
2025-09-05 18:28:12 - ERROR - sidecar.py:286 -  vllm server at localhost:8000 not healthy
2025-09-05 18:28:12 - INFO - sidecar.py:350 -  Waiting 5s before next reconciliation...
2025-09-05 18:28:17 - INFO - sidecar.py:354 -  Periodic reconciliation triggered
2025-09-05 18:28:17 - INFO - sidecar.py:281 -  reconciling model server localhost:8000 with config stored at /config/configmap.yaml
2025-09-05 18:33:18 - ERROR - sidecar.py:286 -  vllm server at localhost:8000 not healthy
2025-09-05 18:33:18 - INFO - sidecar.py:350 -  Waiting 5s before next reconciliation...
2025-09-05 18:33:23 - INFO - sidecar.py:354 -  Periodic reconciliation triggered
2025-09-05 18:33:23 - INFO - sidecar.py:281 -  reconciling model server localhost:8000 with config stored at /config/configmap.yaml

It seems the container is not coming up because its waiting for vllm server at localhost:8000 to be healthy.

[rahulgurnani]

More investigation notes here, deploying with vllm latest without the lora syncer init container succeeds i.e. using the following CPU deployment succeeds:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama3-8b-instruct
spec:
  replicas: 3
  selector:
    matchLabels:
      app: vllm-llama3-8b-instruct
  template:
    metadata:
      labels:
        app: vllm-llama3-8b-instruct
    spec:
      containers:
        - name: lora
          image: "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.10.1" # formal images can be found in https://gallery.ecr.aws/q9t5s3a7/vllm-cpu-release-repo
          imagePullPolicy: IfNotPresent
          command: ["python3", "-m", "vllm.entrypoints.openai.api_server"]
          args:
          - "--model"
          - "Qwen/Qwen2.5-1.5B-Instruct"
          - "--port"
          - "8000"
          - "--enable-lora"
          - "--max-loras"
          - "4"
          - "--lora-modules"
          - '{"name": "food-review-0", "path": "SriSanth2345/Qwen-1.5B-Tweet-Generations", "base_model_name": "Qwen/Qwen2.5-1.5B"}'
          - '{"name": "food-review-1", "path": "SriSanth2345/Qwen-1.5B-Tweet-Generations", "base_model_name": "Qwen/Qwen2.5-1.5B"}'
          env:
            - name: PORT
              value: "8000"
            - name: VLLM_ALLOW_RUNTIME_LORA_UPDATING
              value: "true"
            - name: VLLM_CPU_KVCACHE_SPACE
              value: "4"
          ports:
            - containerPort: 8000
              name: http
              protocol: TCP
          livenessProbe:
            failureThreshold: 240
            httpGet:
              path: /health
              port: http
              scheme: HTTP
            initialDelaySeconds: 5
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 600
            httpGet:
              path: /health
              port: http
              scheme: HTTP
            initialDelaySeconds: 5
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 1
          resources:
             limits:
               cpu: "12"
               memory: "9000Mi"
             requests:
               cpu: "12"
               memory: "9000Mi"
          volumeMounts:
            - mountPath: /data
              name: data
            - mountPath: /dev/shm
              name: shm
            - name: adapters
              mountPath: "/adapters"
      restartPolicy: Always
      schedulerName: default-scheduler
      terminationGracePeriodSeconds: 30
      volumes:
        - name: data
          emptyDir: {}
        - name: shm
          emptyDir:
            medium: Memory
        - name: adapters
          emptyDir: {}
        - name: config-volume

[nirrozenbaum]

@rahulgurnani that's a very interesting finding.
I assume the same is correct for GPU deployment, and more generally sounds like the bug in on our side (in lora syncer) and not in vllm images.

I'm guessing that something in the API has changed after vllm version 0.8.5, and then lora-syncer is not working properly with that.

I think it might be worth opening a dedicate issue (which should remain open) to describe the issues you're seeing.
we should track this in an issue and close the issue when the problem is fixed.