How do I increase the pull timeout in a Kube Quadlet?

[p5]

I am running some services inside Quadlets defined with .kube and YAML files, but am running into issues where startup times out before the images are pulled from their registries.

Even after setting TimeoutStartSec inside the Kube's service definition, Podman is still only allowing 5 minutes to pull the images, which isn't always enough.

Is there some way I am missing to extend this timeout to something like 15m?

Log entry:

 Pulling image //ghcr.io/open-webui/open-webui:latest inside systemd: setting pull timeout to 5m0s

Kube definition:

[Unit]
Description=An Ollama V2 deployment

[Kube]
Yaml=./kube/ollama.yaml

[Service]
TimeoutStartSec=900

[Install]
WantedBy=multi-user.target

Kube YAML:

apiVersion: v1
kind: Pod
metadata:
  labels:
    app: ollama
  name: ollama
spec:
  containers:
  - api:
    name: ollama-api
    image: docker.io/ollama/ollama:latest
    volumeMounts:
    - name: ollama-models
      mountPath: /root/.ollama
    ports:
    - containerPort: 11434
      hostPort: 11434
    resources:
      limits:
        nvidia.com/gpu: 1 # requesting 1 GPU
  - web:
    name: ollama-web
    image: ghcr.io/open-webui/open-webui:latest
    ports:
    - containerPort: 8080
      hostPort: 8080
  volumes:
  - name: ollama-models
    emptyDir: {}

I appreciate your assistance, and look forward to playing with this some more!

[ygalblum]

I'm reading the code here: https://github.com/containers/common/blob/main/libimage/copier.go#L358 and I remember the discussions around it. But, I can't seem to remember (or understand) if this code extends the timeout set in TimeoutStartSec by 5 minutes or if it basically sets the service timeout as such regardless to the original value. @vrothberg Do you remember? In addition, why did we use a these hard coded values?

[Luap99]

https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#TimeoutStartSec=

If a service of Type=notify/Type=notify-reload sends "EXTEND_TIMEOUT_USEC=…", this may cause the start time to be extended beyond TimeoutStartSec=. The first receipt of this message must occur before TimeoutStartSec= is exceeded, and once the start time has extended beyond TimeoutStartSec=, the service manager will allow the service to continue to start, provided the service repeats "EXTEND_TIMEOUT_USEC=…" within the interval specified until the service startup status is finished by "READY=1". (see sd_notify(3)).

I haven't tested it of course but to me this very much read like it should always be at least TimeoutStartSec

[ygalblum]

I haven't tested it either, but since the code starts sending the extend message after 25 seconds and from provided the service repeats "EXTEND_TIMEOUT_USEC=…" within the interval specified until the service startup status is finished, I don't understand the impact of these messages if TimeoutStartSec>300 (the total extension time)

[ygalblum]

@p5 on a side note and mostly out of curiosity. Looking at the pod definition, I assume you expect access only to the UI, right? If so, should you expose the Ollama port to the host?

[p5]

Hey!

Apologies if I am misunderstanding, but the ports defined in the YAML do the same as defining them in the .kube file, so in this case, I am exposing the API (the thing that hosts and exposes the models) on port 11434, and the Web UI on 8080.
Both of these are accessible to users on the host OS however I am not exposing the services to other devices on the network.

You may be referring to the lack of the OLLAMA_BASE_URL environment variable which actually tells the Web UI where to call to access the models, and in this case, you are correct.

[p5]

Oh, you're saying the API shouldn't be exposed at all if it's expected to be used only from the Web UI.

If this were a service exposed to others, I'd agree, but my particular use-case is this will be running on a development machine where I would be learning how to RAG and all the other fancy AI/LLM stuff, so I will be connecting directly to Ollama through something like Langchain on the host.

This is running as a user-scoped service in a rootless container without being exposed to other machines on the network, so in my (non-expert) opinion, it is safe enough from a security standpoint.