For the validator running this on a real quota-enforcing namespace (OpenShift dev cluster included).
Branch 1005-runtime-resources, based on 616778fe.
What you are proving: the two Pods the runtime creates dynamically — the meeting bot and the agent worker — now declare CPU and memory requests and limits, so your namespace admits them instead of rejecting them. Nothing else about the release changes.
What has already been proven, and where — so you can skip re-doing it:
| Already green | Where |
|---|---|
Both profiles admitted under a ResourceQuota requiring all four values, sized independently |
k3d v1.35.5, namespace vexa-quota — see OBSERVATION-BUNDLE-1005.md A1/A2 |
| Image, command, env, labels, tolerations, nodeSelector, workspace mounts all survive | same, A3 |
| Removing one dimension reds admission; restoring it greens the same spawn | same, A4 |
| Offline runtime suite, Helm render, Compose stack, all 34 repo gates | same, A- |
What only your cluster can prove: OpenShift SCC behaviour on the spawned Pods, your real quota headroom, and whether the real bot image lives inside the memory ceiling you set. Read the honest caveats before you start — one of them is likely to bite.
Two values blocks matter. The first is the new one.
# values-quota.yaml
runtime:
backend: k8s
# Sizing for the Pods the runtime SPAWNS. NOT runtime.resources (which sizes the runtime
# Deployment itself). One value per dimension sets BOTH the request and the limit.
workloadResources:
meetingBot:
cpu: 2 # → requests/limits cpu: 2000m
memoryMb: 4096 # → requests/limits memory: 4096Mi
agentWorker:
cpu: 0.5 # → requests/limits cpu: 500m
memoryMb: 1024 # → requests/limits memory: 1024Mihelm upgrade --install vexa deploy/helm/charts/vexa \
-n <your-namespace> --create-namespace \
-f values-quota.yaml \
--set global.imageTag=<TAG> \
--set secrets.adminApiToken=$ADMIN_TOKEN \
--set secrets.internalApiSecret=$INTERNAL_API_SECRETConfirm the sizing reached the runtime before you spawn anything:
kubectl -n <your-namespace> get deploy vexa-runtime -o json \
| jq '.spec.template.spec.containers[0].env[]
| select(.name|test("^RUNTIME_(BOT|AGENT_WORKER)_"))'Expected — four entries, non-empty:
{"name":"RUNTIME_BOT_CPU","value":"2"}
{"name":"RUNTIME_BOT_MEMORY_MB","value":"4096"}
{"name":"RUNTIME_AGENT_WORKER_CPU","value":"0.5"}
{"name":"RUNTIME_AGENT_WORKER_MEMORY_MB","value":"1024"}If they are empty strings, the chart values did not land — the spawned Pods will declare nothing and your namespace will reject them exactly as before. Fix this before going further.
Sizing note, not a formality.
memoryMbis a hard ceiling: a workload exceeding it is OOM-killed. Before this change nothing was enforced, so bots ran unbounded and we have no measured figure to hand you.4096above is deliberately more generous than the chart default (2048). Start high, watch actual usage, then tighten — do not start at the default and discover the ceiling as a dropped meeting.
Trigger one meeting bot (your normal POST /bots path) and one agent dispatch. Then, for each:
kubectl -n <your-namespace> get pods -l runtime.managed=true
kubectl -n <your-namespace> get pod <pod> -o json | jq '{
qos: .status.qosClass,
phase: .status.phase,
resources: .spec.containers[0].resources,
image: .spec.containers[0].image,
command: .spec.containers[0].command,
envCount: (.spec.containers[0].env|length),
labels: .metadata.labels
}'| Pod | Name shape | Expect |
|---|---|---|
| meeting bot | vexa-<meeting-workload-id> |
resources.requests and .limits = {cpu: 2, memory: 4096Mi}; qosClass: Guaranteed; phase: Running; no command (the bot image's own ENTRYPOINT boots it); labels carry runtime.managed=true + runtime.workload_id |
| agent worker | vexa-agent-… |
resources = {cpu: 500m, memory: 1Gi} — different from the bot; qosClass: Guaranteed; command: ["python","-m","worker"] |
2000m/500m display as 2/500m — the API server normalizes; either spelling is the same value.
Quota consumption should move:
kubectl -n <your-namespace> describe resourcequotaBoth classes should appear in requests.cpu / limits.memory usage while running, and drop back when
the workloads finish.
Proves the quota is really enforcing and that the values came from Vexa, not from a LimitRange:
kubectl -n <your-namespace> run quota-probe --image=busybox:1.36 --restart=Never -- sleep 30Expected: rejected — must specify limits.cpu … requests.memory …. If this succeeds, a
LimitRange is defaulting your namespace, and per the issue's closing rule a green run there does not
prove Vexa emitted the resources. Re-check with the jq above: the values must equal what you set in
values-quota.yaml, not the LimitRange defaults.
Please attach all of it — partial reports cost a round trip.
NS=<your-namespace>; POD=<the failing pod>
# 1. Did admission reject the spawn? The runtime logs the API server's own reason verbatim.
kubectl -n $NS logs deploy/vexa-runtime --tail=200 | grep -iE "kubectl create|forbidden|StartFailed"
# 2. Namespace events — quota denials, SCC denials, scheduling failures all land here
kubectl -n $NS get events --sort-by=.lastTimestamp | tail -40
# 3. The Pod as the API server accepted it (or the whole object if it never got created)
kubectl -n $NS get pod $POD -o yaml
# 4. Why it is not running
kubectl -n $NS describe pod $POD
# 5. What the workload itself said
kubectl -n $NS logs $POD --previous --tail=200 || kubectl -n $NS logs $POD --tail=200
# 6. Quota state at the time
kubectl -n $NS describe resourcequota
kubectl -n $NS get limitrange -o yaml # if any — see the LimitRange note aboveReading the three failure shapes:
| Symptom | Almost certainly |
|---|---|
No Pod at all; runtime log says forbidden: failed quota: … must specify … |
the sizing env did not reach the runtime, or one dimension is empty — recheck §1 |
No Pod; message mentions unable to validate against any security context constraint |
SCC, not quota — see caveats below |
Pod exists, Pending, events say Insufficient cpu/memory |
your quota or node headroom is smaller than the size you set — lower cpu/memoryMb or raise the quota |
Pod runs then dies with OOMKilled (exit 137) |
memoryMb is too low for the real workload — raise it |
1 · Spawned-Pod SCC on OpenShift is untested — this is the most likely surprise.
You proved the control plane satisfies your SCC. The bot and agent Pods are different objects:
bare Pods created by the runtime at spawn time, not children of any Deployment, so they inherit
nothing from the control-plane pod templates. A restricted-v2 SCC may additionally require
runAsNonRoot, a dropped capability set, or a seccompProfile on those Pods — this change adds
none of those, and no OpenShift cluster was available to test it. If §2 fails with an SCC message
rather than a quota message, that is a separate, known-open gap: capture the event text and the SCC
name (kubectl -n $NS get pod $POD -o jsonpath='{.metadata.annotations.openshift\.io/scc}') and send
it — it is a follow-up issue, not a defect in this one.
2 · The memory ceiling is a guess until you measure it. No live meeting has run under an enforced
limit. Set generously, observe (kubectl top pod), then tighten.
3 · Enforcement is Kubernetes-only. The docker and process backends accept the same resource intent and do not act on it. Compose and Lite deployments are unchanged; no parity is claimed.
4 · Guaranteed QoS changes scheduling density. Request equals limit by design (runtime.v1 carries
one value per dimension). Reserved capacity is now real — a node that used to fit N unbounded bots
fits floor(allocatable / size) of them.
5 · What the k3d validation substituted. The live legs in the bundle ran busybox:1.36 in place
of the bot and worker images, so the declaration and admission path is proven against a real API
server but the real images were never run inside the limits. Your run is the first that does that —
which is exactly why it is the one that closes the issue.
Re-runnable against any cluster; drop them in a scratch dir.
quota-ns.yaml
apiVersion: v1
kind: Namespace
metadata:
name: vexa-quota
---
apiVersion: v1
kind: ResourceQuota
metadata:
name: require-requests-and-limits
namespace: vexa-quota
spec:
hard:
requests.cpu: "4"
requests.memory: 4Gi
limits.cpu: "4"
limits.memory: 4Gi
pods: "10"quota-validate.py — spawns both profiles through the real kernel
Run from core/runtime as
PYTHONPATH=src uv run python quota-validate.py {green|red-unsized|red-no-memory}.
import json, os, subprocess, sys, time
NS = "vexa-quota"
def kubectl(*args, stdin=None, check=True):
r = subprocess.run(["kubectl", *args], capture_output=True, text=True, input=stdin)
if check and r.returncode != 0:
raise SystemExit(f"kubectl {' '.join(args)} FAILED:\n{r.stderr}")
return r
def pod_json(name):
return json.loads(kubectl("get", "pod", name, "-n", NS, "-o", "json").stdout)
def spawn(profile_name, workload_id):
from runtime_kernel import Runtime
from runtime_kernel.k8s_backend import K8sBackend
from runtime_kernel.models import WorkloadSpec
from runtime_kernel.profiles import apply_command_overrides, default_registry
rt = Runtime(backend=K8sBackend(namespace=NS),
profiles=apply_command_overrides(default_registry()), grace_sec=5.0)
return rt, rt.create(WorkloadSpec(workloadId=workload_id, profile=profile_name,
env={"VEXA_X": "y"}))
def main():
os.environ["BROWSER_IMAGE"] = os.environ["AGENT_WORKER_IMAGE"] = "busybox:1.36"
os.environ["AGENT_IMAGE"] = "busybox:1.36"
os.environ["AGENT_WORKER_COMMAND"] = os.environ["BOT_COMMAND"] = "sleep 120"
mode = sys.argv[1] if len(sys.argv) > 1 else "green"
if mode == "green":
os.environ["RUNTIME_BOT_CPU"] = "1"; os.environ["RUNTIME_BOT_MEMORY_MB"] = "512"
os.environ["RUNTIME_AGENT_WORKER_CPU"] = "0.25"; os.environ["RUNTIME_AGENT_WORKER_MEMORY_MB"] = "256"
elif mode == "red-no-memory":
os.environ["RUNTIME_BOT_CPU"] = "1"; os.environ.pop("RUNTIME_BOT_MEMORY_MB", None)
os.environ["RUNTIME_AGENT_WORKER_CPU"] = "0.25"; os.environ.pop("RUNTIME_AGENT_WORKER_MEMORY_MB", None)
elif mode == "red-unsized":
for k in ("RUNTIME_BOT_CPU", "RUNTIME_BOT_MEMORY_MB",
"RUNTIME_AGENT_WORKER_CPU", "RUNTIME_AGENT_WORKER_MEMORY_MB"):
os.environ.pop(k, None)
print(f"=== MODE: {mode} ===")
for profile, wid in (("meeting-bot", f"mtg-{mode}"), ("agent", f"agent-{mode}")):
kubectl("delete", "pod", f"vexa-{wid}", "-n", NS, "--ignore-not-found",
"--grace-period=0", "--force", check=False)
print(f"\n--- profile={profile} workloadId={wid} ---")
try:
rt, status = spawn(profile, wid)
except Exception as exc:
print(f"SPAWN REJECTED: {type(exc).__name__}: {str(exc)[:600]}")
continue
print(f"kernel state: {status.state.value}")
c = pod_json(f"vexa-{wid}")["spec"]["containers"][0]
print("ADMITTED. container.resources =", json.dumps(c.get("resources", {})))
print("image =", c["image"], "| command =", c.get("command"))
deadline, phase = time.time() + 60, ""
while time.time() < deadline:
phase = pod_json(f"vexa-{wid}")["status"]["phase"]
if phase == "Running":
break
time.sleep(2)
print("phase =", phase, "| qosClass =", pod_json(f"vexa-{wid}")["status"].get("qosClass"))
rt.stop(wid); rt.destroy(wid)
print("terminal:", rt.get(wid).state.value)
if __name__ == "__main__":
main()a3-live.py — proves no generated field is erased
Creates a small PVC, spawns one Pod carrying resources + tolerations + nodeSelector + a two-mount workspace set, and prints what the API server accepted. Full source in the session scratch; the essential call is:
K8sBackend(namespace=NS).start(
wid, Runnable(image="busybox:1.36", command=["sleep", "120"]), env,
Resources(cpu=0.5, memoryMb=256),
)with RUNTIME_K8S_TOLERATIONS / RUNTIME_K8S_NODE_SELECTOR set in the process env and
VEXA_WORKSPACE_MOUNT_SOURCE / _TARGET / VEXA_MOUNTS in the workload env. Then read the Pod back
and assert image, command, env, labels, restartPolicy, tolerations, nodeSelector,
volumes, volumeMounts and resources are all present on the accepted object.