User session spawning times out - hook-image-puller permission error

[radhupr]

Bug description

User session spawning is timedout since the pod created for user is in pending state.
hook-image-awaiter pod ends up in error -

2025/06/19 08:49:36 [DEBUG] GET https://kubernetes.default.svc:443/apis/apps/v1/namespaces/default/daemonsets/hook-image-puller
2025/06/19 08:49:36 Can not parse API response as DaemonSet: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"daemonsets.apps \"hook-image-puller\" is forbidden: User \"system:serviceaccount:jupyterlab:hook-image-awaiter\" cannot get resource \"daemonsets\" in API group \"apps\" in the namespace \"default\"","reason":"Forbidden","details":{"name":"hook-image-puller","group":"apps","kind":"daemonsets"},"code":403}

Why is it trying to get resources from default namespace where as it should use the application namespace - here it should be jupyterlab

How to reproduce

Install helm chart with below values file
Login with Azure creds and launch user server
You can see the status of pods in cluster as below

Expected behaviour

User session launched without error

Actual behaviour

User session pod pending status. The events in pod doesnt show anything
Events: <none>

Your personal set up

Kosmotron standalon deployment - v1.32.3

Configuration

kustomization.yaml

namespace: jupyterlab
helmCharts:
  - name: jupyterhub
    repo: https://hub.jupyter.org/helm-chart/
    version: 4.1.0
    releaseName: jupyterhub
    valuesFile: values.yaml
    kubeVersion: 1.32.3

values.yaml

---
hub:
  extraEnv:
    OAUTH_CLIENT_ID:
      valueFrom:
        secretKeyRef:
          key: client-id
          name: workbench-azure-oauth
    OAUTH_CLIENT_SECRET:
      valueFrom:
        secretKeyRef:
          key: client-secret
          name: workbench-azure-oauth
  config:
    AzureAdOAuthenticator:
      oauth_callback_url: https://jupyterhub.company.com/hub/oauth_callback
      tenant_id: XX-XXX-XXXX
      allow_all: false
      admin_groups:
        - XX-XXX-XXXX  # r-pckube-contributor
      allowed_groups:
        - XX-XXX-XXXX  # Team Presales Analytics Engineering.
        - XX-XXX-XXXX  # r-pckube-contributor
      manage_groups: true
      scope: ["openid", "profile", "email", "User.Read", "GroupMember.Read.All"]
    JupyterHub:
      authenticator_class: azuread
  resources:
    limits:
      memory: 2Gi
    requests:
      cpu: 100m
      memory: 128Mi

proxy:
  chp:
    resources:
      limits:
        memory: 2Gi
      requests:
        cpu: 100m
        memory: 128Mi

  traefik:
    resources:
      limits:
        memory: 2Gi
      requests:
        cpu: 100m
        memory: 128Mi

singleuser:
  networkTools:
    resources:
      limits:
        cpu: 2
        memory: 2Gi
      requests:
        cpu: 10m
        memory: 100Mi
  allowPrivilegeEscalation: false
  cpu:
    limit: 2
    # guarantee: 0.1
  memory:
    limit: 2G
  uid: 1000
  fsGid: 100
  storage:
    type: dynamic
    static:
      pvcName:
      subPath: "{username}"
    capacity: 3Gi   # user home volume
    extraVolumes:
      - name: shared-data
        persistentVolumeClaim:
          claimName: shared-data-pvc
    extraVolumeMounts:
      - mountPath: /home/jovyan/shared
        name: shared-data
  image:
    # name: glcr.b-data.ch/jupyterlab/r/tidyverse
    # tag: "4.4.2"
    name: quay.io/bedata/jupyterlab/r/tidyverse
    tag: "4.5.0-devtools-docker-linux-amd64"
  cmd: start-singleuser.sh

# scheduling relates to the user-scheduler pods and user-placeholder pods.
scheduling:
  userScheduler:
    resources:
      limits:
        memory: 2Gi
      requests:
        cpu: 100m
        memory: 128Mi
  userPlaceholder:
    resources:
      limits:
        memory: 2Gi
      requests:
        cpu: 100m
        memory: 128Mi

# prePuller relates to the hook|continuous-image-puller DaemonsSets
prePuller:
  resources:
    limits:
      memory: 2Gi
    requests:
      cpu: 100m
      memory: 128Mi
  hook:
    resources:
      limits:
        memory: 2Gi
      requests:
        cpu: 100m
        memory: 128Mi
    serviceAccount:
      create: true
      name:
      annotations: {}

ingress:
  enabled: true
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-production
    nginx.ingress.kubernetes.io/proxy-body-size: "1000m"
  ingressClassName: nginx
  hosts:
    - jupyterhub.company.com
  pathSuffix:
  pathType: Prefix
  tls:
    - secretName: jupyterhub-tls
      hosts:
        - jupyterhub.company.com

policyexception for kyverno

---
apiVersion: kyverno.io/v2
kind: PolicyException
metadata:
  name: jupyterlab
  namespace: kyverno-policies
spec:
  exceptions:
    - policyName: require-run-as-non-root-user
      ruleNames:
        - run-as-non-root-user
    - policyName: require-run-as-nonroot
      ruleNames:
        - run-as-non-root
    - policyName: disallow-capabilities-strict
      ruleNames:
        - require-drop-all
        - adding-capabilities-strict
    - policyName: disallow-capabilities
      ruleNames:
        - adding-capabilities
    - policyName: disallow-privilege-escalation
      ruleNames:
        - privilege-escalation
    - policyName: disallow-privileged-containers
      ruleNames:
        - privileged-containers
    - policyName: restrict-seccomp-strict
      ruleNames:
        - check-seccomp-strict
  match:
    any:
      - resources:
          kinds:
            - Pod
          selector:
            matchLabels:
              app.kubernetes.io/component: singleuser-server
              component: singleuser-server
              # app.kubernetes.io/name: jupyterhub
          namespaces: [jupyterlab]

[manics]

Can you share the full helm command you use to deploy the chart?

[radhupr]

@manics I use kustomize and argoCD for deploying the app. The kustomization.yaml is available under configuration section above.