Postprocessing error with nats

[suse-coder]

When I update from one version to another I get this error and restart of postprocessing pod does not work (with nats persistence enabled this does not happen):

{"level":"error","service":"postprocessing","error":"nats: API error: code=404 err_code=10059 description=stream not found","time":"2026-02-24T08:43:20Z","line":"github.com/opencloud-eu/opencloud/services/postprocessing/pkg/service/service.go:439","message":"failed to get consumer"}

What could be the reason. The nats container was also restarted but i up. I have no persistence in nats enabled (but arent the sessions with new container version so or so out of date) and should it not recover from that?

nats:

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2026-02-24T08:46:30Z"
  generateName: nats-89d5c584-
  generation: 1
  labels:
    app: nats
    app.kubernetes.io/instance: opencloud
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: opencloud-microservices
    app.kubernetes.io/version: 5.1.0
    helm.sh/chart: opencloud-microservices-0.3.32
    pod-template-hash: 89d5c584
  name: nats-89d5c584-h4qw7
  namespace: opencloud
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: nats-89d5c584
    uid: 187c0a8a-103c-483d-b9cd-3e731bddd0a3
  resourceVersion: "173487"
  uid: 0d8a93ef-de53-4086-a54c-ffd546dc0504
spec:
  containers:
  - args:
    - server
    command:
    - opencloud
    env:
    - name: OC_RUN_SERVICES
      value: nats
    - name: NATS_LOG_COLOR
      value: "false"
    - name: NATS_LOG_LEVEL
      value: debug
    - name: NATS_LOG_PRETTY
      value: "false"
    - name: NATS_DEBUG_PPROF
      value: "false"
    - name: NATS_NATS_HOST
      value: 0.0.0.0
    - name: NATS_NATS_PORT
      value: "9233"
    - name: NATS_DEBUG_ADDR
      value: 0.0.0.0:9234
    - name: OC_TRANSFER_SECRET
      valueFrom:
        secretKeyRef:
          key: transfer-secret
          name: transfer-secret
    - name: OC_JWT_SECRET
      valueFrom:
        secretKeyRef:
          key: jwt-secret
          name: jwt-secret
    - name: OC_MACHINE_AUTH_API_KEY
      valueFrom:
        secretKeyRef:
          key: machine-auth-api-key
          name: machine-auth-api-key
    - name: OC_SYSTEM_USER_ID
      valueFrom:
        secretKeyRef:
          key: user-id
          name: storage-system
    image: opencloudeu/opencloud-rolling:5.1.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 3
      httpGet:
        path: /healthz
        port: metrics-debug
        scheme: HTTP
      initialDelaySeconds: 60
      periodSeconds: 20
      successThreshold: 1
      timeoutSeconds: 10
    name: nats
    ports:
    - containerPort: 9233
      name: nats
      protocol: TCP
    - containerPort: 9234
      name: metrics-debug
      protocol: TCP
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /healthz
        port: metrics-debug
        scheme: HTTP
      initialDelaySeconds: 5
      periodSeconds: 5
      successThreshold: 1
      timeoutSeconds: 5
    resources: {}
    securityContext:
      readOnlyRootFilesystem: true
      runAsGroup: 1000
      runAsNonRoot: true
      runAsUser: 1000
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/lib/opencloud
      name: nats-data
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-4dpcp
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: talos-246-pvk
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 1000
    fsGroupChangePolicy: OnRootMismatch
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - emptyDir: {}
    name: nats-data
  - name: kube-api-access-4dpcp
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2026-02-24T08:46:32Z"
    observedGeneration: 1
    status: "True"
    type: PodReadyToStartContainers
  - lastProbeTime: null
    lastTransitionTime: "2026-02-24T08:46:30Z"
    observedGeneration: 1
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2026-02-24T08:46:38Z"
    observedGeneration: 1
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2026-02-24T08:46:38Z"
    observedGeneration: 1
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2026-02-24T08:46:30Z"
    observedGeneration: 1
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://f126084f4f1483676e704fbab1ec6f9d740ad897da28cf8877cc15810579805f
    image: docker.io/opencloudeu/opencloud-rolling:5.1.0
    imageID: docker.io/opencloudeu/opencloud-rolling@sha256:6229e937e32e5e114ddac81af7e851fb9346a818c58a36b784529d5503530f93
    lastState: {}
    name: nats
    ready: true
    resources: {}
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2026-02-24T08:46:31Z"
    user:
      linux:
        gid: 1000
        supplementalGroups:
        - 1000
        uid: 1000
    volumeMounts:
    - mountPath: /var/lib/opencloud
      name: nats-data
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-4dpcp
      readOnly: true
      recursiveReadOnly: Disabled
  hostIP: 192.168.200.52
  hostIPs:
  - ip: 192.168.200.52
  observedGeneration: 1
  phase: Running
  podIP: 10.244.0.89
  podIPs:
  - ip: 10.244.0.89
  qosClass: BestEffort
  startTime: "2026-02-24T08:46:30Z"

postprocessing:

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2026-02-24T08:46:53Z"
  generateName: postprocessing-85c79dff78-
  generation: 1
  labels:
    app: postprocessing
    app.kubernetes.io/instance: opencloud
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: opencloud-microservices
    app.kubernetes.io/version: 5.1.0
    helm.sh/chart: opencloud-microservices-0.3.32
    pod-template-hash: 85c79dff78
  name: postprocessing-85c79dff78-ssfss
  namespace: opencloud
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: postprocessing-85c79dff78
    uid: 96978665-8b23-41c5-854f-30b783db907a
  resourceVersion: "173609"
  uid: 983727a1-6abc-40ec-9388-d158e95a0a0e
spec:
  containers:
  - args:
    - server
    command:
    - opencloud
    env:
    - name: MICRO_REGISTRY
      value: nats-js-kv
    - name: MICRO_REGISTRY_ADDRESS
      value: nats:9233
    - name: OC_EVENTS_ENDPOINT
      value: nats:9233
    - name: OC_PERSISTENT_STORE
      value: nats-js-kv
    - name: OC_PERSISTENT_STORE_NODES
      value: nats:9233
    - name: OC_RUN_SERVICES
      value: postprocessing
    - name: POSTPROCESSING_LOG_COLOR
      value: "false"
    - name: POSTPROCESSING_LOG_LEVEL
      value: debug
    - name: POSTPROCESSING_LOG_PRETTY
      value: "false"
    - name: POSTPROCESSING_DEBUG_PPROF
      value: "false"
    - name: POSTPROCESSING_DEBUG_ADDR
      value: 0.0.0.0:9255
    - name: POSTPROCESSING_WORKERS
      value: "3"
    - name: OC_TRANSFER_SECRET
      valueFrom:
        secretKeyRef:
          key: transfer-secret
          name: transfer-secret
    - name: OC_JWT_SECRET
      valueFrom:
        secretKeyRef:
          key: jwt-secret
          name: jwt-secret
    - name: OC_MACHINE_AUTH_API_KEY
      valueFrom:
        secretKeyRef:
          key: machine-auth-api-key
          name: machine-auth-api-key
    - name: OC_SYSTEM_USER_ID
      valueFrom:
        secretKeyRef:
          key: user-id
          name: storage-system
    image: opencloudeu/opencloud-rolling:5.1.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 3
      httpGet:
        path: /healthz
        port: metrics-debug
        scheme: HTTP
      initialDelaySeconds: 60
      periodSeconds: 20
      successThreshold: 1
      timeoutSeconds: 10
    name: postprocessing
    ports:
    - containerPort: 9255
      name: metrics-debug
      protocol: TCP
    resources: {}
    securityContext:
      readOnlyRootFilesystem: true
      runAsGroup: 1000
      runAsNonRoot: true
      runAsUser: 1000
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /etc/opencloud/messaging-system-ca
      name: messaging-system-ca
      readOnly: true
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-xnv6w
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: talos-246-pvk
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 1000
    fsGroupChangePolicy: OnRootMismatch
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - emptyDir: {}
    name: messaging-system-ca
  - name: kube-api-access-xnv6w
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2026-02-24T08:46:55Z"
    observedGeneration: 1
    status: "True"
    type: PodReadyToStartContainers
  - lastProbeTime: null
    lastTransitionTime: "2026-02-24T08:46:53Z"
    observedGeneration: 1
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2026-02-24T08:46:55Z"
    observedGeneration: 1
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2026-02-24T08:46:55Z"
    observedGeneration: 1
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2026-02-24T08:46:53Z"
    observedGeneration: 1
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://1d9cace38a72990321fea1d5955a782da38514fafdbbe2682ea3b19a9615a293
    image: docker.io/opencloudeu/opencloud-rolling:5.1.0
    imageID: docker.io/opencloudeu/opencloud-rolling@sha256:6229e937e32e5e114ddac81af7e851fb9346a818c58a36b784529d5503530f93
    lastState: {}
    name: postprocessing
    ready: true
    resources: {}
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2026-02-24T08:46:54Z"
    user:
      linux:
        gid: 1000
        supplementalGroups:
        - 1000
        uid: 1000
    volumeMounts:
    - mountPath: /etc/opencloud/messaging-system-ca
      name: messaging-system-ca
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-xnv6w
      readOnly: true
      recursiveReadOnly: Disabled
  hostIP: 192.168.200.52
  hostIPs:
  - ip: 192.168.200.52
  observedGeneration: 1
  phase: Running
  podIP: 10.244.0.23
  podIPs:
  - ip: 10.244.0.23
  qosClass: BestEffort
  startTime: "2026-02-24T08:46:53Z"

[micbar]

What could be the reason. The nats container was also restarted but i up. I have no persistence in nats enabled (but arent the sessions with new container version so or so out of date) and should it not recover from that?

Nats stores the events and the postprocessing depends on it.

[suse-coder]

So the nats persistence needs to be always on otherwise I need to restart all of it after a update?

[micbar]

Yes. Postprocessing can always be restarted and that should be a cronjob which runs regularly