[Bug]: Not able to use pipelines.cache & pipelines.tmpdir pvc

[pavel-hladik]

Problem

Hi, I've installed version 11.0.0 of blue-agent/blue-agent to k8s v1.29.11, my storageClass is longhorn, which is working without any issue for other apps. The pv & pvc were created correctly and I can attach them to any of pods. But the blue-agent pods reports FailedMount. Without use of cache & tmpdir pvc I can run the blue-agent without issue.

The values.yaml diff:

replicaCount: 1

autoscaling:
  enabled: false
 
pipelines:
  organizationURL: "myorgyrl"
  personalAccessToken: "mypat"
  poolName: "mypoolname"
  cache:
    type: longhorn
  tmpdir:
    type: longhorn

Impact

I can't deploy blue-agent to the k8s cluster.

Short logs

MountVolume.SetUp failed for volume "pvc-9efa4957-15e1-4160-84fa-f84e27001b39" : rpc error: code = Aborted desc = no Pending workload pods for volume pvc-9efa4957-15e1-4160-84fa-f84e27001b39 to be mounted: map[]

Full logs

  Warning  FailedScheduling        4m4s                  default-scheduler        0/6 nodes are available: waiting for ephemeral volume controller to create the persistentvolumeclaim "blue-agent-ff658c9c-gkflm-azp-work". preemption: 0/6 nodes are available: 6 Preemption is not helpful for scheduling.
  Warning  FailedScheduling        3m28s (x2 over 4m3s)  default-scheduler        0/6 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/6 nodes are available: 6 Preemption is not helpful for scheduling.
  Normal   Scheduled               3m23s                 default-scheduler        Successfully assigned blue-agent/blue-agent-ff658c9c-gkflm to k8s-dev-wn1
  Normal   SuccessfulAttachVolume  3m12s                 attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-71a77747-c2ca-4a6c-8109-53e346ae8cb2"
  Normal   SuccessfulAttachVolume  3m12s                 attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-9efa4957-15e1-4160-84fa-f84e27001b39"
  Warning  FailedMount             63s (x9 over 3m11s)   kubelet                  MountVolume.SetUp failed for volume "pvc-71a77747-c2ca-4a6c-8109-53e346ae8cb2" : rpc error: code = Aborted desc = no Pending workload pods for volume pvc-71a77747-c2ca-4a6c-8109-53e346ae8cb2 to be mounted: map[]
  Warning  FailedMount             63s (x9 over 3m11s)   kubelet                  MountVolume.SetUp failed for volume "pvc-9efa4957-15e1-4160-84fa-f84e27001b39" : rpc error: code = Aborted desc = no Pending workload pods for volume pvc-9efa4957-15e1-4160-84fa-f84e27001b39 to be mounted: map[]

Deployment

On-premise k8s cluster

Version

11.0.0

[clemlesne]

I see those related issues: longhorn/longhorn#8072, longhorn/longhorn#8198

Which version of Longhorn do you use? Pre 1.6.1 seems to have issues with that specific volume type.

The deployment is using ephemeral volumes to allow stability, security and performance. This is all standard but not supported everywhere:

https://github.com/clemlesne/blue-agent/blob/main/src/helm/blue-agent/templates/_helpers.tpl#L277-L286

If you are in the latest Longhorn version and still having the issue, can you send me the deployed YAML?

[clemlesne]

Hello, I want to help. Do you still see his mistakes after updating Longhorn?