stackable_secret_operator::csi_server::node: failed to publish volume

[dshershov]

Hello everyone!
I have some troubles with TRINO deploy by kubernetes operator
trino coordinator and worker have some troubles with deploy and have the following errors:

MountVolume.SetUp failed for volume "pvc-<PVC_ID>" : rpc error: code = Unavailable desc = failed to create secret parent dir /mnt/paas/kubernetes/kubelet/pods/<POD_ID>/volumes/kubernetes.io~csi/pvc-<PVC_ID>/mount: No such file or directory (os error 2)

and:

(combined from similar events): Unable to attach or mount volumes: unmounted volumes=[server-tls-mount internal-tls-mount], unattached volumes=[server-tls-mount internal-tls log-config client-tls rwconfig server-tls internal-tls-mount data config catalog log kube-api-access-znlrz]: timed out waiting for the condition

After investigate that issue I moved to secret-operator pod -> secret-operator container logs and saw that:

2023-06-05T15:07:02.298424Z  INFO stackable_secret_operator::csi_server::node: Received NodePublishVolume request volume.path=/mnt/paas/kubernetes/kubelet/pods/<POD_ID>/volumes/kubernetes.io~csi/pvc-<PVC_ID>/mount
2023-06-05T15:07:02.394812Z  WARN stackable_secret_operator::csi_server::node: failed to publish volume error=status: Unavailable, message: "failed to create secret parent dir /mnt/paas/kubernetes/kubelet/pods/<POD_ID>/volumes/kubernetes.io~csi/pvc-<PVC_ID>/mount: No such file or directory (os error 2)", details: [], metadata: MetadataMap { headers: {} }

I would like to discuss ideas about this
Thanks!

[sbernauer]

Hi @dshershov, thanks for reporting this!
Are you using the Release 23.4? What Kubernetes distribution are you using? Maybe some sort of cloud?
Can you please also check for the longest existing prefix in the path /mnt/paas/kubernetes/kubelet/pods/<POD_ID>/volumes/kubernetes.io~csi/pvc-<PVC_ID>/mount. ("How deep do the folders exist"). You need to check that on the Kubernetes node itself.

[dshershov]

Thanks!
Yes, I have a 23.4.1 trino version (because 23.4.0 isn't working) and 23.4.0 version of secret-operator
Kubernetes deployed by HUAWEI Cloud with the next version - 1.23
/mnt/paas/kubernetes/kubelet/pods/<POD_ID>/volumes/kubernetes.io~csi/pvc-<PVC_ID>/ - this is a valid path with vol_data.json file only

[sbernauer]

Thanks for the background information!
It may be the case that the secret-operator is using the wrong kubelet directory (the default is /var/lib/kubelet).
Can you please try to install the secret-operator with helm (see https://docs.stackable.tech/home/stable/secret-operator/installation.html#_helm), normally this should be
helm repo add stackable https://repo.stackable.tech/repository/helm-stable/
helm install secret-operator stackable/secret-operator --version 23.4.1 --set kubeletDir=/mnt/paas/kubernetes/kubelet
The important part is the --set kubeletDir=/mnt/paas/kubernetes/kubelet, which sets the non-default directory.

[dshershov]

It's really working for me!
Thanks a lot!
But, unfortunately, don't know if it's related or not with that, but I've moved to the next step with deploying service and stuck with that error into the trino container:

copying /stackable/config to /stackable/rwconfig
OpenJDK 64-Bit Server VM warning: Option UseBiasedLocking was deprecated in version 15.0 and will likely be removed in a future release.
Error opening zip file or JAR manifest missing : /stackable/jmx/jmx_prometheus_javaagent-0.16.1.jar
Error occurred during initialization of VM
agent library failed to init: instrument

[sbernauer]

Glad it worked! Have raised a PR to document this: stackabletech/secret-operator#281
Regarding your problem: Can you please paste the TrinoCluster object you are deploying to the Cluster?

[dshershov]

Sure!

apiVersion: trino.stackable.tech/v1alpha1
kind: TrinoCluster
metadata:
  name: trino
  namespace: dwh
spec:
  image:
    productVersion: "414"
    stackableVersion: "0.0.0-dev"
  clusterConfig:
    catalogLabelSelector:
      matchLabels:
        trino: trino
  coordinators:
    roleGroups:
      default:
        replicas: 1
        config:
          resources:
            cpu:
              min: 300m
              max: "2"
            memory:
              limit: 8Gi
  workers:
    roleGroups:
      default:
        replicas: 1
        config:
          resources:
            cpu:
              min: 300m
              max: "2"
            memory:
              limit: 16Gi
---
apiVersion: trino.stackable.tech/v1alpha1
kind: TrinoCatalog
metadata:
  name: hive
  namespace: dwh
  labels:
    trino: trino
spec:
  connector:
    hive:
      metastore:
        configMap: hive

[sbernauer]

Please change spec.image.stackableVersion to "23.4" (you have to include the quotes, otherwise yaml thinks it's a number), so that it matches the operator version you are using ;)
This should resolve your problem, we changed things in our Trino docker image

[dshershov]

hmm

Back-off pulling image "trino:414-stackable23.4"
trino:414-stackable23.4 not found

[sbernauer]

Oh, in 23.4 we only supported Trino 403. 414 will be in 23.7. So in the meantime please use

spec:
  image:
    productVersion: "403"
    stackableVersion: "23.4"

[dshershov]

yeap, it's really working!
Thanks a lot, Sebastian!

[sbernauer]

Awesome! You're welcome :)

[dshershov]

Hello everyone! Here we go again :)
This message only for information for everyone who will face the problem with listner and kafka operators like this issue with secret operator and KubeletDir
We are working on stackable Kafka operator and found the same issue like secret operator which said about "doesn't existing file", but now pods said about "failed to write content: File exists (os error 17)" kafka-pods
And for fixing this issue we need to add similar parameter to listner like upper - kubeletDir
thanks for a watching)

[sbernauer]

Yes, that makes perfect sense, you need to configure it for listener the same way as you do your secret.
Many thanks for pointing that out!