Unable to attach or mount volumes

[kalined]

Hi all,

Maybe somebody knows what is the main reason for such warning messages?

Events:
  Type     Reason       Age                    From               Message
  ----     ------       ----                   ----               -------
  Normal   Scheduled    5m49s                  default-scheduler  Successfully assigned actions-runner-system/controller-manager-6f8b85fb6-d8jkm to hostwroker1.saitama.net
  Warning  FailedMount  17m                  kubelet            Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[kube-api-access-v48zj secret tmp cert]: timed out waiting for the condition
  Warning  FailedMount  8m32s                kubelet            Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[tmp cert kube-api-access-v48zj secret]: timed out waiting for the condition
  Warning  FailedMount  6m18s (x5 over 19m)  kubelet            Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[secret tmp cert kube-api-access-v48zj]: timed out waiting for the condition
  Warning  FailedMount  103s (x2 over 4m1s)  kubelet            Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[cert kube-api-access-v48zj secret tmp]: timed out waiting for the condition
  Warning  FailedMount  86s (x18 over 21m)   kubelet            MountVolume.SetUp failed for volume "cert" : secret "actions-runner-controller-serving-cert" not found

I was using helm chart, during installation have configured couple of things:

helm upgrade --install --namespace actions-runner-system --create-namespace --set=dockerRegistryMirror=my-docker.my.artifactory.com --set=image.repository=my-docker.my.artifactory.com/summerwind/actions-runner-controller --set=image.actionsRunnerRepositoryAndTag=my-docker.my.artifactory.com/summerwind/actions-runner:latest --set=metrics.proxy.image.repository=my-docker.my.artifactory.com/brancz/kube-rbac-proxy --set=authSecret.create=true --set=authSecret.app_id="8" --set=authSecret.github_app_installation_id="5" --wait actions-runner-controller joined-helm/actions-runner-controller

because my K3S cluster doesn't have access to the internet:

1. image: my-docker.my.artifactory.com/summerwind/actions-runner-controller:v0.27.0
2. image: my-docker.my.artifactory.com/brancz/kube-rbac-proxy:v0.10.0

my.artifactory.com is accessible from all K3S hosts.

Additional info:

1. Namespaces

NAME                          STATUS   AGE
actions-runner-system         Active   79m
cattle-fleet-system           Active   12d
cattle-impersonation-system   Active   12d
cattle-system                 Active   12d
cert-manager                  Active   12d
default                       Active   12d
github-runners-small          Active   32m
kube-node-lease               Active   12d
kube-public                   Active   12d
kube-system                   Active   12d
local                         Active   12d

2. actions-runner-system namespace pods

NAME                                 READY   STATUS              RESTARTS   AGE
controller-manager-6f8b85fb6-d8jkm   0/2     ContainerCreating   0          18m

4. Secrets

Thought to create a secret manually, so deleted controller-manager and created it with:

$ kubectl create secret generic controller-manager \
    -n actions-runner-system \
    --from-literal=github_app_id=${GITHUB_APP_ID} \
    --from-literal=github_app_installation_id=${GITHUB_APP_INSTALLATION_ID} \
    --from-file=github_app_private_key=${GITHUB_APP_PRIVATE_KEY_FILEPATH}

Result:

NAME                                              TYPE                 DATA   AGE
controller-manager                                Opaque               3      26m
sh.helm.release.v1.actions-runner-controller.v1   helm.sh/release.v1   1      38m

Error during installation:

Internal error occurred: failed calling webhook "[webhook.cert-manager.io](http://webhook.cert-manager.io/)": failed to call webhook: Post "[https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s](https://cert-manager-webhook.cert-manager.svc/mutate?timeout=10s)[":](https://cert-manager-webhook.cert-manager.svc/mutate?timeout=10s%22:) context deadline exceeded

Thanks in advance,

[kalined]

The issue was with the cert-manager.

[maeghan-porter]

If you shared what specifically was wrong it could help other people coming across this.