"CrashLoopBackOff" issue after injecting Linkerd.

[nm158b]

Hi, I am running linkerd version stable-2.10.2 on Azure Kubernetes service (AKS). While I am trying to inject linkerd to a particular service, it is exiting with "CrashLoopBackOff" error. The other services are running fine, this service has a dependency, it is a multi-container service with a sidecar container running along with main container.

>>kubectl get pods 
NAME                                 READY   STATUS             RESTARTS   AGE
gpsworkflow-cdb7fbcf6-87wmz          1/3     CrashLoopBackOff   6667       14d

>>kubectl describe pod
State:          Terminated
    Reason:       Completed
    Exit Code:    0
Events:
  Type     Reason   Age                    From     Message
  ----     ------   ----                   ----     -------
  Normal   Pulled   46m (x3975 over 14d)   kubelet  Container image "cr.l5d.io/linkerd/proxy:stable-2.10.2" already present on machine
  Warning  BackOff  77s (x96762 over 14d)  kubelet  Back-off restarting failed container

>> kubectl logs gpsworkflow-cdb7fbcf6-87wmz
error: a container name must be specified for pod gpsworkflow, choose one of: [gpsworkflow openrestyssc linkerd-proxy] or one of the init containers: [linkerd-init]

>> kubectl logs gpsworkflow-cdb7fbcf6-87wmz -c gpsworkflow

>> kubectl logs gpsworkflow-cdb7fbcf6-87wmz -c gpsworkflow --previous
Error from server (BadRequest): previous terminated container "gpsworkflow" in pod "gpsworkflow-cdb7fbcf6-87wmz" not found


>> kubectl logs gpsworkflow-cdb7fbcf6-87wmz -c openrestyssc
2021/09/08 15:10:20 [notice] 11#11: using the "epoll" event method
2021/09/08 15:10:20 [notice] 11#11: openresty/1.13.6.1
2021/09/08 15:10:20 [notice] 11#11: built by gcc 6.3.0 (Alpine 6.3.0)
2021/09/08 15:10:20 [notice] 11#11: OS: Linux 5.4.0-1034-azure
2021/09/08 15:10:20 [notice] 11#11: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2021/09/08 15:10:20 [notice] 11#11: start worker processes
2021/09/08 15:10:20 [notice] 11#11: start worker process 12
2021/09/08 15:10:20 [notice] 11#11: start worker process 13

>>kubectl logs gpsworkflow-cdb7fbcf6-87wmz-c linkerd-proxy
time="2021-09-08T15:12:16Z" level=info msg="running version stable-2.10.2"
time="2021-09-08T15:12:16Z" level=info msg="Found pre-existing key: /var/run/linkerd/identity/end-entity/key.p8"
time="2021-09-08T15:12:16Z" level=info msg="Found pre-existing CSR: /var/run/linkerd/identity/end-entity/csr.der"
[     0.002256s] ERROR ThreadId(01) linkerd_app::env: Could not read LINKERD2_PROXY_IDENTITY_TOKEN_FILE: No such file or directory (os error 2)
[     0.002285s] ERROR ThreadId(01) linkerd_app::env: LINKERD2_PROXY_IDENTITY_TOKEN_FILE="/var/run/secrets/kubernetes.io/serviceaccount/token" is not valid: InvalidTokenSource
Invalid configuration: invalid environment variable

Any help would be appreciated!

[kleimkuhler]

@nm158b Are you managing your certificates through cert-manager? If so, can you provide the steps that you took to set that up? I ask because the errors in the proxy indicate those resources have already been created.

[nm158b]

@kleimkuhler we are not maintaining certificates through cert manager. Not sure how the the proxy is indicating resources already created

[kleimkuhler]

Yea I'm not sure why they would already exist after injecting the pod. Either way, the key and CSR will still both be recreated so I don't think the fact that they exist is the actual problem here.

Does restarting the pods do anything, or does it result in the same exact logs?

[nm158b]

yes it shows the same exact logs for all the containers.

[kleimkuhler]

When inspecting the pod's yaml can you see if there is a kube-api-access-... volume that is getting mounted to /var/run/secrets/kubernetes.io/serviceaccount/token? I'd expect to see something like this in the pod yaml:

volumes:
- name: kube-api-access-mb8z6
  projected:
    defaultMode: 420
    sources:
    - serviceAccountToken:
        expirationSeconds: 3607
        path: token

I'm not sure right now why it's failing to find that file.

[nm158b]

@kleimkuhler Yes we did not have this volume mounted to the desired path. And linkerd-proxy is trying to find this token file. We have modified deployment.yaml file according to fix this issue. Now it is resolved, thank you so much for helping us.

Can you please take a look that this #6879.