demo app not inject

[withlin]

Bug Report

What is the issue?

demo app not inject

How can it be reproduced?

//kg all in linkerd ns
[devops_root@devops-dev-01 ~]$ kg all
NAME                                          READY   STATUS    RESTARTS   AGE
pod/linkerd-controller-9d76fc775-q747t        2/2     Running   0          113m
pod/linkerd-destination-6548979954-xpq2t      2/2     Running   0          113m
pod/linkerd-grafana-977c68ff-px56p            2/2     Running   0          113m
pod/linkerd-identity-6c765dd9ff-zqfbl         2/2     Running   0          113m
pod/linkerd-prometheus-7658bf7c5-9xd2v        2/2     Running   0          113m
pod/linkerd-proxy-injector-7b7bc648f4-8vfjl   2/2     Running   0          113m
pod/linkerd-sp-validator-64b7c9b85c-7dzfx     2/2     Running   0          113m
pod/linkerd-tap-7f6878b46d-rzz6q              2/2     Running   0          113m
pod/linkerd-web-5cc8b8dc77-n8vq4              2/2     Running   0          111m

NAME                             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE
service/linkerd-controller-api   ClusterIP   10.106.234.118   <none>        8085/TCP                        113m
service/linkerd-dst              ClusterIP   10.98.220.9      <none>        8086/TCP                        113m
service/linkerd-grafana          ClusterIP   10.106.236.63    <none>        3000/TCP                        113m
service/linkerd-identity         ClusterIP   10.100.235.189   <none>        8080/TCP                        113m
service/linkerd-prometheus       ClusterIP   10.109.251.200   <none>        9090/TCP                        113m
service/linkerd-proxy-injector   ClusterIP   10.101.223.100   <none>        443/TCP                         113m
service/linkerd-sp-validator     ClusterIP   10.108.181.96    <none>        443/TCP                         113m
service/linkerd-tap              ClusterIP   10.111.126.8     <none>        8088/TCP,443/TCP                113m
service/linkerd-web              NodePort    10.97.231.202    <none>        8084:31267/TCP,9994:31093/TCP   113m

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/linkerd-controller       1/1     1            1           113m
deployment.apps/linkerd-destination      1/1     1            1           113m
deployment.apps/linkerd-grafana          1/1     1            1           113m
deployment.apps/linkerd-identity         1/1     1            1           113m
deployment.apps/linkerd-prometheus       1/1     1            1           113m
deployment.apps/linkerd-proxy-injector   1/1     1            1           113m
deployment.apps/linkerd-sp-validator     1/1     1            1           113m
deployment.apps/linkerd-tap              1/1     1            1           113m
deployment.apps/linkerd-web              1/1     1            1           113m

NAME                                                DESIRED   CURRENT   READY   AGE
replicaset.apps/linkerd-controller-9d76fc775        1         1         1       113m
replicaset.apps/linkerd-destination-6548979954      1         1         1       113m
replicaset.apps/linkerd-grafana-977c68ff            1         1         1       113m
replicaset.apps/linkerd-identity-6c765dd9ff         1         1         1       113m
replicaset.apps/linkerd-prometheus-7658bf7c5        1         1         1       113m
replicaset.apps/linkerd-proxy-injector-7b7bc648f4   1         1         1       113m
replicaset.apps/linkerd-sp-validator-64b7c9b85c     1         1         1       113m
replicaset.apps/linkerd-tap-7f6878b46d              1         1         1       113m
replicaset.apps/linkerd-web-5cc8b8dc77              1         1         1       111m
replicaset.apps/linkerd-web-7b5f68d58               0         0         0       113m

NAME                              SCHEDULE      SUSPEND   ACTIVE   LAST SCHEDULE   AGE
cronjob.batch/linkerd-heartbeat   46 6 * * *    False     0        103m            113m

//install demo app
curl -sL https://run.linkerd.io/emojivoto.yml \
  | kubectl apply -f -


//and then inject anno
kubectl get -n emojivoto deploy -o yaml \
  | linkerd inject - \
  | kubectl apply -f -


//kg all
[devops_root@devops-dev-01 ~]$ kg all
NAME                            READY   STATUS    RESTARTS   AGE
pod/emoji-845bdc66b7-ll7k8      1/1     Running   0          71s
pod/vote-bot-7c55876b86-ncbpb   1/1     Running   0          71s
pod/voting-b4689988d-w6qlb      1/1     Running   0          71s
pod/web-67ff67d47b-6bg4g        1/1     Running   0          71s

NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
service/emoji-svc    ClusterIP   10.100.125.194   <none>        8080/TCP,8801/TCP   52m
service/voting-svc   ClusterIP   10.100.238.0     <none>        8080/TCP,8801/TCP   52m
service/web-svc      ClusterIP   10.100.248.81    <none>        80/TCP              52m

NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/emoji      1/1     1            1           52m
deployment.apps/vote-bot   1/1     1            1           52m
deployment.apps/voting     1/1     1            1           52m
deployment.apps/web        1/1     1            1           52m

NAME                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/emoji-845bdc66b7      1         1         1       71s
replicaset.apps/vote-bot-7c55876b86   1         1         1       71s
replicaset.apps/voting-b4689988d      1         1         1       71s
replicaset.apps/web-67ff67d47b        1         1         1       71s


//kd pod ..
[devops_root@devops-dev-01 ~]$  kd  pod emoji-845bdc66b7-ll7k8
Name:         emoji-845bdc66b7-ll7k8
Namespace:    emojivoto
Priority:     0
Node:         ali-tekton-ci-devops-dev-01/172.18.17.86
Start Time:   Mon, 14 Sep 2020 16:07:13 +0800
Labels:       app=emoji-svc
              pod-template-hash=845bdc66b7
              version=v10
Annotations:  linkerd.io/inject: enabled
              ovn.kubernetes.io/allocated: true
              ovn.kubernetes.io/cidr: 10.16.0.0/16
              ovn.kubernetes.io/gateway: 10.16.0.1
              ovn.kubernetes.io/ip_address: 10.16.0.58
              ovn.kubernetes.io/logical_switch: ovn-default
              ovn.kubernetes.io/mac_address: 00:00:00:46:04:25
              ovn.kubernetes.io/network_types: geneve
              ovn.kubernetes.io/routed: true
Status:       Running
IP:           10.16.0.58
IPs:
  IP:           10.16.0.58
Controlled By:  ReplicaSet/emoji-845bdc66b7
Containers:
  emoji-svc:
    Container ID:   docker://8cd7df450e29832869c257b0d2b88ab426bb09c26a0e3c333c0ddbcf245a56c1
    Image:          buoyantio/emojivoto-emoji-svc:v10
    Image ID:       docker-pullable://buoyantio/emojivoto-emoji-svc@sha256:7863269527baef7a797f3f33c42a6b000860ee54114cbff4a5534bf875df9b05
    Ports:          8080/TCP, 8801/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Mon, 14 Sep 2020 16:07:15 +0800
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:  100m
    Environment:
      GRPC_PORT:  8080
      PROM_PORT:  8801
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from emoji-token-84r79 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  emoji-token-84r79:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  emoji-token-84r79
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From                                  Message
  ----    ------     ----  ----                                  -------
  Normal  Scheduled  118s  default-scheduler                     Successfully assigned emojivoto/emoji-845bdc66b7-ll7k8 to ali-tekton-ci-devops-dev-01
  Normal  Pulled     116s  kubelet, ali-tekton-ci-devops-dev-01  Container image "buoyantio/emojivoto-emoji-svc:v10" already present on machine
  Normal  Created    116s  kubelet, ali-tekton-ci-devops-dev-01  Created container emoji-svc
  Normal  Started    116s  kubelet, ali-tekton-ci-devops-dev-01  Started container emoji-svc


//linkerd.io/inject: enabled,but not inject.

Logs, error output, etc

[devops_root@devops-dev-01 ~]$  k logs linkerd-proxy-injector-7b7bc648f4-lcpz7 -c  proxy-injector
time="2020-09-14T09:01:56Z" level=info msg="running version stable-2.8.1"
time="2020-09-14T09:01:56Z" level=info msg="waiting for caches to sync"
time="2020-09-14T09:01:56Z" level=info msg="caches synced"
time="2020-09-14T09:01:56Z" level=info msg="starting admin server on :9995"
time="2020-09-14T09:01:56Z" level=info msg="listening at :8443"
2020/09/14 09:02:46 http: TLS handshake error from 127.0.0.1:43340: remote error: tls: bad certificate
2020/09/14 09:02:46 http: TLS handshake error from 127.0.0.1:43336: remote error: tls: bad certificate
2020/09/14 09:02:46 http: TLS handshake error from 127.0.0.1:43344: remote error: tls: bad certificate
2020/09/14 09:02:46 http: TLS handshake error from 127.0.0.1:43348: remote error: tls: bad certificate

linkerd check output

kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ controller pod is running
√ can initialize the client
√ can query the control plane API

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ control plane PodSecurityPolicies exist

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust anchor

linkerd-api
-----------
√ control plane pods are ready
√ control plane self-check
√ [kubernetes] control plane can talk to Kubernetes
√ [prometheus] control plane can talk to Prometheus
√ tap api service is running

linkerd-version
---------------
√ can determine the latest version
√ cli is up-to-date

control-plane-version
---------------------
√ control plane is up-to-date
√ control plane and cli versions match

linkerd-addons
--------------
√ 'linkerd-config-addons' config map exists

linkerd-grafana
---------------
√ grafana add-on service account exists
√ grafana add-on config map exists
√ grafana pod is running

Status check results are √

Environment

• Kubernetes Version:

$ k version --short
Client Version: v1.19.1
Server Version: v1.19.1

• Cluster Environment: (GKE, AKS, kops, ...)
• Host OS:

centos7 Linux 3.10.0-957.21.3.el7.x86_64 #1 SMP Tue Jun 18 16:35:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

• Linkerd version:

Client version: stable-2.8.1
Server version: stable-2.8.1

Possible solution

Additional context

[ihcsim]

2020/09/14 09:02:46 http: TLS handshake error from 127.0.0.1:43340: remote error: tls: bad certificate

Something is not right with your proxy injector installation, causing the K8s api-server to not be able to communicate with the injector webhook. What commands did you use to install Linkerd? Any error logs in the K8s api-server? If this is a non-prod environment, the easiest thing to do is to uninstall Linkerd and re-install it.

[captainbeardo]

Here is a response that I got on another forum with the exact same issue. Uninstalling and re-installing did not resolve the issue. The only thing that worked for me was going back to a 1.18 based system.

[withlin]

Here is a response that I got on another forum with the exact same issue. Uninstalling and re-installing did not resolve the issue.

Yes,re-install did not reslove the issue.

The only thing that worked for me was going back to a 1.18 based system.

i try rollback k8s from 1.19 to 1.18

[ihcsim]

@captainbeardo Thanks for the pointer. Sounds like this is the same issue as #4918, which has been fixed in the latest edge release.