Linkerd2 tap pod CrashLoopBackOff

[Aisuko]

Bug Report

What is the issue?

│ tap time="2020-05-28T03:28:20Z" level=info msg="running version stable-2.7.1"                                                                                                                                                                          │
│ tap time="2020-05-28T03:28:20Z" level=info msg="Using trust domain: cluster.local"                                                                                                                                                                     ││ tap time="2020-05-28T03:28:20Z" level=fatal msg="failed to load [extension-apiserver-authentication] config: configmaps \"extension-apiserver-authentication\" is forbidden: User \"system:serviceaccount:linkerd:linkerd-tap\" cannot get resource \" │
│ configmaps\" in API group \"\" in the namespace \"kube-system\""

How can it be reproduced?

We use golang to execute Linkerd command and deploy Linkerd to the cluster

Logs, error output, etc

(If the output is long, please create a gist and
paste the link here.)

linkerd check output

version stable-2.7.1

Environment

• Kubernetes Version:
• Cluster Environment: (GKE, AKS, kops, ...)
• Host OS:
• Linkerd version: stable-2.7.1

Possible solution

Additional context

[grampelberg]

Does extension-apiserver-authentication exist in your cluster? Take a look at https://discourse.linkerd.io/t/error-no-client-ca-cert-available-for-apiextension-server/947/4 for solutions.

[nveenjain]

I ran kubectl -n kube-system get cm/extension-apiserver-authentication -ojsonpath='{.data.requestheader-client-ca-file}'
and there was a certificate as output...
Using minikube here... Any suggestions?

[grampelberg]

I would look at the RBAC next then. Maybe an uninstall and reinstall would help. The ClusterRole for linkerd-tap most definitely allows access to that configmap.

[Aisuko]

Is there any Kubernetes resource that can help check this issue?. I mean is there some resources may not deploy or other

[Aisuko]

The resource name and contents we applied
log2.log

[grampelberg]

Your error says system:serviceaccount:default:linkerd-tap, which would mean this is happening from the default namespace. But all your data says that it was run from the linkerd namespace. Any ideas why default is in there at all?

[Aisuko]

The log is the linkerd2 deployed under the linkerd namespace. I had change the namespace

[Aisuko]

➜  meshery git:(master) ✗ kubectl get secret linkerd-tap-tls  -n linkerd -oyaml
apiVersion: v1
data:
  crt.pem: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFRENDQWZpZ0F3SUJBZ0lRVGo2N241OEZ4MmtROVU0dEYvVG1OREFOQmdrcWhraUc5dzBCQVFzRkFEQWkKTVNBd0hnWURWUVFERXhkc2FXNXJaWEprTFhSaGNDNXNhVzVyWlhKa0xuTjJZekFlRncweU1EQTFNalV3T0RNeApNemhhRncweU1UQTFNalV3T0RNeE16aGFNQ0l4SURBZUJnTlZCQU1URjJ4cGJtdGxjbVF0ZEdGd0xteHBibXRsCmNtUXVjM1pqTUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUFvbWkrbEkrdWNuTWoKSCtKN2Zubyt2R2IzTXk2aU05cGVQOFpsNVREcGZXQmRWRGo2Z1lRSDBSVmJIaXZaYzFSZFZsS0ljSmErWDRaMApMSkpJampFOU5PU2hNandkazdRQmtIdmdKYWJCN2FTZjRvSjE4bE1KaThlU1V6MlVKcHh0UlkwbHNpN080WFR2CkpkVHV6R3RHeWdvMDVMOW1kOHNXMXJsUWFNOXZhdmZITUlRdGZnbmtacUpTNEtqN3paVHA1SjlyQmZ6WWhDN3IKZDZNNlRJNVd4OEd2UkVtSW53eUhjSVNuSjZSSVlMdVkyK3RKbk9SbTBKQjdlMEUvWVR5MkJIaE5JMFUyR29uMQptOVNKTWdFNlZ6UlZIbFVrRlMvVHRhRVNiRXg5MTF0aVUzYUVtOWdLUTUxWE5sazdxMG84NE5sMjRjb0xndDYyCnIvWTR4Tm5VR1FJREFRQUJvMEl3UURBT0JnTlZIUThCQWY4RUJBTUNBcVF3SFFZRFZSMGxCQll3RkFZSUt3WUIKQlFVSEF3RUdDQ3NHQVFVRkJ3TUNNQThHQTFVZEV3RUIvd1FGTUFNQkFmOHdEUVlKS29aSWh2Y05BUUVMQlFBRApnZ0VCQUlyNVYyaElsVExydHlEU0dLdTNNTVlYZkR5dktTWnFST1pYN1gzeHdsdUR6MGYzSkpDNXNMOUwycStYCjhac2l5L3kySTFsS2wrUXBmcE5SUWtybGg3VEg3UGU5U1MvdGVjc0VNWVF5QXEwbVppSFpWNWI4Q3dUOXJUckgKNzhtbnRXa0dyblV1cXlDNDVwN0RyaFdmUWRFS0Z1b003dTU1d1JFaGpCSTFqcFN1ZmlKZGtyazRsR3RpNnpJZQozeVk1d3Jjd0R6ZWhiY0hzWWVLNFFYMmtvVDdYWEttSjlQd3p3N0ozWWRuOFUrbTZyOVovUElZUjZ0WmxrWCtZCld2MUdMS2l2VTJxU2hHKzFQZmdSbmtsamVBeHBFN3NZdFc1UU1IV0M2amNnWjY0K01vMXFNUmNwc0R1WWFSVGoKS0hLVy9ScHBFVmdrbHVHa1lUa3NNWXRQTEZJPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
  key.pem: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb2dJQkFBS0NBUUVBb21pK2xJK3Vjbk1qSCtKN2Zubyt2R2IzTXk2aU05cGVQOFpsNVREcGZXQmRWRGo2CmdZUUgwUlZiSGl2WmMxUmRWbEtJY0phK1g0WjBMSkpJampFOU5PU2hNandkazdRQmtIdmdKYWJCN2FTZjRvSjEKOGxNSmk4ZVNVejJVSnB4dFJZMGxzaTdPNFhUdkpkVHV6R3RHeWdvMDVMOW1kOHNXMXJsUWFNOXZhdmZITUlRdApmZ25rWnFKUzRLajd6WlRwNUo5ckJmelloQzdyZDZNNlRJNVd4OEd2UkVtSW53eUhjSVNuSjZSSVlMdVkyK3RKCm5PUm0wSkI3ZTBFL1lUeTJCSGhOSTBVMkdvbjFtOVNKTWdFNlZ6UlZIbFVrRlMvVHRhRVNiRXg5MTF0aVUzYUUKbTlnS1E1MVhObGs3cTBvODRObDI0Y29MZ3Q2MnIvWTR4Tm5VR1FJREFRQUJBb0lCQUZDcnFLYVFhLzFMZFpINwo1STJoV1JJcUx2WE1YeU5VZjJZOFBSYkg0ZUsyQUFXNnU2dWprbUpuc2pmZFhnaXNTaVI3b0d3VVpJRjhmcTQ1CkJ6TWlBQjRLeG9yMlUxYXZuZXRQckxubkNBTHJ0OE5DUENYZTJ5b0VPb21BVjl1Sjdia1RjRHIraHlKeisrb1YKeDN2Z2VXd1gvOGZ6em9rSVp0TEhmOHBmNXg4Qkk3anA3SmlnVXNUVllCUWsyUmxBTkhwaFJTbDI3eUxyZ3liUQpxUU5MdldocGtpSlhOQXF4bDdoOFllYlVXZjIrQTN6TWdHMFl1cDByQUtPMXNRQkJhbmRsR0VTNVUrRVp1bWV2CkIzbDJOMVlaYmV6aXkrMXl4MCtxNUx1Rk1zakxBaEIvZ2lXYzVrZHUyQ1pGWXpPMWwyYWVJR1VpVExLUUpWS2IKU1ErYlMxVUNnWUVBMVZaejl0cDY1SUd2Tm8zRFd1QjNCWUtyY2srTnlicG1SZXh2Q2xOcGltNzNQcFQ3ZXFyZQpHUmhvS2tyVkg1TnRmZGQxSEJRZXZHRk1Gd1MxZy9jSTJ0bU9hRGZ1NjJldlRiYlNvT0pzSzJtZ3RpNGZOc1RDCjhyMC84dlBpWTNQeVdNTkx1cDZYZ2RIYlpHUkF3b1U4RE5XZHlDTWpPdi84L1pHZDYreVcxaDhDZ1lFQXd1TVMKRExRRXp2eDhXd2N1bVFHZHQ2R2RhRHBHVVNlTTJQVG9ISWlVT0JkMndRWnR1R0NsV1VLS0RCMEFlZitBYVEzcQpFOEpIODlyTHhqOWVZOVV5dmRvTThuaDJKUDZxeUhqbmRaSGZCWkU0OXB6WWRLOVEybUJwZEtxd3NGZE5IUEcxCmU2bHhxekYxczVXbDNiTEE3ZzM1T3huaXEyTE1KUm80SjRQR1hzY0NnWUJma1NEVm5qMmJDM1VmN1JWc0trMW4KRElnYzV3SlBNYzhLMThWT1ROdGtoaHovemVGeHJBVmVISTdkRXpTeHRtdEFDanFOMzMxOVNlQzZGd1B4SDhLNQp0dWVTVEN3ZERBOTlzcjVjRDlJQkNJcFQwZ2x0a1dyN0xrZnRYODgwV2daZ09Pdzlwem1ia0JURnZoczhpRTdFCk9SVTdTSEpKeGM3aEVGQThGWG5rWlFLQmdDNm9Ra2Y4U1RKc3pERFcxVmNZUmZRU295YXkzT2JNK3pGdDRLd0UKSHNnWEJmMU1WclJQbndaeG9hOG91bFo2enZsWHRYU3BqS1JhKzNJbzlhWWRBcHJsT3poSmtxOWtzT3B4MTZnVQpGanEwKzlnKzBLdU8xV0tTdG15UitOOGU1MzFJZU5SMFNweGJxUHdubGFBd0RYUndyL3h3UkE0WUJ4dlMzU0xJCnI1TTlBb0dBTzRuaEgrMStmeDdRK2p3amlJR29qbHAybURKVHJpUGFyN1B1a2RTaVlEb1hsM00xNGpvTm04V1QKYkhmSU1YN3JmaVBHVkZTZHN2bEExbXdkRVhDMWxhV2xaQTc2MTlCZUVQNkRhVzZrd2ZSajMrYXpJaC9ES3JyMQpQK3NrTHJIMFBXaXkzVUszQSswQ2JLa0RLN0x5MWFOMSszcnNZdUNhZmkyNzBnRmpLazA9Ci0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==
kind: Secret
metadata:
  annotations:
    linkerd.io/created-by: linkerd/cli stable-2.7.1
  creationTimestamp: "2020-05-25T08:31:42Z"
  labels:
    linkerd.io/control-plane-component: tap
    linkerd.io/control-plane-ns: linkerd
  name: linkerd-tap-tls
  namespace: linkerd
  resourceVersion: "231348"
  selfLink: /api/v1/namespaces/linkerd/secrets/linkerd-tap-tls
  uid: dd7d2d5a-71ab-4d95-9ff0-15f4aa25b2af
type: Opaque


➜  meshery git:(master) ✗ kubectl get secret linkerd-tap-token-rf58d  -n linkerd -oyaml
apiVersion: v1
data:
  ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN3akNDQWFxZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFTTVJBd0RnWURWUVFERXdkcmRXSmwKTFdOaE1CNFhEVEl3TURVeU5EQTJOVFl3TWxvWERUTXdNRFV5TWpBMk5UWXdNbG93RWpFUU1BNEdBMVVFQXhNSAphM1ZpWlMxallUQ0NBU0l3RFFZSktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQU1SSXlzNTR6ZUdSCnhpc21mTkJxMWRUR25hdW5iYXhjQUl6c0huczFrRFNxbGluQ2M0RjFIL2d6KzFTcUszMmZLa1FZNjZuYnNTRDkKZGk3Um1YRWRVeVdtNnZSVXR1RGgydnpNN0RidCtvUzU1U2RMcmswUjg2akpQOWpVeWRoeHZseU9tOFNVNTlrQQpYY0xJVThZTWo5aFFpUCtHNDlSdCs0ZTNZZFBNTk4yZE1EQklEcmJiNFZJOXh6Q0I3NDRlZlBXVzFaWk5JdDRnCm1uYVpCaGI0OXJxNFRMOXF3WmxqZDZ1RWVVZm9ZL2VFSlFVQjBGZ0VoVk12SDlyWkhGNHY5RVc1amJXTXBFNUcKVEh2bzdRMzI1T2M4UDNjK2Vvcmo3aHYrSDZoaGV6aWRKR3R5NFZ2NElQVGU5VXZZUEtJTmUzWjlSL0xmNlgzRApSVU9qaFdways4MENBd0VBQWFNak1DRXdEZ1lEVlIwUEFRSC9CQVFEQWdLa01BOEdBMVVkRXdFQi93UUZNQU1CCkFmOHdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBRllPWjViR0M4RTBVOERCS0lIYzZtNHRFSjEveWRQb0RmdUwKaXRGWTJwRGJCV3hXZlJ2WThuRFRoZmdDTGxQNk82eTBTOWE0dW5FVHU1dW12VjUzd213cGRkQUFzVmxjK3FRSApQVi90TVJlMEtaUnIwMEptbnpDa1VqK3FobXY1eWszaUtRMDJvZko2RjRXQjR0NTZ6U2VrZDE1Rmp6QnhYc1VzCmJnbTM0SFdBbWFaOVBlT1hndStuQ1Vud09jelcvbUovb28yYkRneStTQ0V2YnFxanRwMzFXb1VGRlBqS2JZVDcKMkIyUzI0RTFZQ3o2WWxKOGgzcVZwdEFMYWNUa3JKYUt3NXduVkxWcHNhVWl3dkFzMXJ4ZExQd3RlblpESVJwMgpDUTR5VXpyOG1ZZEtvQUkrNEhjQlhZVmlzdGlFYUNVUHRwZ1dqUS9URFlvQWxRMW03amc9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
  namespace: bGlua2VyZA==
  token: ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNklsZzVjRTlMYm5ndE9VVmFWVWhhTVdzMVZqUTVTVVY0TXpWVmJuaDVabm93VWs5dmRYVnlXSGRzYW04aWZRLmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUpzYVc1clpYSmtJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5elpXTnlaWFF1Ym1GdFpTSTZJbXhwYm10bGNtUXRkR0Z3TFhSdmEyVnVMWEptTlRoa0lpd2lhM1ZpWlhKdVpYUmxjeTVwYnk5elpYSjJhV05sWVdOamIzVnVkQzl6WlhKMmFXTmxMV0ZqWTI5MWJuUXVibUZ0WlNJNklteHBibXRsY21RdGRHRndJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5elpYSjJhV05sTFdGalkyOTFiblF1ZFdsa0lqb2laak5pTVRZM01ERXRNV05oTmkwMFlqZGtMVGczWm1JdE1UQXlNVGxoWkdOa1ltVTBJaXdpYzNWaUlqb2ljM2x6ZEdWdE9uTmxjblpwWTJWaFkyTnZkVzUwT214cGJtdGxjbVE2YkdsdWEyVnlaQzEwWVhBaWZRLlRJdUVRaV9NeVFiaWpYeWNWQVlfNXpQRE5LN21XdDN4emZ4djNjdmZQcW1BX3lzTzIzZWg2b29mVGpoVGJXeVRqU2k3Sl9FT2lFSllldV9JSWd4Y1RkSE9Tdnl2LU4yQU1vbFNBb3E5TkxTYUNWeGNxdVI4bFNvWUJaUjJFdllCaTRNdjd0NzFGVnkwYmdYQlk4dS1KXzFYbmVsMDVKYU5YdmVTem1XS3ZvcEZkTS05OENIYWptZkdZOWhlY1ZFczNzcXdYd2xGa2RLOGVGR19TS1R3ei1VakJRckJxa3M2ZXcxSTVKOTRzUl96cDd2UGpOcEdLcWszcTNRaHhROV9XV2hCeXU5V2lFMzh5R01UcVdLNWR1aVVvcS1aVURkaFk4NXpWNUhRTFhVdjBOS0RESDN2eXdheHVxLTJHTHJrSWdLRW5VbFFCdVJLRGJfYk9TM2Y4dw==
kind: Secret
metadata:
  annotations:
    kubernetes.io/service-account.name: linkerd-tap
    kubernetes.io/service-account.uid: f3b16701-1ca6-4b7d-87fb-10219adcdbe4
  creationTimestamp: "2020-05-25T08:31:42Z"
  name: linkerd-tap-token-rf58d
  namespace: linkerd
  resourceVersion: "231345"
  selfLink: /api/v1/namespaces/linkerd/secrets/linkerd-tap-token-rf58d
  uid: 2dbc9554-3977-4e39-8f44-86c7659813d3
type: kubernetes.io/service-account-token


➜  meshery git:(master) ✗ kubectl get cm linkerd-config -n linkerd -oyaml
apiVersion: v1
data:
  global: |
    {"linkerdNamespace":"linkerd","cniEnabled":false,"version":"stable-2.7.1","identityContext":{"trustDomain":"cluster.local","trustAnchorsPem":"-----BEGIN CERTIFICATE-----\nMIIBgzCCASmgAwIBAgIBATAKBggqhkjOPQQDAjApMScwJQYDVQQDEx5pZGVudGl0\neS5saW5rZXJkLmNsdXN0ZXIubG9jYWwwHhcNMjAwNTI1MDgzMTI4WhcNMjEwNTI1\nMDgzMTQ4WjApMScwJQYDVQQDEx5pZGVudGl0eS5saW5rZXJkLmNsdXN0ZXIubG9j\nYWwwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAAR06U2/YSMINsxjUOwOfQIzfyDV\npOm2b4DsNZRDJ4ZsKrjUGMAGw5wTvtQGUrkD+D9y62nOS+V1eYTud6BvrBzxo0Iw\nQDAOBgNVHQ8BAf8EBAMCAQYwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMC\nMA8GA1UdEwEB/wQFMAMBAf8wCgYIKoZIzj0EAwIDSAAwRQIhAKbay+o35m55eAWZ\nGXYgZwR5mTzN8hZisZIivbxIiDRoAiAwPxS4PKFln+ryJt9X5T7N0uQRpLcpt+bS\nl3mbE9p5Ng==\n-----END CERTIFICATE-----\n","issuanceLifetime":"86400s","clockSkewAllowance":"20s","scheme":"linkerd.io/tls"},"autoInjectContext":null,"omitWebhookSideEffects":false,"clusterDomain":"cluster.local"}
  install: '{"cliVersion":"stable-2.7.1","flags":[]}'
  proxy: |
    {"proxyImage":{"imageName":"gcr.io/linkerd-io/proxy","pullPolicy":"IfNotPresent"},"proxyInitImage":{"imageName":"gcr.io/linkerd-io/proxy-init","pullPolicy":"IfNotPresent"},"controlPort":{"port":4190},"ignoreInboundPorts":[],"ignoreOutboundPorts":[],"inboundPort":{"port":4143},"adminPort":{"port":4191},"outboundPort":{"port":4140},"resource":{"requestCpu":"","requestMemory":"","limitCpu":"","limitMemory":""},"proxyUid":"2102","logLevel":{"level":"warn,linkerd=info"},"disableExternalProfiles":true,"proxyVersion":"stable-2.7.1","proxyInitImageVersion":"v1.3.2","debugImage":{"imageName":"gcr.io/linkerd-io/debug","pullPolicy":"IfNotPresent"},"debugImageVersion":"stable-2.7.1"}
kind: ConfigMap
metadata:
  annotations:
    linkerd.io/created-by: linkerd/cli stable-2.7.1
  creationTimestamp: "2020-05-25T08:31:42Z"
  labels:
    linkerd.io/control-plane-component: controller
    linkerd.io/control-plane-ns: linkerd
  name: linkerd-config
  namespace: linkerd
  resourceVersion: "231354"
  selfLink: /api/v1/namespaces/linkerd/configmaps/linkerd-config
  uid: 5fcbf117-9d3a-4bff-9b6d-0e2b72ec984e

apiVersion: v1
kind: ServiceAccount
metadata:
  creationTimestamp: "2020-05-25T08:31:42Z"
  labels:
    linkerd.io/control-plane-component: tap
    linkerd.io/control-plane-ns: linkerd
  name: linkerd-tap
  namespace: linkerd
  resourceVersion: "231346"
  selfLink: /api/v1/namespaces/linkerd/serviceaccounts/linkerd-tap
  uid: f3b16701-1ca6-4b7d-87fb-10219adcdbe4
secrets:
- name: linkerd-tap-token-rf58d

[Aisuko]

@grampelberg Sorry to bother you, sir. Is there any API of LInked2 I can use to pre-check and deploy it to the cluster? Yes, finally I find maybe the deploy function may have some issue, so I'd like import linkerd2 API into our adapter

[grampelberg]

You should be able to use the CLI as a library. If some of the functions aren't exported we'd love some PRs to make that a possibility =)

[Aisuko]

Thanks for your quick reply. How could I get more information at the first time about PRs of the exported more function of linkerd2 I mean if there exists any channel of Slack? I have a huge issue need to fix as you know😱😱

[Aisuko]

And I'd like to be a contributor to making this happen quickly.

[ihcsim]

@Aisuko Join the #contributors channel. We can help you there.

[Aisuko]

Hi guys, thanks for your guy's advice and it's my pleasure that has a chance to discussed with your guys. Finally, the issue is at the manifest of split linkerd2, every time when we create the resource( use client-go dynamic client) we split the linkerd2 manifest strings there may lose some CRDs resource. So, I changed the function of split the YAML and modified Linked2 source code to make my life easy.

1. In fact, we're going to use helm v3 in client-go to deployment linekrd2, but linkerd2 used helmv2.16.8(I believe that need to improve)
2. I expose the function of renderMaintest and use the return manifest to deploy Linkerd2(

   linkerd2/cli/cmd/check.go

   Line 171 in 311a97a

   installManifest, err = renderInstallManifest()

   )
3. Although I want to use (

   linkerd2/cli/cmd/check.go

   Line 147 in 311a97a

   func configureAndRunChecks(wout io.Writer, werr io.Writer, stage string, options *checkOptions) error {

   ) function to do the pre-check, but(

   linkerd2/cli/cmd/check.go

   Line 201 in 311a97a

   KubeConfig: kubeconfigPath,

   ) the kubeconfigPath may not make sense for me, because I can only provide the []byte type of the kuebconfig.