Cannot install linkerd-smi

[ingshtrom]

I am trying to install the Linkerd SMI extension, but I am having trouble getting the linkerd-smi binary to connect to my EKS cluster, even though the linkerd binary works just fine.

🚀💥 linkerd smi install
Empty KUBERNETES_EXEC_INFO, defaulting to client.authentication.k8s.io/v1alpha1. This is likely a bug in your Kubernetes client. Please update your Kubernetes client.
Cannot connect to Kubernetes: Get "https://EFFB06168AD5A29FBDA7D46487268B9C.gr7.us-east-1.eks.amazonaws.com/version?timeout=30s": getting credentials: exec plugin is configured to use API version client.authentication.k8s.io/v1beta1, plugin returned version client.authentication.k8s.io/v1alpha1
Validate the install with: linkerd check
exit status 1

If I run something from the linkerd binary, it connects fine

🚀 linkerd check
Linkerd core checks
===================

kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ control plane pods are ready
‼ cluster networks can be verified
    the following nodes do not expose a podCIDR:
        ip-10-128-101-102.ec2.internal
        ip-10-128-102-67.ec2.internal
        ip-10-128-105-123.ec2.internal
        ip-10-128-11-75.ec2.internal
        ip-10-128-120-46.ec2.internal
        ip-10-128-123-243.ec2.internal
        ip-10-128-128-150.ec2.internal
        ip-10-128-14-220.ec2.internal
        ip-10-128-143-4.ec2.internal
        ip-10-128-15-137.ec2.internal
        ip-10-128-156-201.ec2.internal
        ip-10-128-159-188.ec2.internal
        ip-10-128-26-69.ec2.internal
        ip-10-128-29-53.ec2.internal
        ip-10-128-40-249.ec2.internal
        ip-10-128-42-102.ec2.internal
        ip-10-128-43-106.ec2.internal
        ip-10-128-71-191.ec2.internal
        ip-10-128-72-194.ec2.internal
        ip-10-128-75-208.ec2.internal
        ip-10-128-76-128.ec2.internal
        ip-10-128-92-17.ec2.internal
    see https://linkerd.io/2.11/checks/#l5d-cluster-networks-verified for hints

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ proxy-init container runs as root user if docker container runtime is used

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust anchor

linkerd-webhooks-and-apisvc-tls
-------------------------------
√ proxy-injector webhook has valid cert
√ proxy-injector cert is valid for at least 60 days
√ sp-validator webhook has valid cert
√ sp-validator cert is valid for at least 60 days
√ policy-validator webhook has valid cert
√ policy-validator cert is valid for at least 60 days

linkerd-version
---------------
√ can determine the latest version
√ cli is up-to-date

control-plane-version
---------------------
√ can retrieve the control plane version
√ control plane is up-to-date
√ control plane and cli versions match

linkerd-control-plane-proxy
---------------------------
√ control plane proxies are healthy
√ control plane proxies are up-to-date
√ control plane proxies and cli versions match

linkerd-ha-checks
-----------------
‼ pod injection disabled on kube-system
    kube-system namespace needs to have the label config.linkerd.io/admission-webhooks: disabled if injector webhook failure policy is Fail
    see https://linkerd.io/2.11/checks/#l5d-injection-disabled for hints
√ multiple replicas of control plane pods

Linkerd extensions checks
=========================

linkerd-viz
-----------
√ linkerd-viz Namespace exists
√ linkerd-viz ClusterRoles exist
√ linkerd-viz ClusterRoleBindings exist
√ tap API server has valid cert
√ tap API server cert is valid for at least 60 days
√ tap API service is running
√ linkerd-viz pods are injected
√ viz extension pods are running
√ viz extension proxies are healthy
√ viz extension proxies are up-to-date
√ viz extension proxies and cli versions match
√ prometheus is installed and configured correctly
√ can initialize the client
√ viz extension self-check

Status check results are √

The version of my client and server

🚀 kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:46:05Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.15-eks-0d102a7", GitCommit:"a8ff0230f2c155701a76c41fdcbddff180dc1833", GitTreeState:"clean", BuildDate:"2022-02-17T16:47:50Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.24) and server (1.20) exceeds the supported minor version skew of +/-1

I remember when we upgraded our AWS EKS cluster from 1.19 to 1.20 we had to update all of our kubeconfigs because the authorization plugin had a new API to hit in order to get a token for the Kubernetes API Server. At least that is what I think had happened 😅

# this is a `user` defined in my kubeconfig file
  user:
    exec:
-      apiVersion: client.authentication.k8s.io/v1alpha1
+      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - --region
      - us-east-1
      - eks
      - get-token
      - --cluster-name
      - <redacted>
      command: aws
      env: null
      interactiveMode: IfAvailable
      provideClusterInfo: false

How can I work around this? It looks like the kubernetes client in the linkderd-cmi repo is up to date enough, or so I am thinking at first blush.

Is there something that I am missing?

[alpeb]

What version are you on? Latest is

$ linkerd smi version
v0.2.0

Note that still has a k8s client (client-go dependency) on v0.19.3, but I don't see that being a problem.
This still looks like an issue with your kubeconfig (no clue though why it works with the linkerd binary). Can you take another look at your kubeconfig?
Another option is to try running things from the tip of the git branch:

$ go run ./cli install

[ingshtrom]

🚀 linkerd smi version
v0.2.0

Here is what I get when installing from the tip of the git branch

🚀💥 go run ./cli install
... # lots of go modules installed
Cannot find Linkerd: invalid CIDR address:
Validate the install with: linkerd check
exit status 1

So then I run linkerd check with the output from my original comment. I am guessing the issue of nodes not exposing podCIDR properties is breaking the SMI install for some reason

...
linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ control plane pods are ready
‼ cluster networks can be verified
    the following nodes do not expose a podCIDR:
        ip-10-128-101-102.ec2.internal
        ip-10-128-102-67.ec2.internal
        ip-10-128-105-123.ec2.internal
        ip-10-128-11-75.ec2.internal
        ip-10-128-120-46.ec2.internal
        ip-10-128-123-243.ec2.internal
        ip-10-128-128-150.ec2.internal
        ip-10-128-14-220.ec2.internal
        ip-10-128-143-4.ec2.internal
        ip-10-128-15-137.ec2.internal
        ip-10-128-156-201.ec2.internal
        ip-10-128-159-188.ec2.internal
        ip-10-128-26-69.ec2.internal
        ip-10-128-29-53.ec2.internal
        ip-10-128-40-249.ec2.internal
        ip-10-128-42-102.ec2.internal
        ip-10-128-43-106.ec2.internal
        ip-10-128-71-191.ec2.internal
        ip-10-128-72-194.ec2.internal
        ip-10-128-75-208.ec2.internal
        ip-10-128-76-128.ec2.internal
        ip-10-128-92-17.ec2.internal
    see https://linkerd.io/2.11/checks/#l5d-cluster-networks-verified for hints
...

It is weird because the configuration docs for kubelet say the podCIDR property is grabbed from the API Server. Maybe linkerd-smi is not grabbing from the same place in the API server as the kubelet is? Or why wouldn't the Kubelet set the node's podCIDR property after it has queried it from the API server? I don't know enough about kubelet to be able to answer these questions--I am only thinking aloud.

I guess one question I have is whether there is any distribution-specific documentation for Linkerd? Surely people have used Linkerd in EKS successfully 🤔

[alpeb]

Thanks for the detailed feedback 👍
As you can see in the linkerd check output, the podCIDR issue is just a warning, not a fatal error, so checks still pass. But I just realized linkerd-smi is still depending internally on an old linkerd library where that check was still not marked as just a warning and that's why it fails. Until we address that, you should be able to unblock yourself using the --skip-checks flag.