kube-apiserver unreachable from management cluster before CNI installation

[aibarbetta]

What would you like to be documented:
Install a CNI provider is right after workload cluster creation from the management cluster and does not clarify if the CNI installation command should be run from the same place or not.
In the previous step, the user is left with a kubeconfig file to operate the workload cluster from the management cluster; however, kubectl commands to the workload cluster's API hang until they timeout. Is this expected since CNI is not installed yet?

Why is this needed:
To improve the user experience, it would be helpful to document the expected location from which the CNI installation is expected to run to function properly.

[joekr]

Thanks for reaching out. So I want to understand better. Are you saying you are unable to install the CNI on the workload cluster?

My typical flow is:
On the management cluster get the workload cluster config

kubectl get clusters -A                                                                                                                                                                                                                                  
NAMESPACE   NAME                     CLUSTERCLASS   PHASE         AGE    VERSION
default     capoci-example-cluster                  Provisioned   52m

clusterctl get kubeconfig capoci-example-cluster -n default > capoci-example-cluster.kubeconfig

Then install the CNI using this new config

kubectl --kubeconfig=capoci-example-cluster.kubeconfig \
  apply -f https://docs.projectcalico.org/v3.21/manifests/calico.yaml

But you are saying the kubctl command is timing out? Is the cluster publicly accessible? If not are you using any bastions or anything to try to access it?

[aibarbetta]

Are you saying you are unable to install the CNI on the workload cluster?

Yes, on the management cluster I get:

$ clusterctl get kubeconfig iad1 > iad1.kubeconfig

$ kubectl --kubeconfig=iad1.kubeconfig get pods -A --v=6
I1016 17:39:10.744929  244605 cmd.go:527] kubectl command headers turned on
I1016 17:39:10.748811  244605 loader.go:402] Config loaded from file:  iad1.kubeconfig
I1016 17:39:10.749235  244605 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I1016 17:39:10.749248  244605 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I1016 17:39:10.749253  244605 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I1016 17:39:10.749258  244605 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I1016 17:39:10.749264  244605 envvar.go:172] "Feature gate default state" feature="InOrderInformers" enabled=true
I1016 17:39:40.751969  244605 round_trippers.go:632] "Response" verb="GET" url="https://<iad1-apiserver-nlb-public-ip>:6443/api?timeout=32s" status="" milliseconds=30002
E1016 17:39:40.752064  244605 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://<iad1-apiserver-nlb-public-ip>:6443/api?timeout=32s\": dial tcp <iad1-apiserver-nlb-public-ip>:6443: i/o timeout"
I1016 17:39:40.752101  244605 cached_discovery.go:120] skipped caching discovery info due to Get "https://<iad1-apiserver-nlb-public-ip>:6443/api?timeout=32s": dial tcp <iad1-apiserver-nlb-public-ip>:6443: i/o timeout
I1016 17:40:10.753313  244605 round_trippers.go:632] "Response" verb="GET" url="https://<iad1-apiserver-nlb-public-ip>:6443/api?timeout=32s" status="" milliseconds=30001
...
E1016 17:41:40.756184  244605 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://<iad1-apiserver-nlb-public-ip>:6443/api?timeout=32s\": dial tcp <iad1-apiserver-nlb-public-ip>:6443: i/o timeout"
I1016 17:41:40.756199  244605 cached_discovery.go:120] skipped caching discovery info due to Get "https://<iad1-apiserver-nlb-public-ip>:6443/api?timeout=32s": dial tcp <iad1-apiserver-nlb-public-ip>:6443: i/o timeout
I1016 17:41:40.756243  244605 helpers.go:264] Connection error: Get https://<iad1-apiserver-nlb-public-ip>:6443/api?timeout=32s: dial tcp <iad1-apiserver-nlb-public-ip>:6443: i/o timeout
Unable to connect to the server: dial tcp <iad1-apiserver-nlb-public-ip>:6443: i/o timeout

Is the cluster publicly accessible?

It should be. The iad1-apiserver-nlb is in the control-plane-endpoint subnet, and uses NSG control-plane-endpoint, which has the following rule to allow access from everywhere:

Direction	Source Type	Source	Protocol	Source Port Range	Destination Port Range	Description
Ingress	CIDR	0.0.0.0/0	TCP	All	6443	External access to Kubernetes API endpoint

The Kubernetes API is up and works if I use the same kubeconfig from the control-plane instance with server: https://localhost:6443:

[opc@iad1-control-plane-rbptx ~]$ kubectl get pod -A
NAMESPACE     NAME                                               READY   STATUS                  RESTARTS         AGE
kube-system   calico-kube-controllers-7498b9bb4c-g2pvv           0/1     Pending                 0                135m
kube-system   etcd-iad1-control-plane-rbptx                      1/1     Running                 0                175m
kube-system   kube-apiserver-iad1-control-plane-rbptx            1/1     Running                 0                175m
kube-system   kube-controller-manager-iad1-control-plane-rbptx   1/1     Running                 0                175m
kube-system   kube-scheduler-iad1-control-plane-rbptx            1/1     Running                 0                175m

If this isn't expected I'm not sure what is happening :/

[aibarbetta]

This was my bad, I had something wrong in my configuration that caused the kube-proxy to not start in my workload cluster. I can now use kubectl to my workload cluster from a node that is part of my management cluster:

$ kubectl --kubeconfig=iad1.kubeconfig get pods -A
NAMESPACE     NAME                                               READY   STATUS    RESTARTS   AGE
kube-system   coredns-668d6bf9bc-2jrjt                           0/1     Pending   0          2m3s
kube-system   coredns-668d6bf9bc-fvhfh                           0/1     Pending   0          2m3s
kube-system   etcd-iad1-control-plane-8tbjz                      1/1     Running   0          2m28s
kube-system   kube-apiserver-iad1-control-plane-8tbjz            1/1     Running   0          2m28s
kube-system   kube-controller-manager-iad1-control-plane-8tbjz   1/1     Running   0          2m28s
kube-system   kube-proxy-gtzft                                   1/1     Running   0          2m3s
kube-system   kube-scheduler-iad1-control-plane-8tbjz            1/1     Running   0          2m28s

[joekr]

Awesome! So it is working as expected?

Are there things in the docs you still wanted to see updated?

[aibarbetta]

Awesome! So it is working as expected?

Yes, I was able to install the CNI using the kubeconfig from the management cluster.

One question before we close this. The doc says:

After creating a workload cluster, a CNI provider must be installed in the workload cluster. Until you install a a CNI provider, the cluster nodes will not go into the Ready state.

After CNI installation, kubectl get machines still returned Ready=False for the control-plane node. I realized it was because of an etcd health check that is part of the machine readiness gate. The capi-kubeadm-control-plane-controller-manager was failing to do this check.
I traced it back to the workload cluster, which was failing to resolve node names. Features like kubectl logs, and port-forwarding (which I'm guessing is what the capi-kubeadm-control-plane-controller-manager uses to do health checking) were failing with node name resolutions. I was able to fix it by adding --node-ip to the kubelets and restarting them.

Is this expected? Is the OCI Cloud Controller Manager (CCM) required to populate internal IPs in the nodes and enable features like kubectl logs, port-forward, and consequently machine health checking from the management cluster?

[joekr]

CCM should handle all of this automatically and yes I believe it is required. Once you install it does it work as expected?