How to set up the cluster

[phumberdroz]

Hey,

I am currently trying this out and struggle a bit.

I deployed this with roughly these steps:

unset KUBECONFIG
kind delete cluster
kind create cluster
clusterctl init --core cluster-api --bootstrap kubeadm --control-plane kubeadm --infrastructure hetzner:v1.0.0-beta.10
export CLUSTER_NAME="main-cluster"
export HCLOUD_REGION="fsn1"
export CONTROL_PLANE_MACHINE_COUNT=3
export WORKER_MACHINE_COUNT=3
export KUBERNETES_VERSION=1.25.2
export HCLOUD_CONTROL_PLANE_MACHINE_TYPE=cpx31
export HCLOUD_WORKER_MACHINE_TYPE=cpx31
export HCLOUD_TOKEN="<TOKEN>"
export HETZNER_ROBOT_USER="<USER>"
export HETZNER_ROBOT_PASSWORD="<PASS>"
export HETZNER_SSH_PUB_PATH="/home/phumberdroz/.ssh/hetzner.pub"
export HETZNER_SSH_PRIV_PATH="/home/phumberdroz/.ssh/hetzner"
export HCLOUD_SSH_KEY="hetzner-k8s"
export GITHUB_TOKEN="<TOKEN>"
kubectl create secret generic hetzner --from-literal=hcloud=$HCLOUD_TOKEN --from-literal=robot-user=$HETZNER_ROBOT_USER --from-literal=robot-password=$HETZNER_ROBOT_PASSWORD
kubectl create secret generic robot-ssh --from-literal=sshkey-name=cluster --from-file=ssh-privatekey=$HETZNER_SSH_PRIV_PATH --from-file=ssh-publickey=$HETZNER_SSH_PUB_PATH
kubectl patch secret hetzner -p '{"metadata":{"labels":{"clusterctl.cluster.x-k8s.io/move":""}}}'
kubectl patch secret robot-ssh -p '{"metadata":{"labels":{"clusterctl.cluster.x-k8s.io/move":""}}}'
clusterctl generate cluster main-cluster --kubernetes-version v1.25.2 --control-plane-machine-count=3 --worker-machine-count=3 --flavor hetzner-hcloud-control-planes > my-cluster.yaml
kubectl apply -f my-cluster.yaml -f my-nodes.yml

I have 3 Bare Metal Nodes that I hope to use all 3 of them are AX101 nodes in FSN1.
Node 01: fails every couple of tries and then succeeds.
Node 02: never fails and always sets up fine on the first try
Node 03: I have never seen it succeed. (https://gist.github.com/phumberdroz/acb4becc47d3162600f18b6f8127218f)

A couple of questions:

1. which resources should I move from the bootstrap cluster to the workload cluster?
2. What can I do in regards to node-03? Any other logs I can check?

[batistein]

1. You need to move all resources incl. hetznerbaremetalhost objects.
2. You can ssh to the baremetal node and check the logs of cloud-init

[phumberdroz]

regarding 1: I guess this only means manifests of the my-cluster.yaml? Or does this include other files in the bootstrap cluster?
regarding 2:

I found on the rescue system now this piece:

[22:55:35] # use config file /autosetup for autosetup
[22:55:35] # use post-install file /root/post-install.sh
[22:55:35] # OPT_CONFIGFILE:   /autosetup
[22:55:35] # starting installimage
[22:55:35] -------------------------------------
[22:55:36] :   Hardware data:
[22:55:36] :   CPU1: AMD Ryzen 9 5950X 16-Core Processor (Cores 32)
[22:55:36] :   Memory:  128746 MB
[22:55:36] :   Disk /dev/nvme0n1: 3840 GB (=> 3576 GiB)
[22:55:36] :   Disk /dev/nvme1n1: 3840 GB (=> 3576 GiB)
[22:55:36] :   Total capacity 7153 GiB with 2 Disks
[22:55:36] -------------------------------------
[22:55:36] # make clean config
[22:55:36] # SYSTYPE: System Product Name
[22:55:36] # SYSMFC:  ASUS
[22:55:36] # executing autosetup ...
[22:55:36] # SYSTYPE: System Product Name
[22:55:36] # SYSMFC:  ASUS
[22:55:36] # checking if the script is disabled
[22:55:36] # validating config ...
[22:55:36] :   /boot : 1024
[22:55:36] :   / : all
[22:55:36] Image info:
[22:55:36] :   DISTRIB ID:               ubuntu
[22:55:36] :   DISTRIB RELEASE/CODENAME: 2004
[22:55:36] :   Size of the first hdd is: 3840755982336
[22:55:36] :   ERROR: ESP missing or multiple ESP found
[22:55:36] cleaning up
[22:55:36] :   umount: /installimage.xXPbM/hdd: not mounted

The autosetup looks like this:

DRIVE1 /dev/nvme0n1
DRIVE2 /dev/nvme1n1

HOSTNAME bm-main-cluster-md-1-n227j
SWRAID 0

PART /boot ext4 1024M
PART / ext4 all



IMAGE /root/.oldroot/nfs/install/../images/Ubuntu-2004-focal-64-minimal-hwe.tar.gz

[phumberdroz]

In regards to server03: this seems to be the issue: hetzneronline/installimage#63 (comment)