Introduce playbook to update kubernetes cluster from one patch version to another

Author: kishen-v

kubetest2-tf/data/k8s-ansible/docs/update-k8s.md

@@ -0,0 +1,56 @@
+## Update build cluster's Kubernetes version.
+
+This guide covers the steps to update the build cluster's kubernetes version from one patch version to another, or
+across minor versions through a playbook.
+
+This update guide is applicable for HA clusters, and is extensively used to automate and update the nodes with a particular
+version of Kubernetes.
+
+The playbook is written to install the kubeadm utility through a package manager post, and then proceeding to perform
+the cluster upgrade. Initially, all the master nodes are updated, followed by which the worker nodes are updated.
+
+
+#### Prerequisites
+```
+   Ansible
+   Kubeconfig of the cluster
+```
+
+#### Steps to follow:
+1. From the k8s-ansible directory, generate the hosts.yml file on which the Kubernetes cluster updates are to be performed.
+   In this case, one can use the `hosts.yml` file under `examples/containerd-cluster/hosts.yml` to contain the IP(s)
+   of the following nodes: Workers and Masters.
+```
+[masters]
+10.20.177.51
+10.20.177.26
+10.20.177.227
+[workers]
+10.20.177.39
+```
+
+The following lines may additionally be needed in hosts.yml file, in case the cluster is associated with a bastion node.
+Here, the private key helps with establishing the SSH connection to the bastion, and X refers to the bastion's IP address.
+```
+[workers:vars]
+ansible_ssh_common_args='-o ProxyCommand="ssh -W %h:%p -i <path-to-private-key> -q root@X" -i <path-to-private-key>'
+
+[masters:vars]
+ansible_ssh_common_args='-o ProxyCommand="ssh -W %h:%p -i <path-to-private-key> -q root@X" -i <path-to-private-key>'
+```
+2. Set the path to the `kubeconfig` of the cluster under `group_vars/all` - under the `kubeconfig_path` variable.
+3. Set the Kubernetes versions in the following variables in `group_vars/all` - under the following variables, for example
+   ```
+   kubernetes_major_minor: "1.32"
+   kubernetes_patch: "2"
+   ```
+3. Once the above are set use the following command to update the nodes -
+   `ansible-playbook -i examples/containerd-cluster/hosts.yml update-k8s-version.yml --extra-vars group_vars/all`
+4. This will proceed to update the nodes. Post update, the same can be verified by executing the kubectl version - under
+   `Server Version` field
+```
+# kubectl version
+Client Version: v1.32.3
+Kustomize Version: v5.5.0
+Server Version: v1.33.1
+```

kubetest2-tf/data/k8s-ansible/docs/update-os.md

@@ -0,0 +1,44 @@
+## Update build cluster's nodes with latest Kernel/patches.
+
+This guide covers the steps to update the build cluster's OS/Kernel/packages to the latest available versions based
+on their availability through package managers. It is necessary to keep the nodes to have the latest security patches
+installed and have the kernel up-to-date.
+
+This update guide is applicable for HA clusters, and is extensively used to automate and perform the rolling updates of
+the nodes.
+
+The strategy used in updating the nodes is by performing rolling-updates to the nodes, post confirming that there are no
+pods in the `test-pods` namespace that generally contains the prow-job workloads. The playbook has the mechanism to wait
+until the namespace is free from running pods, however, there may be a necessity to terminate the boskos-related pods
+as these are generally long-running in nature.
+
+#### Prerequisites
+```
+   ansible
+   private key of the bastion node.
+```
+
+#### Steps to follow:
+1. From the k8s-ansible directory, generate the hosts.yml file on which the OS updates are to be performed.
+   In this case, one can use the hosts.yml file under `examples/containerd-cluster/hosts.yml` to contain the IP(s)
+   of the following nodes - Bastion, Workers and Masters.
+```
+[bastion]
+1.2.3.4
+[masters]
+10.20.177.51
+10.20.177.26
+10.20.177.227
+[workers]
+10.20.177.39
+
+[workers:vars]
+ansible_ssh_common_args='-o ProxyCommand="ssh -W %h:%p -i <path-to-private-key> -q root@X" -i <path-to-private-key>'
+
+[masters:vars]
+ansible_ssh_common_args='-o ProxyCommand="ssh -W %h:%p -i <path-to-private-key> -q root@X" -i <path-to-private-key>'
+```
+2. Set the path to the `kubeconfig` of the cluster under group_vars/all - under the `kubeconfig_path` variable.
+3. Once the above are set use the following command to update the nodes - 
+   `ansible-playbook -i examples/containerd-cluster/hosts.yml update-os-packages.yml --extra-vars group_vars/all`
+4. This will proceed to update the nodes, and reboot them serially if necessary.

kubetest2-tf/data/k8s-ansible/group_vars/all

@@ -94,3 +94,8 @@ cni_plugins_tarball: "cni-plugins-linux-{{ ansible_architecture }}-{{ cni_plugin
 
 # NFS server details
 nfs_directory: "/var/nfsshare"
+
+
+##### Kubernetes version update related fields #####
+kubernetes_major_minor: ""
+kubernetes_patch: ""

kubetest2-tf/data/k8s-ansible/roles/download-k8s/tasks/main.yml

@@ -10,7 +10,7 @@
 - name: Extract the k8s bits
   unarchive:
     src: "{{ src_template }}/{{ item }}"
-    dest: "/usr/local/bin/"
+    dest: "/usr/bin/"
     remote_src: yes
     extra_opts:
       - --strip-components=3
@@ -21,7 +21,7 @@
 - name: Import control plane component container images
   shell: |
     export ARCH={{ ansible_architecture }}
-    ctr -n k8s.io images import "/usr/local/bin/{{ item }}.tar"
+    ctr -n k8s.io images import "/usr/bin/{{ item }}.tar"
     ctr -n k8s.io images ls -q | grep -e {{ item }} | xargs -L 1 -I '{}' /bin/bash -c 'ctr -n k8s.io images tag "{}" "$(echo "{}" | sed s/-'$ARCH':/:/)"'
   with_items:
     - kube-apiserver
@@ -33,7 +33,7 @@
 - name: Import common container images required for setting up cluster
   shell: |
     export ARCH={{ ansible_architecture }}
-    ctr -n k8s.io images import "/usr/local/bin/{{ item }}.tar"
+    ctr -n k8s.io images import "/usr/bin/{{ item }}.tar"
     ctr -n k8s.io images ls -q | grep -e {{ item }} | xargs -L 1 -I '{}' /bin/bash -c 'ctr -n k8s.io images tag "{}" "$(echo "{}" | sed s/-'$ARCH':/:/)"'
   with_items:
     - kube-proxy

kubetest2-tf/data/k8s-ansible/roles/install-k8s/tasks/main.yml

@@ -45,6 +45,20 @@
     dest: /usr/lib/systemd/system/kubelet.service
     mode: '0644'
 
+- name: Ensure destination directory exists
+  ansible.builtin.file:
+    path: /etc/systemd/system/kubelet.service.d
+    state: directory
+    mode: '0755'
+
+# https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/kubelet-integration/#the-kubelet-drop-in-file-for-systemd
+- name: Generate an override configuration to /etc/systemd/system/kubelet.service
+  template:
+    src: override.conf.j2
+    dest: /etc/systemd/system/kubelet.service.d/override.conf
+    mode: '0644'
+
+
 - name: Enable and start kubelet
   systemd:
     name: kubelet

kubetest2-tf/data/k8s-ansible/roles/install-k8s/templates/kubelet.service.j2

@@ -7,7 +7,7 @@ After=network-online.target
 [Service]
 Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
 Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
-ExecStart=/usr/local/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_EXTRA_ARGS {{ kubelet_extra_args }}
+ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_EXTRA_ARGS {{ kubelet_extra_args }}
 Restart=always
 StartLimitInterval=0
 RestartSec=10

kubetest2-tf/data/k8s-ansible/roles/install-k8s/templates/override.conf.j2

@@ -0,0 +1,17 @@
+[Unit]
+Description=kubelet: The Kubernetes Node Agent
+Documentation=https://kubernetes.io/docs/
+Wants=network-online.target
+After=network-online.target
+
+[Service]
+Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
+Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
+ExecStart=
+ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_EXTRA_ARGS {{ kubelet_extra_args }}
+Restart=always
+StartLimitInterval=0
+RestartSec=10
+
+[Install]
+WantedBy=multi-user.target

kubetest2-tf/data/k8s-ansible/roles/reboot-sequentially/tasks/cordon-phase.yaml

@@ -0,0 +1,43 @@
+- name: Resolve Kubernetes node name from inventory IP
+  shell: |
+    kubectl get nodes -o jsonpath="{range .items[*]}{.metadata.name} {.status.addresses[?(@.type=='InternalIP')].address}{'\n'}{end}" --kubeconfig {{ kubeconfig_path }} |\
+    grep {{ inventory_hostname }} | awk '{print $1}'
+  register: node_name
+  delegate_to: "{{ groups['masters'][0] }}"
+
+- name: Cordon the kubernetes node
+  shell: |
+    kubectl cordon {{ node_name.stdout }} --kubeconfig {{ kubeconfig_path }}
+  register: drain_output
+  changed_when: "'already cordoned' not in drain_output.stdout"
+  delegate_to: "{{ groups['masters'][0] }}"
+
+- name: Check and wait if there are any running jobs that need to complete before draining.
+  shell: |
+    kubectl get pods -n test-pods \
+    --kubeconfig {{ kubeconfig_path }} \
+    --field-selector spec.nodeName={{ node_name.stdout }},status.phase=Running \
+    -o go-template={% raw %}'{{range .items}}{{if or (not .metadata.ownerReferences) (ne (index .metadata.ownerReferences 0).kind "DaemonSet")}}{{.metadata.name}}{{"\n"}} {{end}}{{end}}'{% endraw %} \
+    | wc -l
+  register: running_pod_count
+  retries: 360
+  delay: 30
+  until: running_pod_count.stdout | int == 0
+  delegate_to: "{{ groups['masters'][0] }}"
+
+- name: Drain Kubernetes Node
+  shell: |
+    kubectl drain {{ node_name.stdout }} --ignore-daemonsets --delete-emptydir-data --kubeconfig {{ kubeconfig_path }}
+  register: drain_output
+  changed_when: "'already cordoned' not in drain_output.stdout"
+  delegate_to: "{{ groups['masters'][0] }}"
+
+- name: Wait for all pods to be evicted
+  shell: |
+    kubectl get pods -n test-pods --field-selector spec.nodeName={{ node_name.stdout }},status.phase=Running -o go-template='{% raw %}{{range .items}}{{if or (not .metadata.ownerReferences) (ne (index .metadata.ownerReferences 0).kind "DaemonSet")}}{{.metadata.name}}{{"\\n"}}{{end}}{{end}}{% endraw %}' | wc -l
+  register: pods_remaining
+  until: pods_remaining.stdout | int == 0
+  retries: 10
+  delay: 15
+  delegate_to: "{{ groups['masters'][0] }}"
+

kubetest2-tf/data/k8s-ansible/roles/reboot-sequentially/tasks/main.yaml

@@ -1,61 +1,11 @@
 - block:
-    - name: Resolve Kubernetes node name from inventory IP
-      shell: |
-        kubectl get nodes -o jsonpath="{range .items[*]}{.metadata.name} {.status.addresses[?(@.type=='InternalIP')].address}{'\n'}{end}" --kubeconfig {{ kubeconfig_path }} |\
-        grep {{ inventory_hostname }} | awk '{print $1}'
-      register: node_name
-      delegate_to: "{{ groups['masters'][0] }}"
-
-    - name: Cordon the kubernetes node
-      shell: |
-        kubectl cordon {{ node_name.stdout }}
-      register: drain_output
-      changed_when: "'already cordoned' not in drain_output.stdout"
-      delegate_to: "{{ groups['masters'][0] }}"
-
-    - name: Check and wait if there are any running jobs that need to complete before draining.
-      shell: |
-        kubectl get pods -n test-pods \
-        --kubeconfig {{ kubeconfig_path }} \
-        --field-selector spec.nodeName={{ node_name.stdout }},status.phase=Running \
-        -o go-template={% raw %}'{{range .items}}{{if or (not .metadata.ownerReferences) (ne (index .metadata.ownerReferences 0).kind "DaemonSet")}}{{.metadata.name}}{{"\n"}} {{end}}{{end}}'{% endraw %} \
-        | wc -l
-      register: running_pod_count
-      retries: 360
-      delay: 30
-      until: running_pod_count.stdout | int == 0
-      delegate_to: "{{ groups['masters'][0] }}"
-
-    - name: Drain Kubernetes Node
-      shell: |
-        kubectl drain {{ node_name.stdout }} --ignore-daemonsets --delete-emptydir-data --kubeconfig {{ kubeconfig_path }}
-      register: drain_output
-      changed_when: "'already cordoned' not in drain_output.stdout"
-      delegate_to: "{{ groups['masters'][0] }}"
-
-    - name: Wait for all pods to be evicted
-      shell: |
-        kubectl get pods -n test-pods --field-selector spec.nodeName={{ node_name.stdout }},status.phase=Running -o go-template='{% raw %}{{range .items}}{{if or (not .metadata.ownerReferences) (ne (index .metadata.ownerReferences 0).kind "DaemonSet")}}{{.metadata.name}}{{"\\n"}}{{end}}{{end}}{% endraw %}' | wc -l
-      register: pods_remaining
-      until: pods_remaining.stdout | int == 0
-      retries: 10
-      delay: 15
-      delegate_to: "{{ groups['masters'][0] }}"
+    - name: Include the cordon phase tasks to prepare nodes for upgrade
+      include_tasks: cordon-phase.yaml
 
     - name: Reboot node
       reboot:
 
-    - name: Wait for node to become Ready
-      shell: |
-        kubectl get node {{ node_name.stdout }} --kubeconfig {{ kubeconfig_path }} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'
-      register: node_status
-      until: node_status.stdout == "True"
-      retries: 20
-      delay: 15
-      delegate_to: "{{ groups['masters'][0] }}"
-
-    - name: Uncordon the node
-      shell: kubectl uncordon {{ node_name.stdout }} --kubeconfig {{ kubeconfig_path }}
-      delegate_to: "{{ groups['masters'][0] }}"
+    - name: Include the uncordon phase tasks post upgrade of node
+      include_tasks: uncordon-phase.yaml
 
   when: reboot_check is defined and reboot_check.rc == 1

kubetest2-tf/data/k8s-ansible/roles/reboot-sequentially/tasks/uncordon-phase.yaml

@@ -0,0 +1,13 @@
+- name: Wait for node to become Ready
+  shell: |
+    kubectl get node {{ node_name.stdout }} --kubeconfig {{ kubeconfig_path }} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'
+  register: node_status
+  until: node_status.stdout == "True"
+  retries: 20
+  delay: 15
+  delegate_to: "{{ groups['masters'][0] }}"
+
+- name: Uncordon the node
+  shell: kubectl uncordon {{ node_name.stdout }} --kubeconfig {{ kubeconfig_path }}
+  delegate_to: "{{ groups['masters'][0] }}"
+