[QA integration tests] 'ERROR: failed to create cluster: failed to join node with kubeadm' during k8s cluster creation via kind

[andrewluckyguy]

Describe the bug

We observed the error during k8s cluster creation:

./kind create cluster --image kindest/node:v1.32.2 --config qubership-test-pipelines/kind-configs/kind-multi-node.yaml --wait 5m

I0617 13:38:11.104739     174 kubelet.go:214] [kubelet-start] Checking for an existing Node in the cluster with name "kind-worker3" and status "Ready"
I0617 13:38:11.105854     174 round_trippers.go:560] GET https://kind-external-load-balancer:6443/api/v1/nodes/kind-worker3?timeout=10s 404 Not Found in 1 milliseconds
I0617 13:38:11.105929     174 kubelet.go:229] [kubelet-start] Stopping the kubelet
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is not healthy after 4m0.00110275s
Unfortunately, an error has occurred:
	The HTTP call equal to 'curl -sSL http://127.0.0.1:10248/healthz' returned error: Get "http://127.0.0.1:10248/healthz": context deadline exceeded
This error is likely caused by:
	- The kubelet is not running
	- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
	- 'systemctl status kubelet'
	- 'journalctl -xeu kubelet'
The HTTP call equal to 'curl -sSL http://127.0.0.1:10248/healthz' returned error: Get "http://127.0.0.1:10248/healthz": context deadline exceeded

using the following kind config
https://github.com/Netcracker/qubership-test-pipelines/blob/test_opensearch/kind-configs/kind-multi-node.yaml

Link to pipeline
https://github.com/Netcracker/qubership-opensearch/actions/runs/15346413241/job/44261098625

To Reproduce

Re-run
https://github.com/Netcracker/qubership-opensearch/actions/runs/15346413241

Version

No response

Logs

No response

Additional information

No response

[RomanB89]

Confirmed issue with manual kind startup with same parameters as well
In kind logs following error was found:
dynamic_serving_content.go:144] "Failed to watch cert and key file, will retry later" err="error creating fsnotify watcher: too many open files"
Applied
echo fs.inotify.max_user_watches=655360 | sudo tee -a /etc/sysctl.conf
echo fs.inotify.max_user_instances=1280 | sudo tee -a /etc/sysctl.conf

Reference:
kubernetes-sigs/kind#1652

[RomanB89]

Please re-run pipeline and reopen this issue if need

[RomanB89]

Pipeline successful run:
https://github.com/Netcracker/qubership-opensearch/actions/runs/15346413241/job/44327797354