Explore on how to fetch logs from machines when cluster creation fails

[Amulyam24]

/kind feature
/area provider/ibmcloud

Describe the solution you'd like
In E2E, when a cluster creation fails, we currently don't have a way to fetch the logs(Kubelet, cloud-init etc..) which would help us to understand the root cause.

This issue is to explore any mechanisms that would make it possible.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[Amulyam24]

/remove-lifecycle rotten

[carmal891]

/assign

[carmal891]

@Amulyam24 @Karthik-K-N
I'm made progress on enhancing our e2e test logging to better debug workload cluster bootstrap failures. However I have a few queries, please guide.

As confirmed earlier, the objective here is to collect the following data :

1. Cloud-init logs
2. Kubelet logs
3. Node status
4. Pod status in kube-system
5. Pod logs from kube-system
6. CCM logs

Current Findings

1. Above 3-6 can already be retrieved via the Kubernetes APIs using the existing e2e test framework. An extension to the existing function will suffice (https://github.com/kubernetes-sigs/cluster-api-provider-ibmcloud/blob/main/test/e2e/e2e_test.go#L192)
2. The main challenge is collecting cloud-init logs and kubelet logs, which are only available on the workload cluster nodes.

Options explored for fetching cloud-init logs and kubelet logs

1. Must-gather

• Requires a working API server, which we cannot rely on at time of failure
• It is OpenShift-specific and depends on the oc which is not the cli used in the kind test container

2. SSH into nodes

• It's possible to extend the existing e2e test function to SSH into workload machines and fetch logs directly using the keys defined here (https://github.com/kubernetes-sigs/cluster-api-provider-ibmcloud/blob/main/scripts/ci-e2e.sh#L126)
• I've tried and confirmed this on my local machine.

Current Challenge
Above option 2 works fine when accessing the nodes from the host running the tests. However, the e2e tests run inside a Kind container, while the SSH keys are only exported on the host/CI node.

Queries

1. Is it reasonable to proceed with the SSH-based approach for collecting cloud-init and kubelet logs?
2. If yes, what is the recommended way to inject or mount the SSH keys from the host/CI node into the Kind test container so the test container can use them?