fix: don't error if pod has no containerStatuses (#4471)

[ianroberts]

Summary

kill-host-pods.py iterates over all pods in all namespaces looking for pods that are running on the current node. But if any pods in the iteration do not have containerStatuses in their pod status structure (e.g. if they're Pending waiting for a valid node to schedule to) then this currently causes the script to crash with a KeyError. This PR changes the code to treat a missing containerStatuses the same as an empty one, so the pod is correctly skipped rather than the whole script crashing.

Closes #4471

Changes

• changed pod["status"]["containerStatuses"] (which raises an error if containerStatuses is missing) to pod["status"].get("containerStatuses") (which simply returns None).

Testing

Tested locally with a copy of the script, I believe the change is trivial enough not to require automated tests beyond this.

Checklist

• Read the contributions page.
• Submitted the CLA form, if you are a first time contributor.
• The introduced changes are covered by unit and/or integration tests.

Notes

[neoaggelos]

LGTM, thanks a lot for the catch @ianroberts !

[neoaggelos]

Would you also mind signing the CLA? And I believe we should also backport this change to older branches as well

[ianroberts]

Would you also mind signing the CLA? And I believe we should also backport this change to older branches as well

I did just sign the CLA, though the email address I've used is a secondary email on my launchpad account (iroberts-usfd) rather than the preferred one.

[neoaggelos]

The CLA check fails with:

Checking the following users on Launchpad:
- ian@$DOMAIN ✕ (has no Launchpad account)

Can you amend the commit and use the email that you signed the CLA with?

[ianroberts]

The CLA check fails with:

Checking the following users on Launchpad:
- ian@$DOMAIN ✕ (has no Launchpad account)

Can you amend the commit and use the email that you signed the CLA with?

That's what I mean - the email I signed the CLA with is ian@$DOMAIN, and that email address is associated with my launchpad account, but it was a secondary rather than primary email address.

I've just edited my launchpad profile to swap the primary and secondary addresses, can you run the CLA checker again now?

[ianroberts]

Sorry, my bad, I had added the address to my Ubuntu one account but not directly to my Launchpad account, it should be there now. But I'm not showing as a member of https://launchpad.net/~contributor-agreement-canonical, maybe I need to fill in the CLA again now the right emails are all linked?

[neoaggelos]

Yes, can you please sign the CLA with the email address used in the commit? Thank you!

[ianroberts]

I already had, twice... just filled in the form again, let's see if it takes this time.

Edit: I see it's at least found the matching LP account for my email now.

[ianroberts]

This bug is still present - I've just tried again to sign the CLA, can you see whether it has succeeded this time?

[jmrr]

I'd like to chip in with some collateral damage this issue has produced in our setup.

We've had a major outage in production and almost another one (we caught this early and knew how to intervene), due to:

1. A snap refresh caused the creation of "${SNAP_DATA}/var/lock/snapdata-mounts-need-reload" flag file to trigger a calico-node DaemonSet restart
2. Lack of containerStatuses key caused the KeyError exception in the kill-host-pods.py script described in this PR.
3. Script fails with error, preventing microk8s-resources/default-hooks/reconcile.d/10-pods-restart outer script from removing the "${SNAP_DATA}/var/lock/snapdata-mounts-need-reload" file
4. dqlite was flooded with events due to the calico-node pods restart loop
5. At some point, dqlite couldn't cope, corrupted, and the whole cluster performance degraded.

[ianroberts]

Rebased on latest master

[ianroberts]

This has been kind of superseded (at least in 1.32) by #4819 - this bug is still present in kill-host-pods.py but the way it is called from the default hook scripts in 1.32 will not trigger the issue like it did in earlier versions, since the new version of the hook explicitly filters for pods that belong to the current node and so should never hit a pod that is pending waiting to be scheduled. It may be worth back-porting that hooks change to earlier versions too, since it can significantly reduce the number of pods that each iteration of kill-host-pods has to page through.

[ianroberts]

I say only "kind of" superseded, because the default hook scripts in 1.32 have been changed to only look for pods on the current node, but these are only the default scripts that get installed if there are no hooks already present in /var/snap/microk8s/common from a previous version. So it's only truly fixed in a clean install of 1.32; for users who want to upgrade from an earlier version it is necessary to manually copy in the updated hooks from /snap/microk8s/current to /var/snap/microk8s/common in order to benefit from the fix.

[HomayoonAlimohammadi]

Thanks a lot for the PR @ianroberts! LGTM overall, just a minor nit and we can merge it.

[HomayoonAlimohammadi]

@ianroberts That's right, sorry I committed the change since your response was not loaded for some reason.
You're right, your approach catches both. AFAIK that shouldn't ever happen, but it's a nice defensive programming style.

[HomayoonAlimohammadi]

I believe the failing addon tests are not related to this PR. Should we merge this one or wait for a fix to rebase this PR on top of it @louiseschmidtgen?

[louiseschmidtgen]

@HomayoonAlimohammadi The issue is resolved can you re-run the workflow please?