Recreate old nodes based on new images

[otavio]

I have some outdated nodes that are not updating on their own. The node fails as:

Retrieving: k3s-selinux-1.2-2.sle.noarch (Rancher K3s Common (stable)) (373/373),  20.5 KiB
Retrieving: k3s-selinux-1.2-2.sle.noarch.rpm [...done (785 B/s)]
k3s-selinux-1.2-2.sle.noarch.rpm:
    Header V3 RSA/SHA256 Signature, key ID 925ea29ae257814a: NOKEY
    V3 RSA/SHA256 Signature, key ID 925ea29ae257814a: NOKEY

warning: /var/tmp/AP_0xoGm5sW/k3s-selinux-1.2-2.sle.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID e257814a: NOKEY
Looking for gpg key ID E257814A in cache /var/cache/zypp/pubkeys.

New repository or package signing key received:

  Repository:       Rancher K3s Common (stable)
  Key Fingerprint:  C8CF F216 4551 26E9 B9C9 18BE 925E A29A E257 814A
  Key Name:         Rancher (CI) <[email protected]>
  Key Algorithm:    RSA 3072
  Key Created:      Tue Mar 10 22:43:06 2020
  Key Expires:      (does not expire)
  Subkey:           AA7E9EC8FE21FDCF 2020-03-10 [does not expire]
  Rpm Name:         gpg-pubkey-e257814a-5e6817fa

    Note: Signing data enables the recipient to verify that no modifications occurred after the data
    were signed. Accepting data with no, wrong or unknown signature can lead to a corrupted system
    and in extreme cases even to a system compromise.

    Note: A GPG pubkey is clearly identified by its fingerprint. Do not rely on the key's name. If
    you are not sure whether the presented key is authentic, ask the repository provider or check
    their web site. Many providers maintain a web page showing the fingerprints of the GPG keys they
    are using.

Do you want to reject the key, or trust always? [r/a/?] (r): r
k3s-selinux-1.2-2.sle.noarch (Rancher K3s Common (stable)): Signature verification failed [4-Signatures public key is not available]
Abort, retry, ignore? [a/r/i] (a): a
Problem occurred during or after installation or removal of packages:
Installation has been aborted as directed.
Please see the above error message for a hint.
2023-05-18 14:23:09 Application returned with exit status 8.
ERROR: zypper up on /.snapshots/4/snapshot failed with exit code 8!
Use '--interactive' for manual problem resolution.
Removing snapshot #4...

Our goal is to reconstruct these nodes using the latest snapshots. Is it possible to evacuate the old nodes, cordon them, and then destroy and rebuild them?

[otavio]

Is there a way to do this?

[Keyruu]

I have nodes that unfortunately don't have the right stuff installed for longhorn. I don't really know why that is. And some Nodes get rate limited for registry.k8s.io. I would just like to replace the nodes that are broken. How do I do that?

[mysticaltech]

@otavio Just upgrade, then create new nodepool (symetrical), then scale the old ones to 0 after draining each nodes.

More on this #729 and #794