Verify multi-arch builder SSH host keys

[jlebon]

Right now, we're trusting the builder's host keys on first use. Since there's no persistence between runs, we trust on first use on each run. We should instead be verifying host keys. There's tricky details there on how to implement this of course. See conversation in containers/common#1216 (comment) and following.

[dustymabe]

Copying from containers/common#1216 (comment):

But yeah, we should probably tighten this up anyway. It wouldn't be too hard for our wrapper code to first add the known hosts entries (from e.g. a configmap) before calling podman remote. The harder part is reprovisioning these remote machines with the same host keys each time (secret provisioning in general is an issue). It'd probably be easier I think to just have them naturally rotate everytime we reprovision and update the configmap or equivalent. So we'd at least reduce it to a single TOFU per reprovision.

[dustymabe]

The harder part is reprovisioning these remote machines with the same host keys each time (secret provisioning in general is an issue).

Yes. This is the hard part. We could store the hostkeys in s3 and generate an expiring pre-signed URL to use in the Ignition conifg. Which makes the window of time someone would have to both access the Ignition config (with the pre-signed URL) and the actual S3 content backed by the URL much smaller.

[dustymabe]

When re-working this we could consider using a CA to establish trust too: https://jameshfisher.com/2018/03/16/how-to-create-an-ssh-certificate-authority/