SSH problems with 1Password

[cro]

Greetings,

I've had a running cluster created with this module for about 4 months. It has been fantastic, thank you to the effort that has gone into it. I've had very little trouble with the cluster itself, and I've been super pleased with the cost of running a 6 node cluster (3 control, 3 worker) on Hetzner vs Digital Ocean or Vultr.

I noticed today that I cannot seem to ssh into any of my nodes. I validated that the keys are all correct. I managed to get logged in via the console & started sshd in debug mode in the foreground. In the log messages it said

mm_answer_keyallowed: publickey authentication ED25519 key not allowed

(may be paraphrased, the default Hetzner console doesn't allow copy/paste).

That led me to make sure the system crypto policies were set properly and it appears that ed25519 keys are allowed there.

Has anyone else seen this or have ideas on how to troubleshoot it?

[mysticaltech]

@cro How can you login via the console, I do not think that is possible. Really weird.

Is your cluster still running?

Honestly, it's the first time I hear of that issue.

Please post screenshots.

[cro]

Yes, my cluster is still running fine.

You can login to the console if you change the root password via:

kubectl debug node/k3s-control-plane-node-<id> -it --image=opensuse/leap
chroot /host
passwd root
systemctl restart sshd

This is also how I ran sshd in the foreground with debug turned on (/usr/sbin/sshd -ddd ...) to see the message about ED25519 keys not being accepted.

Because of the nature of the debug container you can't do a transactional-update to change /etc/rancher/config/k3s.yaml, which was what I was trying to do originally before going down this rabbit hole 😁 .

But for obvious reasons I'd prefer to be able to ssh into the nodes.

Edited terminal session for example:

$ ssh [email protected] -i ~/.ssh/id_k3shetz -o StrictHostKeyChecking=no
Warning: Permanently added '5.xxx.xxx.xxx' (ED25519) to the list of known hosts.
Received disconnect from 5.xxx.xxx.xxx port 22:2: Too many authentication failures
Disconnected from 5.xxx.xxx.xxx port 22

and screenshot from console showing the "not allowed" message:

[mysticaltech]

@cro Great debugging, and thanks for the info, I had no idea we could set a password this way!

What is happening is that your IP, somehow, got banned. MicroOS uses Tallow (like fail2ban), just find a way to connect from another IP (via a VPN for instance) and unban yourself first.

If you find how to unban with Tallow, please share it here.

Somehow, your current SSH key is not recognized, please list it when you connect via console, see if it the same public key in there, maybe you used another one and forgot. It can be found in ~/.ssh/authorized_keys I believe.

[cro]

Well, now I'm super confused. I see no trace of tallow on these machines, and the 30 March MicroOS changelog says tallow has been removed (excerpt from https://lists.opensuse.org/archives/list/[email protected]/thread/NZOHGNE4Q6JVAJJ5MG73RIP6PKDE5JOK/):

==== systemd-presets-branding-MicroOS ====

- Remove tallow (package got dropped long ago)
- Enable lastlog2-import

I'll keep digging. It's probably something simple.

[mysticaltech]

Damn, ok! Anyways, login via the console and see the authorized keys if yours is in there or not.

[mysticaltech]

Also worth a try, upgrade to latest 2.x, see readme for how to. They create new nodepools, then when up and running effectively doubling your cluster size, drain old nodes and set their respective counts to 0 then apply again, this should solve the problem the easy way!

[mysticaltech]

Also, make sure you are on a unix system, this project doesn't work on Windows command line, but does work from WSL.

[cro]

OK, it was something simple. Well, maybe not simple, but at least it makes sense. I use 1Password and have enabled their ssh-agent integration. Something with that integration is currently broken badly; I'm opening a ticket with them right now. Anyway, if I ssh from any of my other Linux or FreeBSD machines it works fine using the key (using the -a flag so it doesn't try to talk to the 1P ssh-agent). So nothing is wrong with the terraform module or the k8s clusters it deploys!

[mysticaltech]

I should have remembered, a user had this very issue about a year ago. Glad you finally figured it out! And thanks for the info about tallow not being used anymore, I will investigate more.