How to ssh into control node?

[schlichtanders]

Hi there,
I guess this is super simple. If create the cluster with my private key and public key, but now I am unable to connect to the nodes

when trying

ssh -i ~/.ssh/myprivatekey MYIPADDRESS

I get

Received disconnect from MYIPADDRESS port 22:2: Too many authentication failures

Any help is highly appreciated. @pschiffe ? (I saw that you also had ssh problem)

[schlichtanders]

I was able to manage it by adding the following ssh config and connect to root user

ssh -i ~/.ssh/myprivatekey  -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o 'IdentitiesOnly yes' -o PubkeyAuthentication=yes root@MYIPADDRESS

I have not enough experience with ssh - why are these extra options needed?

[pschiffe]

They are needed only when connecting to the host for the first time non-interactively. By default you have to confirm the identity of the remove host manually. If you are using these options all the time, you are making ssh prone to the middle man attack. Ideally just use ssh -i ~/.ssh/myprivatekey root@MYIPADDRESS. If that doesn't work, add -v option and check the output.

[pschiffe]

Too many authentication failures is related to this option: https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/blob/master/locals.tf#L540
With ssh -v you can see more details about what's going on. Maybe you are missing user? ssh -i ~/.ssh/myprivatekey root@MYIPADDRESS

[schlichtanders]

Thank you very much for your help.

Here is the output of ssh -i ~/.ssh/MYPRIVATEKEY root@MYIP -vv

OpenSSH_9.1p1, OpenSSL 3.0.7 1 Nov 2022
debug1: Reading configuration data /home/ssahm/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 5: Applying options for *
debug2: resolve_canonicalize: hostname MYIP is address
debug1: Connecting to MYIP [MYIP] port 22.
debug1: Connection established.
debug1: identity file /home/ssahm/.ssh/MYPRIVATEKEY type 3
debug1: identity file /home/ssahm/.ssh/MYPRIVATEKEY-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.9
debug1: compat_banner: match: OpenSSH_8.9 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to MYIP:22 as 'root'
debug1: load_hostkeys: fopen /home/ssahm/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: [email protected],curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,[email protected],zlib
debug2: compression stoc: none,[email protected],zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,[email protected],diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,[email protected]
debug2: compression stoc: none,[email protected]
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: [email protected]
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519
debug1: load_hostkeys: fopen /home/ssahm/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'MYIP' is known and matches the ED25519 host key.
debug1: Found key in /home/ssahm/.ssh/known_hosts:20
debug2: ssh_set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug2: ssh_set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: get_agent_identities: bound agent to hostkey
debug1: get_agent_identities: agent returned 2 keys
debug1: Will attempt key: MYKEYCOMMENT RSA 
debug1: Will attempt key: MYEMAIL ED25519 
debug1: Will attempt key: /home/ssahm/.ssh/MYPRIVATEKEY ED25519 
debug2: pubkey_prepare: done
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,[email protected],ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected]>
debug1: kex_input_ext_info: [email protected]=<0>
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering public key: MYKEYCOMMENT RSA 
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Offering public key: MYEMAIL ED25519 
debug2: we sent a publickey packet, wait for reply
Received disconnect from MYIP port 22:2: Too many authentication failures
Disconnected from MYIP port 22

do you see something weird? for me it looks like almost everything worked

[pschiffe]

The server is allowing only 2 auth attempts due to MaxAuthTries 2 config. In the logs you can see:

debug1: Will attempt key: MYKEYCOMMENT RSA 
debug1: Will attempt key: MYEMAIL ED25519 
debug1: Will attempt key: /home/ssahm/.ssh/MYPRIVATEKEY ED25519 

The ~/.ssh/MYPRIVATEKEY is on the third place, so it never gets to it. Check these files:

debug1: Reading configuration data /home/ssahm/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 5: Applying options for *

if there isn't any option forcing MYKEYCOMMENT RSA and MYEMAIL ED25519 keys.

[schlichtanders]

/etc/ssh/ssh_config contains the following default

Host *
AddressFamily any
GlobalKnownHostsFile /etc/ssh/ssh_known_hosts /etc/ssh/ssh_known_hosts2

XAuthLocation /nix/store/024fmk79gj336f7l5f84b5q2pzzj5q6v-xauth-1.1/bin/xauth


ForwardX11 no

/home/ssahm/.ssh/config is practically empty

[pschiffe]

Interesting. What do you see in the output of command ssh-add -L, how many keys? Does the login work with this command? ssh -i ~/.ssh/MYPRIVATEKEY -o 'IdentitiesOnly yes' root@MYIP What is you local OS setup?

[schlichtanders]

Thank you very much. Yes this works! is this safe to use?

---

I am running on NixOS, which indeed has sometimes special (but really cool) ways to do things.

I have two configs enabled to combine ssh with my KDE keystore ( see here for a description of one option - they adapt system wide config for ssh in the end):

programs.ssh.startAgent = true;
programs.ssh.askPassword = "${pkgs.ksshaskpass}/bin/ksshaskpass";

So a bit of setup, but not too much.

[pschiffe]

Yep, this is fine and correct approach in your case. IdentitiesOnly=yes tells ssh to use only explicitly set identities (keys), not the default ones coming from your system. Enjoy!

[schlichtanders]

Awesome! thank you very much

[mysticaltech]

@schlichtanders @pschiffe If you folks see ways to enhance the current SSH flows, PR are always welcome!