ssh-copy-id failing: Too many arguments

[basbebe]

Prerequisites

• I have updated to the latest version using nix run --refresh github:nix-community/nixos-anywhere
• I have reproduced the issue with the --debug flag
• I have searched existing issues to make sure this isn't a duplicate

Bug Description

I'm trying to run nixos-anywhere but I keep getting Too many arguments errors by ssh-copy-id which seem to have to do with the usage of the -i flag.

It works when I return to OpenSSH 9.6

I haven't figured out yet if this has to do with how nixos-anywhere uses ssh-copy-id or if it's a bug in ssh-copy-id.

Steps to Reproduce

ssh-copy-id -xn -i /tmp/test-nixos-anywhere ubuntu@10.10.10.210

If I omit the -i flag and its argument, ssh-copy-id works.

Debug Logs

[…]
+ sshUser=ubuntu
+ sshHost=10.10.10.210
+ [[ 1 != 1 ]]
+ uploadSshKey
+ local sshCopyHome=/home/basbebe
+ mkdir -p /home/basbebe/.ssh/
+ [[ -n '' ]]
+ ssh-keygen -t ed25519 -f /tmp/tmp.uBH8XwkMQb/nixos-anywhere -P '' -C nixos-anywhere
+ step Uploading install SSH keys
+ echo '### Uploading install SSH keys ###'
### Uploading install SSH keys ###
+ [[ n == y ]]
+ HOME=/home/basbebe
+ ssh-copy-id -o IdentitiesOnly=no -o ConnectTimeout=10 -o IdentitiesOnly=yes -i /tmp/tmp.uBH8XwkMQb/nixos-anywhere -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o AddKeysToAgent=yes ubuntu@10.10.10.210
/run/current-system/sw/bin/ssh-copy-id: ERROR: Too many arguments.  Expecting a target hostname, got:

Usage: /run/current-system/sw/bin/ssh-copy-id [-h|-?|-f|-n|-s|-x] [-i [identity_file]] [-t target_path] [-F ssh_config] [[-o ssh_option] ...] [-p port] [user@]hostname
        -f: force mode -- copy keys without trying to check if they are already installed
        -n: dry run    -- no keys are actually copied
        -s: use sftp   -- use sftp instead of executing remote-commands. Can be useful if the remote only allows sftp
        -x: debug      -- enables -x in this shell, for debugging
        -h|-?: print this help
+ sleep 3
[…]

Command Used

nixos-anywhere --flake .#nix-host "ubuntu@10.10.10.210" --debug

Target System

Cloud server (AWS, DigitalOcean, etc.)

NixOS Version

25.11

Environment Information

• Host OS:
• Nix version:
• Target architecture:

Configuration Files

Additional Context

No response

[adamarbour]

I ran into the same issue. This appears to be an issue with openssh 10

[sedlund]

the reproducer mentioned works ok for me on linux x86 25.11 and unstable.

ssh-keygen -t ed25519 -f /tmp/test-nixos-anywhere -P '' -C nixos-anywhere
ssh-copy-id -xn -i /tmp/test-nixos-anywhere sedlund@localhost

[adamarbour]

Here is my test ...

ssh-copy-id -xn -i ~/.ssh/id_ed25519 root@111.222.333.444

Here is the output

+ getopts io:p:F:t:fnsxh? OPT
+ DRY_RUN=1
+ getopts io:p:F:t:fnsxh? OPT
+ [  ]
+ SEEN_OPT_I=yes
+ eval nextarg=${3}
+ nextarg=/home/aarbour/.ssh/id_ed25519
+ [ 3 = 4 ]
+ echo io:p:F:t:fnsxh?
+ tr -d :
+ expr -- /home/aarbour/.ssh/id_ed25519 : -[iopFtfnsxh?-]
+ OPTARG=/home/aarbour/.ssh/id_ed25519
+ OPTIND=4
+ use_id_file /home/aarbour/.ssh/id_ed25519
+ L_ID_FILE=/home/aarbour/.ssh/id_ed25519
+ [ -z /home/aarbour/.ssh/id_ed25519 ]
+ expr /home/aarbour/.ssh/id_ed25519 : .*\.pub$
+ PUB_ID_FILE=/home/aarbour/.ssh/id_ed25519.pub
+ [  ]
+ dirname /home/aarbour/.ssh/id_ed25519.pub
+ basename /home/aarbour/.ssh/id_ed25519.pub .pub
+ PRIV_ID_FILE=/home/aarbour/.ssh/id_ed25519
+ ErrMSG=+ :
+ ErrMSG=+ :
+ GET_ID=cat "/home/aarbour/.ssh/id_ed25519.pub"
+ getopts io:p:F:t:fnsxh? OPT
+ shift 2
+ [ 2 = 0 ]
+ [ 2 != 1 ]
+ printf %s: ERROR: Too many arguments.  Expecting a target hostname, got: %s\n\n /run/current-system/sw/bin/ssh-copy-id
/run/current-system/sw/bin/ssh-copy-id: ERROR: Too many arguments.  Expecting a target hostname, got:

+ usage
+ printf Usage: %s [-h|-?|-f|-n|-s|-x] [-i [identity_file]] [-t target_path] [-F ssh_config] [[-o ssh_option] ...] [-p port] [user@]hostname\n /run/current-system/sw/bin/ssh-copy-id
Usage: /run/current-system/sw/bin/ssh-copy-id [-h|-?|-f|-n|-s|-x] [-i [identity_file]] [-t target_path] [-F ssh_config] [[-o ssh_option] ...] [-p port] [user@]hostname
+ printf \t-f: force mode -- copy keys without trying to check if they are already installed\n
	-f: force mode -- copy keys without trying to check if they are already installed
+ printf \t-n: dry run    -- no keys are actually copied\n
	-n: dry run    -- no keys are actually copied
+ printf \t-s: use sftp   -- use sftp instead of executing remote-commands. Can be useful if the remote only allows sftp\n
	-s: use sftp   -- use sftp instead of executing remote-commands. Can be useful if the remote only allows sftp
+ printf \t-x: debug      -- enables -x in this shell, for debugging\n
	-x: debug      -- enables -x in this shell, for debugging
+ printf \t-h|-?: print this help\n
	-h|-?: print this help
+ exit 1

However without passing in the key ... so no identify file

ssh-copy-id -xn root@111.222.333.444

Results...

+ getopts io:p:F:t:fnsxh? OPT
+ DRY_RUN=1
+ getopts io:p:F:t:fnsxh? OPT
+ shift 1
+ [ 1 = 0 ]
+ [ 1 != 1 ]
+ USER_HOST=root@111.222.333.444
+ quote root@111.222.333.444
+ printf %s\n root@111.222.333.444
+ sed -e s/'/'\\''/g
+ OPTS_USER_HOST='root@111.222.333.444'
+ SSH_OPTS='root@111.222.333.444'
+ eval set -- 'root@111.222.333.444'
+ set -- root@111.222.333.444
+ eval ssh-add -L
+ ssh-add -L
+ [ -z ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGDmKBN5jUyczgWuYmTdtcrtYY+us0nW2kEqHz7nhF0P aarbour ]
+ printf %s: INFO: Source of key(s) to be installed: %s\n /run/current-system/sw/bin/ssh-copy-id ssh-add -L
/run/current-system/sw/bin/ssh-copy-id: INFO: Source of key(s) to be installed: ssh-add -L
+ eval ssh-add -L
+ ssh-add -L
+ [ -z ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGDmKBN5jUyczgWuYmTdtcrtYY+us0nW2kEqHz7nhF0P aarbour ]
+ mktemp -d /home/aarbour/.ssh/ssh-copy-id.XXXXXXXXXX
+ SCRATCH_DIR=/home/aarbour/.ssh/ssh-copy-id.Y43bQgdpLk
+ [ /home/aarbour/.ssh/ssh-copy-id.Y43bQgdpLk ]
+ [ -d /home/aarbour/.ssh/ssh-copy-id.Y43bQgdpLk ]
+ chmod 0700 /home/aarbour/.ssh/ssh-copy-id.Y43bQgdpLk
+ SCRATCH_CLEANUP=rm -rf "/home/aarbour/.ssh/ssh-copy-id.Y43bQgdpLk"
+ trap rm -rf "/home/aarbour/.ssh/ssh-copy-id.Y43bQgdpLk" EXIT TERM INT QUIT
+ ssh -a -x -v -o PreferredAuthentications=, -o ControlPath=none root@111.222.333.444
+ sed -ne s/.*remote software version //p
+ REMOTE_VERSION=OpenSSH_9.6p1 Ubuntu-3ubuntu13
+ populate_new_ids 0
+ [  ]
+ printf %s: INFO: attempting to log in with the new key(s), to filter out any that are already installed\n /run/current-system/sw/bin/ssh-copy-id
/run/current-system/sw/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
+ eval ssh-add -L
+ ssh-add -L
+ filter_ids 0
+ L_SUCCESS=0
+ assert_scratch_ok filter_ids
+ [ /home/aarbour/.ssh/ssh-copy-id.Y43bQgdpLk ]
+ [ -d /home/aarbour/.ssh/ssh-copy-id.Y43bQgdpLk ]
+ [ -w /home/aarbour/.ssh/ssh-copy-id.Y43bQgdpLk ]
+ return 0
+ L_TMP_ID_FILE=/home/aarbour/.ssh/ssh-copy-id.Y43bQgdpLk/popids_tmp_id
+ L_OUTPUT_FILE=/home/aarbour/.ssh/ssh-copy-id.Y43bQgdpLk/popids_output
+ eval set -- 'root@111.222.333.444'
+ set -- root@111.222.333.444
+ read -r ID
+ printf %s\n ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGDmKBN5jUyczgWuYmTdtcrtYY+us0nW2kEqHz7nhF0P aarbour
+ ssh -a -x -i /home/aarbour/.ssh/ssh-copy-id.Y43bQgdpLk/popids_tmp_id -o ControlPath=none -o LogLevel=INFO -o PreferredAuthentications=publickey -o IdentitiesOnly=yes root@111.222.333.444 exit
+ [ 255 = 0 ]
+ [  ]
+ grep Permission denied /home/aarbour/.ssh/ssh-copy-id.Y43bQgdpLk/popids_output
+ cat /home/aarbour/.ssh/ssh-copy-id.Y43bQgdpLk/popids_tmp_id
+ read -r ID
+ [  ]
+ NEW_IDS=ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGDmKBN5jUyczgWuYmTdtcrtYY+us0nW2kEqHz7nhF0P aarbour
+ expr ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGDmKBN5jUyczgWuYmTdtcrtYY+us0nW2kEqHz7nhF0P aarbour : ^ERROR:
+ [ -z ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGDmKBN5jUyczgWuYmTdtcrtYY+us0nW2kEqHz7nhF0P aarbour ]
+ printf %s\n ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGDmKBN5jUyczgWuYmTdtcrtYY+us0nW2kEqHz7nhF0P aarbour
+ wc -l
+ printf %s: INFO: %d key(s) remain to be installed -- if you are prompted now it is to install the new keys\n /run/current-system/sw/bin/ssh-copy-id 1
/run/current-system/sw/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
+ [ 1 ]
+ printf %s\n ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGDmKBN5jUyczgWuYmTdtcrtYY+us0nW2kEqHz7nhF0P aarbour
+ wc -l
+ ADDED=1
+ [ 1 ]
+ cat
=-=-=-=-=-=-=-=
Would have added the following key(s):

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGDmKBN5jUyczgWuYmTdtcrtYY+us0nW2kEqHz7nhF0P aarbour
=-=-=-=-=-=-=-=
+ rm -rf /home/aarbour/.ssh/ssh-copy-id.Y43bQgdpLk

[adamarbour]

ok .. so I was determined to get to the root of the issue. I have not found the WHY but I have found the WHAT.

readlink -f /bin/sh
/nix/store/3vby9nf3vjvl45ja9bcpwcn9p2cv3gh9-dash-0.5.13.1/bin/dash

Does not work:

/bin/sh -x "$(command -v ssh-copy-id)" -xn -i ~/.ssh/id_ed25519 root@111.222.333.444

Works!!!:

bash -x "$(command -v ssh-copy-id)" -xn -i ~/.ssh/id_ed25519 root@111.222.333.444

[sedlund]

Interesting!

How did dash become your /bin/sh ?

After review of ssh-copy-id's history, it has kept #!/bin/sh, even though its handling of optional -i arguments with getopts is not POSIX-compliant and requires a bashism to work correctly.

In nixos-anywhere, we could invoke ssh-copy-id with bash directly (as you have shown) to avoid this issue.

[adamarbour]

So... I don't know if we want to consider it a bug. I did not realize that I had left a configuration in place in my environment that had dash as the bin/sh - I was tinkering and left some configs around. I also did not realize that something like this would bubble up. I removed the configuration and all is fine.

[basbebe]

I also have Dash as my default /bin/sh.

I'd personally agree that this should be handled upstream, not necessarily by nixos-anywhere.

[sedlund]

I think the upstream is https://gitlab.com/phil_hands/ssh-copy-id .

[Sepero]

I was having cryptic errors about ssh-copy-id while running nixos-anywhere. Thankfully I found this thread.

I had dash set to environment.binsh. After removing it locally, I was able to run nixos-anywhere without errors again. Thank you

[sedlund]

upstream issue: https://gitlab.com/phil_hands/ssh-copy-id/-/issues/5