Deployer started failing. sign_and_send_pubkey errors and others

[Glenf]

Hi,

I've been using Deployer quite successfully bout month and a half (migrated from a bit broken Envoy setup). But within the previous two weeks it started failing, and I don't know wether this is a Deployer issue or some other. I've been debugging this now quite some time and I've narrowed it down to some sort of ssh agent forwarding issue.

I have set up bot Deployer and Envoy to debug this issue. Deployer has following task:

task('test:git', function () {
    run('echo $SSH_AUTH_SOCK', real_time_output: true);
    run('ssh-add -l', real_time_output: true);
    run('ssh -T [email protected]', real_time_output: true);
});

and Envoy has similar setup with

@task('test:git')
    echo $SSH_AUTH_SOCK
    ssh-add -l
    ssh -T [email protected]
@endtask

Running Deployer gives following output:

[staging] /tmp/ssh-xeMsBbhkX1/agent.3058185
[staging] 256 SHA256:<MyAwesomeKeyHere> XXXXXXXXXXXX (ED25519)
[staging] sign_and_send_pubkey: signing failed for ED25519 "XXXXXXXXXXXX" from agent: communication with agent failed
[staging] [email protected]: Permission denied (publickey).
[staging]  error  in deploy.php on line 70:
[staging] run ssh -T [email protected]
[staging] err sign_and_send_pubkey: signing failed for ED25519 "XXXXXXXXXXXX" from agent: communication with agent failed
[staging] err [email protected]: Permission denied (publickey).
[staging] exit code 255 (Unknown error)
ERROR: Task test:git failed!

Envoy task gives following (working) output:

[staging]: /tmp/ssh-zcFaJ211yC/agent.3075821
[staging]: 256 SHA256:<MyAwesomeKeyHere> XXXXXXXXXXXX (ED25519)
[staging]: Welcome to GitLab, @Glen_f!

These test were run on MacOS Ventura, M1 Apple against Ubuntu 20.04 server. Same behaviour can be seen with Intel Mac running Monterey.

To make things a bit weirder, when I run this through GitLab CI with Docker image built on php:8.2.5-cli, Deployer ssh portion works correctly. This would be a reasonable workaround but deploy fails elsewhere. It's probably something I should look more deeper into, but for some reason release_or_current_path gives always current and never release when deploying. release_or_current_path works correctly when testing from local machine.

I'd prefer to keep Deployer as my deployment solution as it gives a bit more pleasing DX than Envoy. Any ideas or suggestions where I should try next?

[antonmedv]

Each run() call is in separate session. So no ssh agent keys persisted.

Merge all three lines in single run() call.

[Glenf]

Persistent ssh keys isn't the issue here. Each session forwards agent correctly but for some reason ssh and git commands on remote fail to communicate with the agent. Even with

task('test:git', function () {
   run('ssh-add -l && ssh -T [email protected]', real_time_output: true);
});

first part lists keys correctly (connection to agent is good) but second part gives communication with agent failed error.

[antonmedv]

I see. This is strange behaviour. Try to add ssh -v for git

[Glenf]

I think I'll overcome this issue by using access token for git. It doesn't really solve the pubkey error, but allows me to continue using Deployer.