Capistrano SSH Agent during Deploy

[Phillita]

Hey All,

tldr
Is there a recommended way to setup an ssh-agent to connect to a generated ssh key on deploy?
Is there a recommended way to work with the SSH_AUTH_SOCK?

I've been working on integrating Shipit into my org and have an issue with Capistrano and the ssh-agent.

Here is my setup:

• Shipit lives on an Ubuntu Server (22.04.5 LTS)
• The server has an ssh key generated
• GitHub has the SSH key authenticated via our SSO
• On deploy
  • Shipit is able to clone the repo and begin the capistrano deploy steps
  • It fails when it hits the git:check step which checks if the remote server can pull in the repository

  git:check
      01 git ls-remote [email protected]:org/repo.git HEAD
      01 [email protected]: Permission denied (publickey).
      01 fatal: Could not read from remote repository.
      01 
      01 Please make sure you have the correct access rights
      01 and the repository exists.

What we've found is that the server has no ssh-agent connected to the deploy steps.
I found this issue which is similar to my setup but unfortunately went unanswered. I'm trying to avoid the deploy override if possible.

I saw in the templates for shipit that there is an env variable for the SSH_AUTH_SOCK but I'm unsure how to use that to connect an ssh-agent with Shipit.

Any help getting this set up would be much appreciated! I know this isn't exactly a part of Shipit; it is more about integration.

Thank you!

[casperisfine]

It's been a long time I worked with this, but maybe what you are missing is SSH agent forwarding?

https://docs.github.com/en/authentication/connecting-to-github-with-ssh/using-ssh-agent-forwarding

[Phillita]

When I deploy Shipit to its host server, ssh forwarding is set up and running as expected. I'm using capistrano to deploy Shipit from my local.

When Shipit deploys an application to another server, it no longer has an ssh-agent running and can't forward it's own ssh config to the deploying app. I can mimic this behaviour by turning off forwarding locally and then creating an ssh connection to the server.

I may have something wrong with my setup, but I've created the server with its own SSH key to use with deploying. I had assumed that Shipit would start its own SSH agent and connect with this key.

Is Shipit supposed to use the originally forwarded ssh config from when it was deployed to a server?

[casperisfine]

Is Shipit supposed to use the originally forwarded ssh config from when it was deployed to a server?

No, you are supposed to setup the server on which Shipit runs on so that it can SSH to the other servers it needs to deploy to.

The safest way being to install and configure ssh-agent on the shipit server.

[Phillita]

tldr;

I've setup an ssh-agent through systemd and added the production.env.SSH_AUTH_SOCK in the secrets.yml.

---

For anyone else dealing with this, here is what I've learned.

• Shipit spawns a PTY session per command sent to the machine in the deploy steps.
• This PTY session uses bin/sh to run the command
  • Environment commands such as starting an ssh-agent do not persist between commands
• On Ubuntu, bin/sh is a symlink to the shell environment. In new versions of the OS, this will use dash
  • Dash has no config options and doesn't run startup scripts
  • Historical versions of Ubuntu used bash in a no-config mode (no .bashrc or the like)
• Switching bin/sh to bash doesn't help, it will start by default in the --norc mode and not run startup config
  • The switch can be done with sudo dpkg-reconfigure dash

It would be awesome if Shipit could start a single shell session that maintained output from the previous steps. This would allow me to start an ssh-agent per deploy that could then be shutdown once the deploy is complete.

# deployable_app/shipit.yml
deploy:
  pre:
    - eval "$(ssh-agent -s)"
    - ssh-add
  post:
    - ssh-agent -k

For now, I had to setup a persistent ssh-agent that each application could reference using the SSH_AUTH_SOCK.

# shipit/config/secrets.yml
production:
  ...
  env:
    SSH_AUTH_SOCK: 'path/to/known.socket'