Gitea/Forgejo rootless image with podman SSH container passthrough

[CobyPear]

Problem:

I am trying to self host Forgejo (a fork of Gitea: same docs for installation) using podman and the rootless image of Forgejo. I've gotten almost everything working except for the ssh passthrough. I am able to ssh from the host machine, but when I try sshing from a different client, I am met with an error from the passthrough command that the container trying to run the exec command is not found (see the SSH Container Passthrough below to understand what I'm talking about here, sorry I can't explain it better!). With the default names, the error looks like this in the verbose output of the client's ssh command: Error: no container with name or ID "gitea" found: no such container And the logs in sshd on the host show that the public key was accepted and the AuthorizedKeyCommand was run but then something fails:

Dec 28 16:29:07 REDACTED sshd[968243]: AuthorizedKeysCommand /usr/bin/podman exec -i gitea /usr/local/bin/gitea keys -e forgejo -u forgejo -t ssh-ed25519 -k REDACTED failed, status 125
Dec 28 16:29:07 REDACTED sshd[968243]: Accepted publickey for forgejo from ***.***.*.* port 64435 ssh2: ED25519 SHA256:REDACTED
Dec 28 16:29:07 REDACTED sshd[968268]: Received disconnect from ***.***.*.* port 64435:11: disconnected by user
Dec 28 16:29:07 REDACTED sshd[968268]: Disconnected from user forgejo ***.***.*.* port 64435

I have gotten all of this working in the past including the ssh passthrough, but I distinctly remember starting the container as root, which seems to defeat the whole purpose!

If this is not possible with rootless podman, I would love to know so I can maybe add a note to the Gitea docs to save others from this frustration.

Machine info

OS: Asahi Linux - Arch flavor (arm)

$ podman -v
podman version 4.5.1

Various other potentially relevant config:

Caddy reverse proxy:

	git.server.home {
		reverse_proxy localhost:3000
	}

Pi-hole DNS routing git.server.home on my network to localhost of the host machine

Steps to reproduce

Assuming you are the user 1000:1000 let's call it "admin", and you create a user 1001:1001 called "forgejo"

1. Create a dir to store the files mkdir ~/forgejo
2. create some files touch docker-compose.yml .env
3. Populate the .env:

USER_UID=1001
USER_GID=1001

FORGEJO__database__DB_TYPE=postgres
FORGEJO__database__HOST=db:5432
FORGEJO__database__NAME=forgejo
FORGEJO__database__USER=forgejo
FORGEJO__database__PASSWD=supersecurepasswordright?
POSTGRES_USER=forgejo
POSTGRES_PASSWORD=supersecurepasswordright?
POSTGRES_DB=forgejo

FORGEJO_VERSION=1.21.3-0-arm64-rootless

4. Populate the docker-compose.yml

version: "3"

networks:
  forgejo:
    external: false

volumes:
  forgejo-data:
    driver: local
  forgejo-config:
    driver: local
  forgejo-postgres:
    driver: local


services:
  server:
    image: codeberg.org/forgejo/forgejo:${FORGEJO_VERSION}
    container_name: git-pleasant
    user: ${USER_UID}
    environment:
      - USER_UID=${USER_UID}
      - USER_GID=${USER_GID}
      - FORGEJO__database__DB_TYPE=${FORGEJO__database__DB_TYPE}
      - FORGEJO__database__HOST=${FORGEJO__database__HOST}
      - FORGEJO__database__NAME=${FORGEJO__database__NAME}
      - FORGEJO__database__USER=${FORGEJO__database__USER}
      - FORGEJO__database__PASSWD=${FORGEJO__database__PASSWD}
    networks:
      - forgejo
    volumes:
      - forgejo-data:/var/lib/gitea
      - forgejo-config:/etc/gitea
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro

    ports:
      - "3000:3000"
      - "2222:2222"
    depends_on:
      - db

  db:
    image: docker.io/library/postgres:14
    container_name: git-db
    environment:
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: ${POSTGRES_DB}
    networks:
      - forgejo
    volumes:
      - forgejo-postgres:/var/lib/postgresql/data
    ports:
	    # I have something on 5432 already, so using 5431. This shouldn't matter much for this issue however.
      - "5431:5432"

5. Create the volumes:

$ podman volume create forgejo-postgres
$ podman volume create forgejo-data
$ podman volume create forgejo-config

6. Configure the SSH Passthrough: This is straight from the Gitea docs. Might need to scroll down from the linked heading to the bottom: https://docs.gitea.com/installation/install-with-docker-rootless Instructions included here as well in case they change in the docs.

SSH Container Passthrough

Since SSH is running inside the container, SSH needs to be passed through from the host to the container if SSH support is desired. One option would be to run the container SSH on a non-standard port (or moving the host port to a non-standard port). Another option which might be more straightforward is to forward SSH commands from the host to the container. This setup is explained in the following.

This guide assumes that you have created a user on the host called git with permission to run docker exec, and that the Gitea container is called gitea. You will need to modify that user's shell to forward the commands to the sh executable inside the container, using docker exec.

First, create the file /usr/local/bin/gitea-shell on the host, with the following contents:

#!/bin/sh
/usr/bin/docker exec -i --env SSH_ORIGINAL_COMMAND="$SSH_ORIGINAL_COMMAND" gitea sh "$@"

Note that gitea in the docker command above is the name of the container. If you named yours differently, don't forget to change that.

You should also make sure that you’ve set the permissions of the shell wrapper correctly:

sudo chmod +x /usr/local/bin/gitea-shell

Once the wrapper is in place, you can make it the shell for the git user:

sudo usermod -s /usr/local/bin/gitea-shell git

Now that all the SSH commands are forwarded to the container, you need to set up the SSH authentication on the host. This is done by leveraging the SSH AuthorizedKeysCommand to match the keys against those accepted by Gitea. Add the following block to /etc/ssh/sshd_config, on the host:

Match User git
	AuthorizedKeysCommandUser git 
	AuthorizedKeysCommand /usr/bin/docker exec -i gitea /usr/local/bin/gitea keys -c /etc/gitea/app.ini -e git -u %u -t %t -k %k

(From 1.16.0 you will not need to set the -c /etc/gitea/app.ini option.)

All that is left to do is restart the SSH server:

sudo systemctl restart sshd

Notes

This isn't actually using the docker SSH - it is simply using the commands around it. You could theoretically not run the internal SSH server.

7. Because the user inside the container might not have access to the volumes, podman unshare chown -R 1001 /path/to/volumes will be necessary. To find the volume path I used podman volume inspect $VOLUME_NAME
   1. You might need to podman unshare mkdir -p /path/to/data/volume/_data/git to get around Gitea rootless setup error: mkdir: can't create directory '/var/lib/gitea/git': Permission denied go-gitea/gitea#22322
8. Now you should be ready to start the container! podman-compose up -d
9. Visit (in my case) git.server.home and complete the installation. I added an admin user and kept pretty much everything else as default.
10. Create a repo. Grab the SSH link
11. On the host, make an ssh key, add it to forgejo via the UI
12. Try cloning from the host machine. It should work using the :2222 port
13. On another machine, make an ssh key and add it to forgejo via the UI
14. Try cloning from that machine, you may need to drop the :2222 port from the machine. You can also try ssh -Tv [email protected] to see the error.

What else have you tried?

1. userns: keepid
2. using the "forgejo" user for the postgres container.
   1. This worked after giving forgejo permissions to the postgres volume with podman unshare, but I don't think this is necessary or helped.
3. Adding the "forgejo" user to the same group as "admin"
4. plugging in the "admin" username in the AuthorizedKeyCommand: I thought this might let the command be run as the admin user which would fix it, apparently not!

[zenlord]

@CobyPear I don't see the environment variables related to the SSH server in your setup. Did you set these?

--env DISABLE_SSH=false \
--env START_SSH_SERVER=true \
--env SSH_PORT=3022 \
--env SSH_DOMAIN=git.server.home \

BTW: the default port number is 3022 - if you want to use 2222, you are responsible for configuring your instance correctly.

I just set up a Forgejo instance with rootless podman (slirp4netns network) and I didn't do anything else than use the above settings. Of course, you also need to paste your user's SSH pubkey in the Forgejo user panel. Lastly, if you want to test the connection, make sure to enable sufficient level of verbosity because otherwise yo will not see the message welcoming you and then closing the connection, as Forgejo SSH does not provide shell access.

[CobyPear]

Hi @zenlord , thanks for the reply. I have the ssh settings configured in the app.ini. I am able to make an ssh connection (you can see the public key is accepted in the output above), my problem is the passthrough command fails because the user doesn't have permissions to the container. So that is the root of my question: what am I doing wrong/not doing that the passthrough command fails in this way? How did you set up your user to run the container?

[zenlord]

Hi @CobyPear TBH, I haven't done anything special to set up "SSH passthrough"... It just worked once I got the hang of it (I am pretty new to git commands).
This is how I set up my pod in which I added a container for postgresql, a container for forgejo and a container for forgejo-runner:

podman pod create \
--dns <IP4 addr> \
--network slirp4netns:allow_host_loopback=true,port_handler=slirp4netns,enable_ipv6=true,outbound_addr6=<IP6 addr> \
--hostname "${podname}" \
--userns=keep-id:uid=1000,gid=1000 \
--publish 6000:3000 \
--publish 6022:3022 \
--replace \
"${podname}"

Is it possible that you forgot to map the uid and gid to your user? The forgejo container creates a user (1000/1000) to run the app, so with the userns parameter, you map the user inside the container to the user running the container (/pod) on the host. That user on the host should then of course have R/W access to the volume where you store the git data.