image-builder fails with 'OCI runtime create failed' error

[jgallucci32]

Describe the bug

The image-builder pod fails to start with the following error message when using Docker CE on RHEL7

Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:301: running exec setns process for init caused \"exit status 41\"": unknown

Steps to reproduce

1. Install Gitpod self-hosted on a Red Hat Enterprise Linux 7 host with Docker CE
2. Deploy self-hosted install from helm chart

Expected behavior

The pod image-builder should be active.

Additional information

Docker version 19.03.13, build 4484c46d9d
Red Hat Enterprise Linux 7.8 (3.10.0-1127.el7.x86_64)

Example repository

Gitpod Helm Chart v0.4

[jgallucci32]

As a workaround, I was able to get the pod to start by removing --userns-remap=default from the entrypoint of the image-builder pod.

[csweichel]

Thanks for reporting this.
We haven't tested Gitpod Self-Hosted on any Red Hat platform, but it would seem that out of the box they're incompatible with Docker's user namespace remapping feature.

I'm happy to see you've already found the workaround. Beware: this gives all users of your installation potentially privileged access to your host (albeit wrapped in a couple of linux namespaces). For trying out Gitpod, even for small teams where you trust everyone using it, that's totally fine though.

[jgallucci32]

@csweichel Thanks for confirming. I am going to attempt to do the following from the RHEL guide which says it is supported since RHEL 7.4 according to this document.

Add option to kernel

grubby --args="namespace.unpriv_enable=1" \
   --update-kernel=/boot/vmlinuz-$(uname -r)

Modify sysctl.conf

echo "user.max_user_namespaces=15076" >> /etc/sysctl.conf

Assign users and groups to be mapped by user namespaces

echo dockremap:808080:1000 >> /etc/subuid
echo dockremap:808080:1000 >> /etc/subgid

I will report back if this is successful or not.

[jgallucci32]

Well that was easy, running the above commands seems to have resolved the issue. I was able to restart the image-builder pod with the --userns-remap=default flag back in and it worked first try. I will keep monitoring for other issues but it appears to be working just the same as it was before.

Suggest this gets added to the documentation as a prerequisite for using RHEL/CentOS.

[akosyakov]

Self hosted docs are open sourced here: https://github.com/gitpod-io/website/tree/master/src/docs/self-hosted If someone would like to make a PR.

[jgallucci32]

PR created for documentation https://github.com/gitpod-io/website/pull/811

NOTE: I created a separate page for other settings such as configuring for Rancher Kubernetes. This way the page can expand in the future for other vendor-specific implementations.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.