No network connectivity in worker containers when runtime=guardian

[funkypenguin]

Hey folks!

We run Concourse in a Kubernetes cluster, all is well. We're now trying to connect a remote worker on Ubuntu 20.04, which has been relatively straightforward, except for one issue...

The worker registers with TSA fine, and is able to be used to run jobs based on tags, but... when the worker uses "guardian" as its runtime (the default), the resulting container has no network access. Thus, it can't even pull resource images from Docker Hub.

Here's an example of the task output of a job delegated to the remote worker:

resource script '/opt/resource/check []' failed: exit status 1

stderr:
�[31mERRO�[0m[0208] checking origin index.docker.io failed: get remote image: Get "https://index.docker.io/v2/": dial tcp: lookup index.docker.io on 8.8.4.4:53: read udp 10.80.0.2:51810->8.8.4.4:53: i/o timeout 

Curiously, changing the runtime to containerd or houdini resolves this problem. I.e., the fault is specific to how guardian / garden is being used as a runtime.

My worker is started using the following commandline:

./bin/concourse worker --tsa-public-key=elpenguino-tsa-public.key-pub --tsa-worker-private-key=worker.key --tsa-host=<TSA IP>:2222 --team=main --tag=windows-worker --work-dir=/tmp/concourse --runtime=guardian --ephemeral --garden-config=garden-config.ini

And garden-config.ini looks like this:

[server]
dns-server = 8.8.8.8
dns-server = 8.8.4.4
allow-host-access = true

Any ideas what I'm doing wrong?

Thanks!
D

[taylorsilva]

I would try hijacking into that check container that you see failing and poke around from there to see what's up. See what /etc/resolve.conf has in it?

You could also try setting the --garden-dns-proxy-enable which should pass in the contents from the hosts /etc/resolf.conf

[funkypenguin]

Thanks @taylorsilva! When I hijack into the container, I find that I can't even ping my DNS servers (1.1.1.1 in this case), so it's not a matter of /etc/resolv.conf, it's all network connectivity which is affected :(

[funkypenguin]

Followup - the issue seems to have been a conflict between the default iptables rules created by installing Docker, and the rules created by concourse worker / guardian. I stopped Docker, flushed iptables, and now I have connectivity :)