Connection refused when using socket activation

[thyeestes]

As suggested in the Thread I openend in the Traefik repository the issue below might be related to Pasta or Podman, hence I open a discussion here as well.

What did you do?
I upgraded my Traefik container to v3.1 to enable socket activation. I removed PublishPort from my container file and created two socket files( one for HTTP and one for HTTPS). When my Traefik container is started I can acces my containers from my LAN, but Uptime Kuma cannot reach the websites proxied by Traefik.
I also tested with a other container running a browser but got the same result. All containers are on the same Podman network.

Turning off socket activation solves the issue.

What did you see instead?
Kuma reports an error: connect ECONNREFUSED 192.168.1.100:443 (where as 192.168.1.100 is the IP address of my host running Podman)

I also notice that there are more processes listening on port 80 and 443, so maybe a connection is made to the wrong listener.

Ports listening without socket activation:
rootlessp 589917 thyestes 11u IPv6 2247907 0t0 TCP *:80 (LISTEN)
rootlessp 589917 thyestes 12u IPv6 2247908 0t0 TCP *:443 (LISTEN)

Ports listening with socket activation:
systemd 775 thyestes 34u IPv4 2564674 0t0 TCP *:80 (LISTEN)
systemd 775 thyestes 35u IPv4 2566204 0t0 TCP *:443 (LISTEN)
conmon 666883 thyestes 3u IPv4 2566204 0t0 TCP *:443 (LISTEN)
conmon 666883 thyestes 4u IPv4 2564674 0t0 TCP *:80 (LISTEN)
traefik 666896 thyestes 3u IPv4 2564674 0t0 TCP *:80 (LISTEN)
traefik 666896 thyestes 5u IPv4 2566204 0t0 TCP *:443 (LISTEN)

What version of Traefik are you using?
Version: 3.1.0
Codename: comte
Go version: go1.22.5
Built: 2024-07-15T14:44:04Z
OS/Arch: linux/amd64

What is your environment & configuration?
[Container]
Image=docker.io/library/traefik:v3.1
ContainerName=traefik
Network=proxy.network
Volume=/storage/traefik/acme.json:/acme.json
Volume=%t/podman/podman.sock:/var/run/docker.sock

Add more configuration information here.
Client: Podman Engine
Version: 5.1.2
API Version: 5.1.2
Go Version: go1.21.12
Built: Thu Jul 11 15:01:37 2024
OS/Arch: linux/amd64

I hope there is someone, who can shine a light on this.

[Luap99]

Did you allow the ports in your firewall? Can you connect from localhost, i.e. curl 127.0.0.1?

[thyeestes]

@eriksjolund I tried switching to Pasta again and added the option you described above. Result is still that Kuma is not able to connect.

[eriksjolund]

@thyeestes The required pasta version is from last week (passt-0^20240821.g1d6142f) . Did you try with it?

[thyeestes]

Yep that's the version (passt 2024_08_21.1d6142f)

To be sure. I need to replace 11.11.11.11 with the IP address of my host machine, don't I?

[eriksjolund]

No, use 11.11.11.11 or some other IP address except the IP address of your host machine.
I just tried using the IP address of my host machine, but then it doesn't work.

Another thing: Maybe the file /etc/hosts in the Uptime Kuma container needs to be modified so that
the hostnames that Uptime Kuma monitors get resolved to 11.11.11.11?

[thyeestes]

Okay tried that, also updated my host file in the Kuma container, however Kuma stil tries to connect to my real host IP address.

[fsdrw08]

same here, I use podman socket activation + traefik, I put a hashicorp vault server behind it, and config vault as a OIDC provider, then deploy another app hashicorp nomad in the same podman pasta network, when I try to connect from nomad to vault through traefik port 443 which listen by https socket, nomad shows 443: connect: connection refused, but when I try to connect to the vault with the same https url from my laptop which in the same LAN, it works

[eriksjolund]

@fsdrw08 Did you connect using host.containers.internal?

Another possibility is to use --add-host my-vault.example.com:host-gateway and connect to my-vault.example.com

(I'm assuming my-vault.example.com resolves in DNS to your host's main network interface IP address)

References:

https://docs.podman.io/en/latest/markdown/podman-run.1.html#add-host-hostname-hostname-ip

https://blog.podman.io/2024/10/podman-5-3-changes-for-improved-networking-experience-with-pasta/

Outbound TCP/UDP connections to the host's main network interface (e.g eth0)

[fsdrw08]

thanks, it works, btw, I am using podman kube play to deploy the app, it seems also works to set host-gateway in the hostAliases.ip filed

hostAliases:
  - ip: "host-gateway"
    hostnames:
      - "vault.example.com"

[eriksjolund]

I investigated some more:
I think there are two ways a container on the custom network can access the traefik container, either by connecting to the socket-activated sockets (Alternative 2 below) or by connecting to sockets that traefik has created by itself (Alternative 1 below).

Alternative 1: create extra socket and use NetworkAlias=

Alternative 2: use AddHost= and host-gateway

An example of Alternative 1 is example2 where traefik in addition to inheriting socket-activated sockets from its parent process also creates its own listening sockets on the custom network. The traefik container is configured with NetworkAlias= (one for each proxied backend server). One advantage with Alternative 1 is that the download speed is higher when a container downloads files from the traefik container.

[thyeestes]

If you add hostnames in your Traefik quadlet, you have to list them for all your containers don't you? Which is quite cumbersome when you have 20+ containers

[eriksjolund]

If you add hostnames in your Traefik quadlet, you have to list them for all your containers don't you?

yes

Which is quite cumbersome when you have 20+ containers

yes, that is a disadvantage.