IPv6, NAT64: Rails cannot connect to IPv4 through working NAT64 | experience report with bugfix

[tillcarlos]

Hey Devops Friends

A friend urged me to document my last two days of pain. This is our way to running our own infrastructure on ipv6. I am not the main admin here, but debugged this issue. I think it'd be good for people who are stuck at the same place.

Feel free to close / move this if it doesn't belong here.

Situation:

I run a SaaS with one Hetzner server + sqlite3 + ipv4. It runs great. I met a German guy who told me that IPv6 would be a good thing. So we switched it over to IPv6 and ran into a weird issue which needs some context.

This is our setup:

• Several hetzner servers behind a hetzner firewall
• Wireguard VPN, which does the IPv4 to v6 bridge (where I live IPv4 is hardly available)
• One hetzner server, IPv6 only, Rails 8.

I also have an admin in my company who set this all up.

How we set up ipv6

We have a .kamal/hooks/docker-setup that goes into each server and does this:

# For > /etc/docker/daemon.json
{
  "ipv6": true,
  "experimental": true,
  "ip6tables": true,
  "fixed-cidr-v6": "2001:db8:1::/64"
}

# enable ipv6 forwarding
sysctl -w net.ipv6.conf.all.forwarding=1
sysctl -w net.ipv6.conf.default.forwarding=1

# persist sysctl settings
For > /etc/sysctl.d/99-ipv6.conf
  net.ipv6.conf.all.forwarding=1
  net.ipv6.conf.default.forwarding=1

docker network create --ipv6 --subnet=fd00:dead:beef::/48 --gateway=fd00:dead:beef::1 kamal 

It was running nicely until we tried to send an email via sendgrid. They don't have ipv6 (apparently email providers dislike ipv6).

Main issue: Sendgrid wouldn't send emails:

ENETUNREACH - Network is unreachable.

That's when we created a NAT64 server - separate Hetzner box. I didn't set it up, but it worked well. We added the NAT64 to the /etc/resolv.conf of the host.

The rails container still have this DNS set. You are not supposed to change it (like run with --dns flag), and with a custom network you cannot even change it. Fine, because it forwards to the host anyway:

rails@2a01:xxx:xxx:xxx::1-dcf5a7aacff1:/rails$ cat /etc/resolv.conf
# Generated by Docker Engine.
# This file can be edited; Docker Engine will not make further changes once it
# has been modified.

nameserver 127.0.0.11
search Home
options edns0 ndots:0

Where is the error?

We tried to DNS-resolve sendgrid and both time (host + container) we GET THE SAME RESPONSE.

Obvious, because the DNS just forwards to the host machine:

rails@dba1f084d9d1:/rails$ dig smtp.sendgrid.com

; <<>> DiG 9.18.33-1~deb12u2-Debian <<>> smtp.sendgrid.com
...
;smtp.sendgrid.com.		IN	A

;; ANSWER SECTION:
smtp.sendgrid.com.	1789	IN	CNAME	smtp.sendgrid.net.
smtp.sendgrid.net.	79	IN	A	52.57.139.126
...

;; Query time: 3 msec
;; SERVER: 127.0.0.11#53(127.0.0.11) (UDP)
;; WHEN: Mon Feb 17 10:51:57 UTC 2025
;; MSG SIZE  rcvd: 169

By getting help from my German IPv6-buddy, we found the main difference:

This is a curl to sendgrid from the host mathine:

root@52b89e9408bf 05:06:53 /srv ❯ curl -vvv http://smtp.sendgrid.com:25
*   Trying [2a01:4f8:c17:bcde:64:0:36e4:2758]:25...
•⁠  ⁠Connected to smtp.sendgrid.com (2a01:4f8:c17:bcde:64:0:36e4:2758) port 25 (#0)
	⁠GET / HTTP/1.1
	⁠Host: smtp.sendgrid.com:25
	⁠User-Agent: curl/7.88.1
	⁠Accept: /
> 
•⁠  ⁠Received HTTP/0.9 when not allowed
•⁠  ⁠Closing connection 0
curl: (1) Received HTTP/0.9 when not allowed

And this is on the rails container:

rails@2a01:4f8:1c0c:4002::1-fd4a2e9f8dc2:/rails$  curl -vvv http://smtp.sendgrid.com:587
*   Trying 18.197.194.208:587...
* connect to 18.197.194.208 port 587 failed: Network is unreachable
*   Trying 54.228.39.88:587...
* connect to 54.228.39.88 port 587 failed: Network is unreachable
*   Trying 52.57.139.126:587...
*   Trying [2a01:4f8:1c1e:a2b6:64:0:6c80:e5b8]:587...
* Connected to smtp.sendgrid.com (2a01:4f8:1c1e:a2b6:64:0:6c80:e5b8) port 587 (#0)
> GET / HTTP/1.1
> Host: smtp.sendgrid.com:587
> User-Agent: curl/7.88.1
> Accept: */*
>
* Received HTTP/0.9 when not allowed
* Closing connection 0
curl: (1) Received HTTP/0.9 when not allowed
rails@2a01:4f8:1c0c:4002::1-fd4a2e9f8dc2:/rails$

Culprit:

The container is TRYING IPV4 FIRST! Why the heck is that?

This blog post finally helped me: https://chameth.com/ipv6-docker-routing/

Apparently the IPv4 is preferred, because we are talking to a IPv4 DNS (the one inside docker at 127.0.11)

We can set the preferences in a file we didn't know existed. Here in the Dockerfile:

RUN printf '%s\n' \
  'label ::1/128       0' \
  'label ::/0          1' \
  'label 2002::/16     2' \
  'label ::/96         3' \
  'label ::ffff:0:0/96 4' \
  'label fec0::/10     5' \
  'label fc00::/7      6' \
  'label 2001:0::/32   7' \
  'label fd00::/8      1' \
  >> /etc/gai.conf

That did the trick.

If you find anything here confusing, please send me a message and I'll improve this text.

[djmb]

Thanks for this! It's not reporting a Kamal issue though, so I'll convert it to a discussion instead.