Difficulties that I encountered when running Bottlerocket with only IPv6 connectivity

[stefansundin]

Hello,

Due to the cost of IPv4 on AWS these days, I wanted to try running Bottlerocket ECS with only IPv6. There are currently some issues doing so, and I thought I'd open this discussion in case anyone else is looking for this information.

In case it matters, my VPC does have IPv4, so it is not an IPv6-only subnet. It's just that the instance does not have a public IPv4 address, and there is no NAT.

The first issue is that the default ECR Public hostname (public.ecr.aws) is IPv4 only. However, hidden in the user guide is a dualstack hostname (ecr-public.aws.com). So this means that you can fix the admin and control containers by using the following userdata:

# These versions were accurate for Bottlerocket version 1.45.0

[settings.host-containers.admin]
enabled = true
source = "ecr-public.aws.com/bottlerocket/bottlerocket-admin:v0.12.2"

[settings.host-containers.control]
enabled = true
source = "ecr-public.aws.com/bottlerocket/bottlerocket-control:v0.8.7"

I haven't found any information explaining why this was rolled out on a separate hostname, or if they're going to update the default hostname presented on the ECR Public website. No idea. Why AWS always decides to roll out IPv6 on new "dualstack" endpoints will never stop to confound me. This will be very annoying unless AWS decides to add IPv6 to the default hostname at a later date. Would be nice to know.

Anyway, now that that's fixed, we can try to get the ECS agent working.

Looking in /var/log/ecs/ecs-agent.log we can see the following log:

level=error time=2025-08-24T08:59:41Z msg="Error registering container instance" error="RequestError: send request failed\ncaused by: Post \"https://ecs.us-west-2.amazonaws.com/\": dial tcp 18.246.117.146:443: i/o timeout"

According to the amazon-ecs-agent repository, there's an environment variable ECS_INSTANCE_IP_COMPATIBILITY that can be set to ipv6 to force dualstack hostnames for the ECS service endpoints.

Running systemctl status ecs reveals that the agent is configured in /etc/systemd/system/ecs.service.d/10-base.conf, including environment variables. I added Environment=ECS_INSTANCE_IP_COMPATIBILITY="ipv6" to the file to test and then reloaded the systemd service.

This didn't work. Looking further, it seems that this configuration is very new, released in 1.98.0 in the last few weeks.

# /usr/bin/amazon-ecs-agent --version
Amazon ECS Agent:
	Version: 1.91.2
	Commit: b7e96508

This is from March this year, so it'll probably be a while before Bottlerocket updates to the 1.98.0.

I had one last crazy idea that I wanted to try. What if I could redirect the IPv4 traffic to the IPv6 endpoint?

$ dig ecs.us-west-2.amazonaws.com A +short
18.246.117.146
$ dig ecs.us-west-2.api.aws AAAA +short
2600:1f70:4000:40:4054:de09:724a:c660

In the admin container, I run:

sudo yum install socat
sudo socat TCP4-LISTEN:12345,fork TCP6:[2600:1f70:4000:40:4054:de09:724a:c660]:443

Then in the Bottlerocket OS (sudo sheltie), I set up the iptables rule with:

iptables -t nat -A OUTPUT -d 18.246.117.146 -p tcp --dport 443 -j DNAT --to-destination 127.0.0.1:12345

level=info time=2025-08-25T02:28:08Z msg="Registering Instance with ECS"
level=info time=2025-08-25T02:28:08Z msg="Remaining memory" remainingMemory=412
level=info time=2025-08-25T02:28:38Z msg="Registered container instance with cluster!"
level=info time=2025-08-25T02:28:38Z msg="Instance registration completed successfully"

Amazing, I didn't think that would work.

However, later in the log there are errors connecting to ecs-t-11.us-west-2.amazonaws.com and ecs-a-11.us-west-2.amazonaws.com. I haven't been able to find the dualstack hostnames for these, if they exist. They are mentioned in this documentation.

Anyway, this is where I decided to give up.

It would be nice if IPv6-only was supported out of the box with no extra configuration. The ECR Public hostname is very annoying. Other than that, I think setting ECS_INSTANCE_IP_COMPATIBILITY might not be necessary, since its description says that it detect IPv6 automatically.

That's it. Hope this helps someone.