Add new stack parameter for enabling dualstack docker

Adds new `DockerNetworkingProtocol` Cloud Formation Parameter, for
opting in to starting containers with both ipv4 and ipv6 addresses. This
might be useful when running the elastic stack on dual stack and
ipv6-only subnets.

Prior to this change, elastic stack instances could run on dual stack
and ipv6-only VPC subnets and many things continue to work as normal.
However, docker builds and containers do not - containers are started
with private ipv4 addresses and no ipv6 addresses. Attempts to connect
to global or VPC ipv6 addresses from inside a container fail. When this
new parameter is set to `dualstack`, containers will be assigned an ipv6
address and depending on the configuration of the VPC it's running in,
containers may be able to connect to the outside world over ipv6.

-- dualstack subnets

When the new parameter is set to `dualstack` and the instances are running
on a dualstack VPC subnet, we expect:

* containers are started with a private ipv4
* containers are started with an ipv6 from 2001:db8::/32
* connecting to external resources over ipv4 works, with docker
  performing NAT from the container private ipv4 to the EC2 instance
  AWS assigned ipv4
* connecting to external resources over ipv6 works, with docker
  performing NAT from the container private ipv6 to the EC2 instance
  AWS assigned ipv6. NATing ipv6 feels weird, but it's the way docker
  25.x works

-- ipv6-only subnets

When the new parameter is set to dualstack and the instances are running
on a ipv6-only VPC subnet, here's what should happen:

* containers are started with a private ipv4
* containers are started with an ipv6 from 2001:db8::/32
* connecting to external resources over ipv4 will fail with an error
* connecting to external resources over ipv6 works, with docker
  performing NAT from the container private ipv6 to the EC2 instance
  AWS assigned ipv6. NATing ipv6 feels weird, but it's the way docker
  25.x works
* It's very likely your ipv6 subnet will have DNS64 enabled. In this
  case, DNS hostnames that return only A addresses will have a fake
  AAAA record added to the response. This record points to an AWS NAT
  gateway. The container should use the fake ipv6 address and the AWS
  NAT gateway will translate to ipv4 (for a fee).

-- why 2001:db8::/32?

https://chameth.com/ipv6-docker-routing/ is a good explaination of the
issue.

Given docker is NATing the containers ipv6 traffic, the natural ipv6
range to use would be fd00::/8 - the ULA range
(https://en.wikipedia.org/wiki/Unique_local_address). That range will
partially work, however on a dualstack subnet where containers have
working ipv4 and ipv6 addresss, using fd00::/8 will make most requests
default to ipv4 instead of ipv6. That's particularly undesirable in a CI
environment on AWS because ipv4 traffic almost certainly goes via a NAT
gateway with per Gb fees, and CI often generates a lot of AWS ingress
traffic.

By using 2001:db8::/32 the dualstack containers should default to using
ipv6 in most cases where they have the choice of either protocol. This
may avoid significant NAT gateway fees.

2001:db8::/32 is technically reserved for documentation use only. The
cost benefits are significant though, and docker is using them privately
so there should be no negative impact.

-- why opt-in?

It would be nice to just auto-detect when the instances have a valid
ipv6 addresses assigned and automatically start docker in dualstack
mode. However, there is significant potential to change the behavior of
networking in the container and might have surprising side effects. This
prosposes starting opt-in so we can start to build some experience with
dualstack docker and AWS. Eventually, it would be great to make it Just
Work.

Author: James Healy

packer/linux/conf/bin/bk-configure-docker.sh

@@ -69,6 +69,36 @@ else
   echo User namespace remapping not configured.
 fi
 
+# One day we can auto-detect whether the instance is v4-only, dualstack or v6-only. To avoid a
+# breaking change though, we'll default to ipv4 only and users can opt into v6 support. The elastic
+# stack has always defaulted to v4-only so this ensuring no breaking behaviour.
+# v6-only is currently not an option because docker doesn't support it, but maybe one day....
+echo Customising docker network configuration...
+
+if [[ "${DOCKER_NETWORKING_PROTOCOL}" == "ipv4" ]]; then
+  # This is the default
+  cat <<<"$(
+    jq \
+      '."default-address-pools"=[{"base":"172.17.0.0/12","size":20},{"base":"192.168.0.0/16","size":24}]' \
+      /etc/docker/daemon.json
+  )" >/etc/docker/daemon.json
+elif [[ "${DOCKER_NETWORKING_PROTOCOL}" == "dualstack" ]]; then
+  # Using v6 inside containers requires DOCKER_EXPERIMENTAL. This is configured
+  # further down
+  DOCKER_EXPERIMENTAL=true
+  cat <<<"$(jq '.ipv6=true' /etc/docker/daemon.json)" >/etc/docker/daemon.json
+  cat <<<"$(jq '."fixed-cidr-v6"="2001:db8:1::/64"' /etc/docker/daemon.json)" >/etc/docker/daemon.json
+  cat <<<"$(jq '.ip6tables=true' /etc/docker/daemon.json)" >/etc/docker/daemon.json
+  cat <<<"$(
+    jq \
+      '."default-address-pools"=[{"base":"172.17.0.0/12","size":20},{"base":"192.168.0.0/16","size":24},{"base":"2001:db8:2::/104","size":112}]' \
+      /etc/docker/daemon.json
+  )" >/etc/docker/daemon.json
+else
+  # docker 25.0 doesn't support ipv6 only, so we don't support it either
+  true
+fi
+
 if [[ "${DOCKER_EXPERIMENTAL:-false}" == "true" ]]; then
   echo Configuring experiment flag for docker daemon...
   cat <<<"$(jq '.experimental=true' /etc/docker/daemon.json)" >/etc/docker/daemon.json
@@ -85,13 +115,6 @@ else
   echo Instance storage not configured.
 fi
 
-echo Customising docker IP address pools...
-cat <<<"$(
-  jq \
-    '."default-address-pools"=[{"base":"172.17.0.0/12","size":20},{"base":"192.168.0.0/16","size":24}]' \
-    /etc/docker/daemon.json
-)" >/etc/docker/daemon.json
-
 echo Cleaning up docker images...
 systemctl start docker-low-disk-gc.service
 

templates/aws-stack.yml

@@ -462,6 +462,14 @@ Parameters:
       - "false"
     Default: "true"
 
+  DockerNetworkingProtocol:
+    Type: String
+    Description: Which IP version to enable for docker containers and building docker images. Only applies to Linux instances, not Windows.
+    AllowedValues:
+      - "ipv4"
+      - "dualstack"
+    Default: "ipv4"
+
   EnableSecretsPlugin:
     Type: String
     Description: Enables s3-secrets plugin for all pipelines
@@ -1205,6 +1213,7 @@ Resources:
                   <powershell>
                   $Env:DOCKER_USERNS_REMAP="${EnableDockerUserNamespaceRemap}"
                   $Env:DOCKER_EXPERIMENTAL="${EnableDockerExperimental}"
+                  $Env:DOCKER_NETWORKING_PROTOCOL="${DockerNetworkingProtocol}"
                   powershell -file C:\buildkite-agent\bin\bk-configure-docker.ps1 >> C:\buildkite-agent\elastic-stack.log
 
                   $Env:BUILDKITE_STACK_NAME="${AWS::StackName}"
@@ -1260,6 +1269,7 @@ Resources:
                   #!/bin/bash -v
                   DOCKER_USERNS_REMAP=${EnableDockerUserNamespaceRemap} \
                   DOCKER_EXPERIMENTAL=${EnableDockerExperimental} \
+                  DOCKER_NETWORKING_PROTOCOL=${DockerNetworkingProtocol} \
                   BUILDKITE_ENABLE_INSTANCE_STORAGE="${EnableInstanceStorage}" \
                     /usr/local/bin/bk-configure-docker.sh
                   --==BOUNDARY==