How to update MTU of AWS EC2 running bottlerocket os?

[tskinner-oppfi]

I'd like to migrate to bottlerocket os for my EKS clusters but I currently change the MTU via a user data script using commands described here https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_mtu.html#set_mtu

How can I change the MTU of bottlerocket instances?

[stmcginnis]

Great question @tskinner-oppfi. My first thought, as an equivalent to the Amazon Linux instructions, would be to create a bootstrap container to make the same ip link set calls. But my experiments with that show this particular modification sadly will not work.

This could be a race between the bootstrap container setting the value, and the MTU being assigned via DHCP. With EC2, the default ends up being option interface-mtu 9001;. So even though I was able to see the value changed by my bootstrap container, when I then checked it after initialization completed it was back to 9001.

I think to support this properly Bottlerocket might need a new setting exposed to override this. If you would like to see this capability, would you mind submitting a feature request issue? I'm not sure when it could be added, but at least that would help track the request and see if anyone else is also looking for this capability to help prioritize things.

Notes

Just in case anyone is curious, here is what I did to change this setting with a bootstrap container. This will not work, but maybe it can give some clues.

First I created a script to be called by the container called bootstrap-script.sh:

#!/usr/bin/env bash

sudo ip link set dev eth0 mtu 1600

ip link show

Then created a Dockerfile with:

FROM public.ecr.aws/amazonlinux/amazonlinux:latest
COPY ./bootstrap-script.sh /
RUN chmod +x /bootstrap-script.sh
# Both sudo and ip are needed to make the `ip link set` call work properly
RUN yum install -y sudo iproute
ENTRYPOINT ["/bootstrap-script.sh"]

Built the container and published to a private ECR registry. Then set a test node's user data to have:

[settings.bootstrap-containers.bootstrap]
source = "123456789.dkr.ecr.us-east-2.amazonaws.com/bootstrap:latest"
mode = "always"

I could see in the system log (journalctl -u bootstrap-containers@bootstrap) that the script ran and successfully set the eth0 interface MTU to 1600:

eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1600 qdisc mq state UP mode DEFAULT group default qlen 1000

But by the time the rest of the system initialization happened and I was able to connect and run sheltie to do ip link show on the running system, that value was then back to the default 9001:

eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000

[tskinner-oppfi]

Thanks! created a new issue #3341

[MrFishFinger]

we had a similar issue, and were able to workaround it with a k8s daemonset that uses a pod with elevated permissions to reconfigure systemd-networkd on the bottlerocket host.

basically, it creates a networkd override config file (20-override-mtu.conf) on the host, then restarts the networkd daemon via nsenter:

echo -e "[Link]\nMTUBytes=$REQ_MTU\n\n[DHCPv4]\nUseMTU=false\n\n[IPv6AcceptRA]\nUseMTU=false" > /host/etc/systemd/network/10-eth0.network.d/20-override-mtu.conf
nsenter --target 1 --mount systemctl restart systemd-networkd

its definitely a hack though, so would be happy to see a proper config option exposed in bottlerocket 🤞 (#3341)!

---

full manifest below:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: host-mtu-manager
  namespace: default
spec:
  selector:
    matchLabels:
      app: host-mtu-manager
  template:
    metadata:
      labels:
        app: host-mtu-manager
    spec:
      hostNetwork: true
      hostPID: true
      tolerations:
        - effect: NoSchedule
          operator: Exists
      containers:
        - name: host-mtu-manager
          image: public.ecr.aws/docker/library/busybox:latest
          securityContext:
            capabilities:
              add:
                - SYS_ADMIN
                - SYS_PTRACE
            seLinuxOptions:
              type: super_t
          command:
            - /bin/sh
            - -c
            - |
              export REQ_MTU=1460
              export SLEEP_TIME=30
              
              while true; do
                CURRENT_MTU=$(ip link show eth0 | awk '/mtu/ {print $5}')
    
                if [ "$CURRENT_MTU" != "$REQ_MTU" ]; then
                  echo "Current MTU ($CURRENT_MTU) does not match desired MTU ($REQ_MTU). Updating..."
                  echo -e "[Link]\nMTUBytes=$REQ_MTU\n\n[DHCPv4]\nUseMTU=false\n\n[IPv6AcceptRA]\nUseMTU=false" > /host/etc/systemd/network/10-eth0.network.d/20-override-mtu.conf
                  nsenter --target 1 --mount systemctl restart systemd-networkd
                  sleep 1
                  UPDATED_MTU=$(ip link show eth0 | awk '/mtu/ {print $5}')
                  echo "MTU updated to '$UPDATED_MTU'"
                else
                  echo "MTU is already set to '$REQ_MTU'. No update needed."
                fi
    
                echo "Sleeping for $SLEEP_TIME seconds..."
                sleep $SLEEP_TIME
              done
          volumeMounts:
            - name: host
              mountPath: /host
              readOnly: false
      volumes:
        - name: host
          hostPath:
            path: /
            type: Directory