EKS: Baking AMI (pre-pulling container images before instance boot)

[awoimbee]

Hi,
I wanted to share small elements of my journey trying to bake a bottlerocket AMI. I'm posting here because I didn't find any "prior art", if you have a better way, please share !

From my experience, with bottlerocket you're better off using an EBS snapshot of /dev/xvdb rather than using an AMI.

I'm using EKS 1.22, these steps will likely differ in EKS 1.23 (because of dockershim deprecation).
Bottlerocket version: v1.6.X
EDIT: we're now in 2025 (EKS 1.31) and I'm basically still doing the same thing.

The user-data for the ami builder (the instance you'll launch, snapshot the EBS volume, then shutdown):

[settings]
[settings.host-containers.admin]
enabled = true

Launch the instance. Once you're ssh'ed into the admin container, here's how to interact with the local container registry used by kubelet: sudo sheltie ctr -a /run/dockershim.sock -n k8s.io i ls.

Here's how to pull from a private ECR repo:

# install aws cli into the container
sudo yum install unzip -y
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip -q awscliv2.zip
sudo ./aws/install
# get credentials and save them to a config file
printf "[Credentials.\"xxxxxx.dkr.ecr.us-east-1.amazonaws.com\"]\nUsername = 'AWS'\nPassword = '$(aws ecr get-login-password --region us-east-1)'\n" > reg_cfg.toml
# use host-ctr to pull images, it handles private ECR registries
sudo sheltie host-ctr \
        -s /run/dockershim.sock \
        -n k8s.io pull-image \
        --registry-config /proc/${PPID}/root/home/ec2-user/reg_cfg.toml \
        --source MY_IMAGE:MY_TAG

Once you're done, get the volume id of the device /dev/xvdb using aws ec2 describe-instances. Then shutdown the instance and use aws ec2 create-snapshot. You now have an EBS snapshot of the /dev/xvdb device (the one that contains the container images used by bottlerocket) ! Update your EKS worker node lanch template to add something like:

block_device_mappings {
    device_name = "/dev/xvdb"
    ebs {
      snapshot_id = MY_NEWLY_CREATED_SNAPSHOT_ID
    }
  }

[webern]

Thank you for sharing this!

[mamoit]

Dear @awoimbee...
I can't thank you enough!

[ggupta-98]

Hi @awoimbee I am doing the same thing for a GPU based image, the image is getting cached but its not moving to running state instead its going to the following state,

root           1  1.3  3.3 48040744 2172680 ?    Dsl  16:04   0:08 /usr/bin/python3 /usr/local/bin/flask run --host=0.0.0.0 --port=8001

have you ever faced this error?

[awoimbee]

Hi, this thread is about customization at the OS level and you're talking about an issue in your container by showing us a log from your app ?
The only think I can say is that when you spin up a new instance the EBS volume is "cold" (slow), that might explain the long container start, but idk

[ggupta-98]

Actually I followed your approach for caching the image in bottlerocket, it worked fine for CPU based image for me but for a GPU based image its failing for me, its going to running state and the image is also detected but its accepting any traffic from outside the only difference I found between CPU and GPU based pods was that the port was not allocated to the container as well. Have you faced any similar situation?

[awoimbee]

Nope, never had any such issue, I build the snapshot on a CPU instance and use it on GPU instances fine

[ggupta-98]

okay thanks for the help, will try to figure it out