Add optional embedding of container images into DinD:

This helps use cases where images already existing
in the DinD cache is needed. Air gap envs, latency
constrained/concerned envs, etc.

Signed-off-by: Jacob Weinstock <[email protected]>

Author: jacobweinstock

.gitignore

@@ -12,3 +12,5 @@ cache/
 *.swp
 .idea
 kernel/Dockerfile.autogen.*
+images/hook-embedded/images/*
+!images/hook-embedded/images/.keep

README.md

@@ -160,9 +160,17 @@ The `gha-matrix` CLI command prepares a set of JSON outputs for GitHub Actions m
 - `DOCKER_ARCH` is used by the `linuxkit-containers` command to build the containers for the specified architecture.
 - `DO_PUSH`: `yes` or `no`, will push the built containers to the OCI registry; defaults to `no`.
 
+### Embedding container images into the DinD (docker-in-docker), also known as [hook-docker](images/hook-docker/), container
+
+For use cases where having container images already available in Docker is needed, the following steps can be taken to embed container images into hook-docker (DinD):
+
+1. Create a file named `images.txt` in the [images/hook-embedded/](images/hook-embedded/) directory.
+1. Populate this `images.txt` file with the list of images to be embedded. See [images/hook-embedded/images.txt.example](images/hook-embedded/images.txt.example) for details on the required file format.
+1. Change directories to [images/hook-embedded/](images/hook-embedded/) and run the [pull-images.sh](images/hook-embedded/pull-images.sh) script. Read the comments at the top of the script for more details.
+1. Change directories to the root of the HookOS repository and run `sudo ./build.sh build ...` to build the HookOS kernel and ramdisk. FYI, `sudo` is needed as DIND changes file ownerships to root.
+
 ### Build system TO-DO list
 
-- [ ] Update to Linuxkit 1.2.0 and new linuxkit pkgs; this might lead into the containerd vs dind;
 - [ ] `make debug` functionality (sshd enabled) was lost in the Makefile -> bash transition;
 
 [formats]: https://github.com/linuxkit/linuxkit/blob/master/README.md#booting-and-testing

bash/hook-lk-containers.sh

@@ -13,6 +13,7 @@ function build_all_hook_linuxkit_containers() {
 	build_hook_linuxkit_container hook-mdev HOOK_CONTAINER_MDEV_IMAGE
 	build_hook_linuxkit_container hook-containerd HOOK_CONTAINER_CONTAINERD_IMAGE
 	build_hook_linuxkit_container hook-runc HOOK_CONTAINER_RUNC_IMAGE
+	build_hook_linuxkit_container hook-embedded HOOK_CONTAINER_EMBEDDED_IMAGE
 }
 
 function build_hook_linuxkit_container() {

bash/linuxkit.sh

@@ -70,7 +70,8 @@ function linuxkit_build() {
 			HOOK_CONTAINER_MDEV_IMAGE="${HOOK_CONTAINER_MDEV_IMAGE}" \
 			HOOK_CONTAINER_CONTAINERD_IMAGE="${HOOK_CONTAINER_CONTAINERD_IMAGE}" \
 			HOOK_CONTAINER_RUNC_IMAGE="${HOOK_CONTAINER_RUNC_IMAGE}" \
-			envsubst '$HOOK_VERSION $HOOK_KERNEL_IMAGE $HOOK_KERNEL_ID $HOOK_KERNEL_VERSION $HOOK_CONTAINER_IP_IMAGE $HOOK_CONTAINER_BOOTKIT_IMAGE $HOOK_CONTAINER_DOCKER_IMAGE $HOOK_CONTAINER_MDEV_IMAGE $HOOK_CONTAINER_CONTAINERD_IMAGE $HOOK_CONTAINER_RUNC_IMAGE' \
+			HOOK_CONTAINER_EMBEDDED_IMAGE="${HOOK_CONTAINER_EMBEDDED_IMAGE}" \			
+			envsubst '$HOOK_VERSION $HOOK_KERNEL_IMAGE $HOOK_KERNEL_ID $HOOK_KERNEL_VERSION $HOOK_CONTAINER_IP_IMAGE $HOOK_CONTAINER_BOOTKIT_IMAGE $HOOK_CONTAINER_DOCKER_IMAGE $HOOK_CONTAINER_MDEV_IMAGE $HOOK_CONTAINER_CONTAINERD_IMAGE $HOOK_CONTAINER_RUNC_IMAGE $HOOK_CONTAINER_EMBEDDED_IMAGE' \
 			> "hook.${inventory_id}.yaml"
 
 	declare -g linuxkit_bin=""

images/hook-embedded/Dockerfile

@@ -0,0 +1,5 @@
+FROM scratch
+ENTRYPOINT []
+WORKDIR /
+COPY ./images/ /
+CMD []

images/hook-embedded/images.txt.example

@@ -0,0 +1,9 @@
+# This is an example file. It explains the required format.
+# For the actual file, you must remove all the comments.
+# The format is source image, a single space, optional additional tag of the source image.
+#<source image> <optional additional tag of the source image>
+# for example:
+quay.io/tinkerbell/tink-worker:v0.10.0
+quay.io/tinkerbell/tink-worker:v0.10.0 tink-worker:v0.10.0
+quay.io/tinkerbell/actions/image2disk embedded/actions/image2disk
+quay.io/tinkerbell/actions/cexec 127.0.0.1/embedded/actions/cexec

images/hook-embedded/images/.keep

@@ -0,0 +1,59 @@
+#!/bin/bash
+
+# This script is used to pull 
+# This script is used to build an image that is embedded in HookOS.
+# The image contains the /var/lib/docker directory which has pulled images
+# from the images.txt file. When HookOS boots up, the DinD container will
+# have all the images in its cache.
+
+# The purpose of doing this is so that EKS Anywhere doesn't have to set registry
+# and registry credentials in each Hardware object.
+
+# In my testing the initramfs for HookOS with these embedded images was about 334MB.
+# The machine booting HookOS needed 6GB of RAM to boot up successfully.
+
+set -euo pipefail
+
+function main() {
+    local dind_container="$1"
+    local images_file="$2"
+    # as this function maybe called multiple times, we need to ensure the container is removed
+    trap "docker rm -f ${dind_container} &> /dev/null" RETURN
+    # we're using set -e so the trap on RETURN will not be executed when a command fails
+    trap "docker rm -f ${dind_container} &> /dev/null" EXIT
+    # start DinD container
+    # In order to avoid the src bind mount directory (./images/) ownership from changing to root
+    # we don't bind mount to /var/lib/docker in the container because the DinD container is running as root and
+    # will change the permissions of the bind mount directory (images/) to root.
+    echo -e "Starting DinD container"
+    echo -e "-----------------------"
+    docker run -d --rm --privileged --name "${dind_container}" -v ${PWD}/images/:/var/lib/docker-embedded/ -d docker:dind
+
+    # wait until the docker daemon is ready
+    until docker exec "${dind_container}" docker info &> /dev/null; do
+        sleep 1
+    done
+
+    # pull images from list
+    # this expects a file named images.txt in the same directory as this script
+    # the format of this file is line separated: <image> <optional tag>
+    #
+    # the || [ -n "$first_image" ] is to handle the last line of the file that doesn't have a newline.
+    while IFS=" " read -r first_image image_tag || [ -n "$first_image" ] ; do
+        echo -e "----------------------- $first_image -----------------------"
+        docker exec "${dind_container}" docker pull $first_image
+        if [[ $image_tag != "" ]]; then
+            docker exec "${dind_container}" docker tag $first_image $image_tag
+        fi
+    done < "${images_file}"
+
+    # remove the contents of /var/lib/docker-embedded so that any previous images are removed. Without this it seems to cause boot issues.
+    docker exec "${dind_container}" sh -c "rm -rf /var/lib/docker-embedded/*"
+    # We need to copy /var/lib/docker to /var/lib/docker-embedded in order for HookOS to use the Docker images in its build.
+    docker exec "${dind_container}" sh -c "cp -a /var/lib/docker/* /var/lib/docker-embedded/"
+}
+
+arch="amd64"
+dind_container_name="hookos-dind-${arch}"
+images_file="images.txt"
+main "${dind_container_name}" "${images_file}" "${arch}"

images/hook-embedded/pull-images.sh

@@ -9,21 +9,36 @@
 # - HOOK_CONTAINER_MDEV_IMAGE: ${HOOK_CONTAINER_MDEV_IMAGE}
 # - HOOK_CONTAINER_CONTAINERD_IMAGE: ${HOOK_CONTAINER_CONTAINERD_IMAGE}
 # - HOOK_CONTAINER_RUNC_IMAGE: ${HOOK_CONTAINER_RUNC_IMAGE}
+# - HOOK_CONTAINER_EMBEDDED_IMAGE: ${HOOK_CONTAINER_EMBEDDED_IMAGE}
 # - Other variables are not replaced: for example this is a literal dollarsign-SOMETHING: $SOMETHING and with braces: ${SOMETHING}
 
 kernel:
   image: "${HOOK_KERNEL_IMAGE}"
-  cmdline: "this_is_not_used=at_at_all_in_hook command_line_is_determined_by=ipxe"
+  cmdline: "this_is_not_used=at_all_in_hook command_line_is_determined_by=ipxe"
 
 init:
-  # this sha is the first with cgroups v2 as the default
-  - linuxkit/init:8a7b6cdb89197dc94eb6db69ef9dc90b750db598
+  # this init container sha has support for volumes
+  - linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
   - "${HOOK_CONTAINER_RUNC_IMAGE}"
   - "${HOOK_CONTAINER_CONTAINERD_IMAGE}"
   - linuxkit/ca-certificates:v1.0.0
   - linuxkit/firmware:24402a25359c7bc290f7fc3cd23b6b5f0feb32a5 # "Some" firmware from Linuxkit pkg; see https://github.com/linuxkit/linuxkit/blob/master/pkg/firmware/Dockerfile
 
+volumes:
+  - name: embedded-images
+    image: "${HOOK_CONTAINER_EMBEDDED_IMAGE}"
+
 onboot:
+  - name: embedded-images
+    image: alpine
+    binds:
+      - /var/run/images:/var/run/images
+      - embedded-images:/images
+    command: [ "sh", "-xc", "mv /images/* /var/run/images/" ]
+    runtime:
+      mkdir:
+        - /var/run/images
+
   - name: rngd1
     image: linuxkit/rngd:v1.0.0
     command: [ "/sbin/rngd", "-1" ]
@@ -100,6 +115,8 @@ services:
     devices:
     - path: all
       type: b
+    - path: all
+      type: c
     - path: "/dev/console"
       type: c
       major: 5
@@ -187,6 +204,8 @@ services:
     devices:
     - path: all
       type: b
+    - path: all
+      type: c
 
   - name: hook-bootkit
     image: "${HOOK_CONTAINER_BOOTKIT_IMAGE}"
@@ -219,8 +238,14 @@ services:
       mkdir:
         - /var/lib/dhcpcd
 
-#dbg  - name: sshd
-#dbg    image: linuxkit/sshd:v1.0.0
+#SSH_SERVER  - name: sshd
+#SSH_SERVER    image: linuxkit/sshd:v1.0.0
+#SSH_SERVER    binds.add:
+#SSH_SERVER      - /etc/profile.d/local.sh:/etc/profile.d/local.sh
+#SSH_SERVER      - /root/.ssh/authorized_keys:/root/.ssh/authorized_keys
+#SSH_SERVER      - /usr/bin/nerdctl:/usr/bin/nerdctl
+#SSH_SERVER      - /:/host_root
+
 
 files:
   - path: etc/profile.d/local.sh
@@ -299,10 +324,10 @@ files:
       ttyUSB1
       ttyUSB2
 
-#dbg  - path: root/.ssh/authorized_keys
-#dbg    source: ~/.ssh/id_rsa.pub
-#dbg    mode: "0600"
-#dbg    optional: true
+#SSH_SERVER  - path: root/.ssh/authorized_keys
+#SSH_SERVER    source: ~/.ssh/id_rsa.pub
+#SSH_SERVER    mode: "0600"
+#SSH_SERVER    optional: true
 
 trust:
   org: