Nerdctl in EKS pod not working

[pabardz]

Hi!

I want to use nerdctl in an EKS pod to build images and run containers. So far I can interact with containerd, I can nerdctl --namespace=k8s.io ps -a, pull an image but that's it. I can't run a container nor build an image.

EKS version: 1.28

root@test-nerdctl:/# nerdctl version
Client:
 Version:	v2.0.0-beta.5
 OS/Arch:	linux/amd64
 Git commit:	9236b9370203b7f3274ae8d1417cc6ddcbdd4511
 buildctl:
  Version:	v0.13.2
  GitCommit:	2e18d709fefdcc2db20853ee241c75b058189d39

Server:
 containerd:
  Version:	1.7.11
  GitCommit:	64b8a811b07ba6288238eefc14d898ee0b5b99ba
 runc:
  Version:	1.1.12
  GitCommit:	v1.1.12-0-g51d5e946

root@test-nerdctl:/# nerdctl info
Client:
 Namespace:	default
 Debug Mode:	false

Server:
 Server Version: 1.7.11
 Storage Driver: overlayfs
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Log: fluentd journald json-file syslog
  Storage: native overlayfs
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 5.10.217-205.860.amzn2.x86_64
 Operating System: Ubuntu 22.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.816GiB
 Name: test-nerdctl

How the nerdctl pod is configured

apiVersion: v1
kind: Pod
metadata:
  name: test-nerdctl
spec:
  volumes:
   - name: containerdsock
     hostPath:
       path: /run/containerd/containerd.sock
       type: Socket
   - name: containerddir
     hostPath:
       path: /run/containerd
       type: Directory
   - name: containerdlib
     hostPath:
       path: /var/lib/containerd
       type: Directory
   - name: tmp
     hostPath:
       path: /tmp
       type: Directory
  containers:
  - image: ghcr.io/containerd/nerdctl:latest
    name: test-nerdctl
    resources: {}
    command: ["sleep","infinity"]
    securityContext:
      privileged: true
    volumeMounts:
      - mountPath: /run/containerd/containerd.sock
        name: containerdsock
      - mountPath: /run/containerd
        name: containerddir
      - mountPath: /var/lib/containerd
        name: containerdlib
        readOnly: false
      - mountPath: /tmp
        name: tmp
        readOnly: false
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

Build an image

root@test-nerdctl:/# nohup buildkitd &
[1] 198

>Dockerfile cat <<EOF
FROM library/alpine:3.16.2

RUN echo hello > /tmp/hello.txt
EOF

nerdctl build -t mytestimage .
[+] Building 0.4s (5/5) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                  0.0s
 => => transferring dockerfile: 97B                                                                                                                                                   0.0s
 => [internal] load metadata for docker.io/library/alpine:3.16.2                                                                                                                      0.3s
 => [internal] load .dockerignore                                                                                                                                                     0.0s
 => => transferring context: 2B                                                                                                                                                       0.0s
 => CACHED [1/2] FROM docker.io/library/alpine:3.16.2@sha256:65a2763f593ae85fab3b5406dc9e80f744ec5b449f269b699b5efd37a07ad32e                                                         0.0s
 => => resolve docker.io/library/alpine:3.16.2@sha256:65a2763f593ae85fab3b5406dc9e80f744ec5b449f269b699b5efd37a07ad32e                                                                0.0s
 => ERROR [2/2] RUN echo hello > /tmp/hello.txt                                                                                                                                       0.1s
------
 > [2/2] RUN echo hello > /tmp/hello.txt:
------
Dockerfile:3
--------------------
   1 |     FROM library/alpine:3.16.2
   2 |
   3 | >>> RUN echo hello > /tmp/hello.txt
   4 |
--------------------
error: failed to solve: process "/bin/sh -c echo hello > /tmp/hello.txt" did not complete successfully: failed to create shim task: failed to mount rootfs component: no such file or directory: unknown

Run a pod

root@test-nerdctl:/# nerdctl run -it ubuntu sh
FATA[0000] failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/nerdctl/1935db59/containers/default/29854b8d548f1c761a84287a000d1cba75f24132a137f828ef8fe722bec28804/resolv.conf" to rootfs at "/etc/resolv.conf": stat /var/lib/nerdctl/1935db59/containers/default/29854b8d548f1c761a84287a000d1cba75f24132a137f828ef8fe722bec28804/resolv.conf: no such file or directory: unknown

I have no clue what is going one, I read a lot of discussions about those issues, it seems I have something not working with volume?

Thanks for the help!

[AkihiroSuda]

You can try just jailbreaking to the host with hostPID: true, and run nerdctl there: https://github.com/justincormack/nsenter1

No need to specify volumeMounts then.

[pabardz]

Hey! thanks for the answer.

Sadly, I tried that but I got:

sh-4.2# nerdctl run alpine sh
FATA[0000] failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: time="2024-06-21T18:50:37Z" level=fatal msg="failed to call cni.Setup: plugin type=\"portmap\" failed (add): incompatible CNI versions; config is \"1.0.0\", plugin supports [\"0.1.0\" \"0.2.0\" \"0.3.0\" \"0.3.1\" \"0.4.0\"]"
Failed to write to log, write /var/lib/nerdctl/1935db59/containers/k8s.io/05d6ce634b1af311889123eb4c60767663fc50efc4bc7f63675f1a661df13ccf/oci-hook.createRuntime.log: file already closed: unknown

apiVersion: v1
kind: Pod
metadata:
  name: test-containerd
spec:
  hostPID: true
  containers:
  - image: ghcr.io/containerd/nerdctl:latest
    name: test-containerd
    resources: {}
    command: ["sleep", "infinity"]
    securityContext:
      privileged: true

In the pod:

root@test-containerd:/# nsenter -t 1 -m -u -n -i sh
sh-4.2# nerdctl run alpine sh
FATA[0000] failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: time="2024-06-21T18:53:09Z" level=fatal msg="failed to call cni.Setup: plugin type=\"portmap\" failed (add): incompatible CNI versions; config is \"1.0.0\", plugin supports [\"0.1.0\" \"0.2.0\" \"0.3.0\" \"0.3.1\" \"0.4.0\"]"
Failed to write to log, write /var/lib/nerdctl/1935db59/containers/k8s.io/cb4688e3cd9ca4d07895e3bb0985645b2d6468d819601f100b20518b1ae2a631/oci-hook.createRuntime.log: file already closed: unknown

Did I miss something?

[AkihiroSuda]

Please try installing the latest CNI plugins