containerd 2.0.0 does not load the containerd-shim-spin plugin 0.15.1

[rgl]

containerd 2.0.0 does not load the containerd-shim-spin plugin 0.15.1 using the instructions at https://github.com/spinkube/containerd-shim-spin?tab=readme-ov-file#installing-the-containerd-shim-spin-on-kubernetes-nodes.

maybe this plugin needs to support the new containerd version?

not sure if this matters, but dumping the containerd config emits these warnings:

# containerd config dump >config
WARN[0000] Ignoring unknown key in TOML for plugin       error="strict mode: fields in the document are missing in the target struct" key=containerd plugin=io.containerd.grpc.v1.cri
WARN[0000] Ignoring unknown key in TOML for plugin       error="strict mode: fields in the document are missing in the target struct" key="containerd runtimes" plugin=io.containerd.grpc.v1.cri
WARN[0000] Ignoring unknown key in TOML for plugin       error="strict mode: fields in the document are missing in the target struct" key="containerd runtimes spin" plugin=io.containerd.grpc.v1.cri

[kate-goldenring]

does not load the containerd-shim-spin plugin 0.15.1

@rgl what behavior are you seeing and which k8s distro are you using (unless you are running directly with ctr)? Can you share the subset of your containerd config with the spin shim configured (cat /path/to/containerd/config.toml | grep -C 5 spin)?

[rgl]

I'm using raw containerd, with the same configuration as containerd 1.7.23 (which works).

Here's the requested config.toml snippet:

$ cat /etc/containerd/config.toml | grep -C 5 spin
    returns = 'application/vnd.oci.image.layer.v1.tar+gzip'
    path = 'ctd-decoder'
    args = ['--decryption-keys-path', '/etc/containerd/ocicrypt/keys']
    env = ['OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf']

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.spin]
  runtime_type = "io.containerd.spin.v2"

The full config is at the bottom of this comment.

But, for some reason (the only clue I have are the containerd config dump warnings mentioned on the previous comment), containerd 2.0.0 does not load the containerd-shim-spin plugin 0.15.1 (also happens with 0.17.0). which means that when I try to start a wasm container as:

$ cat cri-spin-http-go-example.pod.yml
metadata:
  uid: cri-spin-http-go-example
  name: cri-spin-http-go-example
  namespace: default
log_directory: /var/log/cri/cri-spin-http-go-example

$ cat cri-spin-http-go-example.web.ctr.yml
metadata:
  name: web
image:
  image: ghcr.io/rgl/spin-http-go-example:0.3.1
command:
  - /
log_path: web.log

$ crictl runp --runtime spin cri-spin-http-go-example.pod.yml

it fails with:

    linux: ++ crictl runp --runtime spin cri-spin-http-go-example.pod.yml
    linux: E1109 14:59:41.416057    5616 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = unable to get OCI runtime for sandbox \"ff46473b4ff16bef72420d50427701a7c24d54fd1c7a7ca238b02b44e978e19e\": no runtime for \"spin\" is configured"
    linux: time="2024-11-09T14:59:41Z" level=fatal msg="run pod sandbox: rpc error: code = Unknown desc = unable to get OCI runtime for sandbox \"ff46473b4ff16bef72420d50427701a7c24d54fd1c7a7ca238b02b44e978e19e\": no runtime for \"spin\" is configured"

Full containerd 2.0.0 config.toml

version = 3
root = '/var/lib/containerd'
state = '/run/containerd'
temp = ''
plugin_dir = ''
disabled_plugins = []
required_plugins = []
oom_score = 0
imports = []

[grpc]
  address = '/run/containerd/containerd.sock'
  tcp_address = ''
  tcp_tls_ca = ''
  tcp_tls_cert = ''
  tcp_tls_key = ''
  uid = 0
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216

[ttrpc]
  address = ''
  uid = 0
  gid = 0

[debug]
  address = ''
  uid = 0
  gid = 0
  level = ''
  format = ''

[metrics]
  address = ''
  grpc_histogram = false

[plugins]
  [plugins.'io.containerd.cri.v1.images']
    snapshotter = 'overlayfs'
    disable_snapshot_annotations = true
    discard_unpacked_layers = false
    max_concurrent_downloads = 3
    image_pull_progress_timeout = '5m0s'
    image_pull_with_sync_fs = false
    stats_collect_period = 10

    [plugins.'io.containerd.cri.v1.images'.pinned_images]
      sandbox = 'registry.k8s.io/pause:3.10'

    [plugins.'io.containerd.cri.v1.images'.registry]
      config_path = ''

    [plugins.'io.containerd.cri.v1.images'.image_decryption]
      key_model = 'node'

  [plugins.'io.containerd.cri.v1.runtime']
    enable_selinux = false
    selinux_category_range = 1024
    max_container_log_line_size = 16384
    disable_apparmor = false
    restrict_oom_score_adj = false
    disable_proc_mount = false
    unset_seccomp_profile = ''
    tolerate_missing_hugetlb_controller = true
    disable_hugetlb_controller = true
    device_ownership_from_security_context = false
    ignore_image_defined_volumes = false
    netns_mounts_under_state_dir = false
    enable_unprivileged_ports = true
    enable_unprivileged_icmp = true
    enable_cdi = true
    cdi_spec_dirs = ['/etc/cdi', '/var/run/cdi']
    drain_exec_sync_io_timeout = '0s'
    ignore_deprecation_warnings = []

    [plugins.'io.containerd.cri.v1.runtime'.containerd]
      default_runtime_name = 'runc'
      ignore_blockio_not_enabled_errors = false
      ignore_rdt_not_enabled_errors = false

      [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes]
        [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
          runtime_type = 'io.containerd.runc.v2'
          runtime_path = ''
          pod_annotations = []
          container_annotations = []
          privileged_without_host_devices = false
          privileged_without_host_devices_all_devices_allowed = false
          base_runtime_spec = ''
          cni_conf_dir = ''
          cni_max_conf_num = 0
          snapshotter = ''
          sandboxer = 'podsandbox'
          io_type = ''

          [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
            BinaryName = ''
            CriuImagePath = ''
            CriuWorkPath = ''
            IoGid = 0
            IoUid = 0
            NoNewKeyring = false
            Root = ''
            ShimCgroup = ''

    [plugins.'io.containerd.cri.v1.runtime'.cni]
      bin_dir = '/opt/cni/bin'
      conf_dir = '/etc/cni/net.d'
      max_conf_num = 1
      setup_serially = false
      conf_template = ''
      ip_pref = ''
      use_internal_loopback = false

  [plugins.'io.containerd.gc.v1.scheduler']
    pause_threshold = 0.02
    deletion_threshold = 0
    mutation_threshold = 100
    schedule_delay = '0s'
    startup_delay = '100ms'

  [plugins.'io.containerd.grpc.v1.cri']
    disable_tcp_service = true
    stream_server_address = '127.0.0.1'
    stream_server_port = '0'
    stream_idle_timeout = '4h0m0s'
    enable_tls_streaming = false

    [plugins.'io.containerd.grpc.v1.cri'.x509_key_pair_streaming]
      tls_cert_file = ''
      tls_key_file = ''

  [plugins.'io.containerd.image-verifier.v1.bindir']
    bin_dir = '/opt/containerd/image-verifier/bin'
    max_verifiers = 10
    per_verifier_timeout = '10s'

  [plugins.'io.containerd.internal.v1.opt']
    path = '/opt/containerd'

  [plugins.'io.containerd.internal.v1.tracing']

  [plugins.'io.containerd.metadata.v1.bolt']
    content_sharing_policy = 'shared'

  [plugins.'io.containerd.monitor.container.v1.restart']
    interval = '10s'

  [plugins.'io.containerd.monitor.task.v1.cgroups']
    no_prometheus = false

  [plugins.'io.containerd.nri.v1.nri']
    disable = false
    socket_path = '/var/run/nri/nri.sock'
    plugin_path = '/opt/nri/plugins'
    plugin_config_path = '/etc/nri/conf.d'
    plugin_registration_timeout = '5s'
    plugin_request_timeout = '2s'
    disable_connections = false

  [plugins.'io.containerd.runtime.v2.task']
    platforms = ['linux/amd64']

  [plugins.'io.containerd.service.v1.diff-service']
    default = ['walking']
    sync_fs = false

  [plugins.'io.containerd.service.v1.tasks-service']
    blockio_config_file = ''
    rdt_config_file = ''

  [plugins.'io.containerd.shim.v1.manager']
    env = []

  [plugins.'io.containerd.snapshotter.v1.blockfile']
    root_path = ''
    scratch_file = ''
    fs_type = ''
    mount_options = []
    recreate_scratch = false

  [plugins.'io.containerd.snapshotter.v1.btrfs']
    root_path = ''

  [plugins.'io.containerd.snapshotter.v1.devmapper']
    root_path = ''
    pool_name = ''
    base_image_size = ''
    async_remove = false
    discard_blocks = false
    fs_type = ''
    fs_options = ''

  [plugins.'io.containerd.snapshotter.v1.native']
    root_path = ''

  [plugins.'io.containerd.snapshotter.v1.overlayfs']
    root_path = ''
    upperdir_label = false
    sync_remove = false
    slow_chown = false
    mount_options = []

  [plugins.'io.containerd.snapshotter.v1.zfs']
    root_path = ''

  [plugins.'io.containerd.tracing.processor.v1.otlp']

  [plugins.'io.containerd.transfer.v1.local']
    max_concurrent_downloads = 3
    max_concurrent_uploaded_layers = 3
    config_path = ''

[cgroup]
  path = ''

[timeouts]
  'io.containerd.timeout.bolt.open' = '0s'
  'io.containerd.timeout.metrics.shimstats' = '2s'
  'io.containerd.timeout.shim.cleanup' = '5s'
  'io.containerd.timeout.shim.load' = '5s'
  'io.containerd.timeout.shim.shutdown' = '3s'
  'io.containerd.timeout.task.state' = '2s'

[stream_processors]
  [stream_processors.'io.containerd.ocicrypt.decoder.v1.tar']
    accepts = ['application/vnd.oci.image.layer.v1.tar+encrypted']
    returns = 'application/vnd.oci.image.layer.v1.tar'
    path = 'ctd-decoder'
    args = ['--decryption-keys-path', '/etc/containerd/ocicrypt/keys']
    env = ['OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf']

  [stream_processors.'io.containerd.ocicrypt.decoder.v1.tar.gzip']
    accepts = ['application/vnd.oci.image.layer.v1.tar+gzip+encrypted']
    returns = 'application/vnd.oci.image.layer.v1.tar+gzip'
    path = 'ctd-decoder'
    args = ['--decryption-keys-path', '/etc/containerd/ocicrypt/keys']
    env = ['OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf']

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.spin]
  runtime_type = "io.containerd.spin.v2"

[kate-goldenring]

The no runtime for \"spin\" is configured error often occurs if the shim is not at the expected path. runtime_type = "io.containerd.spin.v2" tells containerd that the shim should be on PATH. Alternatively, you can specify the precise path with runtime_type = "/path/to/containerd-shim-spin-v2. Could you check that the binary is in the expected location, moving it to usr/bin is one option.

Another possible test is to see if just using ctr to run it works:

sudo ctr image pull $image
sudo ctr run --rm --net-host --runtime io.containerd.spin.v2 $image $name bogus-arg

[rgl]

I've found the problem (and included the change in my example). containerd 2 changed the way a runtime is registered in its configuration file, which now defaults to config version 3.

with containerd 2, we have to use:

version = 3
[plugins."io.containerd.cri.v1.runtime".containerd.runtimes.spin]
  runtime_type = "io.containerd.spin.v2"

with containerd 1, we have to use:

version = 2
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.spin]
  runtime_type = "io.containerd.spin.v2"

I think the documentation at https://github.com/spinkube/containerd-shim-spin?tab=readme-ov-file#installing-the-containerd-shim-spin-on-kubernetes-nodes should be updated to distinguish containerd 1 vs 2.

[kate-goldenring]

@rgl thank you for taking the time to find the solution. I agree documentation should be updated. If you're interested in putting up the PR, happy to review it. Otherwise, i can look at updating it soon

[shirok1]

Hello, I tried using k3s v1.32.2+k3s1 and kwasmOperator.installerImage=ghcr.io/spinkube/containerd-shim-spin/node-installer:v0.18.0, but what written to /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl was still the old format (containerd 2 wasn't detected). Should node-installer/script/installer.sh reflect changes here?

[shirok1]

One more strange thing is that although k3s can actually detect /opt/kwasm/bin and generate following if config.toml.tmpl is not existing:

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.'spin']
  runtime_type = "io.containerd.spin.v2"

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.'spin'.options]
  BinaryName = "/opt/kwasm/bin/containerd-shim-spin-v2"
  SystemdCgroup = true

But containerd won't recognize the BinaryName and say failed to create containerd task: failed to start shim: failed to resolve runtime path: runtime \"io.containerd.spin.v2\" binary not installed \"containerd-shim-spin-v2\": file does not exist". My workaround was ln -s /opt/kwasm/bin/containerd-shim-spin-v2 /usr/local/bin/containerd-shim-spin-v2. How about do this in the installer.sh?

[kate-goldenring]

My workaround was ln -s /opt/kwasm/bin/containerd-shim-spin-v2 /usr/local/bin/containerd-shim-spin-v2. How about do this in the installer.sh?

@shirok1 that seems like a reasonable approach to me. Would you be interested in putting in the change?

[shirok1]

@shirok1 that seems like a reasonable approach to me. Would you be interested in putting in the change?

My opinion is that the script shouldn't install binary to /opt/kwasm/bin but /usr/local/bin, since Kwasm's upstream install script seems to be not actively maintained.

Plus the .tmpl shouldn't be created for k3s (since it will be detected once it is in $PATH or /opt/kwasm/bin). We might need to check the version of k3s including this detection and see if it fits in the "support policy range".

https://github.com/k3s-io/k3s/blob/7fed27f62efa0a2cdb8c3962e51a43e2204e4018/pkg/agent/containerd/runtimes.go#L83-L96

Also a detection for version = 3 could be added for rest of the distros for the new format in containerd 2.

[Mossaka]

@shirok1 , what you proposed sounds great to me. We always welcome new contributions, would you be interested in taking this task and a Pull Request?

[shirok1]

#301