k3s containerd configuration faulty if shim exists in filesystem outside of kwasm dir

[kate-goldenring]

K3s recently added support for configuring containerd to work with shims. Ironically, this makes it harder to use kwasm.

Here k3s now checks if the shim exists in the filesystem. If it find it, it then assumes it is on the users path, updating the containerd.toml.tmpl config to set runtime_type = "io.containerd.spin.v2". This seems to wire containerd to look for the shim on path even if it tries to qualify where with the options section next:

# /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."spin"]
  runtime_type = "io.containerd.spin.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."spin".options]
  BinaryName = "/opt/kwasm/bin/containerd-shim-spin-v2"
  SystemdCgroup = true

The node installer then sees that the config template was updated and opts not to update it with the path to the binary at /opt/kwasm. The runtime_type should be specified as:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.spin]
    runtime_type = "/opt/kwasm/bin/containerd-shim-spin-v2"

When you go to run your app, you get the error:

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: failed to start shim: failed to resolve runtime path: runtime "io.containerd.spin.v2" binary not installed "containerd-shim-spin-v2": file does not exist: unknown

Workaround

1. Option 1: Update the /var/lib/rancher/k3s/agent/etc/containerd/config.toml file to point to the shim under the kwasm directory:

# /var/lib/rancher/k3s/agent/etc/containerd/config.toml
....

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.spin]
    runtime_type = "/opt/kwasm/bin/containerd-shim-spin-v2"

2. Add the /opt/kwasm directory to path

[asteurer]

Tried running SpinKube on K3S this evening, and got a slightly different error when deploying a spinapp. Unsure if it's related:

Failed to create pod sandbox: rpc error: code = Unknown desc = unable to get OCI runtime for sandbox "...": no runtime for "spin" is configured

I updated the /var/lib/rancher/k3s/agent/etc/containerd/config.toml file and added /opt/kwasm to the path per the workaround suggested above and restarted the nodes, but am not seeing any changes to the error message.

If there are additional workarounds I can try, I would be grateful for suggestions