Virtualkubelet openstack-zun - unknown service runtime.v1alpha2.RuntimeService #16
Description
Hello everyone,
We have got a problem about our zun-capsule deployment.
Our infrastructure run on :
RedHat Enterprise Linux 9
Openstack Epoxy (2025.1) - deployed by Kolla-Ansible.
Docker version 29.1.3
containerd containerd.io v2.2.1
We use docker and containerd with Kata-containers (version 3.22.0).
When we create a container, it works :
openstack appcontainer list
+--------------------------------------+-------+----------------------------------------+---------+------------+--------------+-------+
| uuid | name | image | status | task_state | addresses | ports |
+--------------------------------------+-------+----------------------------------------+---------+------------+--------------+-------+
| 0a3c648a-16b7-4da8-b2c5-7ee8df809d11 | nginx | localhost:4000/nginx:latest | Created | None | 172.16.32.60 | [80] |
+--------------------------------------+-------+----------------------------------------+---------+------------+--------------+-------+
But when we create a capsule with my deployment :
cat capsule.yaml
capsuleVersion: beta
kind: capsule
metadata:
name: init-demo
spec:
containers:
- image: localhost:4000/nginx:latest
volumeMounts:
- name: workdir
mountPath: /usr/share/nginx/html
# These containers are run during capsule initialization
InitContainers:
- image: localhost:4000/busybox:latest
command:
- wget
- "-O"
- "/work-dir/index?html"
- https://www.openstack.org/
volumeMounts:
- name: workdir
mountPath: "/work-dir"
volumes:
- name: workdir
cinder:
size: 1
And we execute it :
zun capsule-create -f capsule.yaml
Request to create capsule has been accepted.
We have got this error in zun-compute.log :
2026-01-06 15:48:42.841 7 INFO zun.compute.claims [None req-fb629952-8733-4a72-b379-d025bd0d90f7 2fdb0efc642840118c2c721bff42fde7 d6aff92a161440ceb5370252a8ba84b1 default - - -] Attempting claim: memory 0, cpu 0.00 CPU, disk 0
2026-01-06 15:48:42.841 7 INFO zun.compute.claims [None req-fb629952-8733-4a72-b379-d025bd0d90f7 2fdb0efc642840118c2c721bff42fde7 d6aff92a161440ceb5370252a8ba84b1 default - - -] Total memory: 386343 MB, used: 0.00 MB
2026-01-06 15:48:42.842 7 INFO zun.compute.claims [None req-fb629952-8733-4a72-b379-d025bd0d90f7 2fdb0efc642840118c2c721bff42fde7 d6aff92a161440ceb5370252a8ba84b1 default - - -] memory limit not specified, defaulting to unlimited
2026-01-06 15:48:42.842 7 INFO zun.compute.claims [None req-fb629952-8733-4a72-b379-d025bd0d90f7 2fdb0efc642840118c2c721bff42fde7 d6aff92a161440ceb5370252a8ba84b1 default - - -] Total vcpu: 16 VCPU, used: 0.00 VCPU
2026-01-06 15:48:42.842 7 INFO zun.compute.claims [None req-fb629952-8733-4a72-b379-d025bd0d90f7 2fdb0efc642840118c2c721bff42fde7 d6aff92a161440ceb5370252a8ba84b1 default - - -] vcpu limit not specified, defaulting to unlimited
2026-01-06 15:48:42.843 7 INFO zun.compute.claims [None req-fb629952-8733-4a72-b379-d025bd0d90f7 2fdb0efc642840118c2c721bff42fde7 d6aff92a161440ceb5370252a8ba84b1 default - - -] Total disk: 40 GB, used: 0.00 GB
2026-01-06 15:48:42.843 7 INFO zun.compute.claims [None req-fb629952-8733-4a72-b379-d025bd0d90f7 2fdb0efc642840118c2c721bff42fde7 d6aff92a161440ceb5370252a8ba84b1 default - - -] disk limit not specified, defaulting to unlimited
2026-01-06 15:48:42.843 7 INFO zun.compute.claims [None req-fb629952-8733-4a72-b379-d025bd0d90f7 2fdb0efc642840118c2c721bff42fde7 d6aff92a161440ceb5370252a8ba84b1 default - - -] Claim successful
2026-01-06 15:48:44.728 7 ERROR zun.compute.manager [None req-22a8228e-0626-4057-9e52-65091e4c58a6 - - - - - -] Unexpected exception: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNIMPLEMENTED
details = "unknown service runtime.v1alpha2.RuntimeService"
debug_error_string = "UNKNOWN:Error received from peer {grpc_message:"unknown service runtime.v1alpha2.RuntimeService", grpc_status:12, created_time:"2026-01-06T15:48:44.728089896+01:00"}"
>: grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNIMPLEMENTED
details = "unknown service runtime.v1alpha2.RuntimeService"
debug_error_string = "UNKNOWN:Error received from peer {grpc_message:"unknown service runtime.v1alpha2.RuntimeService", grpc_status:12, created_time:"2026-01-06T15:48:44.728089896+01:00"}"
>
2026-01-06 15:48:44.728 7 ERROR zun.compute.manager Traceback (most recent call last):
2026-01-06 15:48:44.728 7 ERROR zun.compute.manager File "/var/lib/kolla/venv/lib64/python3.9/site-packages/zun/compute/manager.py", line 370, in _do_container_create_base
2026-01-06 15:48:44.728 7 ERROR zun.compute.manager container = self.capsule_driver.create_capsule(
2026-01-06 15:48:44.728 7 ERROR zun.compute.manager File "/var/lib/kolla/venv/lib64/python3.9/site-packages/zun/container/cri/driver.py", line 52, in create_capsule
2026-01-06 15:48:44.728 7 ERROR zun.compute.manager self._create_pod_sandbox(context, capsule, requested_networks)
2026-01-06 15:48:44.728 7 ERROR zun.compute.manager File "/var/lib/kolla/venv/lib64/python3.9/site-packages/zun/container/cri/driver.py", line 80, in _create_pod_sandbox
2026-01-06 15:48:44.728 7 ERROR zun.compute.manager sandbox_resp = self.runtime_stub.RunPodSandbox(
2026-01-06 15:48:44.728 7 ERROR zun.compute.manager File "/var/lib/kolla/venv/lib64/python3.9/site-packages/grpc/_channel.py", line 1181, in __call__
2026-01-06 15:48:44.728 7 ERROR zun.compute.manager return _end_unary_response_blocking(state, call, False, None)
2026-01-06 15:48:44.728 7 ERROR zun.compute.manager File "/var/lib/kolla/venv/lib64/python3.9/site-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking
2026-01-06 15:48:44.728 7 ERROR zun.compute.manager raise _InactiveRpcError(state) # pytype: disable=not-instantiable
2026-01-06 15:48:44.728 7 ERROR zun.compute.manager grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
2026-01-06 15:48:44.728 7 ERROR zun.compute.manager status = StatusCode.UNIMPLEMENTED
2026-01-06 15:48:44.728 7 ERROR zun.compute.manager details = "unknown service runtime.v1alpha2.RuntimeService"
2026-01-06 15:48:44.728 7 ERROR zun.compute.manager debug_error_string = "UNKNOWN:Error received from peer {grpc_message:"unknown service runtime.v1alpha2.RuntimeService", grpc_status:12, created_time:"2026-01-06T15:48:44.728089896+01:00"}"
2026-01-06 15:48:44.728 7 ERROR zun.compute.manager >
2026-01-06 15:48:44.728 7 ERROR zun.compute.manager
Same when I use virtual-kubelet :
openstack-zun/bin/virtual-kubelet --provider openstack --provider-config /home/myuser/clouds.yaml
ERRO[0000] TLS certificates not provided, not setting up pod http server certPath= keyPath= node=virtual-kubelet operatingSystem=Linux provider=openstack watchedNamespace=
INFO[0000] Initialized node=virtual-kubelet operatingSystem=Linux provider=openstack watchedNamespace=
INFO[0000] Pod cache in-sync node=virtual-kubelet operatingSystem=Linux provider=openstack watchedNamespace=
INFO[0001] starting workers node=virtual-kubelet operatingSystem=Linux provider=openstack watchedNamespace=
INFO[0001] started workers node=virtual-kubelet operatingSystem=Linux provider=openstack watchedNamespace=
INFO[0209] Created pod in provider key=default/nginx-deployment-7d64497959-24mms method=createOrUpdatePod name=nginx-deployment-7d64497959-24mms namespace=default node=virtual-kubelet operatingSystem=Linux phase=Pending pod=nginx-deployment-7d64497959-24mms provider=openstack reason= uid=0fa946a2-c08b-45c6-bc57-d1d984eb3c0c watchedNamespace= workerId=4
INFO[0211] Updated pod in provider key=default/nginx-deployment-7d64497959-24mms method=createOrUpdatePod name=nginx-deployment-7d64497959-24mms namespace=default node=virtual-kubelet operatingSystem=Linux phase=Pending pod=nginx-deployment-7d64497959-24mms provider=openstack reason= uid=0fa946a2-c08b-45c6-bc57-d1d984eb3c0c watchedNamespace= workerId=7
WARN[0216] skipping sync of pod "default/nginx-deployment-7d64497959-24mms" in "Failed" phase key=default/nginx-deployment-7d64497959-24mms method=syncPodInProvider name=nginx-deployment-7d64497959-24mms namespace=default node=virtual-kubelet operatingSystem=Linux phase=Failed provider=openstack reason= uid=0fa946a2-c08b-45c6-bc57-d1d984eb3c0c watchedNamespace= workerId=5
INFO[0217] Created pod in provider key=default/nginx-deployment-7d64497959-dvxn8 method=createOrUpdatePod name=nginx-deployment-7d64497959-dvxn8 namespace=default node=virtual-kubelet operatingSystem=Linux phase=Pending pod=nginx-deployment-7d64497959-dvxn8 provider=openstack reason= uid=26b7faa2-d954-4d99-94d6-c3de25fd195d watchedNamespace= workerId=8
WARN[0221] skipping sync of pod "default/nginx-deployment-7d64497959-dvxn8" in "Failed" phase key=default/nginx-deployment-7d64497959-dvxn8 method=syncPodInProvider name=nginx-deployment-7d64497959-dvxn8 namespace=default node=virtual-kubelet operatingSystem=Linux phase=Failed provider=openstack reason= uid=26b7faa2-d954-4d99-94d6-c3de25fd195d watchedNamespace= workerId=0
My deployment for virtual-kubelet :
cat deployment.yaml
apiVersion: apps/v1 # for version before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
replicas: 1 # tells deployment to run N port matching the template
template:
metadata:
labels:
app: nginx
spec:
tolerations:
- key: "virtual-kubelet.io/provider"
operator: "Equal"
value: "openstack"
effect: "NoSchedule"
containers:
- name: nginx
image: nginx:latest
ports:
- containerPort: 80
We are pretty sure the root cause is about containerd, but we didn't know how to resolve that.
Here our configure.toml of containerd :
cat /etc/containerd/config.toml
version = 3
root = '/var/lib/containerd'
state = '/run/containerd'
temp = ''
disabled_plugins = []
oom_score = 0
imports = ['/etc/containerd/conf.d/*.toml']
[grpc]
address = '/run/containerd/containerd.sock'
tcp_address = ''
tcp_tls_ca = ''
tcp_tls_cert = ''
tcp_tls_key = ''
uid = 0
gid = 42463
max_recv_message_size = 16777216
max_send_message_size = 16777216
tcp_tls_common_name = ''
[ttrpc]
address = ''
uid = 0
gid = 0
[debug]
address = ''
uid = 0
gid = 0
level = 'debug'
format = ''
[metrics]
address = ''
grpc_histogram = false
[plugins]
[plugins.'io.containerd.cri.v1.images']
snapshotter = 'native'
disable_snapshot_annotations = true
discard_unpacked_layers = false
max_concurrent_downloads = 3
concurrent_layer_fetch_buffer = 0
image_pull_progress_timeout = '5m0s'
image_pull_with_sync_fs = false
stats_collect_period = 10
use_local_image_pull = false
[plugins.'io.containerd.cri.v1.images'.pinned_images]
sandbox = 'localhost:4000/k8s/pause:3.10.1'
[plugins.'io.containerd.cri.v1.images'.registry]
config_path = '/etc/containerd/certs.d:/etc/docker/certs.d'
[plugins.'io.containerd.cri.v1.images'.image_decryption]
key_model = 'node'
[plugins.'io.containerd.cri.v1.runtime']
enable_selinux = false
selinux_category_range = 1024
max_container_log_line_size = 16384
disable_apparmor = false
restrict_oom_score_adj = false
disable_proc_mount = false
unset_seccomp_profile = ''
tolerate_missing_hugetlb_controller = true
disable_hugetlb_controller = true
device_ownership_from_security_context = false
ignore_image_defined_volumes = false
netns_mounts_under_state_dir = false
enable_unprivileged_ports = true
enable_unprivileged_icmp = true
enable_cdi = true
cdi_spec_dirs = ['/etc/cdi', '/var/run/cdi']
drain_exec_sync_io_timeout = '0s'
ignore_deprecation_warnings = []
[plugins.'io.containerd.cri.v1.runtime'.containerd]
default_runtime_name = 'runc'
ignore_blockio_not_enabled_errors = false
ignore_rdt_not_enabled_errors = false
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes]
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
runtime_type = 'io.containerd.runc.v2'
runtime_path = ''
pod_annotations = []
container_annotations = []
privileged_without_host_devices = false
privileged_without_host_devices_all_devices_allowed = false
cgroup_writable = false
base_runtime_spec = ''
cni_conf_dir = ''
cni_max_conf_num = 0
snapshotter = ''
sandboxer = 'podsandbox'
io_type = ''
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
BinaryName = ''
CriuImagePath = ''
CriuWorkPath = ''
IoGid = 0
IoUid = 0
NoNewKeyring = false
Root = ''
ShimCgroup = ''
SystemdCgroup = true
[plugins.'io.containerd.cri.v1.runtime'.cni]
bin_dir = ''
bin_dirs = ['/opt/cni/bin']
conf_dir = '/etc/cni/net.d'
max_conf_num = 2
setup_serially = false
conf_template = ''
ip_pref = ''
use_internal_loopback = false
[plugins.'io.containerd.differ.v1.erofs']
mkfs_options = []
enable_tar_index = false
[plugins.'io.containerd.gc.v1.scheduler']
pause_threshold = 0.02
deletion_threshold = 0
mutation_threshold = 100
schedule_delay = '0s'
startup_delay = '100ms'
[plugins.'io.containerd.grpc.v1.cri']
disable_tcp_service = true
stream_server_address = '127.0.0.1'
stream_server_port = '0'
stream_idle_timeout = '4h0m0s'
enable_tls_streaming = false
[plugins.'io.containerd.grpc.v1.cri'.x509_key_pair_streaming]
tls_cert_file = ''
tls_key_file = ''
[plugins.'io.containerd.image-verifier.v1.bindir']
bin_dir = '/opt/containerd/image-verifier/bin'
max_verifiers = 10
per_verifier_timeout = '10s'
[plugins.'io.containerd.internal.v1.opt']
path = '/opt/containerd'
[plugins.'io.containerd.internal.v1.tracing']
[plugins.'io.containerd.metadata.v1.bolt']
content_sharing_policy = 'shared'
no_sync = false
[plugins.'io.containerd.monitor.container.v1.restart']
interval = '10s'
[plugins.'io.containerd.monitor.task.v1.cgroups']
no_prometheus = false
[plugins.'io.containerd.mount-handler.v1.erofs']
[plugins.'io.containerd.nri.v1.nri']
disable = false
socket_path = '/var/run/nri/nri.sock'
plugin_path = '/opt/nri/plugins'
plugin_config_path = '/etc/nri/conf.d'
plugin_registration_timeout = '5s'
plugin_request_timeout = '2s'
disable_connections = false
[plugins.'io.containerd.nri.v1.nri'.default_validator]
enable = false
reject_oci_hook_adjustment = false
reject_runtime_default_seccomp_adjustment = false
reject_unconfined_seccomp_adjustment = false
reject_custom_seccomp_adjustment = false
reject_namespace_adjustment = false
reject_sysctl_adjustment = false
required_plugins = []
tolerate_missing_plugins_annotation = ''
[plugins.'io.containerd.runtime.v2.task']
platforms = ['linux/amd64']
[plugins.'io.containerd.service.v1.diff-service']
default = ['walking']
sync_fs = false
[plugins.'io.containerd.service.v1.tasks-service']
blockio_config_file = ''
rdt_config_file = ''
[plugins.'io.containerd.shim.v1.manager']
env = []
[plugins.'io.containerd.snapshotter.v1.blockfile']
root_path = ''
scratch_file = ''
fs_type = ''
mount_options = []
recreate_scratch = false
[plugins.'io.containerd.snapshotter.v1.btrfs']
root_path = ''
[plugins.'io.containerd.snapshotter.v1.devmapper']
root_path = ''
pool_name = ''
base_image_size = ''
async_remove = false
discard_blocks = false
fs_type = ''
fs_options = ''
[plugins.'io.containerd.snapshotter.v1.erofs']
root_path = ''
ovl_mount_options = []
enable_fsverity = false
set_immutable = false
default_size = ''
[plugins.'io.containerd.snapshotter.v1.native']
root_path = ''
[plugins.'io.containerd.snapshotter.v1.overlayfs']
root_path = ''
upperdir_label = false
sync_remove = false
slow_chown = false
mount_options = []
[plugins.'io.containerd.snapshotter.v1.zfs']
root_path = ''
[plugins.'io.containerd.tracing.processor.v1.otlp']
[plugins.'io.containerd.transfer.v1.local']
max_concurrent_downloads = 3
concurrent_layer_fetch_buffer = 0
max_concurrent_uploaded_layers = 3
check_platform_supported = false
config_path = ''
max_concurrent_unpacks = 1
[cgroup]
path = ''
[timeouts]
'io.containerd.timeout.bolt.open' = '0s'
'io.containerd.timeout.cri.defercleanup' = '1m0s'
'io.containerd.timeout.metrics.shimstats' = '2s'
'io.containerd.timeout.shim.cleanup' = '5s'
'io.containerd.timeout.shim.load' = '5s'
'io.containerd.timeout.shim.shutdown' = '3s'
'io.containerd.timeout.task.state' = '2s'
[stream_processors]
[stream_processors.'io.containerd.ocicrypt.decoder.v1.tar']
accepts = ['application/vnd.oci.image.layer.v1.tar+encrypted']
returns = 'application/vnd.oci.image.layer.v1.tar'
path = 'ctd-decoder'
args = ['--decryption-keys-path', '/etc/containerd/ocicrypt/keys']
env = ['OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf']
[stream_processors.'io.containerd.ocicrypt.decoder.v1.tar.gzip']
accepts = ['application/vnd.oci.image.layer.v1.tar+gzip+encrypted']
returns = 'application/vnd.oci.image.layer.v1.tar+gzip'
path = 'ctd-decoder'
args = ['--decryption-keys-path', '/etc/containerd/ocicrypt/keys']
env = ['OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf']
Thank you very much for all your help.
Respectfully.