Skip to content

Add Containerd 2.2 to k8s 1.35 variants#4801

Draft
ytsssun wants to merge 2 commits intobottlerocket-os:developfrom
ytsssun:containerd-22/bottlerocket
Draft

Add Containerd 2.2 to k8s 1.35 variants#4801
ytsssun wants to merge 2 commits intobottlerocket-os:developfrom
ytsssun:containerd-22/bottlerocket

Conversation

@ytsssun
Copy link
Copy Markdown
Contributor

@ytsssun ytsssun commented Apr 2, 2026

NOTE: Pending core-kit release before we can mark this PR as ready

Issue number:

Related bottlerocket-os/bottlerocket-core-kit#806

Description of changes:

Introduce containerd 2.2 for all variants currently on containerd 2.1 (27 total). Also adds the container-runtime.max-concurrent-unpacks settings migration and wires up the unsupported-setting warning service so that variants on older containerd versions (1.7) log a journal warning when the setting is configured.

Variants bumped to containerd 2.2:

AWS K8s:

  • aws-k8s-1.30-nvidia-fips
  • aws-k8s-1.31-nvidia-fips
  • aws-k8s-1.32-nvidia-fips
  • aws-k8s-1.33, aws-k8s-1.33-fips, aws-k8s-1.33-nvidia, aws-k8s-1.33-nvidia-fips
  • aws-k8s-1.34, aws-k8s-1.34-fips, aws-k8s-1.34-nvidia, aws-k8s-1.34-nvidia-fips
  • aws-k8s-1.35, aws-k8s-1.35-fips, aws-k8s-1.35-nvidia, aws-k8s-1.35-nvidia-fips

AWS ECS:

  • aws-ecs-3, aws-ecs-3-fips, aws-ecs-3-nvidia, aws-ecs-3-nvidia-fips

VMware K8s:

  • vmware-k8s-1.33, vmware-k8s-1.33-fips
  • vmware-k8s-1.34, vmware-k8s-1.34-fips
  • vmware-k8s-1.35, vmware-k8s-1.35-fips

Dev:

  • aws-dev, vmware-dev

The setting is opt-in only — no default value is applied. Users who want to tune concurrent unpacks can explicitly set it via apiclient set container-runtime.max-concurrent-unpacks=<value>.

Depends on:

Testing done:

Conformance tested containerd 2.2 across 20 variant/arch combinations — all passing.

Test Containerd 2.2 variant Containerd 2.1 variant
Setting rendered in config max_concurrent_unpacks = 5 Not rendered ✅
Env file Does not exist ✅ UNSUPPORTED_SETTING=... written ✅
Warning timer Inactive (no env file) ✅ Active, fires on boot + every 6h ✅
Journal warning None ✅ "not supported by the containerd version" ✅

Migration Testing (v1.59.0 → v1.60.0)

Built a custom TUF repo and tested upgrade/downgrade on aws-k8s-1.35 (x86_64) on an EKS 1.35 cluster.

Before upgrade (v1.59.0): containerd 2.1, setting does not exist in the model.

bash-5.2# containerd --version
containerd github.com/containerd/containerd/v2 2.1.6+bottlerocket c74fd8780002eb26bd5940ae339d690d891221c2
bash-5.2# apiclient get os
{
 "os": {
 "arch": "x86_64",
 "build_id": "73b3be44",
 "pretty_name": "Bottlerocket OS 1.59.0 (aws-k8s-1.35)",
 "variant_id": "aws-k8s-1.35",
 "version_id": "1.59.0"
 }
}
bash-5.2# apiclient set settings.container-runtime.max-concurrent-unpacks=4
Failed to change settings: Failed PATCH request to '/settings/keypair?tx=apiclient-set-YqaRzPE6ntefoKmt': Status 400 when PATCHing /settings/keypair?tx=apiclient-set-YqaRzPE6ntefoKmt: Unable to match your input to the data model. We may not have enough type information. Please try the --json input form. Cause: Error during deserialization: unknown field `max-concurrent-unpacks`, expected one of `max-container-log-line-size`, `max-concurrent-downloads`, `concurrent-download-chunk-size`, `concurrent-layer-fetch-buffer`, `enable-unprivileged-ports`, `enable-unprivileged-icmp`, `snapshotter` at line 1 column 46

After upgrade (v1.60.0): containerd 2.2, setting is available and functional.

bash-5.2# containerd --version
containerd github.com/containerd/containerd/v2 2.2.2+bottlerocket 301b2dac98f15c27117da5c8af12118a041a31d9
bash-5.2# apiclient get os
{
 "os": {
 "arch": "x86_64",
 "build_id": "a68d6de8-dirty",
 "pretty_name": "Bottlerocket OS 1.60.0 (aws-k8s-1.35)",
 "variant_id": "aws-k8s-1.35",
 "version_id": "1.60.0"
 }
}
bash-5.2# apiclient set settings.container-runtime.max-concurrent-unpacks=4
bash-5.2# apiclient get settings.container
{
 "settings": {
 "container-runtime": {
 "max-concurrent-unpacks": 4
 }
 }
}

Containerd config reflects the setting:

bash-5.2# cat /etc/containerd/config.toml | grep max_concurrent_unpacks -C 5
[plugins."io.containerd.cri.v1.runtime"]
device_ownership_from_security_context = true
enable_selinux = true

[plugins."io.containerd.transfer.v1.local"]
max_concurrent_unpacks = 4
concurrent_layer_fetch_buffer = 0

[[plugins."io.containerd.transfer.v1.local".unpack_config]]
snapshotter = "overlayfs"
differ = "walking"

After downgrade (signpost rollback-to-inactive): Setting is cleanly removed.

bash-5.2# apiclient get settings.container
{}
[plugins."io.containerd.transfer.v1.local"]
concurrent_layer_fetch_buffer = 0

Unsupported Setting Warning (aws-k8s-1.30, containerd 1.7)

Built aws-k8s-1.30 (x86_64) with the local core-kit to verify the warning infrastructure on variants that remain on older containerd versions.

Setting is accepted but has no effect on containerd config:

bash-5.2# apiclient set container-runtime.max-concurrent-unpacks=4

Env file rendered at /etc/unsupported-settings/max-concurrent-unpacks:

UNSUPPORTED_SETTING=container-runtime.max-concurrent-unpacks
UNSUPPORTED_REASON=This setting requires containerd 2.2 or later.

Timer starts automatically and fires on boot:

● unsupported-setting-warning@max-concurrent-unpacks.timer - Scheduled unsupported setting warning for max-concurrent-unpacks
     Loaded: loaded (/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/unsupported-setting-warning@.timer; disabled; preset: enabled)
     Active: active (waiting) since Tue 2026-04-21 00:38:00 UTC
    Trigger: Tue 2026-04-21 06:38:00 UTC; 5h 59min left
   Triggers: ● unsupported-setting-warning@max-concurrent-unpacks.service

Journal warning:

Apr 21 00:38:00 ip-192-168-21-104.us-west-2.compute.internal root[2113]: The setting container-runtime.max-concurrent-unpacks is not supported and has no effect. This setting requires containerd 2.2 or later.

Timer fires on boot + every 6h (same cadence as existing deprecation-warning infrastructure).

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

@ytsssun ytsssun changed the title Containerd 22/bottlerocket Add Containerd 2.2 to k8s 1.35 variants Apr 3, 2026
@piyush-jena
Copy link
Copy Markdown
Contributor

You can cherry-pick this commit - ec6cc83 and move your migration to v1.60.0

@ytsssun ytsssun force-pushed the containerd-22/bottlerocket branch from 2ce6f0b to 87c758c Compare April 20, 2026 23:17
@ytsssun
Copy link
Copy Markdown
Contributor Author

ytsssun commented Apr 20, 2026

You can cherry-pick this commit - ec6cc83 and move your migration to v1.60.0

Yeah we need to consolidate on the merge. That is OK. I can resolve merge conflict after you merge the PR. Or the other way around depending on which PR gets merged earlier.

@ytsssun ytsssun force-pushed the containerd-22/bottlerocket branch 2 times, most recently from 53a6a9d to d72021c Compare April 21, 2026 19:22
Signed-off-by: Yutong Sun <yutongsu@amazon.com>
@ytsssun ytsssun force-pushed the containerd-22/bottlerocket branch from d72021c to 18b26fd Compare April 22, 2026 05:40
Signed-off-by: Yutong Sun <yutongsu@amazon.com>
@ytsssun ytsssun force-pushed the containerd-22/bottlerocket branch from 18b26fd to 32eb1e6 Compare April 22, 2026 05:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants