Support for container checkpointing

[jicowan]

Is it possible to checkpoint a container running on EKS? If the latest version of EKS variant supports containerd 2.0, it should be theoretically possible. In order to checkpoint a container you need to run cURL -k POST http://localhost:10250/checkpoint/{namespace}/{pod}/{container} from the node. I assume you could do this from a control and/or admin container so long as you passed in the appropriate cert/key. My initial attempts at this failed on Auto Mode node which I know is a slightly different variant than the EKS variant. I basically want to know if this will work on BR and Auto Mode. See https://seifrajhi.github.io/blog/k8s-criu-container-checkpointing/ for reference.

[jicowan]

I've been experimenting with how to implement this on AL2023. The version of containerd that is included in the EKS optimized AMI (1.7) does not expose CheckpointContainer unless it is patched and rebuilt. I assume that this is also an issue on Bottlerocket. When I tried upgrading to a newer version of containerd (2.1.4) I started getting shim errors.

[vigh-m]

Hi, we have a similar issue open asking for support for CRIU at #3803

Bottlerocket currently doesn't support checkpointing. The error I received is below for reference:

% curl -X POST "${API_SERVER}/api/v1/nodes/ip-192-168-95-39.us-west-2.compute.internal/proxy/checkpoint/default/checkpoint-test/busybox" \
-H "Authorization: Bearer ${TOKEN}" \
--insecure
checkpointing of default/checkpoint-test/busybox failed (rpc error: code = Unknown desc = CRIU binary not found or too old (<31600). Failed to checkpoint container "8deae7ee42ca5e56ff55ff3dc6919f2555617ef77c35311df4b726d0fac73735": failed to check for criu version: exec: "criu": executable file not found in $PATH

I had to create a service account on my cluster to get this working with

kubectl create serviceaccount checkpoint-user
kubectl create clusterrolebinding checkpoint-user --clusterrole=cluster-admin --serviceaccount=default:checkpoint-user
TOKEN=$(kubectl create token checkpoint-user)

We recently merged containerd-2.1 in bottlerocket-os/bottlerocket-core-kit#621 which opens up a path to enable this.

[jicowan]

Awesome. I believe you should be able to use kube-proxy instead of using a service account token.