sig auth may have a big say in security of this whole restoration pipeline

Thank you for pointing this out! Security is definitely an important topic that we need to discuss with sig-auth, both for the checkpoint API and the restoration pipeline. The following paper and master thesis describe our recent work on this topic:

• Towards Efficient End-to-End Encryption for Container Checkpointing Systems
• Improving Checkpoint/Restore Functionality in Kubernetes

I added sig auth to the list of stakeholder sigs

this showed up in the sig-auth meeting, we may have missed the discussion around this WG

if this WG is contemplating taking state from a running pod / saving it / letting it be consumed on another node or from another pod or another namespace, then sig-auth is definitely interested in making sure the permissions model around that exists and is ~consistent with similar things Kubernetes does elsewhere (like PVC / snapshots)

We're happy to consult on that, I'm not sure our awareness / involvement rises to the level of sponsoring the WG :)

cc @kubernetes/sig-auth-leads

is charter included into this PR?

now it is, I didn't add it initially as the lifecycle document mentions that it is added later, but looking at the WG PRs it seems to be common to have a charter in the initial PR.

This is a valuable initiative. The charter mentions that the scope includes checkpointing and restoring 'workloads' and providing 'guidance for developers on checkpoint-friendly app design.' Given this focus, it's essential for SIG Apps to be involved as a key stakeholder.

@janetkuo This is a good idea, thank you so much for suggesting it!

Yes, thanks @janetkuo. I added SIG Apps to the proposal.

I agree with Janet here, but please make sure to show up and present the scope of this proposal to one of the future SIG-Apps calls.

there may be API needed to communicate between the app and API server that the checkopoint is requested AND/OR that the app is ready for checkpoint. Something that is beyond just guidance

That is actually something we discussed how to do in containers for years now (outside of Kubernetes). But we never found the right way how to do this. We were looking at kernel interfaces or systemd interfaces because for many applications it could be helpful to free temporary memory to reduce checkpoint size or even drop confidential information. Also after restore it would be good to tell the application that maybe certain cryptographic values need to be reset or regenerated. I will try to include something mentioning this. Thanks.

has there been outreach to these SIGs yet? I see some SIG node participants/leaders but not CLI yet for example

Changing from SIG CLI to SIG API machinery after SIG CLI mentioned that new commands are first introduced via a plugin.

kubernetes/enhancements#5091

I have a simple prototype of a plugin for the kubectl checkpoint command that we used in our demos and can discuss further in the working group.

OK, we should remove the SIG if they are not a stakeholder, or else the WG organizers should reach out to the SIGs.

I have doubts if this incentivize the right behavior and will encourage people to build WG to get a slot in the kubecon

I agree with Antonio, this particular line should be removed, it's sufficient what the previous point shows.

With my SIG-CLI hat, I'm raising similar comment as the other one, this topic wasn't brought to SIG-CLI attention.

cc @kubernetes/sig-cli-leads

I agree with Janet here, but please make sure to show up and present the scope of this proposal to one of the future SIG-Apps calls.

The mailing lists for all WGs/SIGs are part of our managed google groups, with a kubernetes.io domain. So this should be https://groups.google.com/a/kubernetes.io/g/wg-checkpoint-restore once you get the group created we'll be able to provision this for you.

Can you link to those discussions?

I believe most of the previous discussions are linked here: kubernetes/enhancements#2008

Why it has to be a central part of Kubernetes, where multiple external solutions already exists?

i personally like the idea of having it be integrated because then the ecosystem can rely on it. for instance, we could make eviction or preemption less disruptive in kubelet/kueue respectively

Why it has to be a central part of Kubernetes, where multiple external solutions already exists?

Can you be more specific about what already exists? Not sure what you are referring to?

This first thing that I'd like to point out is that there are 2 main use cases:

1. the whole control-plane snapshot
2. workload

Which one this group is planning to cover? As I'm reading this document I'm seeing both used interchangeably which is very confusing. That's why I'd start with clearly drawing the line between the two and properly documenting which one of these two (or both) are you planning to tackle.

What do yo mean by "control-plane-snapshot"?

In our presentations, we use the following diagram to illustrate how checkpoint/restore operations work in Kubernetes:

Reference: Efficient Transparent Checkpointing of AI/ML Workloads in Kubernetes

Why pod checkpointing would have anything to do with scheduling?

Because, as far as I know, a pod is always scheduled on one node. It doesn't sound useful to base the scheduling on the possibility to migrate containers. Container migration is an important first step, but for automatic scheduling decisions, it would make more sense to be able to easily migrate a complete pod.

Our use-case is similar to how CRIU is integrated with Google's Borg ¹ and Microsoft's Singularity ² to enable preemptive and elastic scheduling.

I agree with Antonio, this particular line should be removed, it's sufficient what the previous point shows.