Separate container sandbox lifecycle from that of the processes inside it

[vishh]

As of now, the runtime lifecycle states that there is a Start action which results in creation of a container sandbox and spawning the first process inside the container.
It would be better to separate lifecycle of the container sandbox from that of container processes.
To be more precise, I'm proposing introducing the following oci actions:

1. Create - This action will create a container sandbox. On Linux, this would be all cgroups and namespaces, excepting pid namespace (assuming all cgroups and namespaces were requested as part of the Spec).
2. Start - This action will spawn a new process in the existing container sandbox. On Linux, a new pid namespace will be created for the first process that is being spawned. Subsequent processes will run the same pid namespace.
3. Cleanup - This action will kill all processes in a container sandbox. In the case of Linux, this will result in loss of pid namespace.
4. Delete - This action will delete the container sandbox.

This split will obviate the need for supporting hooks. Users are free to run hooks outside of the runtime as they please.

The existing Spec needs to be split into a Sandbox configuration and a Process configuration.
The process configuration can be reused for the Exec use-case.

FYI: This was discussed during the OCI meeting on 1/12/16 and everyone present agreed to this proposal.

We need to prototype this separation in one or more Operating Systems before requiring it in the Spec.

Input from runtime authors on non-linux platforms will be helpful here.

[ashahab-altiscale]

I'd suggest the prototype should test with usernamespaces, especially in light of bugs like this one:
opencontainers/runc#101
Related patch: https://github.com/opencontainers/runc/pull/105/files

The init process of the container needs to be able to join a pre-created user namespace.
If this does not work, it would not possible to create any namespaces, as user namespaces require all other namespaces to be children of the user namespace.

[vishh]

@ashahab-altiscale: As long as user namespaces don't require a process to pin it, I don't see any issue here.

[wking]

On Tue, Jan 12, 2016 at 03:13:37PM -0800, Vish Kannan wrote:

As of now, the runtime lifecycle states that there is a Start
action which results in creation of a container sandbox and spawning
the first process inside the container. It would be better to
separate lifecycle of the container sandbox from that of container
processes.

This seems like a high-level change. Should discussion happen on the
list 1? There are earlier notes on this split approach in 2, as
well as some previous discussion along these lines in #226. If we
decide to have this conversation here (instead of the list or earlier
issue), I can write up a summary of past points.

This split will obviate the need for supporting hooks. Users are
free to run hooks outside of the runtime as they please.

Yeah, if we have a separate create and exec, we can drop hooks and
close #265 3.

The existing Spec needs to be split into a Sandbox configuration and
a Process configuration.

Not necessarily. ccon-ced (discussed in #226) wraps a runtime that
uses a single spec to do this. To pull that off, you need a spec that
makes it easy to turn on only the functionality you want (e.g. “create
namespaces” vs. “join, [create a new PID namespace,] and exec a
process”).

 Subject: Documenting container lifecycles
 Date: Mon, 14 Sep 2015 20:47:17 -0700
 Message-ID: <20150915034717.GO18018@odin.tremily.us>

[jonboulle]

s/Support/Separate in issue title

It would be better to separate lifecycle of the container sandbox from that of container processes.

(emphasis mine)
more justification/use cases here would be useful, at the very least for posterity

This split will obviate the need for supporting hooks.

To be clear, are you suggesting removing hooks entirely from the spec?

Subsequent processes will run the same pid namespace.

It makes me pretty uneasy to have the start operation be nondeterministic - e.g. if I run two start commands simultaneously it's a race which one's process becomes PID1 (which has non-trivial implications).

[wking]

On Tue, Jan 12, 2016 at 03:56:28PM -0800, Jonathan Boulle wrote:

Subsequent processes will run the same pid namespace.

It makes me pretty uneasy to have the start operation be
nondeterministic - e.g. if I run two start commands simultaneously
it's a race which one's process becomes PID1 (which has non-trivial
implications).

Yeah, -1 to having PID namespace be implicit. If you run starts
(parallel or not) that both create PID namespaces, then they both get
to be PID 1 in their own PID namespace. If you run a start (parallel
or not) that does not create a PID namespace, it does not get to be
PID 1. So I don't see a problem with racing if we have explicit
PID-namespace creation.

On naming, see ‘exec’ vs. ‘start’ here in opencontainers/runc#210.

[vbatts]

On Tue, Jan 12, 2016 at 6:52 PM W. Trevor King notifications@github.com
wrote:

On Tue, Jan 12, 2016 at 03:13:37PM -0800, Vish Kannan wrote:

As of now, the runtime lifecycle states that there is a Start
action which results in creation of a container sandbox and spawning
the first process inside the container. It would be better to
separate lifecycle of the container sandbox from that of container
processes.

This seems like a high-level change. Should discussion happen on the
list [1]?

This is because we're sitting in the face-2-face and working through this
presently.

[wking]

On Tue, Jan 12, 2016 at 04:51:36PM -0800, Vincent Batts wrote:

On Tue, Jan 12, 2016 at 6:52 PM W. Trevor King wrote:

On Tue, Jan 12, 2016 at 03:13:37PM -0800, Vish Kannan wrote:

As of now, the runtime lifecycle states that there is a Start
action which results in creation of a container sandbox and
spawning the first process inside the container. It would be
better to separate lifecycle of the container sandbox from that
of container processes.

This seems like a high-level change. Should discussion happen on
the list [1]?

This is because we're sitting in the face-2-face and working through
this presently.

Yeah, that's fine. But when dumping from the face-to-face into the
archives, I think you should dump into a list post, not into a new
issue (like this one).

[wking]

On Tue, Jan 12, 2016 at 03:29:28PM -0800, Vish Kannan wrote:

@ashahab-altiscale: As long as user namespaces don't require a
process to pin it, I don't see any issue here.

The issue is the order in which you join or create namespaces, which
impacts permissions. For example, an unprivileged user can join a
user namespace they created earlier, and then create a new PID
namespace underneath. But they can't create a new PID namespace
before joining a user namespace they created (or creating a new user
namespace). There are a number of comments to this effect in
opencontainers/runc#105, and also some notes in the “Interaction of
user namespaces and other types of namespaces” section of
user_namespaces(7). For a command line example:

Make a new user namespace:
—
$ unshare --user --map-root-user sh
sh-4.3# echo $$
21171

In a different terminal, trying to create a new PID namespace and join
the user namespace fails:
—
$ unshare --pid nsenter --user=/proc/21171/ns/user --preserve-credentials sh
unshare: unshare failed: Operation not permitted

But trying to join the user namespace and then create a new PID
namespace succeeds:
—
$ nsenter --user=/proc/21171/ns/user --preserve-credentials unshare --pid sh
sh-4.3#

I think @ashahab-altiscale is just suggesting we avoid that issue by
proper namespace join/create ordering when implementing create/exec
tooling.

[ashahab-altiscale]

@wking Exactly. The order of namespace creation is important.

[duglin]

s/Cleanup/Stop/ but aside from that I like the idea of the split.

[duglin]

re: multiple "starts": based on what I've heard I don't think multiple concurrent "starts" would be allowed. I think I'm hearing that people still want a single main process in the container, which means once you've started it your only choice is to use "exec" to get a 2nd process. Once the main proc ended then you can issue "start" again tho.