Exec plugin from container image

[danwinship]

One of the arguments for daemonization (#821) is that copying binaries out of container images onto the local disk is tricky and insecure. So here's an alternative. (Which... is not actually fully an alternative to #821... maybe it would make sense to do both.)

• The runtime creates a directory /var/run/cni at startup.
• The user uses kubelet/CRI/whatever to start up a "network plugin" container. The definition of this container/pod includes mounting a single file out of the container image into /var/run/cni on the host. Something like /var/run/cni/mynetwork.conf.
• The elements of plugins in this CNI conf file may have a containerPath element (in addition to type), which refers to a binary in the container's filesystem.
• The runtime chooses a default CNI plugin via whatever means it likes (just like today). (This may involve combining data from /var/run/cni and /etc/cni/net.d, for instance, so that everything still works just like it used to for systems where there is no network plugin using this new system.)
• Everything else works mostly just like before, except that for CNI conf files that use containerPath, the runtime executes the CNI binary out of the container's mount namespace, rather than in the host mount namespace. (Since the runtime is the one that mounted the CNI conf files out of the container(s) into /var/run/cni in the first place, it knows which mount namespaces are associated with which conf files.)
• Runtimes that support this feature are required to also support STATUS, and must consider the network plugin ready only when it responds successfully to a STATUS call. (It can't consider the network ready as soon as the conf file appears, because it's guaranteed the conf file will appear before the plugin is even running.)
• When the container exits, the mount for its conf file in /var/run/cni is automatically destroyed... which is nice in some ways, but it will screw up the network status during container restarts, so we need to figure out something there...
• A network plugin that wants to remain backward compatible with runtimes that don't support this new feature can just check whether it has received any STATUS calls yet at the point when it has finished starting up, and if the answer is "no", then it can assume the runtime is ignoring its /var/run/cni conf file, and instead copy a traditional conf file and binary out into the host filesystem at that point just like it would have before.

The combination of using /var/run for the conf file and using the container image for the binary, means the network plugin doesn't need to modify anything actually on the host filesystem, which may help people who want immutable host filesystems.

[danwinship]

The runtime creates a directory /var/run/cni at startup.

(I should have said, but part of the proposal is also to fix the fact that we are making plugins dynamically add files to parts of the root fs that admins ought to be able to make read-only.)

(Since the runtime is the one that mounted the CNI conf files out of the container(s) into /var/run/cni in the first place, it knows which mount namespaces are associated with which conf files.)

I'm hiding a subtlety here: my original idea was that the plugin would still write its CNI conf file out normally, and just include something like containerImage: blah/myplugin:v1.0.3 or mountNamespace: /proc/1234/ns/mnt or filesystem: /var/lib/containers/... or whatever, but it turns out to actually be really hard for code running inside a container to figure out a way to describe its filesystem accurately and unambiguously to code outside the container, so it made more sense to come up with a way to let the runtime figure it out automatically.

When the container exits, the mount for its conf file in /var/run/cni is automatically destroyed... which is nice in some ways, but it will screw up the network status during container restarts, so we need to figure out something there...

One possibility (suggested by Casey) would be to keep the conf file as something that was manually-created and then mount a different file as a way of telling the runtime what mount namespace to use. (Perhaps it could even mount the CNI binary itself (in which case there's no need to explicitly specify the path to the binary in the config), but it would be understood that the runtime isn't supposed to execute it directly, it needs to execute it in the right mount namespace.)