Refactor container runtime interface

[yuvipanda]

Our current container runtime interface has these as separate things:

1. Build
2. Run
3. Push
4. List images
5. Inspect images

However, the following combinations are what happen a lot of the time:

1. Build + Run (no push)
2. Build + Push (No run)
3. List Images

This was fine when we were doing this with the old docker build api, but I think causes issues now with docker buildx.

1. The buildkit build cache is different from the image store now, so we must pass --load to buildx during build so our run finds the image without having to pull
2. However, when we are only doing build + push, this adds unnecessary extra complexity that will literally never be used. It's a copy (IIRC) from buildkit cache to the image store, except the image store will never actually be used. This takes up disk space and nothing else. This is primarily a problem on binderhub installs
3. I'd like us to eventually support running buildkit directly (which removes the need for a privileged pod), and that would only support build + push. This is actually the ideal for binderhub.

Proposal

I propose we simply add 'push' and 'run' as boolean args to build and deprecate the run and push methods. Additionally, if this feels weird, we can rename build to be something else.

Runtimes that don't support run (like buildkit) can simply error out when run is True.

[yuvipanda]

Opening as an issue instead of a PR particularly to get feedback from @manics who is the only other person I know who manages a container runtime right now :)

[manics]

I think this is fine! I vaguely recall thinking about this when working on the original Container Engine abstraction PR, but wanted to minimise changes.

[yuvipanda]

I had to revert the buildx deployment on mybinder.org (jupyterhub/mybinder.org-deploy#3228) because the node's disk was consistently being filled up.

[yuvipanda]

The buildx setup is back on mybinder.org

[yuvipanda]

I spent some time trying to do this, particularly with passing push to build. The major issue here is around how authentication to docker push is handled. docker login modifies DOCKER_CONFIG, and the existing .login method we use does not. So we have to figure out how to make sure that:

1. The credentials passed to repo2docker don't leak on to ~/.docker/config
2. The credentials already in ~/.docker/config are still considered for the push

[manics]

Can we override the config file path? https://forums.docker.com/t/config-json-stored-in-other-directory-not-docker/136641/2

[yuvipanda]

yeah, that's what i'm exploring now.

[yuvipanda]

#1421 changes push to use the CLI, and explores how we can use docker login to meet the goals set out in #1414 (comment). This seems to work fine in my testing!