Feature request description

Most usage of Podman in HPC environments focuses on a single container for each user. This container is run only once and provides the environment jobs are launched in. Each compute job is run in the existing container via a podman exec command. Unfortunately, podman exec can become a bottleneck when many jobs are being launched for the same user simultaneously. We provide an API for interacting with exec sessions (IE, commands run by podman exec) including the ability to list them, inspect their status, and remove them. This means that each podman exec command must take a lock for the container in question, update the database with information about a new exec session that has been created, and then unlock the container; further database updates and locking of the container are necessary to update state when the exec session is started, and again when it finishes. The use of this exclusive lock and the amount of database I/O involved has been identified as a large part of the bottleneck with attempting to launch concurrent podman exec processes.

Suggest potential solution

We want to introduce a new argument to podman exec, podman exec --no-session, which will eliminate database session tracking and minimize (but, likely, not entirely remove) use of the container lock. We would also like to eliminate the usage of conmon to monitor the exec session, and instead to directly invoke the OCI runtime from Podman. This should have further benefits by reducing time to start and minimizing resource consumption of exec sessions. Behaviorally, podman exec --no-session will act like a normal podman exec but will not perform writes to the database. The exec session will not be tracked as a result and will be inaccessible via API but this is acceptable for HPC usage. The --no-session flag should be implemented for local Podman only, and not the remote client, where it does not make sense. All flags of the podman exec command should be supported except --detach and --detach-keys. For implementation, I feel that this should be a new API - perhaps called ExecNoSession - implemented in container_exec.go in Libpod