preserve-fds behavior

[spoutn1k]

Hello everyone,

I am trying to run containerized MPI applications using podman. To setup execution, MPI libraries communicate via PMI, often using file descriptors inherited from the manager process.

$ mpirun -np 4 env | grep PMI_FD
PMI_FD=6
PMI_FD=8
PMI_FD=13
PMI_FD=17

Now using --preserve-fds fails for me, as some of those file descriptors are non-sequential: the process with the fd 17 might not have a fd 15.

I found this check:

podman/cmd/podman/containers/exec.go

Lines 134 to 138 in f918a94

	for fd := 3; fd < int(3+execOpts.PreserveFDs); fd++ {
	if !rootless.IsFdInherited(fd) {
	return errors.Errorf("file descriptor %d is not available - the preserve-fds option requires that file descriptors must be passed", fd)
	}
	}

And I was wondering why the file passed file descriptors had to be

• contiguous;
• all accessible

and if this was a limitation set by a backend of some sort.

Thanks !

[vrothberg]

Thank you for reaching out, @spoutn1k!

I honestly don't know why. @giuseppe do you?

[eriksjolund]

A side-note: The problem with non-continuous FD:s with mpirun was encountered before
#10409 (comment)

[spoutn1k]

Thanks for your answers ! Sorry I did not see this issue, but his solution is more of a workaround; I am working with multiple implementations of the MPI standard and not all of them implement this flag.

[giuseppe]

if the FDs are not sequential we can cause weird issues in the OCI runtime, since opened files from the OCI runtime will fill these spaces and they could more easily leak into the container.

I think we could relax the Podman check, but wouldn't be also possible to address this issue by opening /dev/null and dup its fd to fill these spaces?

EDIT: and it is also trickier to handle in Podman and in conmon as we must close these fds (as likely they were open by Podman and conmon itself) before executing conmon and the OCI runtime.

[spoutn1k]

Are saying this should be handled externally ? There is one mpirun implementation per library, the chances of them actually fixing this issue are slim.

Why should we close these fds if they plug possible leakage ?

[giuseppe]

Yes I am suggesting to handle this externally before it gets to Podman.

[spoutn1k]

My project will allow me to circumvent that, but should it not be considered a Podman bug still ? If some MPI implementations create this situation, other software might too.

Both Singularity and Shifter allow passing file descriptors without even using an option. Should Podman not be able to as well ?

[giuseppe]

you can, with --preserve-fds but they should be contiguous. If there are holes, they can be filled with dummy fds, like opening /dev/null

[spoutn1k]

$ cat show_fds.py 
#!/usr/bin/env python3

from pathlib import Path

fd_files = Path('/proc/self/fd').glob('*')
fds = set()

for file in fd_files:
    if not file.exists():
        continue

    try:
        fds.add(int(file.name))
    except ValueError:
        continue

print(f"{fds} - missing {set(range(max(fds))) - fds}")

$ mpirun -np 4 ./show_fds.py 
{0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 12, 13, 14, 17, 19} - missing {6, 10, 15, 16, 18}
{0, 1, 2, 3, 4, 5, 7, 9, 11, 12, 13, 14, 16, 17, 18, 21, 23} - missing {6, 8, 10, 15, 19, 20, 22}
{0, 1, 2, 3, 4, 5, 7, 9, 11, 12, 14, 16, 17, 18, 20, 21, 22, 25, 27} - missing {6, 8, 10, 13, 15, 19, 23, 24, 26}
{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 13, 15} - missing {10, 12, 14}

There is no filling with dummy fds using MPI, and even less having a exact value passed to --preserve-fds, as you can see the amount and numbers of file descriptors passed to each process is arbitrary.

$ mpirun -np 4 podman --preserve-fds={19 ? 23 ? 27 ? 15 ?}

Meanwhile:

$ mpirun -np 4 singularity exec ~/Images/test.simg ./show_fds.py 
{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 13, 15} - missing {10, 12, 14}
{0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 12, 13, 14, 17, 19} - missing {6, 10, 15, 16, 18}
{0, 1, 2, 3, 4, 5, 7, 9, 11, 12, 14, 16, 17, 18, 20, 21, 22, 25, 27} - missing {6, 8, 10, 13, 15, 19, 23, 24, 26}
{0, 1, 2, 3, 4, 5, 7, 9, 11, 12, 13, 14, 16, 17, 18, 21, 23} - missing {6, 8, 10, 15, 19, 20, 22}