Fflush(stdout) doesn't work as expected.

[WSaffery]

Thank you for taking the time to submit an issue!

Background information

What version of Open MPI are you using? (e.g., v4.1.6, v5.0.1, git branch name and hash, etc.)

v5.0.2

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

Installed using DNF on Fedora 40 (KDE Spin)

If you are building/installing from a git clone, please copy-n-paste the output from git submodule status.

N/A

Please describe the system on which you are running

• Operating system/version: Fedora 40 (KDE Spin)
• Computer hardware: Framework 13
• Network type: N/A

---

Details of the problem

It appears to me that fflush(stdout) has no effect, where previously it would cause the stdout buffer to flush as one might expect.

#include <mpi.h>
#include <stdio.h>

void main(int argc, char** argv)
{
	int num_procs, rank;
        MPI_Init(&argc, &argv);
        MPI_Comm_size(MPI_COMM_WORLD, &num_procs);
	MPI_Comm_rank(MPI_COMM_WORLD, &rank);

	if (rank == 0)
	{
		printf("N = ");
		fflush(stdout); // this should flush "N = " to the screen, prompting the user to input
		int n;
		scanf("%d", &n);
		printf("bye\n");
	}

	MPI_Finalize();
}

Will hang with no output, instead of printing "N = " before running scanf at the root (rank 0 process).

On the other hand if I include a print containing a newline character, the stdout buffer is automatically flushed without issue.

#include <mpi.h>
#include <stdio.h>

void main(int argc, char** argv)
{
	int num_procs, rank;
        MPI_Init(&argc, &argv);
        MPI_Comm_size(MPI_COMM_WORLD, &num_procs);
	MPI_Comm_rank(MPI_COMM_WORLD, &rank);

	if (rank == 0)
	{
		printf("N = ");
		printf("\n"); // this DOES flush the "N = ", but so SHOULD fflush(stdout); !
		int n;
		scanf("%d", &n);
		printf("bye\n");
	}

	MPI_Finalize();
}

Programs were executed using mpirun -np 8 <binary_path>.

A zip file containing an equivalent project to these markdown blocks can be found here.

• PS: It seems to me (based on delays from printing) that fflush(stdout) is completely broken (i.e. all the output is just flushed at MPI_Finalize, and an earlier fflush(stdout) has no actual effect), and I expect this is to do with output redirection done internally in mpirun, running our binaries without mpirun doesn't have this issue. But writing code that clearly verifies this is harder, and this is just what I have right now.

[ggouaillardet]

FWIW the issue does not occur with Open MPI v4.1.x
@rhc54 that suggests prrte (unless PMIx does all/most of the work) is involved here.

[rhc54]

Most likely in PMIx (src/common/pmix_iof.c) as it does the actual output. This is one of those "you cannot win" scenarios. Some users complain if the output comes in incomplete lines because of the inevitable interleaving that occurs, while others complain about this scenario. ORTE opted for one side, PMIx/PRRTE for the other. 🤷‍♂️

There is no way for us to know that someone called "fflush", so I'm not sure how one can resolve it. However, I'm happy to consider a PR that attempts to resolve it. I personally won't have time to look into better solutions before late this year (retirement is busy).

[ggouaillardet]

@rhc54 thanks for the quick reply, as far as I am concerned, I am fine with "it's not a bug, it's a feature" since I believe apps should either read from the command line or an input file, and not from stdin

[rhc54]

I suppose you could add a cmd line option to permit interleaving so that someone who needs/wants to output partial lines can do so. I have no issues with it - you'd just need to pass that down to PMIx so it doesn't buffer output.

[WSaffery]

Thanks everyone for your help everyone,

I was a little flustered when writing this report because I relied on the behaviour from the former version for an assignment I had already submitted before discovery, and I wasn't exactly sure whether I'd misremembered or if this behaviour was from an update.

Obviously that external pressure shouldn't play a disproportionate in how you architect your application. I can see that the uni student who relies on stdin for input and stdout for output is a pretty special case. So I don't entirely disagree with @ggouaillardet.

A little reading seems to indicate to me that at least for gnu c, stdout is line-buffered by default, and fflush or scanf will both cause it to flush early. As such I'm not sure why changing this behaviour (I suppose by adding an additional buffer around the output of processes? I'm not actually sure what's being done) would prevent interleaving- but I'm also not exactly sure what the buffering behaviour of PMIx is.

It would be nice if things worked as I expected, but from just editing this project and not external ones it seems more like a trade-off between two imperfect solutions as mentioned by @rhc54. A parameter could be a helpful middle ground, but this can probably be closed if there isn't will for it.

[rhc54]

A little reading seems to indicate to me that at least for gnu c, stdout is line-buffered by default, and fflush or scanf will both cause it to flush early. As such I'm not sure why changing this behaviour (I suppose by adding an additional buffer around the output of processes? I'm not actually sure what's being done) would prevent interleaving- but I'm also not exactly sure what the buffering behaviour of PMIx is.

Well, that's another case of "you cannot win". The problem is that people complain if their process dies and they lose their last output because it is being buffered by libc (which makes debugging nearly impossible). So you have to disable that buffering to "display the last output".

Interleaving means that we output a partial line from one process (because that is all we read from the pipe at that moment), and then output text from some other process, and then output the rest of the line when the pipe delivers it from the first process. Now multiply that by thousands of processes and you can hopefully understand the complaint. Line buffering in the runtime allows us to remedy that problem while still providing the "last output" when we flush our buffers on process termination.

Of course, we also get complaints if the runtime daemon itself fails while holding some output - so you do your best to not allow that to happen.

Like I said, you cannot win regardless of which behavior you pick. Every runtime has to make the choice, and every runtime gets criticized for that choice. 🤷‍♂️

The cmd line option does provide some middle ground - though we'll hear the complaints when someone sets the option and gets a mess of interleaved output they cannot understand. 😄

[awni]

Was there ever a command added to control the output buffering behavior? I'm running into the same issue after recently upgrading openmpi.

[FreddieWitherden]

Our users have been encountering similar issues since upgrading to v5. In particular, it breaks the progress bar functionality of our code (which relies on the root rank writing to stderr with explicit flushes but no newlines to update the progress). As far as I can tell --stream-buffering=0 should resolve this since it explicitly states in the documentation that:

Adjust buffering for stdout/stderr. Allowable values:

    0: unbuffered

    1: line buffered

    2: fully buffered

However, it looks as of v5 we get line buffering irrespective of what is passed. Our current guidance is to swap out to mpich.

[awni]

It worked for me to pass --output :raw to mpirun

[rhc54]

Switching to MPICH is fine - two implementations exist for a reason. Or you could just ask mpirun for help:

$ mpirun --help output
...
Supported qualifiers include "NOCOPY" (do not copy the output to the
stdout/err streams), and "RAW" (do not buffer the output into complete
lines, but instead output it as it is received).

I can't speak to what is or isn't in the OMPI docs - I don't maintain those things. Sadly, the OMPI community doesn't put a lot of emphasis on supporting the runtime in general, including the runtime doc entries. I explained the rationale above, and users have in general been able to "find" the correct option.

So I would recommend you first turn to the hierarchical "help" system when trying to find the right command line option to get what you want. I spent a fair amount of time on that, so hopefully it is a little more complete than the docs.