MPI run

[Hasupama]

Hi

In my code I am using mpi command and printing a value at a node in 2d domain. If it is mpirun-3 then the same value prints in 3 times.
ex:

 2V at time 1s 
 2V at time 1s
 2V at time 1s

 2.1V at time 2s
 2.1V at time 2s
 2.1V at time 2s

I need to assign results of the current iteration (1s) to initial values of next iteration(2s), so order needs to be kept. But sometimes this order changes as,
ex:

 2V at time 1s 
 2.1V at time 2s
 2.1V at time 2s
 2V at time 1s
 2V at time 1s
 2.1V at time 2s

Does the Printing order is changing mean, before executing the code for the whole domain in the current time step, the next time step starts?

How can I change it as the next time step can start only if the current time step finishes(then the values should be printed as the first examples)?

I have tried the command ' mpiexec -n' as well. But the same issue occured.

Appreciate your comments!

[connorjward]

Please can you share the code that you are running?

MPI ranks run effectively in isolation to each other so, unless you have collective MPI operations (e.g. barrier), the ordering between ranks is non-deterministic.

[Hasupama]

from firedrake import *
import time as comp_time
t = 0.0  #STARTING TIME
tMax = 10e-6 
Dt = Constant(2e-6)  
start_time = comp_time.time()
while t <= tMax:

	elapsed = int(comp_time.time() - start_time)
	e_h, e_m, e_s = int(elapsed/3600), int(elapsed % 3600 / 60), int( ( elapsed % 3600 ) % 60 )

	print('time: %.3f ns solved in %.0f h %.0f min %.0f s ' % (t, e_h, e_m, e_s)  )

	t += Dt(0) 

In this code there is nothing to solve and I just print the time values. But the order is mixed as;

time: 0.000 ns solved in 0 h 0 min 2 s 
time: 0.000 ns solved in 0 h 0 min 3 s 
time: 2000.000 ns solved in 0 h 0 min 3 s 
time: 0.000 ns solved in 0 h 0 min 2 s 
time: 2000.000 ns solved in 0 h 0 min 2 s 
time: 4000.000 ns solved in 0 h 0 min 3 s 
time: 0.000 ns solved in 0 h 0 min 2 s

I expected that the printing order appear as after finishing all the 'time: 0.000 ns solved in 0 h 0 min 2 s ', it starts to print 'time: 2000.000 ns solved in 0 h 0 min 3 s' . When I followed https://www.firedrakeproject.org/demos/parprint.py.html , I got only one print for each time value with the order. But I am confused why the above print command doesnot give the order.

[wence-]

There is no imposed ordering between the different processes when you print, so data can come out in any order. In particular, when running in parallel there is often some buffering that occurs such that some processes may appear to print after others.

You can sometimes avoid this by saying print(..., flush=True), but even then, output can be mixed up.

The parallel printing demo shows one way to print using PETSc facilities: the prints are now collective on a communicator and only rank 0 on that communicator prints.

If you really want everyone to print and you want deterministic ordering, then use PETSc.Sys.syncPrint. This buffers up (in rank order over a communicator) the prints from each process until you call PETSc.Sys.syncFlush on the same communicator:

from firedrake.petsc import PETSc
from firedrake import *
comm = COMM_WORLD
while ...:
    PETSc.Sys.syncPrint("whatever", comm=comm)
    PETSc.Sys.syncFlush(comm=comm)

[Hasupama]

I understood. Thank you Wence and Connorjward!