Barrier in MPI_Finalize

[nysal]

I had posted this a while back in the devel mailing list - https://www.open-mpi.org/community/lists/devel/2015/10/18145.php

This issue can be easily reproduced in both master and v2.x using the ibm/random/ticket-1944-test4.c test case. Just reduce the sm BTL's fifo size:

$ /u/jnysal/ompi-idbg-v2.x-test/bin/mpirun  -np 4 --mca pml ob1 --mca btl self,sm --mca btl_sm_fifo_size 512 ticket-1944-test4
1 finish send
0 start recv
0 finish recv
<hang>

As expected adding an MPI_Barrier before MPI_Finalize in the test case fixes the issue. The pmix fence does not progress MPI.

[nysal]

@jsquyres @rhc54 Comments ?

[bosilca]

Now that the RTE does not call opal_progress we need indeed a way to synchronize the processes before tearing down the communication infrastructure. Adding a barrier is a workable solution.

[rhc54]

Actually, I'm not sure why this is true. What happens is that we call PMIx_Fence_nb in MPI_Finalize, and then loop opal_progress while we wait for the non-blocking fence to complete. So things should be progressing just fine.

Unless someone removed that logic?? I can take a look in a bit.

[nysal]

It looks like the blocking version is used. Quoting from ompi_mpi_finalize():

242     opal_pmix.fence(NULL, 0);
243 
244     /* check for timing request - get stop time and report elapsed
245      time if so */
246     OPAL_TIMING_MSTOP(&tm);
247     OPAL_TIMING_DELTAS(ompi_enable_timing, &tm);
248     OPAL_TIMING_REPORT(ompi_enable_timing_ext, &tm);
249     OPAL_TIMING_RELEASE(&tm);

[rhc54]

Yes, someone removed the logic - I've replaced it here:

#1577

Please check it out

[nysal]

@rhc54 Thanks, that fixes the issue for me

[bosilca]

👍

[rhc54]

Fixed by 7373111