ember mpi custom communicator resulting in deadlock

[mdafibuzzaman]

For our simulation , we need multiple custom mpi communicators. One custom communicator along columns and one along rows. I am using enQ_commsplit for creating custom communicators. The issue is , the collective routines works fine when the process ranks are in consecutive order in a communicator. But when they are not, the system gets into a deadlock. I am sharing an example -

Suppose, we have 6 processors. We will be creating 3 column communicators and 3 row communicators each having 2 processors in it. Processes in column communicator will have consecutive order e.g - (0,1) in one communicator , (2,3) in the next one and (4,5) in the next one. When I create communicators in this way , all the collective routines run fine.

The row communicator is created having (0,5) in one communicator , (2,1) in the next communicator and (4,3) in the next communicator. When I split the ranks in this way , the collective routines get into a deadlock.

Is the enQ_commsplit only supposed to work when the process ranks are in consecutive order? If not , what can I do to use collectives along row communicators?

Thanks in advance.

[jjwilke]

Is it possible to post a minimal reproducer code? And Python file?
The FFT motif should already be doing this.

[mdafibuzzaman]

enQ_commCreate is used in FFT where we have to provide an array of group ranks that we want to put in a communicator. But we need enQ_commSplit for our simulation. I am attaching a dummy code that I built on embercomm.cc in the mpi/motif folder. In this code we create col custom communicators and row custom communicators. I am also attaching an image showing how our row comms and col comms and the ranks look like.

I am attaching a defaultSim_comm.py file which will be convenient.

communication_issue.zip

Hope these helps. Thank you again.

[mdafibuzzaman]

Replacing the existing comm motif(embercomm.cc and embercomm.h) with this one and rebuilding the ember motif will be enough to produce the issue.