test: fix tests not working with high nranks and change GA to test that

[mrava87]

This PR fixes a number issues that arose when trying to increase the number of ranks in the test suite to 9 or 16.

As a consquence, I have also decided to modify the GA build to run tests with 2,4,9 ranks instead of 2,3,4

[mrava87]

@rohanbabbar04 no idea why the openmpi runs with N=9 stall, but those with intelmpi and mpich don't... any idea?

note that without the proposed changes some of the tests failed for N=9 for me locally... most are tests I added with the mask and add some clear if/else statement on the number of ranks and did not stop if a different one was passed, and another one was for the solvers due to some stochasticity in the random (if I run it independently, instead as part of the entire test suite, they were passing for N=9)

PS: in my local conda env, I used install_conda and got mpi4py installed with MPICH (which seems to be the default if not specifying anything... so I guess I can try openmpi and debug locally..

[mrava87]

This works locally, so not really sure what is happening in the CI...

[rohanbabbar04]

Hmm, let me give it a look

[mrava87]

Hmm, let me give it a look

@rohanbabbar04 did you manage to have some time to take a look at this?

[rohanbabbar04]

Hmm, let me give it a look

@rohanbabbar04 did you manage to have some time to take a look at this?

Oops, I missed this one. I will look at it during the weekend to find out why it is happening, and let you know

[rohanbabbar04]

Just added a minor change to the OpenMPI execution by using ^openib to exclude InfiniBand, since GitHub Actions runners may stall/show error messages when running with higher ranks. I think we should add it since it will be required as well.

I will now merge this 🙂

[mrava87]

Great, thanks a lot @rohanbabbar04 for investigating this 😀