Choosing MPI procs for lfric_atm, multi-grid

[mcdalvi]

Hi,
Is there a thumb rule or 'ready reckoner' for selecting the number of MPI tasks that can be used for a given LFRic multigrid configuration ?

I am attempting different values based on (a) multiple of 6 and (b) divisible by 'C' number (nproc_y=1) but keep getting errors of the type:
ERROR: Global mesh (multigrid_XX) does not produce even partitions with the requested partition strategy

Thanks

[iboutle]

What resolution are you trying to run Mohit? You need a factor of 8 and a factor of 6 - I'm not sure being divisible by the C number is required. For C64, 384 processors is basically the only thing worth doing. For C144, 1944 would be the maximum possible, and then you could do 972 or 486 for example.

[mcdalvi]

Thanks Ian,
This is for the C144 configuration. From what I have seen so far the model divides the total processor number into 6 (faces) and tries to further assign equal X x Y processors for each face. So, for C144 with 1944 processors this will be 324 for each face, then allocated as 18x18 in X and Y directions respectively.
Further, in case of 4-level multi-grid the respective domains seem to be set up as C144, C72, C36 and C18 and each C number has to be cleanly divisible by the 18 processors in X and Y direction.
So, yes I can see the difficulty in arriving at a number higher than 1944 that can meet these criteria. I have tried 18x36 x 6 = 3888 but the model divides each face into 24x27 so fails the 2nd criteria.

[iboutle]

Yep, values any higher than 1944 for C144 will have to be done with openMP (when we have that).

It's worth adding that C144 doesn't work on the ex1a currently, as it hits the same xios out-of-memory issues that we are having with lots of jobs!