[mpich romio 73a3eba] darray needs a resize in the block()/cyclic() subtype creation

> Pulled in from mpich romio, branch "main".
> Their commit message is below.
>
> This is part of a batch of commits from the
> following set of PRs:
>  *  pmodels/mpich#4943
>     -- darray fix which contains a flatten fix
>         73a3eba
>         c4b5762
>  *  pmodels/mpich#4995
>     -- write strided fix
>         bbb5210
>  *  pmodels/mpich#5100
>     -- build fix for -Wpedantic
>         ad0e435
>  *  pmodels/mpich#5099
>     -- build fix, they had let file-system=...gpfs bit rot
>         e1d42af
>         313289a
>         83bbb82
>  *  pmodels/mpich#5150
>     -- build fix, configure-related _GNU_SOURCE
>         a712b56
>         5a036e7
>  *  pmodels/mpich#5184
>     -- build fix, continuation of _GNU_SOURCE fix
>         d97c4ee

When Type_create_darray() uses the block()/cyclic() function to
construct a subtype for whatever dimension it's processing, it
needs to resize the resulting type before it goes into the type
construction as the input for processing the next dimension.

The same problem is in the main path's type creation, and in romio's
mirroring of it.

Gist for a testcase:
https://gist.github.com/markalle/940de93d64fd779e304ee124855b8e6a

The darray_bug_romio.c testcase creates a darray using
* 4 ranks in a 2x2 grid
* looking at the type made for rank 0
* inner dimension: 4 ints distributed block over 2 ranks with 2 items each
* outer dimension: 6 of the above distributed cyclic over 2 ranks with 2 items each

The type created by processing the first dimension should look like this
[ x x . . ]

And then distributing those for the second dimension becomes

[ x x x x ]
[ . . . . ]
[ . . . . ]
[ . . . . ]
[ x x x x ]
[ . . . . ]

Going to the MPI standard to justify why the first layout is right,
it's where the definiton of the cyclic() function has a ub_marker
of gsize*ex, eg 4 ints for that first type.

Signed-off-by: Mark Allen <[email protected]>

Author: markalle

3rd-party/romio341/adio/common/ad_darray.c

@@ -196,6 +196,13 @@ static int MPIOI_Type_block(int *array_of_gsizes, int dim, int ndims, int nprocs
     if (mysize == 0)
         *st_offset = 0;
 
+    MPI_Aint ex;
+    MPI_Type_extent(type_old, &ex);
+    MPI_Datatype type_tmp;
+    MPI_Type_create_resized(*type_new, 0, array_of_gsizes[dim] * ex, &type_tmp);
+    MPI_Type_free(type_new);
+    *type_new = type_tmp;
+
     return MPI_SUCCESS;
 }
 
@@ -293,5 +300,11 @@ static int MPIOI_Type_cyclic(int *array_of_gsizes, int dim, int ndims, int nproc
     if (local_size == 0)
         *st_offset = 0;
 
+    MPI_Aint ex;
+    MPI_Type_extent(type_old, &ex);
+    MPI_Type_create_resized(*type_new, 0, array_of_gsizes[dim] * ex, &type_tmp);
+    MPI_Type_free(type_new);
+    *type_new = type_tmp;
+
     return MPI_SUCCESS;
 }