[mtt] mpich_tests_mpi_pt2pt fails with enabled lanes

[amaslenn]

Configuration

OMPI: 3.1.0rc4
MOFED: MLNX_OFED_LINUX-4.3-1.0.1.0
Nodes: hercules x47 (ppn=32(x47), nodelist=clx-hercules-[005,008,081-125])
Job: ucx

MTT log: http://e2e-gw.mellanox.com:4080//mnt/lustre/users/mtt/scratch/ucx_ompi/20180427_020635_16343_83128_clx-hercules-005/html/test_stdout_CKV8QI.txt

Test passes without -x UCX_MAX_EAGER_LANES=3 -x UCX_MAX_RNDV_LANES=3.

Cmd:
mpirun -np 2 -mca btl self --tag-output --timestamp-output -mca pml ucx -mca coll '^hcoll' --bind-to core -x UCX_NET_DEVICES=mlx5_0:1,mlx5_2:1 -x UCX_IB_REG_METHODS=rcache,direct -x UCX_TLS=all -x UCX_RC_TM_ENABLE=n -mca pmix_base_async_modex 1 -mca mpi_add_procs_cutoff 0 -mca pmix_base_collect_data 0 -x UCX_MAX_EAGER_LANES=3 -x UCX_MAX_RNDV_LANES=3 --map-by node /mnt/lustre/users/mtt/scratch/ucx_ompi/20180427_020635_16343_83128_clx-hercules-005/installs/mOeN/tests/mpich_tests/mpich-mellanox.git/test/mpi/pt2pt/sendrecv1

Output:

...
Sun Apr 29 16:27:34 2018[1,0]<stdout>:Data in target buffer did not match for destination datatype recv struct (64 nblock 4096 blocklen 40960 stride 0 lb) and source datatype MPI_DOUBLE, count = 262144
Sun Apr 29 16:27:34 2018[1,0]<stdout>:Data expected = 35 but got p[0,3065] = 2d
Sun Apr 29 16:27:34 2018[1,0]<stdout>:Data expected = 34 but got p[0,3065] = 2c
Sun Apr 29 16:27:34 2018[1,0]<stdout>:Data expected = 33 but got p[0,3065] = 2b
Sun Apr 29 16:27:34 2018[1,0]<stdout>:Data expected = 32 but got p[0,3065] = 2a
Sun Apr 29 16:27:34 2018[1,0]<stdout>:Data expected = 31 but got p[0,3065] = 29
Sun Apr 29 16:27:34 2018[1,0]<stdout>:Data expected = 30 but got p[0,3065] = 28
Sun Apr 29 16:27:34 2018[1,0]<stdout>:Data expected = 2f but got p[0,3066] = 27
Sun Apr 29 16:27:34 2018[1,0]<stdout>:Data expected = 2e but got p[0,3066] = 26
Sun Apr 29 16:27:34 2018[1,0]<stdout>:Data expected = 2d but got p[0,3066] = 25
Sun Apr 29 16:27:39 2018[1,0]<stdout>: Found 40 errors

[hoopoepg]

@yosefe issue is caused because OMPI can't process out-of-order portion of data on generic datatype (pml_ucx_generic_datatype_unpack callback used to handle custom data).

disable m-rail on for such cases?

[yosefe]

do you mean OMPI does not really support "offset" parameter for unpack?
e.g that opal_convertor_set_position(&convertor->opal_conv, &offset); does not work?

[hoopoepg]

as I can see it shift position to forward only

[shamisp]

Adding George to confirm this open mpi limitation.

…

On Thu, May 3, 2018 at 14:56 Yossi Itigin ***@***.***> wrote: do you mean OMPI does not really support "offset" parameter for unpack? e.g that opal_convertor_set_position(&convertor->opal_conv, &offset); does not work? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2568 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACIe2CjR0RLOClNXO1sfHZVOa_3rii2yks5tuvB6gaJpZM4TrzQv> .

[shamisp]

@bosilca can you please take a look

[bosilca]

Multi-rail is supported by the OB1 PML and as far as we know the code is working.

To answer @hoopoepg question moving the convertor backward is done in opal_convertor_set_position_nocheck (opal/datatype/opal_convertor.c), by rewinding the convertor at the beginning and then moving forward to the desired position. However, such rewinding only works on predefined datatype boundaries, condition respected by the pack side as well. In other words, if you use OPAL pack/unpack things work flawlessly, but if you only use one side and do the other manually then you should respect the predefined types boundaries (and alignments).

[yosefe]

@bosilca we use only opal_convertor on both sides so i think datatype boundary should be ok
IIUC, we should use opal_convertor_set_position_nocheck() instead of opal_convertor_set_position(), in pml_ucx code (https://github.com/open-mpi/ompi/blob/master/ompi/mca/pml/ucx/pml_ucx_datatype.c#L88), right?

[bosilca]

The nocheck function avoids some consistency checkings (position outside bounds, convertor already at the right position), and should not be used directly. The current usage in pml_ucx looks OK to me.

Another possible source of errors is the fact that the convertor can pack/unpack less data than requested (which explains why max_data argument is a pointer).

[hoopoepg]

as I can see opal_convertor_unpack doesn't support out-of-order data at all: it uses call opal_convertor_get_current_pointer to get pointer for every portion of data, but opal_convertor_get_current_pointer just returns dest_ptr + converted_amount + lb and returned pointer could not handle out-of-order data