Openmpi 3.1.4 segv in point to point communications using CM.

[mwheinz]

Thank you for taking the time to submit an issue!

Background information

What version of Open MPI are you using? (e.g., v1.10.3, v2.1.0, git branch name and hash, etc.)

v3.1.4

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

built from a git clone of v3.1.4.

Please describe the system on which you are running

• Operating system/version: RHEL 7.5/7.6
• Computer hardware: Xeon E6-2699
• Network type: OPA 100

---

Details of the problem

Our internal acceptance tests for changing the version of openmpi we ship includes the test programs shipped with the mpich package. (https://www.mpich.org/2019/06/07/mpich-3-3-1-released/).

Nearly all of the ~400 test programs work fine, but 15 crash, all in the same way, all PT2PT tests using MPI_Bsend.

Using a debug build of ompi 3.1.4, I get the following stack trace:

/usr/mpi/gcc/openmpi-3.1.4rc2-hfi/bin/mpirun -H hdsmpriv01,hdsmpriv02 --allow-run-as-root --mca mtl ofi --mca mtl_ofi_provider_include sockets --mca osc_rdma_mtls ofi  -map-by ppr:1:node test/mpi/pt2pt/bsend1

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `test/mpi/pt2pt/bsend1'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f3467be24ca in opal_thread_add_32 (delta=-1, addr=0x1)
    at ../../../../opal/threads/thread_usage.h:144
144	OPAL_THREAD_DEFINE_ATOMIC_ADD(int32_t, 32)
Missing separate debuginfos, use: debuginfo-install glibc-2.17-222.el7.x86_64 infinipath-psm-3.3-26_g604758e_open.2.el7.x86_64 libatomic-4.8.5-28.el7.x86_64 libgcc-4.8.5-28.el7.x86_64 libibverbs-15-6.el7.x86_64 libnl3-3.2.28-4.el7.x86_64 librdmacm-15-6.el7.x86_64 libuuid-2.23.2-52.el7.x86_64 numactl-libs-2.0.9-7.el7.x86_64 zlib-1.2.7-17.el7.x86_64
(gdb) where
#0  0x00007f3467be24ca in opal_thread_add_32 (delta=-1, addr=0x1)
    at ../../../../opal/threads/thread_usage.h:144
#1  wait_sync_update (updates=1, status=0, sync=0x1) at ../../../../opal/threads/wait_sync.h:112
#2  ompi_request_complete (with_signal=true, request=0x24a2580)
    at ../../../../ompi/request/request.h:447
#3  mca_pml_cm_send (buf=<optimized out>, count=7, datatype=<optimized out>, dst=<optimized out>, 
    tag=<optimized out>, sendmode=<optimized out>, comm=0x607520 <ompi_mpi_comm_world>) at pml_cm.h:361
#4  0x00007f3478d68f62 in PMPI_Bsend (buf=0x7fff40f01230, count=7, type=0x606220 <ompi_mpi_char>, 
    dest=<optimized out>, tag=<optimized out>, comm=0x607520 <ompi_mpi_comm_world>) at pbsend.c:80
#5  0x0000000000401d15 in main (argc=1, argv=0x7fff40f01368) at bsend1.c:52

[mwheinz]

Looks like the problem only occurs when CM is being used. Switching to ob1 for pt2pt prevents the crash:

[RHEL7.5 hdsmpriv01 20190802_1252 OMPI_TESTS]# /usr/mpi/gcc/openmpi-3.1.4rc2-hfi/bin/mpirun -H hdsmpriv01,hdsmpriv02 --allow-run-as-root --mca mtl ofi --mca pml ob1 --mca mtl_ofi_provider_include sockets --mca osc_rdma_mtls ofi  -map-by ppr:1:node test/mpi/pt2pt/bsend1
 No Errors
[RHEL7.5 hdsmpriv01 20190802_1253 OMPI_TESTS]# 

[bosilca]

The sync is atomically swapped to 1 when the request is marked as completed. The fact that the sync is already set to 1 (according to the backtrace) indicates that somehow the MPI request has already been marked as completed. That a starting point to find the culprit.

[mwheinz]

The sync is atomically swapped to 1 when the request is marked as completed. The fact that the sync is already set to 1 (according to the backtrace) indicates that somehow the MPI request has already been marked as completed. That a starting point to find the culprit.

Is it possible a request structure is being re-used without being cleared out?

[bosilca]

I don't know the CM mechanism for handling requests, but if CM does the right thing (aka. call OMPI_REQUEST_INIT before using a request) then no, it is not possible (the sync is stored as a void* in the req_complete field).

[mwheinz]

The sync is atomically swapped to 1 when the request is marked as completed. The fact that the sync is already set to 1 (according to the backtrace) indicates that somehow the MPI request has already been marked as completed. That a starting point to find the culprit.

@bosilca - I'm not sure that's right. The value of sync isn't 1, the address of sync is 1. That's what's causing the segv. The implication is that the request pointer itself is corrupt.

Maybe this line is the problem?

 ompi_wait_sync_t *tmp_sync = (ompi_wait_sync_t *) OPAL_ATOMIC_SWAP_PTR(&request->req_complete, REQUEST_COMPLETED);

Isn't that going to return a pointer with the value of the contents of request->req_complete? Shouldn't req_complete point to a ompi_wait_sync_t structure and not be an integer value? Or is that field being overloaded?

[ggouaillardet]

TL;DR 3285325 has to be backported, I'll issue a PR shortly.

Here is what happens :

ompi_request_complete() in invoked a first time by mtl/ofi here

main
  PMPI_Bsend (pbsend.c:80)
    mca_pml_cm_send (pml_cm.h:361)
      ompi_mtl_ofi_isend (mtl_ofi.h:381)
        ompi_mtl_ofi_send_start (mtl_ofi.h:304)
          ompi_mtl_ofi_isend_callback (mtl_ofi.h:221)
            mca_pml_cm_send_request_completion (pml_cm_sendreq.c:59)
              ompi_request_complete (request.h:442)

and then a second time by pml/cm here

main
  PMPI_Bsend (pbsend.c:80)
    mca_pml_cm_send (pml_cm.h:361)
      ompi_request_complete (request.h:440)

and based on the code, ompi_request_complete() is not supposed to be called twice since
the req_complete_cb would be invoked twice, and then
OPAL_ATOMIC_CMPSET_PTR() fails (since req_complete is already REQUEST_COMPLETED) and we end up calling wait_sync_update() with tmp_sync=REQUEST_COMPLETED that is 0x1.

            if(!OPAL_ATOMIC_CMPSET_PTR(&request->req_complete, REQUEST_PENDING, REQUEST_COMPLETED)) {
                ompi_wait_sync_t *tmp_sync = (ompi_wait_sync_t *) OPAL_ATOMIC_SWAP_PTR(&request->req_complete,
                                                                                       REQUEST_COMPLETED);
                /* In the case where another thread concurrently changed the request to REQUEST_PENDING */
                if( REQUEST_PENDING != tmp_sync )
                    wait_sync_update(tmp_sync, 1, request->req_status.MPI_ERROR);

[mwheinz]

@ggouaillardet

TL;DR 3285325 has to be backported, I'll issue a PR shortly.

Okay - thanks. That looks like it could be cherry-picked right into 3.1.4?

[mwheinz]

@ggouaillardet - a quick spot check does indicate that patch fixes the problem.

[mwheinz]

A thorough test indicates that cherry-picking patch 3285325 resolved all pt2pt issues I was experiencing.

[mwheinz]

@ggouaillardet Have you opened the issue to backport this? Do you want me to do it?

[ggouaillardet]

I previously issued #6862