Petsc test failing: possible MPI_REQUEST_FREE issue

[jsquyres]

According to Eric Chamberland in http://www.open-mpi.org/community/lists/devel/2016/07/19210.php, he's getting a failure in a petsc test. Here's the backtrace:

*** Error in `/pmi/cmpbib/compilation_BIB_dernier_ompi/COMPILE_AUTO/GIREF/bin/Test.ProblemeGD.opt': free(): invalid pointer: 0x00007f9ab09c6020 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7277f)[0x7f9ab019b77f]
/lib64/libc.so.6(+0x78026)[0x7f9ab01a1026]
/lib64/libc.so.6(+0x78d53)[0x7f9ab01a1d53]
/opt/openmpi-2.x_opt/lib/openmpi/mca_pml_ob1.so(+0x172a1)[0x7f9aa3df32a1]
/opt/openmpi-2.x_opt/lib/libmpi.so.0(MPI_Request_free+0x4c)[0x7f9ab0761dac]
/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(+0x4adaf9)[0x7f9ab7fa2af9]
/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(VecScatterDestroy+0x68d)[0x7f9ab7f9dc35]
/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(+0x4574e7)[0x7f9ab7f4c4e7]
/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(VecDestroy+0x648)[0x7f9ab7ef28ca]
/pmi/cmpbib/compilation_BIB_dernier_ompi/COMPILE_AUTO/GIREF/lib/libgiref_opt_Petsc.so(_Z15GIREFVecDestroyRP6_p_Vec+0xe)[0x7f9abc9746de]
/pmi/cmpbib/compilation_BIB_dernier_ompi/COMPILE_AUTO/GIREF/lib/libgiref_opt_Petsc.so(_ZN12VecteurPETScD1Ev+0x31)[0x7f9abca8bfa1]
/pmi/cmpbib/compilation_BIB_dernier_ompi/COMPILE_AUTO/GIREF/lib/libgiref_opt_Petsc.so(_ZN10SolveurGCPD2Ev+0x20c)[0x7f9abc9a013c]
/pmi/cmpbib/compilation_BIB_dernier_ompi/COMPILE_AUTO/GIREF/lib/libgiref_opt_Petsc.so(_ZN10SolveurGCPD0Ev+0x9)[0x7f9abc9a01f9]
/pmi/cmpbib/compilation_BIB_dernier_ompi/COMPILE_AUTO/GIREF/lib/libgiref_opt_Formulation.so(_ZN10ProblemeGDD2Ev+0x42)[0x7f9abeeb94e2]
/pmi/cmpbib/compilation_BIB_dernier_ompi/COMPILE_AUTO/GIREF/bin/Test.ProblemeGD.opt[0x4159b9]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f9ab014ab25]
/pmi/cmpbib/compilation_BIB_dernier_ompi/COMPILE_AUTO/GIREF/bin/Test.ProblemeGD.opt[0x4084dc]

@hjelmn @bosilca Could you have a look?

[ericch1]

Ok, thanks for your reply.

I will report bugs I have here for the 2.0x master branch (from ompi.git repo)... Maybe when things will stable, I should return to the ompi-release.git repo?

Thanks,
Eric

[jsquyres]

@ericch1 Just to be clear on our versioning system:

1. The ompi repo contains our master development branch. It is currently marked as version 3.0.0a1.
2. The ompi-release repo contains all of our release branches (e.g., v2.x, ...etc.).

We did it this way because github didn't used to have per-branch ACLs. Now they do, and we anticipate merging ompi-release back into ompi sometime soon. However, we're also in the middle of moving all of our infrastructure from one hosting provider to another, and that has a deadline associated with it, so merging ompi-release back into ompi may take a back seat for a little while.

If we can figure out this problem on master and merge the fix over to the release branch for the v2.0.1 release, that would be great. I know @hjelmn was poking into this today.

[ericch1]

Hi Jeff,

please have a look at the following exchange with Matthew Knepley from PETSc:

http://lists.mcs.anl.gov/pipermail/petsc-users/2016-July/029911.html

In this report, I used the ompi-release/v2.x branch... But it looks like the same MPI_Request_free problem...

Eric

[ggouaillardet]

@ericch1 per the message on the ML

On Mon, Jul 25, 2016 at 12:44 PM, Eric Chamberland <
Eric.Chamberland at giref.ulaval.ca> wrote:

> Ok,
>
> here is the 2 points answered:
>
> #1) got valgrind output... here is the fatal free operation:
>

Okay, this is not the MatMult scatter, this is for local representations of
ghosted vectors. However, to me
it looks like OpenMPI mistakenly frees its built-in type for MPI_DOUBLE.


> ==107156== Invalid free() / delete / delete[] / realloc()
> ==107156==    at 0x4C2A37C: free (in
> /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
> ==107156==    by 0x1E63CD5F: opal_free (malloc.c:184)
> ==107156==    by 0x27622627: mca_pml_ob1_recv_request_fini
> (pml_ob1_recvreq.h:133)
> ==107156==    by 0x27622C4F: mca_pml_ob1_recv_request_free
> (pml_ob1_recvreq.c:90)
> ==107156==    by 0x1D3EF9DC: ompi_request_free (request.h:362)
> ==107156==    by 0x1D3EFAD5: PMPI_Request_free (prequest_free.c:59)
> ==107156==    by 0x14AE3B9C: VecScatterDestroy_PtoP (vpscat.c:219)
> ==107156==    by 0x14ADEB74: VecScatterDestroy (vscat.c:1860)
> ==107156==    by 0x14A8D426: VecDestroy_MPI (pdvec.c:25)
> ==107156==    by 0x14A33809: VecDestroy (vector.c:432)
> ==107156==    by 0x10A2A5AB: GIREFVecDestroy(_p_Vec*&)
> (girefConfigurationPETSc.h:115)
> ==107156==    by 0x10BA9F14: VecteurPETSc::detruitObjetPETSc()
> (VecteurPETSc.cc:2292)
> ==107156==    by 0x10BA9D0D: VecteurPETSc::~VecteurPETSc()
> (VecteurPETSc.cc:287)
> ==107156==    by 0x10BA9F48: VecteurPETSc::~VecteurPETSc()
> (VecteurPETSc.cc:281)
> ==107156==    by 0x1135A57B: PPReactionsAppuiEL3D::~PPReactionsAppuiEL3D()
> (PPReactionsAppuiEL3D.cc:216)
> ==107156==    by 0xCD9A1EA: ProblemeGD::~ProblemeGD() (in
> /home/mefpp_ericc/depots_prepush/GIREF/lib/libgiref_dev_Formulation.so)
> ==107156==    by 0x435702: main (Test.ProblemeGD.icc:381)
> ==107156==  Address 0x1d6acbc0 is 0 bytes inside data symbol
> "ompi_mpi_double"
> --107156-- REDIR: 0x1dda2680 (libc.so.6:__GI_stpcpy) redirected to
> 0x4c2f330 (__GI_stpcpy)
> ==107156==
> ==107156== Process terminating with default action of signal 6 (SIGABRT):
> dumping core
> ==107156==    at 0x1DD520C7: raise (in /lib64/libc-2.19.so)
> ==107156==    by 0x1DD53534: abort (in /lib64/libc-2.19.so)
> ==107156==    by 0x1DD4B145: __assert_fail_base (in /lib64/libc-2.19.so)
> ==107156==    by 0x1DD4B1F1: __assert_fail (in /lib64/libc-2.19.so)
> ==107156==    by 0x27626D12: mca_pml_ob1_send_request_fini
> (pml_ob1_sendreq.h:221)
> ==107156==    by 0x276274C9: mca_pml_ob1_send_request_free
> (pml_ob1_sendreq.c:117)
> ==107156==    by 0x1D3EF9DC: ompi_request_free (request.h:362)
> ==107156==    by 0x1D3EFAD5: PMPI_Request_free (prequest_free.c:59)
> ==107156==    by 0x14AE3C3C: VecScatterDestroy_PtoP (vpscat.c:225)
> ==107156==    by 0x14ADEB74: VecScatterDestroy (vscat.c:1860)
> ==107156==    by 0x14A8D426: VecDestroy_MPI (pdvec.c:25)
> ==107156==    by 0x14A33809: VecDestroy (vector.c:432)
> ==107156==    by 0x10A2A5AB: GIREFVecDestroy(_p_Vec*&)
> (girefConfigurationPETSc.h:115)
> ==107156==    by 0x10BA9F14: VecteurPETSc::detruitObjetPETSc()
> (VecteurPETSc.cc:2292)
> ==107156==    by 0x10BA9D0D: VecteurPETSc::~VecteurPETSc()
> (VecteurPETSc.cc:287)
> ==107156==    by 0x10BA9F48: VecteurPETSc::~VecteurPETSc()
> (VecteurPETSc.cc:281)
> ==107156==    by 0x1135A57B: PPReactionsAppuiEL3D::~PPReactionsAppuiEL3D()
> (PPReactionsAppuiEL3D.cc:216)
> ==107156==    by 0xCD9A1EA: ProblemeGD::~ProblemeGD() (in

do you have any idea on how the MPI_Request was created ?
e.g. MPI_Send_init, MPI_Isend, MPI_Ibcast or other ?

[knepley]

MPI_Isend() and MPI_Irecv()

[ggouaillardet]

got it, thanks !
I can try to build a simple reproducer with that.

btw, can you confirm the destructor is invoked before MPI_Finalize() ?

[ericch1]

Yes, I can confirm this:

yes the destructor is invoked before MPI_Finalize...

Eric

[hjelmn]

Are you sure about that? From what I can tell the to->requests and from->requests arrays only hold requests allocated through MPI_*_init.

-Nathan

On Jul 26, 2016, at 08:12 AM, Matthew Knepley notifications@github.com wrote:

MPI_Isend() and MPI_Irecv()
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[ericch1]

100% sure: no more, no less.

Eric

[knepley]

Nathan,

I think you are right, that I was wrong. The Isend and Irecv seem to be
just calculating setup information. Then the
stored pattern is setup by the _init calls. Here is a pointer to the code
https://bitbucket.org/petsc/petsc/src/e99b8dcdee95b77e1530d3c4d61c134af21db400/src/vec/vec/utils/vpscat.c?at=master&fileviewer=file-view-default#vpscat.c-2654

Sorry about that.

Thanks,

 Matt

On Tue, Jul 26, 2016 at 7:58 AM, Eric Chamberland notifications@github.com
wrote:

100% sure: no more, no less.

Eric

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#1875 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAjoiWVdUCc-_TZtlEDOf6Z0DkWUHC01ks5qZiClgaJpZM4JMi_z
.

What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

[hjelmn]

We did fix a leak in MPI_Start/MPI_Startall in 2.0.0. The initial request passed to MPI_Start/MPI_Startall was being leaked. This was leaving an extra retain on both the communicator and datatype. Its very possible this bug is in Open MPI and has been for many years but was masked by the leak. It will probably take us at least a couple of days to figure it out.

-Nathan

On Jul 26, 2016, at 09:44 AM, Matthew Knepley notifications@github.com wrote:

Nathan,

I think you are right, that I was wrong. The Isend and Irecv seem to be
just calculating setup information. Then the
stored pattern is setup by the _init calls. Here is a pointer to the code
https://bitbucket.org/petsc/petsc/src/e99b8dcdee95b77e1530d3c4d61c134af21db400/src/vec/vec/utils/vpscat.c?at=master&fileviewer=file-view-default#vpscat.c-2654

Sorry about that.

Thanks,

Matt

On Tue, Jul 26, 2016 at 7:58 AM, Eric Chamberland notifications@github.com
wrote:

100% sure: no more, no less.

Eric

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#1875 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAjoiWVdUCc-_TZtlEDOf6Z0DkWUHC01ks5qZiClgaJpZM4JMi_z
.

What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[ggouaillardet]

i tried to be creative but was unable to build a reproducer.
what i found is a memory leak
if we MPI_Recv_init(...) followed by MPI_Request_free(...), then the datatype is not OBJ_RELEASE'd
(and i guess the same thing happen for the communicator)

at this stage, i cannot exclude the extra RELEASE or the missing RETAIN does not occur "somewhere else" in the code, and the crash in the destructor is just a consequence of that.

note it is possible to mpirun --mca mpi_param_check 1 to be 100% sure the destructor is not invoked after MPI_Finalize().