OMPI 5.0.8 occasionally segfaults upon start (with analysis and fix)

[dl1ycf]

I have had occasional segfaults of prte when starting jobs, and debugged this. This is may analysis and how I fixed it (the errors are in the prte as bundled with openmpi 5.0.8)

FIRST PROBLEM:

file 3rd-party/prrte/src/mca/plm/base/plm_base_launch_support.c, in function
prte_plm_base_daemon_callback:

The vector prte_node_topologies may be filled with entries t for which t->topo == NULL, e.g.
if only a signature is added to the topology array. This causes a segfault later on when
either prte_hwloc_base_setup_summary() or hwloc_cpuset_t prte_hwloc_base_filter_cpus()
is called with a NULL pointer. A quick-and-dirty fix is to catch this case in the file

3rd-party/prrte/src/hwloc/hwloc_base_util.c

by adding the "catch" at the two of these two functions, that is:

---

hwloc_cpuset_t prte_hwloc_base_setup_summary(hwloc_topology_t topo)
{
if (topo == NULL) {
hwloc_cpuset_t bogus = hwloc_bitmap_alloc();
hwloc_bitmap_fill(bogus);
return bogus;
}

---

hwloc_cpuset_t prte_hwloc_base_filter_cpus(hwloc_topology_t topo)
{
if (topo == NULL) {
hwloc_cpuset_t bogus = hwloc_bitmap_alloc();
hwloc_bitmap_fill(bogus);
return bogus;
}

(the if clauses are added). Note that I do not care much about the return value, since I do not use
CPU pinning. Note that the return value of prte_hwloc_base_setup_summary is mostly not used
anyway (this is a memory leak). This fix at least makes it work for me, it cures a symptom not the
cause.

SECOND PROBLEM:

The vector prte_node_topologies may contain two different entries t which have the same pointer
t->topo. This leads to a segfault in prte_finalize() when the second of these two entries is released.
I had no time to dig into this deeply, an even quicker-and-dirtier fix is to just commented out the
release at the end of

prte_finalize() in file 3rd-party/prrte/src/runtime/prte_finalize.c:

---

for (n = 0; n < prte_node_topologies->size; n++) {
    topo = (prte_topology_t *) pmix_pointer_array_get_item(prte_node_topologies, n);
    if (NULL == topo) {
        continue;
    }
    pmix_pointer_array_set_item(prte_node_topologies, n, NULL);

    //PMIX_RELEASE(topo);
}
PMIX_RELEASE(prte_node_topologies);

---

(Note the line PMIX_RELEASE(topo) is commented out).

I know the "true" solution looks different but this is to share my work in the hope it might be
useful for others.

Yours,

C. van Wüllen

[rhc54]

Part of the problem/frustration at times is the time lag between OMPI releases and changes to the runtime. OMPI doesn't release on RTE fixes - it only releases based on MPI layer changes. Which is why they have the "external RTE" option so people can update that way.

I'll take a gander, but I believe these issues have been resolved several months ago. Got a few other things on the fire, so it may be a bit before I can verify or address it.

[dl1ycf]

What you say is true, and this is why I posted this HERE and not in the prrte repo. I still find this is useful
info for openmpi maintainers and supporters who now may know where some of the "segfault" complaints
may come from.

I agree that the current prrte snapshot in OMPI is buggy. For example, the function prte_hwloc_base_setup_summary()
is used as a void function in ./3rd-party/prrte/src/mca/plm/base/plm_base_launch_support.c, but it allocates
a bitmap (and returns it) in ./3rd-party/prrte/src/hwloc/hwloc_base_util.c which makes the memory leak I was
referring to, and this clearly demonstrates the prte snapshot is not "settled down".

Originally suspected that using a prrte version that is newer than the OMPI version may also cause problems,
therefore I compiles OMPI with the "bundled" snapshot of prrte but this may change in the future.

[rhc54]

where some of the "segfault" complaints may come from.

Haven't been hearing any until now 🤷‍♂️

clearly demonstrates the prte snapshot is not "settled down".

I've never seen any software that is "settled down" - there are bugs in every piece of software I've seen in my 50+ years in the business. Not saying we shouldn't try. Just being realistic about throwing stones.

using a prrte version that is newer than the OMPI version may also cause problems

There is no issue with using newer PRRTE versions than what is bundled - it is the first recommendation given. Not saying there won't be bugs there too - I'm sure there are.

[rhc54]

Doing a quick triage, I find that pretty much everything you mention should have been resolved - in some cases, the fix was committed nearly a year ago. The fix for the "memory leak" you mentioned was considered too disruptive to backport to an existing release branch, and not deemed critical enough to warrant the effort. I very much doubt those decisions will be changed at this point.

Perhaps the next steps should be to:

• have you try using the head of PRRTE's master branch and see what, if any, of these problems persist? Can be fixed looking towards the future since we are already in the PRRTE v4 series, and not much is going back to the v3 series.

• have you provide a little more info on exactly what you are doing that triggers these problems? Not hearing it from anywhere else, so I'm assuming you are doing something a little different, perhaps in a different environment than others are using.

[dl1ycf]

Compiling openpmix and prrte from the master branches, and using that for OMPI 5.0.8, occasionally leads to the error message

[lc01:31954] PMIX ERROR: PMIX_ERR_NOT_FOUND in file state_dvm.c at line 300

and then it hangs. Using the internal libraries is more convenient since I then have all the executables and shared libs in one place. I have four OMPI universes on my machines (gnu and intel fortran, 32 and 64 bit integers) and these are then self-contained packages.

[rhc54]

Could you please tell us what exactly is your environment, and what you are doing? How many nodes are involved, are they homogeneous in topology (this means hardware and allocation), etc?

Your errors are not being reported by anyone else, yet are hitting in fundamental code paths that are always traversed and so would be hit by everyone. So something appears to be rather different about your situation.

[dl1ycf]

The original problem possibly was the "hetero nodes" problem fixed in prrte in May. Though I did the test on four "homogeneous" nodes, they were not really homogeneous since two of them had NUMA disabled in the BIOS. Enabling NUMA there and repeating the test, the NULL ptr problem was gone since a "new" signature is never found in the daemon cache.
The problem did not occur always, it depends on the order other daemons report their signature.

My feeling is that the problem is the same that lead to prte occasionally "hanging" in OMPI 5.0.3 (this is why I moved to 5.0.8), and then I got occasional segfaults instead of occasional hangs.

Perhaps it was all my fault since on my system one should probably set the prte_hetero_nodes MCA param to "true".

Note I have no time to dig into this further. With NULL ptr "catching" as described above, it works perfectly for me
so I won't touch it for some time now. Feel free to close this "issue".

[dl1ycf]

To close this definitely, I can confirm that with setting prte_hetero_nodes, the original problem does not occur and I can also successfully use OpenMPI (5.0.8) with openpmix and prte "externally" compiled from their master branch.

Perhaps failing to set prte_hetero_nodes should not have these consequences, but at least now it seems to be clear what has happened.

[rhc54]

Agreed - it would be better if we could improve the auto-detection of hetero nodes. I'll continue to ponder the issue. It's a question of efficiency vs accuracy - we could just always return the entire topology to check for homogeneity, but that's a large block of data to return from every node. The summary "signature" we compute works for the more common cases, but as you've seen is far from foolproof.

Thanks for your patience!