PSM2 MTL doesn't work for single node jobs

[hppritcha]

PSM2 MTL has problems with initialization, in particular, in the call to psm2_ep_open when runninig on a single node.

The failure signature is

srun -N 1 -n 1 a.out          
opal13.0Assertion failure at psm_ep.c:832: ep->epid != 0

miniFE.x:52223 terminated with signal 6 at PC=2aaaabd835f7 SP=7fffffffb7d8. Backtrace:
/lib64/libc.so.6(gsignal+0x37)[0x2aaaabd835f7]
/lib64/libc.so.6(abort+0x148)[0x2aaaabd84ce8]
/lib64/libpsm2.so.2(+0x11dea)[0x2aaab97c7dea]
/lib64/libpsm2.so.2(+0x10782)[0x2aaab97c6782]
/lib64/libpsm2.so.2(psm2_ep_open+0x3c5)[0x2aaab97c5165]
/opt/openmpi/1.10/intel/lib/openmpi/mca_mtl_psm2.so(ompi_mtl_psm2_module_init+0x196)[0x2aaab95b25e6]
/opt/openmpi/1.10/intel/lib/openmpi/mca_mtl_psm2.so(+0x2a00)[0x2aaab95b2a00]
/opt/openmpi/1.10/intel/lib/libmpi.so.12(ompi_mtl_base_select+0x9d)[0x2aaaaad6148d]
/opt/openmpi/1.10/intel/lib/openmpi/mca_pml_cm.so(+0x3d9b)[0x2aaab8916d9b]
/opt/openmpi/1.10/intel/lib/libmpi.so.12(mca_pml_base_select+0x433)[0x2aaaaad68643]
/opt/openmpi/1.10/intel/lib/libmpi.so.12(ompi_mpi_init+0x6ac)[0x2aaaaad1a06c]
/opt/openmpi/1.10/intel/lib/libmpi.so.12(MPI_Init+0xe4)[0x2aaaaad3b134]
miniFE.x[0x432da2]
miniFE.x[0x404e1a]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x2aaaabd6fb15]
miniFE.x[0x404c69]

This is with v1.10.1 but also shows up in v2.x.

The issue appeared on devel mail list:
https://www.open-mpi.org/community/lists/devel/2016/04/18762.php

[hppritcha]

Okay the problem is that the PSM2 MTL is setting the PSM2_DEVICES env variable to just self,shm
when running on a single node. There seems to be a bug in the PSM2 installed on our systems that causes psm2_ep_connect to timeout for process 0 with PSM2_DEVICES so set. If I comment out the lines in the MTL which set this env. variable the single node jobs run to completion.

Logfile for when PSM2_DEVICES is not set by PSM2 MTL
logfile.works.txt
Logfile for when PSM2_DEVICES is set by PSM2 MTL
logfile.busted.txt

@matcabral

The psm2 related RPMS on our cluster are:

infinipath-psm-devel-3.3-0.g6f42cdb1bb8.2.el7.x86_64
hfi1-psm-0.7-221.ch6.x86_64
hfi1-psm-devel-0.7-221.ch6.x86_64
infinipath-psm-3.3-0.g6f42cdb1bb8.2.el7.x86_64

[hppritcha]

Per request, modinfo hfi1 output for the cluster.
modinfo.txt

The rpms that appear to be relevant to hfi1 installed on the nodes of the cluster are:
opa-hfi1-0-201603141940.ch6.x86_64
hfi1-psm-0.7-221.ch6.x86_64
hfi1-psm-devel-0.7-221.ch6.x86_64
libhfi1-0.2-2.el7.x86_64
kmod-opa-hfi1-0-201603141940.ch6.x86_64
libhfi1-static-0.2-2.el7.x86_64
hfi1-firmware-0.9-31.noarch

[hppritcha]

Here's a description of the changes made for the rpm list above for this DOE cluster. RPMS with "ch6" in the name have been repackaged for these clusters:

opa-hfi1-0-201603141940.ch6.x86_64 and kmod-opa-hfi1 . code is from https://github.com/01org/opa-hfi1 on 201603141940. we run make on it, and package it up as a weak module. no changes from github

hfi1-psm is built from hfi1-psm-0.7-221.src.rpm that Intel provided

{noformat}
rpm -qpi hfi1-psm-0.7-221.src.rpm
Name : hfi1-psm
Version : 0.7
Release : 221
Architecture: x86_64
Install Date: (not installed)
Group : System Environment/Libraries
Size : 516926
License : GPL
Signature : (none)
Source RPM : (none)
Build Date : Mon Feb 8 14:14:06 2016
Build Host : phbldprivrhel7-2.ph.intel.com
Relocations : (not relocatable)
URL : http://www.intel.com/
Summary : Intel PSM Libraries
Description :
The PSM Messaging API, or PSM API, is Intel's low-level
user-level communications interface for the Truescale
family of products. PSM users are enabled with mechanisms
necessary to implement higher level communications
interfaces in parallel environments.
{noformat}

only change to that rpm is in the spec file, I add %{?dist} to the Release, and added opa-hfi1 to the BuildRequires

[hppritcha]

@matcabral in case you didn't get email about these updates.

[hppritcha]

I was informed customer wants this problem resolved for 1.10.3. The clusters will be put in to production soon and a functional Open MPI 1.10.X is necessary.

[rhc54]

Just scratching my head here for a moment - if you are concerned about the PSM2 MTL, then shouldn't you need the psm2 library rpm? You are only showing the "psm" rpms above.

[rhc54]

Just FWIW: here's a list of the missing rpm's from your system (based on my CentOS7 system):

libpsm2.x86_64                          0.7-4.el7                      @base    
libpsm2-compat.x86_64                   0.7-4.el7                      base     
libpsm2-compat-devel.x86_64             0.7-4.el7                      base     
libpsm2-devel.x86_64                    0.7-4.el7                      base     

[hppritcha]

Ralph,

that information was provided near the beginning of this email thread.
It is also included in the issue 1559. LLNL did some repackaging.

Howard

2016-04-21 10:31 GMT-06:00 rhc54 notifications@github.com:

Just FWIW: here's a list of the missing rpm's from your system (based on
my CentOS7 system):

libpsm2.x86_64 0.7-4.el7 @base
libpsm2-compat.x86_64 0.7-4.el7 base
libpsm2-compat-devel.x86_64 0.7-4.el7 base
libpsm2-devel.x86_64 0.7-4.el7 base

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#1559 (comment)

[matcabral]

@rhc54 it seems there are different names of the RPMs according to to who creates them. The names that @hppritcha provided above (hfi-psm*.rpm) are valid ones and are the ones distributed with intel IFS package.

@hppritcha merging the ompi-devel email thread here, I see you are reproducing two different issues:
a) epid = 0 assert. which I can reproduce and @rhc54 started trying a patch, so we have a root cause and need to polish the fix.
b) in psm2_ep_connect() hang. Which might be related, but still cannot reproduce. Now, I'm looking to have the same driver version. I'll share updates as I have them.

Thanks,

[hppritcha]

Correct about the two issues. Its not clear that they are related at this point.

[hppritcha]

If I simply if def out the PSM2_DEVICES setting code in the PSM2 MTL, all problems with single node jobs vanish, both the hang for multi process cases and the assert for single process case.