Fat SMPs (i.e., systems with nodes containing large numbers of cpus) were failing to start due to connection failures of the opal/pmix support. Root cause was that (a) we were setting the client socket to non-blocking before calling connect, and (b) the server was using the event library to harvest the accepts, and also did the handshake while in that event. So the server would backup beyond the connection backlog limit, and we would fail.

Changing the client to leave its socket as blocking during the connect doesn't solve the problem by itself - you also have to introduce a sleep delay once the backlog is hit to avoid simply machine-gunning your way thru retries. This gets somewhat difficult to adjust as you don't want to unnecessarily prolong startup time.

We've solved this before by adding a listening thread that simply reaps accepts and shoves them into the event library for subsequent processing. This would resolve the problem, but meant yet another daemon-level thread. So I centralized the listening thread support and let multiple elements register listeners on it. Thus, each daemon now has a single listening thread that reaps accepts from multiple sources - for now, the orte/pmix server and the oob/usock support are using it. I'll add in the oob/tcp component later.

This still didn't fully resolve the SMP problem, especially on coprocessor cards (e.g., KNC). Removing the shared memory dstore support helped further improve the behavior - it looks like there is some kind of memory paging issue there that needs further understanding. Given that the shared memory support was about to be lost when I bring over the PMIx integration (until it is restored in that library), it seemed like a reasonable thing to just remove it at this point.

Author: Ralph Castain

opal/mca/dstore/base/dstore_base_frame.c

@@ -1,7 +1,7 @@
 /*
  * Copyright (c) 2010      Cisco Systems, Inc.  All rights reserved. 
  * Copyright (c) 2012-2013 Los Alamos National Security, Inc.  All rights reserved. 
- * Copyright (c) 2014      Intel, Inc. All rights reserved.
+ * Copyright (c) 2014-2015 Intel, Inc. All rights reserved.
  * Copyright (c) 2014-2015 Research Organization for Information Science
  *                         and Technology (RIST). All rights reserved.
  * $COPYRIGHT$
@@ -43,7 +43,6 @@ opal_dstore_base_API_t opal_dstore = {
 opal_dstore_base_t opal_dstore_base = {0};
 
 int opal_dstore_internal = -1;
-int opal_dstore_modex = -1;
 
 static int opal_dstore_base_frame_close(void)
 {

opal/mca/dstore/dstore.h

@@ -43,9 +43,6 @@ BEGIN_C_DECLS
  * as someone figures out how to separate the various
  * datastore channels
  */
-OPAL_DECLSPEC extern int opal_dstore_internal;
-OPAL_DECLSPEC extern int opal_dstore_modex;
-
 OPAL_DECLSPEC extern int opal_dstore_peer;
 OPAL_DECLSPEC extern int opal_dstore_internal;
 OPAL_DECLSPEC extern int opal_dstore_nonpeer;