Atomics broken on ARMv5

[jsquyres]

Per @pterjan note on fe09355:

This breaks openmpi builds in other cases as it is used in various places:

ompi/mca/osc/portals4/osc_portals4_active_target.c: OPAL_THREAD_ADD64(&module->opcount, 1);
ompi/mca/osc/portals4/osc_portals4_active_target.c: OPAL_THREAD_ADD64(&module->opcount, 1);
ompi/mca/osc/pt2pt/osc_pt2pt_passive_target.c: lock->serial_number = OPAL_THREAD_ADD64((int64_t ) &module->lock_serial_number, 1);
ompi/mca/btl/portals4/btl_portals4.c: frag->segments[0].key = OPAL_THREAD_ADD64(&(portals4_btl->portals_rdma_key), 1);
ompi/mca/btl/portals4/btl_portals4.c: frag->segments[0].key = OPAL_THREAD_ADD64(&(portals4_btl->portals_rdma_key), 1);
ompi/mca/mtl/portals4/mtl_portals4_recv.c: ptl_request->opcount = OPAL_THREAD_ADD64((int64_t) &ompi_mtl_portals4.recv_opcount, 1);
ompi/mca/mtl/portals4/mtl_portals4_send.c: ptl_request->opcount = OPAL_THREAD_ADD64((int64_t*)&ompi_mtl_portals4.opcount, 1);

When compiling on armv5 I get:

/home/iurt/rpmbuild/BUILD/openmpi-1.10.0/ompi/.libs/libmpi.so: undefined reference to `OPAL_THREAD_ADD64'

[hjelmn]

My guess is osc/pt2pt is to blame. I will update it in a bit.

[pterjan]

This is unfortunately only true on arm > v6.
I guess I should disable threads on armv5 as there is no smp support
anyway, a patch would still be needed to allow using the simple version
when OMPI_ENABLE_THREAD_MULTIPLE is not enabled:

--- opal/threads/thread_usage.h 2015-12-15 00:10:58.412181433 +0000
+++ opal/threads/thread_usage.h.thr 2015-12-15 00:10:41.521798534 +0000
@@ -126,8 +126,8 @@
}
#endif

-#if OPAL_HAVE_ATOMIC_MATH_64
#if OMPI_ENABLE_THREAD_MULTIPLE
+#if OPAL_HAVE_ATOMIC_MATH_64
static inline int64_t
OPAL_THREAD_ADD64(volatile int64_t *addr, int delta)
{
@@ -141,6 +141,7 @@

 return ret;

}
+#endif
#else
static inline int64_t
OPAL_THREAD_ADD64(volatile int64_t *addr, int delta)
@@ -150,7 +151,6 @@
return ret;
}
#endif
-#endif

#if OMPI_ENABLE_THREAD_MULTIPLE
static inline size_t

On 3 December 2015 at 19:38, Todd Kordenbrock notifications@github.com
wrote:

I don't have an armv5 to test on, but opal_atomic_cmpset_64 is defined in
ARM.asm which means the fallback implementation of opal_atomic_add_64 in
atomic_impl.h should work. And the definition of OPAL_THREAD_ADD64 in
mutex.h really only needs to be protected by OPAL_HAVE_ATOMIC_ADD_64. So I
think this patch should fix it.

diff --git a/opal/threads/mutex.h b/opal/threads/mutex.h
index d10eb93..b7f761b 100644
--- a/opal/threads/mutex.h
+++ b/opal/threads/mutex.h
@@ -280,7 +280,7 @@ OPAL_THREAD_ADD32(volatile int32_t *addr, int delta)
return ret;
}

-#if OPAL_HAVE_ATOMIC_MATH_64
+#if OPAL_HAVE_ATOMIC_ADD_64
static inline int64_t
OPAL_THREAD_ADD64(volatile int64_t *addr, int delta)
{

—
Reply to this email directly or view it on GitHub
#1162 (comment).

[jsquyres]

Per discussion 24 Feb 2016, @hjelmn will look at this.

[amckinstry]

Is there a pointer to the discussion? I've been debugging this in Debian
(https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=816303) where it breaks mpi4py on armel, mips, mipsel, powerpc (ie. basically any 32-bit platform).

[hjelmn]

It was in a conference call. Planning to fix this today.

[amckinstry]

I've been testing a patch (attached).
Can you review it? appears to work where I've tested it.

regards
Alastair

On 08/03/2016 16:20, Nathan Hjelm wrote:

It was in a conference call. Planning to fix this today.

—
Reply to this email directly or view it on GitHub
#1162 (comment).

Alastair McKinstry, alastair@sceal.ie, mckinstry@debian.org, https://diaspora.sceal.ie/u/amckinstry
Misentropy: doubting that the Universe is becoming more disordered.

[amckinstry]

Apologies for the inline patch; the Debian repo is in subversion :-(

--- openmpi-1.10.2.orig/ompi/mca/btl/portals4/btl_portals4.c
+++ openmpi-1.10.2/ompi/mca/btl/portals4/btl_portals4.c
@@ -319,7 +319,11 @@ mca_btl_portals4_prepare_src(struct mca_

         frag->segments[0].base.seg_len = max_data;
         frag->segments[0].base.seg_addr.pval = iov.iov_base;
+#ifdef OPAL_HAVE_ATOMIC_MATH_64
         frag->segments[0].key = OPAL_THREAD_ADD64(&(portals4_btl->portals_rdma_key), 1);
+#else
+        frag->segments[0].key = OPAL_THREAD_ADD32(&(portals4_btl->portals_rdma_key), 1);
+#endif
         frag->base.des_src_cnt = 1;

         /* either a put or get.  figure out which later */
@@ -405,7 +409,11 @@ mca_btl_portals4_prepare_dst(struct mca_

     frag->segments[0].base.seg_len = *size;
     opal_convertor_get_current_pointer( convertor, (void**)&(frag->segments[0].base.seg_addr.pval) );
+#ifdef OPAL_HAVE_ATOMIC_MATH_64
     frag->segments[0].key = OPAL_THREAD_ADD64(&(portals4_btl->portals_rdma_key), 1);
+#else
+    frag->segments[0].key = OPAL_THREAD_ADD32(&(portals4_btl->portals_rdma_key), 1);
+#endif
     frag->base.des_src = NULL;
     frag->base.des_src_cnt = 0;
     frag->base.des_dst = &frag->segments[0].base;
--- openmpi-1.10.2.orig/ompi/mca/mtl/portals4/mtl_portals4_send.c
+++ openmpi-1.10.2/ompi/mca/mtl/portals4/mtl_portals4_send.c
@@ -411,7 +411,11 @@ ompi_mtl_portals4_send_start(struct mca_
     ret = ompi_mtl_datatype_pack(convertor, &start, &length, &free_after);
     if (OMPI_SUCCESS != ret) return ret;

+#ifdef OPAL_HAVE_ATOMIC_MATH_64
     ptl_request->opcount = OPAL_THREAD_ADD64((int64_t*)&ompi_mtl_portals4.opcount, 1);
+#else
+    ptl_request->opcount = OPAL_THREAD_ADD32((int64_t*)&ompi_mtl_portals4.opcount, 1);
+#endif
     ptl_request->buffer_ptr = (free_after) ? start : NULL;
     ptl_request->event_count = 0;
--- openmpi-1.10.2.orig/ompi/mca/osc/portals4/osc_portals4_active_target.c
+++ openmpi-1.10.2/ompi/mca/osc/portals4/osc_portals4_active_target.c
@@ -102,7 +102,11 @@ ompi_osc_portals4_complete(struct ompi_w
                             PTL_SUM,
                             PTL_INT32_T);
             if (ret != OMPI_SUCCESS) return ret;
+#ifdef OPAL_HAVE_ATOMIC_MATH_64
             OPAL_THREAD_ADD64(&module->opcount, 1);
+#else
+            OPAL_THREAD_ADD32(&module->opcount, 1);
+#endif
         }

         ret = ompi_osc_portals4_complete_all(module);
@@ -151,7 +155,11 @@ ompi_osc_portals4_post(struct ompi_group
                             PTL_SUM,
                             PTL_INT32_T);
             if (ret != OMPI_SUCCESS) return ret;
+#ifdef OPAL_HAVE_ATOMIC_MATH_64
             OPAL_THREAD_ADD64(&module->opcount, 1);
+#else
+            OPAL_THREAD_ADD64(&module->opcount, 1);
+#endif
         }
     } else {
         module->post_group = NULL;
--- openmpi-1.10.2.orig/ompi/mca/osc/pt2pt/osc_pt2pt_passive_target.c
+++ openmpi-1.10.2/ompi/mca/osc/pt2pt/osc_pt2pt_passive_target.c
@@ -331,7 +331,11 @@ static int ompi_osc_pt2pt_lock_internal
     lock->target = target;
     lock->lock_acks_expected = 0;
     lock->unlock_acks_expected = 0;
+#ifdef OPAL_HAVE_ATOMIC_MATH_64
     lock->serial_number = OPAL_THREAD_ADD64((int64_t *) &module->lock_serial_number, 1);
+#else
+    lock->serial_number = OPAL_THREAD_ADD32((int64_t *) &module->lock_serial_number, 1);
+#endif
     lock->type = lock_type;
     lock->assert = assert;

[hjelmn]

Not quite correct, the types have to be changed as well.

[hjelmn]

Will post a PR to master in a bit.

[hjelmn]

@regrant Does portals4 support 32-bit platforms? If not we can leave the portals4 component as is.