mpirun is terminated saying stack smashing detected when hostname is long

[ktns]

Background information

What version of Open MPI are you using? (e.g., v4.1.6, v5.0.1, git branch name and hash, etc.)

both

• 4.1.6-7ubuntu2 on ubuntu 24.04
• 4.1.8 built from source on ubuntu 24.04

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

• apt-get install
• built from source tarball

If you are building/installing from a git clone, please copy-n-paste the output from git submodule status.

Please describe the system on which you are running

• Operating system/version: ubuntu 24.04
• Computer hardware: Intel(R) Xeon(R) Gold 6254 / ThinkPad X1 nano gen1
• Network type: n/a

---

Details of the problem

When hostname is longer than 56 characters, mpirun is immediately terminated saying

*** stack smashing detected ***: terminated

I found it in a k8s pod with openmpi 4.1.8 built from source tarball, but I can reproduce it with 4.1.6-ubuntu2 installed via apt-get on ubuntu 24.04 in a docker container.

docker run --rm -h dpzwksxgqlelrdfraudrnnjnpwvsfxgeqozkukzhzlkzfyfotdsuflkja ubuntu:24.04 bash -c 'apt-get update ; apt-get install openmpi-bin -y ; mpirun --allow-run-as-root -np 2 true'
....
*** stack smashing detected ***: terminated
[dpzwksxgqlelrdfraudrnnjnpwvsfxgeqozkukzhzlkzfyfotdsuflkja:00001] *** Process received signal ***
[dpzwksxgqlelrdfraudrnnjnpwvsfxgeqozkukzhzlkzfyfotdsuflkja:00001] Signal: Segmentation fault (11)
[dpzwksxgqlelrdfraudrnnjnpwvsfxgeqozkukzhzlkzfyfotdsuflkja:00001] Signal code:  (128)
[dpzwksxgqlelrdfraudrnnjnpwvsfxgeqozkukzhzlkzfyfotdsuflkja:00001] Failing at address: (nil)
[dpzwksxgqlelrdfraudrnnjnpwvsfxgeqozkukzhzlkzfyfotdsuflkja:00001] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x45330)[0x7b6da6245330]
[dpzwksxgqlelrdfraudrnnjnpwvsfxgeqozkukzhzlkzfyfotdsuflkja:00001] [ 1] /lib/x86_64-linux-gnu/libc.so.6(abort+0x182)[0x7b6da62289a2]
[dpzwksxgqlelrdfraudrnnjnpwvsfxgeqozkukzhzlkzfyfotdsuflkja:00001] [ 2] /lib/x86_64-linux-gnu/libc.so.6(+0x297b6)[0x7b6da62297b6]
[dpzwksxgqlelrdfraudrnnjnpwvsfxgeqozkukzhzlkzfyfotdsuflkja:00001] [ 3] /lib/x86_64-linux-gnu/libc.so.6(+0x136c19)[0x7b6da6336c19]
[dpzwksxgqlelrdfraudrnnjnpwvsfxgeqozkukzhzlkzfyfotdsuflkja:00001] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x137ea4)[0x7b6da6337ea4]
[dpzwksxgqlelrdfraudrnnjnpwvsfxgeqozkukzhzlkzfyfotdsuflkja:00001] [ 5] /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_regx_fwd.so(+0x2404)[0x7b6da3b40404]
[dpzwksxgqlelrdfraudrnnjnpwvsfxgeqozkukzhzlkzfyfotdsuflkja:00001] [ 6] /lib/x86_64-linux-gnu/libopen-rte.so.40(orte_odls_base_default_get_add_procs_data+0x156)[0x7b6da659fce6]
[dpzwksxgqlelrdfraudrnnjnpwvsfxgeqozkukzhzlkzfyfotdsuflkja:00001] [ 7] /lib/x86_64-linux-gnu/libopen-rte.so.40(orte_plm_base_launch_apps+0x1aa)[0x7b6da65a9bba]
[dpzwksxgqlelrdfraudrnnjnpwvsfxgeqozkukzhzlkzfyfotdsuflkja:00001] [ 8] /lib/x86_64-linux-gnu/libevent_core-2.1.so.7(+0x1f2a8)[0x7b6da646e2a8]
[dpzwksxgqlelrdfraudrnnjnpwvsfxgeqozkukzhzlkzfyfotdsuflkja:00001] [ 9] /lib/x86_64-linux-gnu/libevent_core-2.1.so.7(event_base_loop+0x4af)[0x7b6da646ffaf]
[dpzwksxgqlelrdfraudrnnjnpwvsfxgeqozkukzhzlkzfyfotdsuflkja:00001] [10] mpirun(+0x128c)[0x581b3fb3028c]
[dpzwksxgqlelrdfraudrnnjnpwvsfxgeqozkukzhzlkzfyfotdsuflkja:00001] [11] /lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca)[0x7b6da622a1ca]
[dpzwksxgqlelrdfraudrnnjnpwvsfxgeqozkukzhzlkzfyfotdsuflkja:00001] [12] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b)[0x7b6da622a28b]
[dpzwksxgqlelrdfraudrnnjnpwvsfxgeqozkukzhzlkzfyfotdsuflkja:00001] [13] mpirun(+0x1415)[0x581b3fb30415]
[dpzwksxgqlelrdfraudrnnjnpwvsfxgeqozkukzhzlkzfyfotdsuflkja:00001] *** End of error message ***

[rhc54]

Can you build this with --enable-debug and get a real stack trace?

[ktns]

Built from v4.1.8 tarball with

./configure --prefix=/usr --enable-debug

and got the below.

#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x0000750192c4527e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x0000750192c2898f in __GI_abort () at ./stdlib/abort.c:100
#5  0x0000750192c297b6 in __libc_message_impl (fmt=fmt@entry=0x750192dce765 "*** %s ***: terminated\n") at ../sysdeps/posix/libc_fatal.c:134
#6  0x0000750192d36c19 in __GI___fortify_fail (msg=msg@entry=0x750192dce77d "stack smashing detected") at ./debug/fortify_fail.c:24
#7  0x0000750192d37ea4 in __stack_chk_fail () at ./debug/stack_chk_fail.c:24
#8  0x000075019268d0a2 in nidmap_create (pool=0x5eeb9f6e43c0, regex=0x7ffe0fcec348) at regx_fwd.c:304
#9  0x00007501930c6822 in orte_odls_base_default_get_add_procs_data (buffer=0x5eeb9f79f4a0, job=1908277249) at base/odls_base_default_fns.c:172
#10 0x00007501930db934 in orte_plm_base_launch_apps (fd=-1, args=4, cbdata=0x5eeb9f79bdc0) at base/plm_base_launch_support.c:553
#11 0x0000750192f142a8 in ?? () from /lib/x86_64-linux-gnu/libevent_core-2.1.so.7
#12 0x0000750192f15faf in event_base_loop () from /lib/x86_64-linux-gnu/libevent_core-2.1.so.7
#13 0x00005eeb8a6128bd in orterun (argc=5, argv=0x7ffe0fcec8a8) at orterun.c:179
#14 0x00005eeb8a61234d in main (argc=5, argv=0x7ffe0fcec8a8) at main.c:13

Is there anything I can add?

[rhc54]

Pretty trivial fix, but I believe the 4.x series is basically closed at this point. You shouldn't encounter this with the OMPI v5 series, so one solution would be to upgrade. Alternatively, if you want to stick with OMPI v4, then add --mca regx naive to your mpirun cmd line and you should avoid the problem.

[ktns]

Thanks, I'm using a proprietary software that requires 4.1.x so I'll try --mca regx naive.