Benchmark combining MPI and LPF either skips LPF run or returns error

[KADichev]

With attached files, the experiments with LPF do not work:

• with lpf_hook, it seems like the peer list in src/MPI/ibverbs.cpp is empty and no communication is scheduled at all (lpf_put translates to NOP) - example synthetic4.cpp below
• with lpf_exec, an error happens at runtime with the synthetic3.cpp:

build/_deps/lpf-src/build/bin/lpfrun -engine ibverbs -n 2 build/synthetic3 64 100
LPF Backend Error! lpf_exec( LPF_ROOT, LPF_MAX_P, &spmd, args)[srv01:716807] *** Process received signal ***
[srv01:716807] Signal: Aborted (6)
[srv01:716807] Signal code:  (-6)
[srv01:716807] [ 0] linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0x4000304ba7dc]
[srv01:716807] [ 1] /lib/aarch64-linux-gnu/libc.so.6(+0x7f1f0)[0x40003096f1f0]
[srv01:716807] [ 2] /lib/aarch64-linux-gnu/libc.so.6(raise+0x1c)[0x40003092a67c]
[srv01:716807] [ 3] /lib/aarch64-linux-gnu/libc.so.6(abort+0xe4)[0x400030917130]
[srv01:716807] [ 4] build/synthetic3(main+0x130)[0xaaaabdad7ab0]
[srv01:716807] [ 5] /lib/aarch64-linux-gnu/libc.so.6(+0x273fc)[0x4000309173fc]
[srv01:716807] [ 6] /lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0x98)[0x4000309174cc]
[srv01:716807] [ 7] build/synthetic3(_start+0x30)[0xaaaabdad6d30]
[srv01:716807] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node srv01 exited on signal 6 (Aborted).

Compilation as follows:
lpfcxx -engine ibverbs synthetic3.cpp -o synthetic3

Execution as follows:
lpfrun -engine ibverbs synthetic3 64 100

synthetic3.cpp
synthetic4.cpp

[anyzelman]

Thread 1 "a.out" received signal SIGSEGV, Segmentation fault.
0x000055dbd2649188 in spmd (lpf=0x7ffd4e926bc0, pid=2, nprocs=4, args=...) at kiril-benchmark.cpp:69
69	    for (int j=0; j<my_args->iters; j++) {
(gdb) print my_args
$2 = (my_args_t *) 0x0
(gdb) 

It seems like the SPMD section is expecting my_args to be available at all processes, but this requires my_args to be broadcast first -- from the core spec: lpf_exec() will make sure that LPF process 0 will get the contents of the input memory area via a lpf_args_t object as well. [The other processes will have no such guarantee, get NULL as in this example, but in general are invoking UB]

I'll fix this tomorrow and then see if something else breaks down (as this error doesn't seem to match the one reported).

[anyzelman]

After fixing it, I get an unrelated error on ws[03-04] but on srv[01-04] the attached works fine. Changes:

• exec now broadcasts the arguments;
• I removed all reliance on direct MPI calls. It feels risky to me to mix both LPF and MPI in the same code, as LPF talks directly to the ibv driver and probably messes with the state that MPI thinks ibv is in;
• running it with the debug layer also works, and gives as the only warning that some registrations are not freed up. There was a comment about that in the code-- and to clarify, yes, you can deregister without sync, however with the caveat that the removed slot is guaranteed not subject to communication: Deregistration takes effect immediately. Any local or remote communication request using the given memslot in the current superstep is illegal

The attached is the resulting version of synthetic3.cpp

kiril-benchmark.cpp

[anyzelman]

Here is the MPI-only version of synthetic4.cpp, which also works ok. This way the two benchmarks are split into two utilities-- this one for MPI, and the above-attached one for LPF

synthetic4-edited.cpp

[KADichev]

This is not working for me. With the source file kiril-benchmarks.cpp nothing has changed -- I still get no data transfers, and the peer list is still 0 for me.

[KADichev]

All I see is that IBVerbs::put is never ever called, so this is why the peer list is never populated.

[KADichev]

Unrelated, but this line in the initializer list makes no sense:
https://github.com/Algebraic-Programming/LPF/blob/master/src/MPI/ibverbs.cpp#L68

[anyzelman]

Quick reply from holiday (hope this works): did you run it using master or using the zero_engine MR? From: Kiril Dichev ***@***.***> Sent: Monday, 8 September 2025 17:23 To: Algebraic-Programming/LPF ***@***.***> Cc: Albert Jan YZELMAN ***@***.***>; Assign ***@***.***> Subject: Re: [Algebraic-Programming/LPF] Benchmark combining MPI and LPF either skips LPF run or returns error (Issue #62) [https://avatars.githubusercontent.com/u/30658903?s=20&v=4]KADichev left a comment (Algebraic-Programming/LPF#62)<#62 (comment)> This is not working for me. With the source file kiril-benchmarks.cpp nothing has changed -- I still get no data transfers, and the peer list is still 0 for me. — Reply to this email directly, view it on GitHub<#62 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AYG4WZOKLX4OYV5RNFCP4A33RWNLNAVCNFSM6AAAAACE6F475CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTENRWHAZDINZUGI>. You are receiving this because you were assigned.Message ID: ***@***.******@***.***>>

[KADichev]

Hey, enjoy the break :)

I am following the above commands:

lpfcxx -engine ibverbs <>.cpp -o <>

Execution as follows:
lpfrun -engine ibverbs <> 64 100

and relying on the master branch

[KADichev]

For the zero branch and zero engine, I have working versions that work quite well. The beautiful thing about the zero engine is however that it is easy to debug -- lpf_put translates into Zero::put, etc. This is not the case with the master branch and ibverbs engine. If an IBVerbs::put is not called, we need to look deep in the queues being populated, and the IBVerbs::sync call.