MPI_Probe crash with messages from large number of ranks on Aurora

[ryanstocks00]

Instructions for reproducing using default modules on Aurora: (confirmed by @colleeneb)

git clone https://github.com/ryanstocks00/DynaMPI.git
git checkout 2ed2c4a8afb1b6a9dc377c22cd9adf1eabec9aaa
./benchmark/aurora/aurora_compile.sh # (Note you will likely need to do this on a login node the first time as it automatically downloads google test dependencies)
qsub -I -l select=16 -l walltime=30:00 -l filesystems=home:flare -A XXX -q debug-scaling NODE_LIST=16 
mpirun -n 1632 --ppn 102 ./build/benchmark/strong_scaling_distribution_rate --expected_us 1000 --distribution naive --nodes 16

This runs a dynamic master-slave distribution (i.e. all ranks send a message to rank 0 requesting a task, and rank 0 receives and sends a "task" back. The worker then spins for expected_us microseconds before requesting another task). This gives a non-deterministic error such as

x4220c5s7b0n0.hsn.cm.aurora.alcf.anl.gov: rank 0 died from signal 6
x4220c6s0b0n0.hsn.cm.aurora.alcf.anl.gov: rank 118 died from signal 15

Sometimes the output will add a bunch of
Abort(15) on node 1328 (rank 1328 in comm 496): Fatal error in internal_Probe: Other MPI error

Interestingly, reducing the duration of the tasks (e.g. --expected_us 1) which should mean more conflicting messages seems to be more likely to run successfully.

The issue goes away if I use export FI_CXI_RX_MATCH_MODE=software, however then the performance seems to be significantly lacking (not sure if this is due to the matching mode or something else) and up to two orders of magnitude slower than Frontier.

May be related to #7427

[hzhou]

@ryanstocks00 When you use MPI_Probe, the messages arrives as "unexpected" since there is no corresponding MPI_Recv to match with. The hardware matching works best when the Recv is pre-posted so it can't immediately produce a match. When the match fails, I think the current cxi strategy is retry then fallback to software (if the match mode is hybrid). It is known that retries exacerbate congestion and may crash the system. I need understand more on how you use MPI_Probe, but my guess is when you shorten the duration of between-task spin, it let the system process messages more promptly and thus reduce the congestion.

Instead of relying on MPI_Probe, how about replace it with MPI_Irecv(MPI_ANY_SOURCE, MPI_ANY_TAG) with a medium-sized buffer. So instead of "probe", you just MPI_Test on the Irecv request. For messages with large data, you can split the message into a header and a payload. The receiver can issue a separate MPI_Recv once it receives the header. Effectively, you are simulating an active message design with anysource receives. You can pre-issue multiple anysource receives to reduce the amount of unexpected messages to alleviate congestion. And you need re-issue the anysource receives after you processed the previous one.

[colleeneb]

Hi Ryan,

We are also trying to understand why the Frontier results are that much faster. It looks like the libfabric version is the same on Frontier and Aurora (1.22.0) at least.

We found that setting export FI_CXI_DEFAULT_CQ_SIZE=131072 even without setting export FI_CXI_RX_MATCH_MODE=software will get your 16-node case to run without crashing. However, I'm not sure about if the performance is better than FI_CXI_RX_MATCH_MODE=software -- the printed value forthroughput_tasks_per_s with FI_CXI_RX_MATCH_MODE=software was higher than the one with FI_CXI_DEFAULT_CQ_SIZE=131072.

It would be interesting to know if HPE MPICH on Frontier is setting different environment variables than what we have on Aurora. From HPE's docs (https://cpe.ext.hpe.com/docs/latest/mpt/mpich9/intro_mpi.html) it looks like HPE MPICH sets FI_CXI_DEFAULT_CQ_SIZE=131072 by default, but I'm not sure if that's the case on Frontier. I don't have access to Frontier to check, and I'm not sure if it is hard-coded into HPE MPICH (meaning that just checking echo $FI_CXI_DEFAULT_CQ_SIZE may not be enough to see what it really is set to, if HPE MPICH is setting the default internally). One thing you could do to check is if you set export FI_CXI_DEFAULT_CQ_SIZE=1024 (this is the default on Aurora) (or even could push it smaller to see, like =128) on Frontier, does it still work there?

Thanks,
Colleen

[ryanstocks00]

Thanks @hzhou that is indeed a better solution and eliminates the need for MPI_Probe (though at present this is just trying to obtain a naive baseline which I believe it is quite common to implement dynamic task distribution using MPI_Probe). I will implement/test that and get back to you.

@colleeneb It seems frontier is not particularly sensitive to FI_CXI_DEFAULT_CQ_SIZE - even if I push it down to just 1 I don't see a difference in performance so perhaps it's just hardcoded to something?

One aspect that might be quite different to Frontier is that on Aurora I am using 102 ranks per node to fully occupy the available cores whilst on Frontier I am using only 56. If I reduce the number of cores per node from 102 to 56 which is a fairer comparison to frontier the time per shutdown goes down from 30ms to 5.5ms (which I would have expected to be linear - is it likely that having 102 ranks they are conflicting in some way?) and the asymptotic throughput gets to within 50% of Frontier. The time per shutdown is 3 ms on frontier with hardware matching for comparison. Interestingly on 16 nodes on Frontier, it actually gets significantly faster if I specify FI_CXI_RX_MATCH_MODE=software and reduces to just 0.5 ms (though there is a less significant ~2x improvement in the asymptotic task throughput). Are there any other environment variables like this you would recommend playing around with?

[colleeneb]

It seems frontier is not particularly sensitive to FI_CXI_DEFAULT_CQ_SIZE - even if I push it down to just 1 I don't see a difference in performance so perhaps it's just hardcoded to something?

Unexpected, but interesting! Indeed also maybe it's related to the larger number of ranks per node like you mentioned, or some issues from not having the ranks bound to cores explicitly, so one thing to try:

• You can try binding the ranks to CPUs with --cpu-bind. So you can try something like

mpirun -n 1632 --ppn 102 --depth 1 --cpu-bind=depth ./build/benchmark/strong_scaling_distribution_rate --expected_us 1000 --distribution naive --nodes 16 

Or for 56 ranks I see something like

> FI_CXI_RX_MATCH_MODE=software mpirun -n 816 --ppn 56 -cpu-bind=list:1:2:3:4:5:6:7:8:9:10:11:12:13:14:15:16:17:18:19:20:21:22:23:24:25:26:27:28:53:54:55:56:57:58:59:60:61:62:63:64:65:66:67:68:69:70:71:72:73:74:75:76:77:78:79:80 ./build/benchmark/strong_scaling_distribution_rate --expected_us 1000 --distribution naive --nodes 16 
RESULT distributor=naive mode=fixed expected_us=1000 nodes=16 world_size=816 total_tasks=5774989 elapsed_s=10.0008 throughput_tasks_per_s=577453

which at least binds each MPI rank to its own CPU (more examples here: https://docs.alcf.anl.gov/aurora/running-jobs-aurora/#mpi-rank-and-thread-binding-to-cores-and-gpus). This is the best result I saw so far on Aurora at least.

Another thing is that each Aurora node has 8 NICs (4 per socket). 56 ranks at least is a multiple, with 28 ranks per socket. 102 is not, so possibly there is an issue with a non-even split of ranks to NICs for 102 ranks.

Are there any other environment variables like this you would recommend playing around with?

There are various values here, which cray mpich is sets by default and can be different from the libfabric default itself: https://cpe.ext.hpe.com/docs/latest/mpt/mpich9/intro_mpi.html#libfabric-environment-variables-for-hpe-slingshot-nic-slingshot-11
From looking at the software match mode we see:

When running in software endpoint mode the ENV variables FI_CXI_REQ_BUF_SIZE and FI_CXI_REQ_BUF_MIN_POSTED are used to control the size and number of the eager request buffers posted to handle incoming unmatched messages. 

So potentially these are two to try as well.

[colleeneb]

Some progress.

It looks like we can see the CXI that are set at runtime by getting a verbose log from libfabric. This can be done by running with FI_LOG_LEVEL=debug FI_LOG_PROV=cxi as in:

FI_LOG_LEVEL=debug FI_LOG_PROV=cxi mpirun -n 12 ./a.out &> mpich_cxi_frontier.out

With this we can see what values libfabric on Frontier is reading which are different from the libfabric defaults in fi_info -e:

> grep read mpich_cxi_frontier.out | grep 910197 
libfabric:910197:1770398333::core:core:fi_param_get_():405<info> read long var mr_cache_max_size=18446744073709551615
libfabric:910197:1770398333::core:core:fi_param_get_():405<info> read long var mr_cache_max_count=524288
libfabric:910197:1770398333::core:core:fi_param_get_():382<info> read string var mr_cache_monitor=kdreg2
libfabric:910197:1770398333::cxi:core:fi_param_get_():400<info> read bool var ats=0
libfabric:910197:1770398333::cxi:core:fi_param_get_():382<info> read string var rx_match_mode=hardware
libfabric:910197:1770398333::cxi:core:fi_param_get_():405<info> read long var rdzv_threshold=16384
libfabric:910197:1770398333::cxi:core:fi_param_get_():405<info> read long var rdzv_eager_size=2048
libfabric:910197:1770398333::cxi:core:fi_param_get_():405<info> read long var oflow_buf_size=12582912
libfabric:910197:1770398333::cxi:core:fi_param_get_():405<info> read long var oflow_buf_count=3
libfabric:910197:1770398333::cxi:core:fi_param_get_():405<info> read long var req_buf_size=12582912
libfabric:910197:1770398333::cxi:core:fi_param_get_():405<info> read long var req_buf_min_posted=6
libfabric:910197:1770398333::cxi:core:fi_param_get_():405<info> read long var req_buf_max_cached=0
libfabric:910197:1770398333::cxi:core:fi_param_get_():405<info> read long var default_cq_size=131072
libfabric:910197:1770398333::cxi:core:fi_param_get_():405<info> read long var default_tx_size=1024

(thanks to @abagusetty for running to get these on frontier)

When I do the same on Aurora, the only non-default (read) setting is

libfabric:47841:1770407044::core:core:fi_param_get_():382<info> read string var provider=cxi,tcp;ofi_rxm

If we set the same envs as Frontier on Aurora with the benchmark, it does not crash:

>  FI_MR_CACHE_MAX_SIZE=18446744073709551615 FI_MR_CACHE_MAX_COUNT=524288 FI_MR_CACHE_MONITOR=kdreg2 FI_CXI_RX_MATCH_MODE=hardware FI_CXI_RDZV_THRESHOLD=16384 FI_CXI_RDZV_EAGER_SIZE=2048 FI_CXI_OFLOW_BUF_SIZE=12582912  FI_CXI_OFLOW_BUF_COUNT=3 FI_CXI_REQ_BUF_SIZE=12582912 FI_CXI_REQ_BUF_MIN_POSTED=6 FI_CXI_REQ_BUF_MAX_CACHED=0 FI_CXI_DEFAULT_CQ_SIZE=131072 FI_CXI_DEFAULT_TX_SIZE=1024 mpirun -n 816 --ppn 56 -cpu-bind=list:1:2:3:4:5:6:7:8:9:10:11:12:13:14:15:16:17:18:19:20:21:22:23:24:25:26:27:28:53:54:55:56:57:58:59:60:61:62:63:64:65:66:67:68:69:70:71:72:73:74:75:76:77:78:79:80 ./build/benchmark/strong_scaling_distribution_rate --expected_us 100 --distribution naive --nodes 16 
RESULT distributor=naive mode=fixed expected_us=100 nodes=16 world_size=816 total_tasks=2935150 elapsed_s=10.0027 throughput_tasks_per_s=293436

With the logs on, we were also able to confirm that FI_CXI_DEFAULT_CQ_SIZE=131072 was essential for hardware mode. In the verbose log for the crashing version we can see right before the crash:

libfabric:70265:1770408254::cxi:core:cxip_evtq_progress():410<warn> x4703c1s0b0n0: H/W Event Queue overflow detected.

so it’s overflowing the HW event queue on Aurora in hardware match mode.

Since it looks like Frontier is defaulting to hardware match mode, it’s interesting that down-grading FI_CXI_DEFAULT_CQ_SIZE didn’t have an effect. Potentially Cray MPICH has a workaround for it.

@ryanstocks00 do you think you could get the output of your code on Frontier with FI_LOG_LEVEL=debug FI_LOG_PROV=cxi to see if any of the CXI variables are different from what we saw with just a hello world?