profiling with Nsight Sys shows lots of "IB send waits?"

[BenWibking]

Hi,

I've been profiling my AMReX code with Nsight Systems, with --nic-metrics=true. It's a neat feature, but sometimes there are things going on in the network that don't seem to match up with anything else in the timeline. For instance, at the end of a timestep, there are sometimes a large number of "IB send waits" that appear.

What is a "send wait" and what does it mean that I see spikes of ~1e7 of them in the profile view?