How to improve the performance of small GETs on CUDA buffers?

[lyu]

Hi, I'm testing UCX PUTs/GETs on CUDA memory on ORNL Summit, and I'm seeing very low performance for small GETs.

I'm using UCX trunk edacb52, configured with --enable-mt --enable-cma --enable-numa --enable-silent-rules --enable-optimizations --enable-builtin-memcpy --enable-compiler-opt=3 --enable-option-checking --disable-debug --disable-gtest --disable-stats --disable-static --disable-tuning --disable-logging --disable-examples --disable-profiling --disable-assertions --disable-debug-data --disable-params-check --disable-fault-injection --disable-frame-pointer --disable-dependency-tracking --without-go --without-bfd --without-mpi --without-knem --without-java --without-valgrind --with-cache-line-size=128 --with-cuda=/sw/summit/cuda/11.0.3 --with-gdrcopy=/sw/summit/gdrcopy/2.0.

For environment variable, I have set:

UCX_UNIFIED_MODE=y
UCX_MEM_CUDA_HOOK_MODE=reloc
UCX_PROTO_ENABLE=y

With the setup above, I get the following inter-node numbers from ucx_perftest -t ucp_get:

+--------------+--------------+------------------------------+---------------------+-----------------------+
|              |              |       overhead (usec)        |   bandwidth (MB/s)  |  message rate (msg/s) |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
|     Test     | # iterations | 50.0%ile | average | overall |  average |  overall |  average  |  overall  |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
 size=2^00              1000   1064.709  1091.816  1042.950        0.00       0.00         916         959
 size=2^01              1000   1064.734  1090.869  1043.085        0.00       0.00         917         959
 size=2^02              1000   1064.746  1086.616  1051.286        0.00       0.00         920         951
 size=2^03              1000   1064.738  1085.797  1053.508        0.01       0.01         921         949
 size=2^04              1000   1064.787  1086.555  1051.327        0.01       0.01         920         951
 size=2^05              1000   1064.732  1086.126  1052.763        0.03       0.03         921         950
 size=2^06              1000   1064.795  1086.041  1053.084        0.06       0.06         921         950
 size=2^07              1000      3.732     3.744     3.744       32.61      32.61      267102      267102
 size=2^08              1000      3.768     3.776     3.776       64.66      64.66      264842      264842
 size=2^09              1000   1064.715  1087.595  1051.868        0.45       0.46         919         951
 size=2^10              1000   1064.785  1085.300  1054.897        0.90       0.93         921         948

If I add UCX_ZCOPY_THRESH=0 to the environment, I get:

+--------------+--------------+------------------------------+---------------------+-----------------------+
|              |              |       overhead (usec)        |   bandwidth (MB/s)  |  message rate (msg/s) |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
|     Test     | # iterations | 50.0%ile | average | overall |  average |  overall |  average  |  overall  |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
 size=2^00              1000   1064.812  1091.706  1041.900        0.00       0.00         916         960
 size=2^01              1000   1064.852  1085.267  1053.151        0.00       0.00         921         950
 size=2^02              1000   1064.811  1087.108  1051.748        0.00       0.00         920         951
 size=2^03              1000   1064.812  1091.951  1043.279        0.01       0.01         916         959
 size=2^04              1000   1064.781  1086.414  1052.607        0.01       0.01         920         950
 size=2^05              1000   1064.812  1091.902  1043.043        0.03       0.03         916         959
 size=2^06              1000   1064.762  1085.858  1054.326        0.06       0.06         921         948
 size=2^07              1000      3.693     3.703     3.703       32.97      32.97      270060      270060
 size=2^08              1000      3.756     3.761     3.761       64.91      64.91      265883      265883
 size=2^09              1000      3.807     3.811     3.811      128.13     128.13      262406      262406
 size=2^10              1000      3.941     3.964     3.964      246.36     246.36      252274      252274

Adding UCX_PROTO_INFO=y to the environment shows that software emulation (am bcopy?) is used for GET sizes of 0..64, but I cannot find a way to change this behavior or improve its performance. What's the correct setting for Summit?

Any help is appreciated!

[MattBBaker]

We did a lot of work with UCX and dask/ucx-py on summit here: rapidsai/ucx-py#616

I lost my summit access so I can't help directly anymore.