Can't force SHARP with score map tuning

[james-sungjae-lee]

Hi,

I’m currently testing SHARP in UCC. I’m using HPC-X v2.23, which includes UCC v1.4.4. With HPC-X, my software stack is OSU Micro-Benchmarks, Open MPI v4.1.7, UCC v1.4.4, and UCX v1.19.0, running on a cluster with SHARP-enabled switch.

My goal is to measure pure SHARP performance for Allreduce, Allgather, Reduce-Scatter, and Broadcast. I believe all of these operations have SHARP implementations in UCC.

I tried to force SHARP for all message sizes, but I found that it always uses its internal score map rather than the SHARP backend. For Allreduce and Broadcast, it follows the score map with a 4 KB threshold between UCP and SHARP. For Allgather and Reduce-Scatter, even though the score map threshold is 16 KB, it still seems to always use the UCP backend.

I’m wondering whether this is because my SHARP-forcing settings are incorrect, or because UCC internally decides between UCP and SHARP (or uses only UCP) based on its own selection logic.

I’ve attached my benchmark code and the Allgather output log.

HPC-X setup

export HPCX_HOME=$PWD
source $HPCX_HOME/hpcx-init.sh
hpcx_load

Benchmark Code

I used UCC_TL_SHARP_TUNE env var, but the score map was not changed.

mpirun -np $1 \
    --bind-to core \
    --map-by ppr:1:node \
    --hostfile ./hostfile-2 \
    -x LD_LIBRARY_PATH \
    -x SHARP_COLL_ENABLE_SAT=1 \
    -x SHARP_COLL_LOG_LEVEL=3 \
    -x UCC_CLS=basic \
    -x UCC_TL_SHARP_TUNE=allgather:inf \
    -x UCC_TL_SHARP_DEVICES=mlx5_0 \
    -x UCC_LOG_LEVEL=INFO \
    -x UCC_COLL_TRACE=INFO \
    -x OMPI_UCC_CL_BASIC_TLS=ucp,sharp \
    $HPCX_OSU_DIR/osu_allgather -x 20 -i 200 -m 4:8388608 > log.out

output - score map

You can see TL_SHARP is selected for {16K..inf}

[1765938844.536867] [-:20594:0]        ucc_team.c:471  UCC  INFO  ===== COLL_SCORE_MAP (team_id 32768, size 16) =====
[1765938844.536882] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  Allgather:
[1765938844.536882] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	Host: {0..4095}:TL_UCP:10 {4K..16383}:TL_UCP:10 {16K..inf}:TL_SHARP:10 
[1765938844.536882] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	Cuda: {16K..inf}:TL_SHARP:10 
[1765938844.536882] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	CudaManaged: {16K..inf}:TL_SHARP:10 
[1765938844.536882] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	Rocm: {16K..inf}:TL_SHARP:10 
[1765938844.536882] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	RocmManaged: {16K..inf}:TL_SHARP:10 
[1765938844.536907] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  Allreduce:
[1765938844.536907] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	Host: {0..4094}:CL_HIER:50 {4095..4K}:CL_HIER:50 {4097..inf}:TL_SHARP:10 
[1765938844.536907] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	Cuda: {0..inf}:TL_SHARP:10 
[1765938844.536907] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	CudaManaged: {0..inf}:TL_SHARP:10 
[1765938844.536907] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	Rocm: {0..4K}:CL_HIER:50 {4097..inf}:TL_SHARP:10 
[1765938844.536907] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	RocmManaged: {0..4K}:CL_HIER:50 {4097..inf}:TL_SHARP:10 
[1765938844.536924] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  Alltoall:
[1765938844.536924] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	Host: {0..2063}:TL_UCP:10 {2064..4095}:TL_UCP:10 {4K..inf}:TL_UCP:10 
[1765938844.536930] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  Alltoallv:
[1765938844.536930] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	Host: {0..inf}:TL_UCP:10 
[1765938844.536936] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  Barrier:
[1765938844.536936] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	Host: {0..inf}:CL_HIER:50 
[1765938844.536936] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	Cuda: {0..inf}:TL_SHARP:10 
[1765938844.536936] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	CudaManaged: {0..inf}:TL_SHARP:10 
[1765938844.536936] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	Rocm: {0..inf}:TL_SHARP:10 
[1765938844.536936] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	RocmManaged: {0..inf}:TL_SHARP:10 
[1765938844.536956] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  Bcast:
[1765938844.536956] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	Host: {0..4K}:CL_HIER:50 {4097..32767}:TL_SHARP:10 {32K..inf}:TL_SHARP:10 
[1765938844.536956] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	Cuda: {0..inf}:TL_SHARP:10 
[1765938844.536956] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	CudaManaged: {0..inf}:TL_SHARP:10 
[1765938844.536956] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	Rocm: {0..4K}:CL_HIER:50 {4097..inf}:TL_SHARP:10 
[1765938844.536956] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	RocmManaged: {0..4K}:CL_HIER:50 {4097..inf}:TL_SHARP:10 
[1765938844.536971] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  Fanin:
[1765938844.536971] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	Host: {0..inf}:TL_UCP:10 
[1765938844.536978] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  Fanout:
[1765938844.536978] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	Host: {0..inf}:TL_UCP:10 
[1765938844.536983] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  Gather:
[1765938844.536983] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	Host: {0..inf}:TL_UCP:10 
[1765938844.536990] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  Gatherv:
[1765938844.536990] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	Host: {0..inf}:TL_UCP:10 
[1765938844.536999] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  Reduce:
[1765938844.536999] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	Host: {0..4K}:CL_HIER:50 {4097..32767}:TL_UCP:10 {32K..inf}:TL_UCP:10 
[1765938844.536999] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	Cuda: {0..4K}:CL_HIER:50 
[1765938844.536999] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	CudaManaged: {0..4K}:CL_HIER:50 
[1765938844.536999] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	Rocm: {0..4K}:CL_HIER:50 
[1765938844.536999] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	RocmManaged: {0..4K}:CL_HIER:50 
[1765938844.537017] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  Reduce_scatter:
[1765938844.537017] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	Host: {0..16383}:TL_UCP:10 {16K..inf}:TL_SHARP:10 
[1765938844.537017] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	Cuda: {16K..inf}:TL_SHARP:10 
[1765938844.537017] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	CudaManaged: {16K..inf}:TL_SHARP:10 
[1765938844.537017] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	Rocm: {16K..inf}:TL_SHARP:10 
[1765938844.537017] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	RocmManaged: {16K..inf}:TL_SHARP:10 
[1765938844.537032] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  Reduce_scatterv:
[1765938844.537032] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	Host: {0..inf}:TL_UCP:10 
[1765938844.537037] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  Scatterv:
[1765938844.537037] [-:20594:0] ucc_coll_score_map.c:225  UCC  INFO  	Host: {0..inf}:TL_UCP:10 
[1765938844.537041] [-:20594:0]        ucc_team.c:475  UCC  INFO  ================================================

Output - large message sizes

[1765938844.928066] [-:20594:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Allgather: src={0x14eb5765c000, 262144, int8, Host}, dst={0x14eb47fff000, 4194304, int8, Host}; CL_BASIC {TL_UCP}, team_id 32768
[1765938844.928483] [-:20594:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Barrier; CL_HIER {TL_UCP}, team_id 32768
[1765938844.928489] [-:20594:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Allgather: src={0x14eb5765c000, 262144, int8, Host}, dst={0x14eb47fff000, 4194304, int8, Host}; CL_BASIC {TL_UCP}, team_id 32768
[1765938844.928906] [-:20594:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Barrier; CL_HIER {TL_UCP}, team_id 32768
[1765938844.928914] [-:20594:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Allgather: src={0x14eb5765c000, 262144, int8, Host}, dst={0x14eb47fff000, 4194304, int8, Host}; CL_BASIC {TL_UCP}, team_id 32768
[1765938844.929327] [-:20594:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Barrier; CL_HIER {TL_UCP}, team_id 32768
[1765938844.929334] [-:20594:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Allgather: src={0x14eb5765c000, 262144, int8, Host}, dst={0x14eb47fff000, 4194304, int8, Host}; CL_BASIC {TL_UCP}, team_id 32768
[1765938844.929747] [-:20594:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Barrier; CL_HIER {TL_UCP}, team_id 32768
[1765938844.929753] [-:20594:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Barrier; CL_HIER {TL_UCP}, team_id 32768
[1765938844.929766] [-:20594:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Reduce min root 0: src={0x7ffd0a0d89d0, 1, float64, Host}, dst={0x7ffd0a0d89e8, 1, float64, Host}; CL_HIER {TL_UCP}, team_id 32768
[1765938844.929774] [-:20594:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Reduce max root 0: src={0x7ffd0a0d89d0, 1, float64, Host}, dst={0x7ffd0a0d89e0, 1, float64, Host}; CL_HIER {TL_UCP}, team_id 32768
[1765938844.929781] [-:20594:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Reduce sum root 0: src={0x7ffd0a0d89d0, 1, float64, Host}, dst={0x7ffd0a0d89d8, 1, float64, Host}; CL_HIER {TL_UCP}, team_id 32768
262144                415.87
[1765938844.929794] [-:20594:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Barrier; CL_HIER {TL_UCP}, team_id 32768
[1765938844.929799] [-:20594:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Barrier; CL_HIER {TL_UCP}, team_id 32768
[1765938844.929805] [-:20594:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Allgather: src={0x14eb5765c000, 524288, int8, Host}, dst={0x14eb47fff000, 8388608, int8, Host}; CL_BASIC {TL_UCP}, team_id 32768
[1765938844.932148] [-:20594:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Barrier; CL_HIER {TL_UCP}, team_id 32768
[1765938844.932161] [-:20594:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Allgather: src={0x14eb5765c000, 524288, int8, Host}, dst={0x14eb47fff000, 8388608, int8, Host}; CL_BASIC {TL_UCP}, team_id 32768

Also, here's ucp_info -s result

Default CLs scores: basic=10 hier=50
Default TLs scores: mlx5=1 self=50 sharp=30 shm=100 ucp=10

I'm running this on 16 nodes allocation, not GPU only CPU hosts setting. Please let me know if I'm missing anything or need to provide any additional information.

Here are Log files for Allgather, Allreduce, Broadcast, and Reduce Scatter. Benchmark code is exactly same except collective name.

log-allgather.txt

log-allreduce.txt

log-bcast.txt

log-reduce_scatter.txt

[Sergei-Lebedev]

Hi @james-sungjae-lee

For Allreduce and Broadcast, it follows the score map with a 4 KB threshold between UCP and SHARP.

Could be due to hpcx ucc config file. Please try with OMPI_UCC_CLS=basic or UCC_CONFIG_FILE= to disable CL hier and enable CL basic for small messages

For Allgather and Reduce-Scatter, even though the score map threshold is 16 KB, it still seems to always use the UCP backend.

Why is it 16K? see comment here

ucc/src/components/tl/sharp/tl_sharp_coll.c

Line 15 in 265a048

/* Disable allgather and reduce_scatter for message sizes 0-16k since SHARP SAT is only enabled by default for messages larger than 16k.

You can adjust threshold manually by specifying message range in your SHARP_TUNE variable https://github.com/openucx/ucc/wiki/FAQ#6-what-is-tl-scoring-and-how-to-select-a-certain-tl

Why UCP backend is used?
Could be that even though SHARP was forced by users, we can't actually use it for a particular collective argument because something is not supported. It might be data type; I see that you are running with int8, which is not supported by some switch models.

ucc/src/components/tl/sharp/tl_sharp_team.c

Line 95 in 265a048

#if SHARP_API > SHARP_VERSION(3, 0)

[james-sungjae-lee]

Hi @Sergei-Lebedev, thank you for the quick response.

First, some issues were resolved with your suggestion. I did not realize there was another OMPI variable to control the UCC CL layer. After I set OMPI_UCC_CLS=basic, it switched to CL_BASIC instead of CL_HIER, and I was able to force SHARP for any message size for Allreduce and Broadcast.

For Allreduce and Broadcast, everything is now resolved, but Allgather and Reduce-Scatter still have issues. I have a couple of questions.

1. After I changed the datatype from MPI_CHAR to MPI_INT, Allgather works well for large message sizes, but not for small message sizes (<= 512), where it uses TL_UCP. Do you have any idea?

[1765987365.876987] [-:46697:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Allgather: src={0x149880bb3000, 128, int32, Host}, dst={0x14986bfff000, 2048, int32, Host}; CL_BASIC {TL_UCP}, team_id 32768
[1765987365.877002] [-:46697:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Barrier; CL_BASIC {TL_SHARP}, team_id 32768
[1765987365.877008] [-:46697:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Allgather: src={0x149880bb3000, 128, int32, Host}, dst={0x14986bfff000, 2048, int32, Host}; CL_BASIC {TL_UCP}, team_id 32768
[1765987365.877023] [-:46697:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Barrier; CL_BASIC {TL_SHARP}, team_id 32768
[1765987365.877029] [-:46697:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Barrier; CL_BASIC {TL_SHARP}, team_id 32768
[1765987365.877036] [-:46697:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Reduce min root 0: src={0x7ffe762e3850, 1, float64, Host}, dst={0x7ffe762e3868, 1, float64, Host}; CL_BASIC {TL_UCP}, team_id 32768
[1765987365.877043] [-:46697:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Reduce max root 0: src={0x7ffe762e3850, 1, float64, Host}, dst={0x7ffe762e3860, 1, float64, Host}; CL_BASIC {TL_UCP}, team_id 32768
[1765987365.877048] [-:46697:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Reduce sum root 0: src={0x7ffe762e3850, 1, float64, Host}, dst={0x7ffe762e3858, 1, float64, Host}; CL_BASIC {TL_UCP}, team_id 32768
512                    16.54
[1765987365.877060] [-:46697:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Barrier; CL_BASIC {TL_SHARP}, team_id 32768
[1765987365.877064] [-:46697:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Barrier; CL_BASIC {TL_SHARP}, team_id 32768
[1765987365.877082] [-:46697:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Allgather: src={0x149880bb3000, 256, int32, Host}, dst={0x14986bfff000, 4096, int32, Host}; CL_BASIC {TL_SHARP}, team_id 32768
[1765987365.877887] [-:46697:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Barrier; CL_BASIC {TL_SHARP}, team_id 32768
[1765987365.877918] [-:46697:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Allgather: src={0x149880bb3000, 256, int32, Host}, dst={0x14986bfff000, 4096, int32, Host}; CL_BASIC {TL_SHARP}, team_id 32768

2. After I changed the datatype to MPI_INT, Reduce Scatter’s Barrier call changed to CL_BASIC {TL_SHARP}. However, as before, there is no Reduce Scatter INFO output, which makes it hard to tell which CL or TL UCC is using internally. Why do other collectives show UCC_COLL info, but Reduce Scatter does not? Based on a performance comparison with the official SHARP example, it looks like UCC is still not using SHARP internally even with INT32.

# OSU MPI Reduce_scatter Latency Test v7.5.2
# Datatype: MPI_INT.
# Size       Avg Latency(us)
...
[1765987808.366077] [-:46804:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Barrier; CL_BASIC {TL_SHARP}, team_id 32768
[1765987808.366086] [-:46804:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Barrier; CL_BASIC {TL_SHARP}, team_id 32768
[1765987808.366104] [-:46804:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Reduce min root 0: src={0x7ffdbae53258, 1, float64, Host}, dst={0x7ffdbae53270, 1, float64, Host}; CL_BASIC {TL_UCP}, team_id 32768
[1765987808.366130] [-:46804:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Reduce max root 0: src={0x7ffdbae53258, 1, float64, Host}, dst={0x7ffdbae53268, 1, float64, Host}; CL_BASIC {TL_UCP}, team_id 32768
[1765987808.366138] [-:46804:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Reduce sum root 0: src={0x7ffdbae53258, 1, float64, Host}, dst={0x7ffdbae53260, 1, float64, Host}; CL_BASIC {TL_UCP}, team_id 32768
4                       1.29
[1765987808.366150] [-:46804:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Barrier; CL_BASIC {TL_SHARP}, team_id 32768
[1765987808.366155] [-:46804:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Barrier; CL_BASIC {TL_SHARP}, team_id 32768

Do you have any idea about these two points? Please let me know if you have any thoughts or comments. I really appreciate your help.

FYI, I installed OMB 7.5.2 separately as previous version with HPC-X 2.23 (OMB 7.4) has an issue with -T datatype option.

[Sergei-Lebedev]

Hi @james-sungjae-lee, good to know that allreduce and bcast are now ok.

After I changed the datatype from MPI_CHAR to MPI_INT, Allgather works well for large message sizes, but not for small message sizes (<= 512), where it uses TL_UCP. Do you have any idea?

As I mentioned in a previous message, there is a UCC default threshold of 16KB for SHARP allgather and reduce scatter, and it looks like you haven't changed it because in your run, SHARP starts to work when the allgather dst buffer size is 4096 int32 or 16KB. This threshold is needed because SHARP SAT is required for allgather and reduce scatter, and there is no way to query the SAT threshold set by the user for libsharp. The default SAT threshold is 16KB in SHARP, and that's why it's also 16 KB in UCC TL SHARP. You can adjust the default values manually by setting -x SHARP_COLL_SAT_THRESHOLD=128 -x OMPI_UCC_TL_SHARP_TUNE="allgather:128-inf:200" which enables SHARP allgather for msg > 128B.

However, as before, there is no Reduce Scatter INFO output, which makes it hard to tell which CL or TL UCC is using internally. Why do other collectives show UCC_COLL info, but Reduce Scatter does not?

I think there is a confusion here. What we call reduce_scatter in UCC and SHARP is reduce_scatter_block in MPI terminology, and reduce_scatter in MPI is reduce_scatterv in UCC. Since you are running OSU reduce_scatter benchmark, it translates to UCC reduce_scatterv, which is not supported by SHARP and not supported by OMPI v4 UCC coll component. To run SHARP reduce_scatter, you should use osu_reduce_scatter_block instead. Also, keep in mind that the same 16KB default threshold applies to reduce_scatter.

[james-sungjae-lee]

Hi @Sergei-Lebedev, thank you for your comments.

First, the reduce_scatterv issue is resolved and it is clear now. After I used osu_reduce_scatter_block, which calls blocking Reduce Scatter, it shows Reduce Scatter INFO with SHARP calls. Thank you again for your help.

There is still one more issue, which I believe is the last one. You mentioned two variables, SHARP_COLL_SAT_THRESHOLD and OMPI_UCC_TL_SHARP_TUNE. I think the second one might be a typo, and the correct one is UCC_TL_SHARP_TUNE without OMPI_. Is that right? Also, even when I set both variables, it still goes into the UCP backend with the default score map.

So now it works correctly with the UCC score map, but the problem is that the score map does not change even if I manually set the SAT threshold and SHARP tune. I am checking why the original score map is not changing by analyzing the ucc_coll_score_build_map function. If you have any other suggestions, please let me know.

Here's my mpirun command script.

mpirun -np $1 \
    --bind-to core \
    --map-by ppr:1:node \
    --hostfile ./hostfile-2 \
    -x LD_LIBRARY_PATH \
    -x SHARP_COLL_ENABLE_SAT=1 \
    -x SHARP_COLL_LOG_LEVEL=3 \
    -x SHARP_COLL_SAT_THRESHOLD=128 \
    -x UCC_TL_SHARP_TUNE="reduce_scatter:128-inf:200" \
    -x UCC_CLS=basic \
    -x UCC_TL_SHARP_DEVICES=mlx5_0 \
    -x UCC_LOG_LEVEL=INFO \
    -x UCC_COLL_TRACE=INFO \
    -x OMPI_UCC_CL_BASIC_TLS=sharp,ucp \
    -x OMPI_UCC_CLS=basic \
    $HPCX_OMB_DIR/osu_reduce_scatter_block -x 20 -i 200 -m 4:8388608 -T mpi_int > log-reduce_scatter_block.out

I tried several other options, like UCC_TL_SHARP_TUNE=reduce_scatter:inf with SHARP_COLL_SAT_THRESHOLD=1, but couldn't change the score map.

Here's the score map output

[1766085286.577371] [-:540686:0]        ucc_team.c:471  UCC  INFO  ===== COLL_SCORE_MAP (team_id 32768, size 16) =====
[1766085286.577388] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  Allgather:
[1766085286.577388] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	Host: {0..4095}:TL_UCP:10 {4K..16383}:TL_UCP:10 {16K..inf}:TL_SHARP:10 
[1766085286.577388] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	Cuda: {16K..inf}:TL_SHARP:10 
[1766085286.577388] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	CudaManaged: {16K..inf}:TL_SHARP:10 
[1766085286.577388] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	Rocm: {16K..inf}:TL_SHARP:10 
[1766085286.577388] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	RocmManaged: {16K..inf}:TL_SHARP:10 
[1766085286.577407] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  Allreduce:
[1766085286.577407] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	Host: {0..4095}:TL_SHARP:10 {4K..inf}:TL_SHARP:10 
[1766085286.577407] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	Cuda: {0..inf}:TL_SHARP:10 
[1766085286.577407] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	CudaManaged: {0..inf}:TL_SHARP:10 
[1766085286.577407] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	Rocm: {0..inf}:TL_SHARP:10 
[1766085286.577407] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	RocmManaged: {0..inf}:TL_SHARP:10 
[1766085286.577423] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  Alltoall:
[1766085286.577423] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	Host: {0..2063}:TL_UCP:10 {2064..4095}:TL_UCP:10 {4K..inf}:TL_UCP:10 
[1766085286.577429] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  Alltoallv:
[1766085286.577429] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	Host: {0..inf}:TL_UCP:10 
[1766085286.577437] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  Barrier:
[1766085286.577437] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	Host: {0..inf}:TL_SHARP:10 
[1766085286.577437] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	Cuda: {0..inf}:TL_SHARP:10 
[1766085286.577437] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	CudaManaged: {0..inf}:TL_SHARP:10 
[1766085286.577437] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	Rocm: {0..inf}:TL_SHARP:10 
[1766085286.577437] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	RocmManaged: {0..inf}:TL_SHARP:10 
[1766085286.577453] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  Bcast:
[1766085286.577453] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	Host: {0..32767}:TL_SHARP:10 {32K..inf}:TL_SHARP:10 
[1766085286.577453] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	Cuda: {0..inf}:TL_SHARP:10 
[1766085286.577453] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	CudaManaged: {0..inf}:TL_SHARP:10 
[1766085286.577453] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	Rocm: {0..inf}:TL_SHARP:10 
[1766085286.577453] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	RocmManaged: {0..inf}:TL_SHARP:10 
[1766085286.577469] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  Fanin:
[1766085286.577469] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	Host: {0..inf}:TL_UCP:10 
[1766085286.577474] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  Fanout:
[1766085286.577474] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	Host: {0..inf}:TL_UCP:10 
[1766085286.577480] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  Gather:
[1766085286.577480] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	Host: {0..inf}:TL_UCP:10 
[1766085286.577485] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  Gatherv:
[1766085286.577485] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	Host: {0..inf}:TL_UCP:10 
[1766085286.577491] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  Reduce:
[1766085286.577491] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	Host: {0..32767}:TL_UCP:10 {32K..inf}:TL_UCP:10 
[1766085286.577500] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  Reduce_scatter:
[1766085286.577500] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	Host: {0..16383}:TL_UCP:10 {16K..inf}:TL_SHARP:10 
[1766085286.577500] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	Cuda: {16K..inf}:TL_SHARP:10 
[1766085286.577500] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	CudaManaged: {16K..inf}:TL_SHARP:10 
[1766085286.577500] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	Rocm: {16K..inf}:TL_SHARP:10 
[1766085286.577500] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	RocmManaged: {16K..inf}:TL_SHARP:10 
[1766085286.577523] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  Reduce_scatterv:
[1766085286.577523] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	Host: {0..inf}:TL_UCP:10 
[1766085286.577530] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  Scatterv:
[1766085286.577530] [-:540686:0] ucc_coll_score_map.c:225  UCC  INFO  	Host: {0..inf}:TL_UCP:10 
[1766085286.577535] [-:540686:0]        ucc_team.c:475  UCC  INFO  ================================================

Here's the UCP backend with the threshold

[1766085286.708153] [-:540686:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Reduce_scatter sum: src={0x1480758dd000, 32768, int32, Host}, dst={0x148074ad8000, 2048, int32, Host}; CL_BASIC {TL_UCP}, team_id 32768
[1766085286.708210] [-:540686:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Barrier; CL_BASIC {TL_SHARP}, team_id 32768
[1766085286.708218] [-:540686:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Reduce_scatter sum: src={0x1480758dd000, 32768, int32, Host}, dst={0x148074ad8000, 2048, int32, Host}; CL_BASIC {TL_UCP}, team_id 32768
[1766085286.708279] [-:540686:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Barrier; CL_BASIC {TL_SHARP}, team_id 32768
[1766085286.708286] [-:540686:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Reduce min root 0: src={0x7ffd23cedc58, 1, float64, Host}, dst={0x7ffd23cedc70, 1, float64, Host}; CL_BASIC {TL_UCP}, team_id 32768
[1766085286.708293] [-:540686:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Reduce max root 0: src={0x7ffd23cedc58, 1, float64, Host}, dst={0x7ffd23cedc68, 1, float64, Host}; CL_BASIC {TL_UCP}, team_id 32768
[1766085286.708299] [-:540686:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Reduce sum root 0: src={0x7ffd23cedc58, 1, float64, Host}, dst={0x7ffd23cedc60, 1, float64, Host}; CL_BASIC {TL_UCP}, team_id 32768
131072                 61.22
[1766085286.708309] [-:540686:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Barrier; CL_BASIC {TL_SHARP}, team_id 32768
[1766085286.708315] [-:540686:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Barrier; CL_BASIC {TL_SHARP}, team_id 32768
[1766085286.708333] [-:540686:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Reduce_scatter sum: src={0x1480758dd000, 65536, int32, Host}, dst={0x148074ad8000, 4096, int32, Host}; CL_BASIC {TL_SHARP}, team_id 32768
[1766085286.708987] [-:540686:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Barrier; CL_BASIC {TL_SHARP}, team_id 32768
[1766085286.708996] [-:540686:0]        ucc_coll.c:301  UCC_COLL INFO  coll_init: Reduce_scatter sum: src={0x1480758dd000, 65536, int32, Host}, dst={0x148074ad8000, 4096, int32, Host}; CL_BASIC {TL_SHARP}, team_id 32768