Skip to content

UCP/PROTO: Prefer interfaces closer to memory #10932

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
+16 −2
Open

UCP/PROTO: Prefer interfaces closer to memory #10932

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion src/ucp/proto/proto_common.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -350,7 +350,7 @@ ucp_proto_common_get_lane_perf(const ucp_proto_common_init_params_t *params,
ucs_sys_dev_distance_t distance;
size_t tl_min_frag, tl_max_frag;
uct_perf_attr_t perf_attr;
ucs_sys_device_t sys_dev;
ucs_sys_device_t sys_dev, device_sys_dev;
ucs_status_t status;
char bdf_name[32];

Expand Down Expand Up @@ -427,6 +427,15 @@ ucp_proto_common_get_lane_perf(const ucp_proto_common_init_params_t *params,
ucs_topo_sys_device_get_name(sys_dev),
ucs_topo_sys_device_bdf_name(sys_dev, bdf_name,
sizeof(bdf_name)));

/* Add bandwidth for interfaces very close to the memory. */
device_sys_dev = ucp_proto_common_get_tl_rsc(&params->super, lane)->sys_device;
if ((sys_dev != UCS_SYS_DEVICE_ID_UNKNOWN) &&
(device_sys_dev != UCS_SYS_DEVICE_ID_UNKNOWN) &&
(sys_dev != device_sys_dev) &&
ucs_topo_is_pci_bridge(device_sys_dev, sys_dev)) {
tl_perf->bandwidth *= 1.2;
Copy link
Contributor

@iyastreb iyastreb Oct 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's ok as a quickfix

I agree about taking into account multiple factors when choosing a protocol. This is actually implemented in protocol variants: #10778
The idea is to select lanes by score (not just BW), where both BW and latency contributes. With this approach the better way maybe would be to decrease system latency for iface on the same PCI bridge (or increase if not).

}
}

/* For remote memory access, consider remote system topology distance */
Expand Down
2 changes: 1 addition & 1 deletion src/ucs/sys/topo/base/topo.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -832,7 +832,7 @@ ucs_numa_node_t ucs_topo_sys_device_get_numa_node(ucs_sys_device_t sys_dev)
* @return 1 if the devices share a PCI bridge, 0 otherwise. Returns 0 the
* devices are the same.
*/
static int
int
ucs_topo_is_pci_bridge(ucs_sys_device_t sys_dev1, ucs_sys_device_t sys_dev2)
{
ucs_status_t status;
Expand Down
5 changes: 5 additions & 0 deletions src/ucs/sys/topo/base/topo.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -358,6 +358,11 @@ void ucs_topo_init(void);
*/
void ucs_topo_cleanup(void);

/**
* Check if two system devices share a PCI bridge.
*/
int ucs_topo_is_pci_bridge(ucs_sys_device_t sys_dev1, ucs_sys_device_t sys_dev2);

END_C_DECLS

#endif
Loading