what's the usage of uct_md_attr::reg_cost? how to estimate the memory registeration time

[XCloudAI]

I see below info when running "ucx_info -d -t rc_verbs":

23 # Memory domain: mlx5_bond_0
 24 #     Component: ib
 25 #             register: unlimited, cost: 180 nsec
 26 #           remote key: 24 bytes
 27 #           local memory handle is required for zcopy

What's the usage of cost here? Does it use ib_reg_mr function to estimate the registeration time? What's the size of memory?

[yosefe]

This is estimated time (on average) to register memory, as function of the size. Since this particular memory domain (device) is using a registration cache, the cost is estimated as 180nsec (hardcoded)

[XCloudAI]

@yosefe Is it the average time to register 4KB? If we got the cost e.g. 180 nsec, how to estimate the time to register 32MB space?

[yosefe]

when using registration cache (default is yes), the amortized time (when same buffer is registered multiple times and there is a cache hit) is constant time and does not depend on buffer size.
however, the first time buffer is registered, it will take more time, depending on the system. this time cannot be derived from ucx_info output; need to benchmark it