UCP/RNDV: Throttle rndv fragment requests #11062
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ne mtype requests)
amastbaum
force-pushed
the
add_throttling_to_rndv_fragment_requests
branch
from
2be3040 to
b175673
Compare
amastbaum
added
Ready for Review
and removed
WIP-DNM
amastbaum
commented
Dec 15, 2025
Comment on lines
214
to
250
| ucp_proto_rndv_mtype_fc_increment(ucp_request_t *req) | ||
| { | ||
| ucp_worker_h worker = req->send.ep->worker; | ||
|
|
||
| if (worker->context->config.ext.rndv_ppln_worker_fc_enable) { | ||
| worker->rndv_ppln_fc.active_frags++; | ||
| } | ||
| } | ||
|
|
||
| /** | ||
| * Decrement active_frags counter and reschedule pending request if any. | ||
| */ | ||
| static UCS_F_ALWAYS_INLINE void | ||
| ucp_proto_rndv_mtype_fc_decrement(ucp_request_t *req) | ||
| { | ||
| ucp_worker_h worker = req->send.ep->worker; | ||
| ucp_context_h context = worker->context; | ||
|
|
||
| if (!context->config.ext.rndv_ppln_worker_fc_enable) { | ||
| return; | ||
| } | ||
|
|
||
| ucs_assert(worker->rndv_ppln_fc.active_frags > 0); | ||
| worker->rndv_ppln_fc.active_frags--; | ||
|
|
||
| if (!ucs_queue_is_empty(&worker->rndv_ppln_fc.pending_q)) { | ||
| ucp_request_t *pending_req; | ||
| ucs_queue_elem_t *elem; | ||
|
|
||
| elem = ucs_queue_pull(&worker->rndv_ppln_fc.pending_q); | ||
| pending_req = ucs_container_of(elem, ucp_request_t, | ||
| send.rndv.ppln.queue_elem); | ||
| ucs_callbackq_add_oneshot(&worker->uct->progress_q, pending_req, | ||
| ucp_proto_rndv_mtype_fc_reschedule_cb, | ||
| pending_req); | ||
| } | ||
| } |
Contributor
Author
There was a problem hiding this comment.
maybe it can be named better, or I should divide it differently
amastbaum
commented
Dec 15, 2025
src/ucp/core/ucp_context.c
Outdated
Comment on lines
391
to
394
| {"RNDV_MTYPE_WORKER_MAX_FRAGS", "1024", | ||
| "Maximum number of concurrent mtype fragments per worker\n" | ||
| "(only applies when RNDV_MTYPE_WORKER_FC_ENABLE=y)", | ||
| ucs_offsetof(ucp_context_config_t, rndv_mtype_worker_max_frags), UCS_CONFIG_TYPE_ULUNITS}, |
Contributor
Author
There was a problem hiding this comment.
@shasson5 I am wondering if it's better to define RNDV_MTYPE_WORKER_MAX_FRAG_MEM instead, because the user shouldn't know how the amount of available memory is translated to the number of fragments.
The thing is that if we translate it in the code, it should be defined for each memory type (mem_type fragment size).
nbellalou
previously approved these changes
Jan 1, 2026
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What?
Added a worker-level throttling mechanism for rndv fragment requests (both pipeline and standalone mtype requests).
Why?
Without throttling, rendezvous protocols can spawn an unbounded number of concurrent fragment requests, each allocating staging memory from GPU memory pools. Under heavy workloads, this leads to GPU memory exhaustion (OOM), as tracked in https://nvbugspro.nvidia.com/bug/5359018. The throttling mechanism bounds the number of in-flight fragments per worker, preventing resource exhaustion while allowing work to progress as fragments complete.
How?
The throttling is implemented using a worker-level flow control state (
worker->rndv_ppln_fc) consisting of:active_fragscounter tracking currently in-flight fragment operationspending_qqueue holding throttled requests waiting for resourcesWhen a new fragment request would exceed
UCX_RNDV_PPLN_WORKER_MAX_FRAGS, the request is queued inpending_q. When fragments complete,active_fragsis decremented and one queued request is rescheduled viaucs_callbackq_add_oneshot()to the main progress loop.The mechanism applies to:
rndv_ppln.cmtypeprotocols inrndv_rtr.c,rndv_get.c, andrndv_put.c.