fix: make rollout /run failures recoverable#1788
Open
yuchenwang3 wants to merge 2 commits into
Open
Conversation
nemo-automation-bot
Bot
added
the
community-request
raise_for_status re-raises aiohttp's ClientResponseError unchanged. It carries request_info/history/headers, which are multidict.CIMultiDictProxy objects that don't pickle. When rollout collection runs under Ray the exception is pickled to cross actors and the run dies with "can't pickle CIMultiDictProxy" - so any resource-server 5xx takes down the whole job instead of failing one rollout. Strip the unpicklable fields; keep status/message/response_content. Signed-off-by: yuchenwang3 <yuchenwang3@users.noreply.github.com>
_post_subroutine doesn't retry /run, so a transient 5xx (e.g. a momentarily overloaded code-exec server) aborts the batch. Retry a few times on 5xx and connection errors; leave 4xx alone. Signed-off-by: yuchenwang3 <yuchenwang3@users.noreply.github.com>
yuchenwang3
changed the title
yuchenwang3
force-pushed
the
recoverable-rollout-errors
branch
from
June 26, 2026 22:09
03e29e7 to
22ed1af
Compare
svcnvidia-nemo-ci
added
the
waiting-on-maintainers
Author
|
Ping — waiting on CI authorization since Jun 26. Could a maintainer |
svcnvidia-nemo-ci
added
waiting-on-maintainers
Author
|
@marta-sd sorry for the direct ping — you've been the main recent committer on |
svcnvidia-nemo-ci
added
waiting-on-maintainers
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two issues that together turn one resource-server error into a fatal crash:
raise_for_statusre-raises aiohttp'sClientResponseErrorunchanged. It holdsrequest_info/history/headers, which aremultidict.CIMultiDictProxyobjects that don't pickle. When rollout collection runs under Ray, the exception is pickled to cross actors and the run dies withcan't pickle CIMultiDictProxy— so any resource-server 5xx takes down the whole job instead of failing a single rollout. Strip the unpicklable fields; keepstatus/message/response_content._post_subroutinenever retries/run. A transient 5xx (e.g. a momentarily overloaded code-exec server) aborts the batch. Retry a few times on 5xx/connection errors; leave 4xx alone.How we hit it: under GRPO, a vLLM worker 500s on a NaN logprob (fixed separately in NVIDIA-NeMo/RL#2962). That 500 reaches
raise_for_status, and (1) then crashes the whole rollout collection on the unpicklable exception rather than surfacing the underlying error.