generated from fastai/nbdev_template
-
Notifications
You must be signed in to change notification settings - Fork 2.5k
Open
Description
As part of #5119
Motivation:
generate_rollout_completions() is the primary helper API for rollout function
authors who need to call the inference backend mid-rollout (for multi-turn agent
loops). It currently hard-requires vLLM:
if not trainer.use_vllm:
raise RuntimeError("Custom rollouts require vLLM ...")Scope:
- Replace direct vLLM API calls (
trainer.vllm_generation.vllm_client.generate(),
trainer.vllm_generation.llm.generate()) with calls through the generation
backend interface. - Remove the
if not trainer.use_vllm: raise RuntimeError(...)guard. - Preserve the parameter-building helpers (
_build_base_generation_kwargs,
_build_colocate_sampling_params,_build_server_generation_kwargs) but make
them backend-aware or extract backend-specific ones. - Update function signature if needed to accept a backend-agnostic generation
callable.
CC: @albertvillanova
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels