测试代码中的训练问题

[JiangWu0826]

project: "Trinity-RFT-gsm8k"
name: "qwen2.5-1.5B-gsm8k"
checkpoint_root_dir: ${oc.env:TRINITY_CHECKPOINT_ROOT_DIR,./checkpoints}
algorithm:
  algorithm_type: grpo
  repeat_times: 8   
model:
  model_path: ${oc.env:TRINITY_MODEL_PATH,Qwen/Qwen2.5-1.5B-Instruct}
  max_response_tokens: 1024   
  max_model_len: 1280   
cluster:
  node_num: 1
  gpu_per_node: 8   # 8 GPUs per node
buffer:
  total_epochs: 1   #
  batch_size: 96
  explorer_input:
    taskset:
      name: gsm8k
      storage_type: file
      path: 'openai/gsm8k'
      subset_name: 'main'
      split: 'train'
      format:
        prompt_key: 'question'
        response_key: 'answer'
      rollout_args:
        temperature: 1.0
    eval_tasksets:
    - name: gsm8k-eval
      storage_type: file
      path: 'openai/gsm8k'
      subset_name: 'main'
      split: 'test'
      format:
        prompt_key: 'question'
        response_key: 'answer'
    default_workflow_type: 'math_workflow'     #奖励形式
  trainer_input:
    experience_buffer:
      name: gsm8k_buffer
      storage_type: queue
      path: 'sqlite:///gsm8k.db'
explorer:
  eval_interval: 50
  runner_per_model: 8
  rollout_model:
    engine_num: 4           # 2 个 vLLM 引擎
    tensor_parallel_size: 1
    enable_prefix_caching: false
    enforce_eager: true
    dtype: bfloat16
    seed: 42
synchronizer:   
  sync_method: 'nccl'  
  sync_interval: 1
  sync_timeout: 1200
trainer:
  trainer_type: 'verl' 
  save_interval: 100  
  trainer_config:
    actor_rollout_ref:
      model:
        use_remove_padding: true
      actor:
        use_dynamic_bsz: true
        ppo_max_token_len_per_gpu: 16384
        ulysses_sequence_parallel_size: 1
        optim:
          lr: 1e-5
      ref:
        log_prob_use_dynamic_bsz: ${trainer.trainer_config.actor_rollout_ref.actor.use_dynamic_bsz}
        log_prob_max_token_len_per_gpu: ${trainer.trainer_config.actor_rollout_ref.actor.ppo_max_token_len_per_gpu}
        ulysses_sequence_parallel_size: ${trainer.trainer_config.actor_rollout_ref.actor.ulysses_sequence_parallel_size} # sp size

开发者您好，trinity run --config examples/grpo_gsm8k/gsm8k.yaml，我在运行上面的示例代码中，使用您这边提供的默认参数，但是遇到了No more samples to train. Stopping training.请问这个怎么解决呢。我的GPUs是8张46GB的A40。

[chenyushuo]

这是因为在你使用的设置中，total_epoch=1, batch_size=96，而gsm8k的训练数据总共只有7473条，1个epoch只有7473 // 96 = 77步，这与你所提供的日志一致，如果你需要训练更多步，请调整total_epoch的设置

[JiangWu0826]

好的，谢谢，另外想问一下，训练已经得到了Trinity-RFT/checkpoints/Trinity-RFT-gsm8k/qwen2.5-1.5B-gsm8k/global_step_77权重参数了，那么怎么评估一下我训练模型的测试精度呢，想看一下实际推理效果 Thanks♪(･ω･)ﾉ

[chenyushuo]

如果你想使用Trinity-RFT评测最后一个检查点，你需要设置mode, buffer.explorer_input.eval_tasksets和explorer.bench_on_latest_checkpoint的内容，大致如下：

mode: bench  # 使用benchmark模式
project: "Trinity-RFT-gsm8k"  # project, name, checkpoint_root_dir请保持与训练使用的yaml一致，以保证能够读取相同的checkpoint
name: "qwen2.5-1.5B-gsm8k"
checkpoint_root_dir: ${oc.env:TRINITY_CHECKPOINT_ROOT_DIR,./checkpoints}
...
buffer:
  ...
  explorer_input:
    ...
    eval_tasksets:   # 你给出的yaml使用的是gsm8k的eval数据，可以根据需求添加其他评测数据
    - name: gsm8k-eval
      storage_type: file
      path: 'openai/gsm8k'
      subset_name: 'main'
      split: 'test'
      format:
        prompt_key: 'question'
        response_key: 'answer'
...
explorer:
  ...
  bench_on_latest_checkpoint: true

然后使用trinity run --config命令运行你修改过的yaml。

如果你想在 Trinity-RFT 框架之外加载检查点，请参考文档。

[waltonfuture]

@chenyushuo 作者您好！我对于chord算法里面的step计算不太确定。
我的yaml是：

project: "mix_chord"
name: "mix_chord_med_3.1"
checkpoint_root_dir: ${oc.env:TRINITY_CHECKPOINT_ROOT_DIR,./checkpoints}
algorithm:
  algorithm_type: mix_chord
  repeat_times: 8 # or 16 for better performance in math related tasks
  kl_loss_fn_args:
    kl_coef: 0.05
  sample_strategy_args:
    expert_data_ratio: 0.20
  policy_loss_fn_args: # feel free to change, we encourage you to try out different hyperparameters
    mu_warmup_steps: 0  # 0 for chord-mu and chord-phi
    mu_decay_steps: 0 # 200 for chord-mu and 0 for chord-phi
    mu_peak: 0.5 # 0.9 for chord-mu and 0.1 for chord-phi
    mu_valley: 0.1 # 0.05 for chord-mu and 0.1 for chord-phi
    enable_phi_function: true # false for chord-mu and true for chord-phi
    clip_range: 0.2
    sft_loss_agg_mode: "token-mean"
    use_dynamic_bsz: true
    ppo_mini_batch_size: 576 # 320 = 256 + 64; if you set repeat times = 16, then it shoudle be 32 * 16 + 64
    ppo_micro_batch_size_per_gpu: 8
    ngpus_trainer: 4
    train_batch_size_expert: 64
    train_batch_size_usual: 512 # 32 batchsize * 8 repeat times
model:
  model_path: ${oc.env:TRINITY_MODEL_PATH,/inspire/hdd/project/aiforquantum/zhengkaipeng-240108120123/test/models/Qwen3-4B-Instruct-2507}
  max_response_tokens: 10240
  max_model_len: 11264
cluster:
  node_num: 1
  gpu_per_node: 4
buffer:
  total_epochs: 1
  batch_size: 64
  train_batch_size: 576
  explorer_input:
    taskset:
      name: medmcqa_data_filtered_int
      storage_type: file
      path: ${oc.env:TRINITY_TASKSET_PATH, /inspire/hdd/project/aiforquantum/zhengkaipeng-240108120123/test/codes/ContinualRL/datasets/data/chord/combined_medical_rl_dataset}
      format:
        prompt_key: 'query'
        response_key: 'reference_answer'
      rollout_args:
        temperature: 1.0
        logprobs: 0
      workflow_args:
        with_think: false
    eval_tasksets:
    - name: medmcqa_data_test
      storage_type: file
      path: ${oc.env:TRINITY_TASKSET_PATH, /inspire/hdd/project/aiforquantum/zhengkaipeng-240108120123/test/codes/ContinualRL/datasets/data/chord/medmcqa_test1} 
      repeat_times: 1
      split: 'test'
      format:
        prompt_key: 'query'
        response_key: 'response'
      rollout_args:
        temperature: 1.0
        top_p: 0.7
      default_reward_fn_type: 'math_boxed_reward'
      reward_fn_args: {}
    default_workflow_type: 'medical_boxed_workflow'
  trainer_input:
    experience_buffer:
      name: math_buffer
      storage_type: queue
      path: 'sqlite:///mix_chord_medmcqa_3.1.db'
    auxiliary_buffers:
      sft_dataset:
        total_epochs: 1
        name: SFT_data
        storage_type: file
        schema_type: sft
        path: ${oc.env:TRINITY_SFT_DATASET_PATH,/inspire/hdd/project/aiforquantum/zhengkaipeng-240108120123/test/codes/ContinualRL/datasets/data/chord/combined_medical_sft/rej--Qwen3-235B-A22B-Instruct-2507-FP8}
        split: 'train'
        format:
          prompt_type: messages
          messages_key: 'messages'
explorer:
  eval_interval: 10
  runner_per_model: 8
  rollout_model:
    engine_num: 2
    tensor_parallel_size: 1
    enable_prefix_caching: true
    enforce_eager: false
    dtype: bfloat16
    seed: 42
synchronizer:
  sync_method: 'nccl'
  sync_interval: 1
  sync_timeout: 1200
trainer:
  save_interval: 10
  total_steps: 1000
  trainer_config:
    actor_rollout_ref:
      model:
        use_remove_padding: true
      actor:
        use_dynamic_bsz: true
        ppo_max_token_len_per_gpu: 25600
        ulysses_sequence_parallel_size: 2
        optim:
          lr: 1e-6 # or 5e-6, larger lr with warm up can result in better performance for SFT training.
      ref:
        log_prob_use_dynamic_bsz: ${trainer.trainer_config.actor_rollout_ref.actor.use_dynamic_bsz}
        log_prob_max_token_len_per_gpu: ${trainer.trainer_config.actor_rollout_ref.actor.ppo_max_token_len_per_gpu}
        ulysses_sequence_parallel_size: ${trainer.trainer_config.actor_rollout_ref.actor.ulysses_sequence_parallel_size}
monitor:
  monitor_type: wandb

其中，这里面/inspire/hdd/project/aiforquantum/zhengkaipeng-240108120123/test/codes/ContinualRL/datasets/data/chord/combined_medical_rl_dataset （RL数据）的数据约有25000条 ，/inspire/hdd/project/aiforquantum/zhengkaipeng-240108120123/test/codes/ContinualRL/datasets/data/chord/combined_medical_sft/rej--GLM-4.5-FP8 （SFT数据）约有23000条。发现一共只训练了205个step就结束了。请问这个205是怎么计算得到的呢？谢谢！

[garyzhang99]

@chenyushuo 作者您好！我对于chord算法里面的step计算不太确定。 我的yaml是：

project: "mix_chord"
name: "mix_chord_med_3.1"
checkpoint_root_dir: ${oc.env:TRINITY_CHECKPOINT_ROOT_DIR,./checkpoints}
algorithm:
  algorithm_type: mix_chord
  repeat_times: 8 # or 16 for better performance in math related tasks
  kl_loss_fn_args:
    kl_coef: 0.05
  sample_strategy_args:
    expert_data_ratio: 0.20
  policy_loss_fn_args: # feel free to change, we encourage you to try out different hyperparameters
    mu_warmup_steps: 0  # 0 for chord-mu and chord-phi
    mu_decay_steps: 0 # 200 for chord-mu and 0 for chord-phi
    mu_peak: 0.5 # 0.9 for chord-mu and 0.1 for chord-phi
    mu_valley: 0.1 # 0.05 for chord-mu and 0.1 for chord-phi
    enable_phi_function: true # false for chord-mu and true for chord-phi
    clip_range: 0.2
    sft_loss_agg_mode: "token-mean"
    use_dynamic_bsz: true
    ppo_mini_batch_size: 576 # 320 = 256 + 64; if you set repeat times = 16, then it shoudle be 32 * 16 + 64
    ppo_micro_batch_size_per_gpu: 8
    ngpus_trainer: 4
    train_batch_size_expert: 64
    train_batch_size_usual: 512 # 32 batchsize * 8 repeat times
model:
  model_path: ${oc.env:TRINITY_MODEL_PATH,/inspire/hdd/project/aiforquantum/zhengkaipeng-240108120123/test/models/Qwen3-4B-Instruct-2507}
  max_response_tokens: 10240
  max_model_len: 11264
cluster:
  node_num: 1
  gpu_per_node: 4
buffer:
  total_epochs: 1
  batch_size: 64
  train_batch_size: 576
  explorer_input:
    taskset:
      name: medmcqa_data_filtered_int
      storage_type: file
      path: ${oc.env:TRINITY_TASKSET_PATH, /inspire/hdd/project/aiforquantum/zhengkaipeng-240108120123/test/codes/ContinualRL/datasets/data/chord/combined_medical_rl_dataset}
      format:
        prompt_key: 'query'
        response_key: 'reference_answer'
      rollout_args:
        temperature: 1.0
        logprobs: 0
      workflow_args:
        with_think: false
    eval_tasksets:
    - name: medmcqa_data_test
      storage_type: file
      path: ${oc.env:TRINITY_TASKSET_PATH, /inspire/hdd/project/aiforquantum/zhengkaipeng-240108120123/test/codes/ContinualRL/datasets/data/chord/medmcqa_test1} 
      repeat_times: 1
      split: 'test'
      format:
        prompt_key: 'query'
        response_key: 'response'
      rollout_args:
        temperature: 1.0
        top_p: 0.7
      default_reward_fn_type: 'math_boxed_reward'
      reward_fn_args: {}
    default_workflow_type: 'medical_boxed_workflow'
  trainer_input:
    experience_buffer:
      name: math_buffer
      storage_type: queue
      path: 'sqlite:///mix_chord_medmcqa_3.1.db'
    auxiliary_buffers:
      sft_dataset:
        total_epochs: 1
        name: SFT_data
        storage_type: file
        schema_type: sft
        path: ${oc.env:TRINITY_SFT_DATASET_PATH,/inspire/hdd/project/aiforquantum/zhengkaipeng-240108120123/test/codes/ContinualRL/datasets/data/chord/combined_medical_sft/rej--Qwen3-235B-A22B-Instruct-2507-FP8}
        split: 'train'
        format:
          prompt_type: messages
          messages_key: 'messages'
explorer:
  eval_interval: 10
  runner_per_model: 8
  rollout_model:
    engine_num: 2
    tensor_parallel_size: 1
    enable_prefix_caching: true
    enforce_eager: false
    dtype: bfloat16
    seed: 42
synchronizer:
  sync_method: 'nccl'
  sync_interval: 1
  sync_timeout: 1200
trainer:
  save_interval: 10
  total_steps: 1000
  trainer_config:
    actor_rollout_ref:
      model:
        use_remove_padding: true
      actor:
        use_dynamic_bsz: true
        ppo_max_token_len_per_gpu: 25600
        ulysses_sequence_parallel_size: 2
        optim:
          lr: 1e-6 # or 5e-6, larger lr with warm up can result in better performance for SFT training.
      ref:
        log_prob_use_dynamic_bsz: ${trainer.trainer_config.actor_rollout_ref.actor.use_dynamic_bsz}
        log_prob_max_token_len_per_gpu: ${trainer.trainer_config.actor_rollout_ref.actor.ppo_max_token_len_per_gpu}
        ulysses_sequence_parallel_size: ${trainer.trainer_config.actor_rollout_ref.actor.ulysses_sequence_parallel_size}
monitor:
  monitor_type: wandb

其中，这里面/inspire/hdd/project/aiforquantum/zhengkaipeng-240108120123/test/codes/ContinualRL/datasets/data/chord/combined_medical_rl_dataset （RL数据）的数据约有25000条 ，/inspire/hdd/project/aiforquantum/zhengkaipeng-240108120123/test/codes/ContinualRL/datasets/data/chord/combined_medical_sft/rej--GLM-4.5-FP8 （SFT数据）约有23000条。发现一共只训练了205个step就结束了。请问这个205是怎么计算得到的呢？谢谢！

@waltonfuture 您好，因为您修改了对应SFT数据和Expert Data的数据比例，因此您这里的expert_data_ratio: 0.20也需要修改一下。

原始config中对应的sft数量是64，rl数量是32 * 8，所以expert data ratio是64/(64 + 32 * 8) = 0.2。

而调整了RL对应batchsize为64后，对应的正确expert_data_ratio应该是64/(64 + 64 * 8) = 0.11。
如果做这样的对应修改会正确的在23000/64=360 step得时候停下来。

而因为这里的expert_data_ratio设置成了0.2那么对应的SFT每次会sample 576*0.2=115个，那对应的就在23000/115=200 step左右的时候停下来了。

以及注意到这里的ngpus_trainer: 4也应该修改为ngpus_trainer: 2，虽然理论上loss_scale等比替换对训练影响不大。

我们后续会完善对应的config注释，或者加上更自动化的参数配置。

[waltonfuture]

谢谢您的及时回复！！！