Some issues when reproducing

[szl2001]

Hello, thank you very much for your excellent work! While reproducing your results, I encountered some issues that I would like to ask for your advice on:

1. During reproduction, I found the training progress to be quite slow. Using 2×A100 GPUs and 10 spatial tasks, after 10 hours I only generated about 330 episodes. According to the paper, fine-tuning for 48 hours yielded 10,000 steps (100k episodes?).

2. I noticed that in the current code, openvla-7b is used as the value model, but it seems that critic warmup was not applied. Is that correct?

3. Looking at the code, it seems that the environment initialization procedure during fine-tuning and evaluation is the same. Is there any distinction between the initialization vectors for training vs. evaluation? Or is it possible that the evaluation could also use environments seen during fine-tuning?

[szl2001]

Additionally, I often encounter NCCL multi-GPU communication issues during fine-tuning, which seem to occur due to all-gather timeouts. This happens both when using 8 GPUs and 2 GPUs. May I ask if you have experienced similar issues before? Thank you!

[GuanxingLu]

Additionally, I often encounter NCCL multi-GPU communication issues during fine-tuning, which seem to occur due to all-gather timeouts. This happens both when using 8 GPUs and 2 GPUs. May I ask if you have experienced similar issues before? Thank you!

Got it, the problem is in saving checkpoint logic. I pushed a fix to avoid these sync timeouts, please check it out, thank you!

[GuanxingLu]

Hello, thank you very much for your excellent work! While reproducing your results, I encountered some issues that I would like to ask for your advice on:

1. During reproduction, I found the training progress to be quite slow. Using 2×A100 GPUs and 10 spatial tasks, after 10 hours I only generated about 330 episodes. According to the paper, fine-tuning for 48 hours yielded 10,000 steps (100k episodes?).
2. I noticed that in the current code, openvla-7b is used as the value model, but it seems that critic warmup was not applied. Is that correct?
3. Looking at the code, it seems that the environment initialization procedure during fine-tuning and evaluation is the same. Is there any distinction between the initialization vectors for training vs. evaluation? Or is it possible that the evaluation could also use environments seen during fine-tuning?

1. the step is not episodes in our repo.
2. critic warmup is needed, sorry for the misleading param setup.
3. our RL training uses the same set for training and evaluation... It is weird, we are trying to split out train and valid sets.

[szl2001]

Hello, thank you very much for your excellent work! While reproducing your results, I encountered some issues that I would like to ask for your advice on:

1. During reproduction, I found the training progress to be quite slow. Using 2×A100 GPUs and 10 spatial tasks, after 10 hours I only generated about 330 episodes. According to the paper, fine-tuning for 48 hours yielded 10,000 steps (100k episodes?).
2. I noticed that in the current code, openvla-7b is used as the value model, but it seems that critic warmup was not applied. Is that correct?
3. Looking at the code, it seems that the environment initialization procedure during fine-tuning and evaluation is the same. Is there any distinction between the initialization vectors for training vs. evaluation? Or is it possible that the evaluation could also use environments seen during fine-tuning?

1. the step is not episodes in our repo.
2. critic warmup is needed, sorry for the misleading param setup.
3. our RL training uses the same set for training and evaluation... It is weird, we are trying to split out train and valid sets.

Thank you for your reply, it was very helpful for me. I saw that you mentioned in another issue that you are using global_step, so does the "step" in the paper correspond to the global step in the code? I’m not sure if I understand correctly: if I set num_steps to 128 and rollout_batch_size to 10, then generating one round of episodes would equal 1280 steps?

[szl2001]

Hello, thank you very much for your excellent work! While reproducing your results, I encountered some issues that I would like to ask for your advice on:

1. During reproduction, I found the training progress to be quite slow. Using 2×A100 GPUs and 10 spatial tasks, after 10 hours I only generated about 330 episodes. According to the paper, fine-tuning for 48 hours yielded 10,000 steps (100k episodes?).
2. I noticed that in the current code, openvla-7b is used as the value model, but it seems that critic warmup was not applied. Is that correct?
3. Looking at the code, it seems that the environment initialization procedure during fine-tuning and evaluation is the same. Is there any distinction between the initialization vectors for training vs. evaluation? Or is it possible that the evaluation could also use environments seen during fine-tuning?

1. the step is not episodes in our repo.
2. critic warmup is needed, sorry for the misleading param setup.
3. our RL training uses the same set for training and evaluation... It is weird, we are trying to split out train and valid sets.

When I run the evaluation script, I also encounter the following error. Could this be due to insufficient GPU memory? I have checked system memory, shared memory, and several other settings but haven’t been able to identify the cause.

[2025-09-03 15:32:50,627 C 267234 267234] shared_memory.cc:32: mmap failed
*** StackTrace Information ***
/workspace/S/liyutai/rocky/envs/vlarl/lib/python3.10/site-packages/ray/_raylet.so(+0x115b93a) [0x7fb12125293a] ray::operator<<()
/workspace/S/liyutai/rocky/envs/vlarl/lib/python3.10/site-packages/ray/_raylet.so(_ZN3ray6RayLogD1Ev+0x4c1) [0x7fb121254d31] ray::RayLog::~RayLog()
/workspace/S/liyutai/rocky/envs/vlarl/lib/python3.10/site-packages/ray/_raylet.so(+0x9c5c68) [0x7fb120abcc68] plasma::ClientMmapTableEntry::ClientMmapTableEntry()
/workspace/S/liyutai/rocky/envs/vlarl/lib/python3.10/site-packages/ray/_raylet.so(+0x9c0854) [0x7fb120ab7854] plasma::PlasmaClient::Impl::GetStoreFdAndMmap()
/workspace/S/liyutai/rocky/envs/vlarl/lib/python3.10/site-packages/ray/_raylet.so(_ZN6plasma12PlasmaClient4Impl17HandleCreateReplyERKN3ray8ObjectIDEbPKhPmPSt10shared_ptrINS2_6BufferEE+0x387) [0x7fb120ab87d7] plasma::PlasmaClient::Impl::HandleCreateReply()
/workspace/S/liyutai/rocky/envs/vlarl/lib/python3.10/site-packages/ray/_raylet.so(_ZN6plasma12PlasmaClient4Impl22CreateAndSpillIfNeededERKN3ray8ObjectIDERKNS2_3rpc7AddressEblPKhlPSt10shared_ptrINS2_6BufferEENS_7flatbuf12ObjectSourceEi+0x22b) [0x7fb120ab936b] plasma::PlasmaClient::Impl::CreateAndSpillIfNeeded()
/workspace/S/liyutai/rocky/envs/vlarl/lib/python3.10/site-packages/ray/_raylet.so(_ZN6plasma12PlasmaClient22CreateAndSpillIfNeededERKN3ray8ObjectIDERKNS1_3rpc7AddressEblPKhlPSt10shared_ptrINS1_6BufferEENS_7flatbuf12ObjectSourceEi+0x2b) [0x7fb120ab99bb] plasma::PlasmaClient::CreateAndSpillIfNeeded()
/workspace/S/liyutai/rocky/envs/vlarl/lib/python3.10/site-packages/ray/_raylet.so(_ZN3ray4core29CoreWorkerPlasmaStoreProvider6CreateERKSt10shared_ptrINS_6BufferEEmRKNS_8ObjectIDERKNS_3rpc7AddressEPS4_bb+0xe6) [0x7fb120a4c306] ray::core::CoreWorkerPlasmaStoreProvider::Create()
/workspace/S/liyutai/rocky/envs/vlarl/lib/python3.10/site-packages/ray/_raylet.so(_ZN3ray4core29CoreWorkerPlasmaStoreProvider11WarmupStoreEv+0xa2) [0x7fb120a4e852] ray::core::CoreWorkerPlasmaStoreProvider::WarmupStore()
/workspace/S/liyutai/rocky/envs/vlarl/lib/python3.10/site-packages/ray/_raylet.so(_ZN3ray4core29CoreWorkerPlasmaStoreProviderC2ERKSsSt10shared_ptrINS_6raylet12RayletClientEES4_INS0_16ReferenceCounterEESt8functionIFNS_6StatusEvEEbSA_IFSsvEE+0x2ab) [0x7fb120a4ee7b] ray::core::CoreWorkerPlasmaStoreProvider::CoreWorkerPlasmaStoreProvider()
/workspace/S/liyutai/rocky/envs/vlarl/lib/python3.10/site-packages/ray/_raylet.so(_ZN3ray4core10CoreWorkerC2ENS0_17CoreWorkerOptionsERKNS_8WorkerIDE+0x1d6d) [0x7fb1209d952d] ray::core::CoreWorker::CoreWorker()
/workspace/S/liyutai/rocky/envs/vlarl/lib/python3.10/site-packages/ray/_raylet.so(_ZN3ray4core21CoreWorkerProcessImplC2ERKNS0_17CoreWorkerOptionsE+0x5b3) [0x7fb1209ec193] ray::core::CoreWorkerProcessImpl::CoreWorkerProcessImpl()
/workspace/S/liyutai/rocky/envs/vlarl/lib/python3.10/site-packages/ray/_raylet.so(_ZN3ray4core17CoreWorkerProcess10InitializeERKNS0_17CoreWorkerOptionsE+0x34) [0x7fb1209ed254] ray::core::CoreWorkerProcess::Initialize()
/workspace/S/liyutai/rocky/envs/vlarl/lib/python3.10/site-packages/ray/_raylet.so(+0x74efdf) [0x7fb120845fdf] pyx_pw_3ray_7_raylet_10CoreWorker_1__cinit()
/workspace/S/liyutai/rocky/envs/vlarl/lib/python3.10/site-packages/ray/_raylet.so(+0x750289) [0x7fb120847289] __pyx_tp_new_3ray_7_raylet_CoreWorker()
python(_PyObject_MakeTpCall+0x1d5) [0x56311868d905] _PyObject_MakeTpCall
python(_PyEval_EvalFrameDefault+0x51ee) [0x563118689c8e] _PyEval_EvalFrameDefault
python(_PyFunction_Vectorcall+0x6c) [0x563118694b7c] _PyFunction_Vectorcall
python(_PyEval_EvalFrameDefault+0x1341) [0x563118685de1] _PyEval_EvalFrameDefault
python(_PyFunction_Vectorcall+0x6c) [0x563118694b7c] _PyFunction_Vectorcall
python(_PyEval_EvalFrameDefault+0x2c3e) [0x5631186876de] _PyEval_EvalFrameDefault
python(_PyFunction_Vectorcall+0x6c) [0x563118694b7c] _PyFunction_Vectorcall
python(_PyEval_EvalFrameDefault+0x49b7) [0x563118689457] _PyEval_EvalFrameDefault
python(_PyFunction_Vectorcall+0x6c) [0x563118694b7c] _PyFunction_Vectorcall
python(_PyEval_EvalFrameDefault+0x309) [0x563118684da9] _PyEval_EvalFrameDefault
python(+0x146fb2) [0x56311869ffb2] method_vectorcall
python(PyObject_Call+0xbc) [0x5631186a094c] PyObject_Call
python(_PyEval_EvalFrameDefault+0x2c3e) [0x5631186876de] _PyEval_EvalFrameDefault
python(+0x146fb2) [0x56311869ffb2] method_vectorcall
python(PyObject_Call+0xbc) [0x5631186a094c] PyObject_Call
python(_PyEval_EvalFrameDefault+0x2c3e) [0x5631186876de] _PyEval_EvalFrameDefault
python(_PyFunction_Vectorcall+0x6c) [0x563118694b7c] _PyFunction_Vectorcall
python(_PyEval_EvalFrameDefault+0x1341) [0x563118685de1] _PyEval_EvalFrameDefault
python(_PyFunction_Vectorcall+0x6c) [0x563118694b7c] _PyFunction_Vectorcall
python(_PyEval_EvalFrameDefault+0x2c3e) [0x5631186876de] _PyEval_EvalFrameDefault
python(_PyFunction_Vectorcall+0x6c) [0x563118694b7c] _PyFunction_Vectorcall
python(_PyEval_EvalFrameDefault+0x309) [0x563118684da9] _PyEval_EvalFrameDefault
python(+0x1cca8c) [0x563118725a8c] _PyEval_Vector
python(PyEval_EvalCode+0x87) [0x5631187259d7] PyEval_EvalCode
python(+0x1fceba) [0x563118755eba] run_eval_code_obj
python(+0x1f8343) [0x563118751343] run_mod
python(+0x97098) [0x5631185f0098] pyrun_file.cold
python(_PyRun_SimpleFileObject+0x1bd) [0x56311874bb7d] _PyRun_SimpleFileObject
python(_PyRun_AnyFileObject+0x44) [0x56311874b714] _PyRun_AnyFileObject
python(Py_RunMain+0x31b) [0x563118748a7b] Py_RunMain
python(Py_BytesMain+0x37) [0x563118719117] Py_BytesMain
/lib64/libc.so.6(__libc_start_main+0xe5) [0x7fb398b79d85] __libc_start_main
python(+0x1c002e) [0x56311871902e]

[szl2001]

When I used OpenVLA's official evaluation method to test the merged model, I also observed the performance decline mentioned in other issues, with the accuracy similarly dropping to just over 60%. I understand that you previously mentioned plans to update the curriculum learning strategy, and though it hasn’t been implemented yet, I’m really looking forward to any future code improvements. Thank you so much for your attention and effort!

[GuanxingLu]

Hello, thank you very much for your excellent work! While reproducing your results, I encountered some issues that I would like to ask for your advice on:

1. During reproduction, I found the training progress to be quite slow. Using 2×A100 GPUs and 10 spatial tasks, after 10 hours I only generated about 330 episodes. According to the paper, fine-tuning for 48 hours yielded 10,000 steps (100k episodes?).
2. I noticed that in the current code, openvla-7b is used as the value model, but it seems that critic warmup was not applied. Is that correct?
3. Looking at the code, it seems that the environment initialization procedure during fine-tuning and evaluation is the same. Is there any distinction between the initialization vectors for training vs. evaluation? Or is it possible that the evaluation could also use environments seen during fine-tuning?

1. the step is not episodes in our repo.
2. critic warmup is needed, sorry for the misleading param setup.
3. our RL training uses the same set for training and evaluation... It is weird, we are trying to split out train and valid sets.

Thank you for your reply, it was very helpful for me. I saw that you mentioned in another issue that you are using global_step, so does the "step" in the paper correspond to the global step in the code? I’m not sure if I understand correctly: if I set num_steps to 128 and rollout_batch_size to 10, then generating one round of episodes would equal 1280 steps?

Yes, the generated steps in one evaluation-improving loop are equal to rollout_batch_size * num_steps.