Question about reproduction of Drivepi in open/closed-loop eval

[TabGuigui]

Thank you for sharing the excellent work. I have a few questions when reproducing the Drivepi baseline methods follow the instructions.

1. Open-loop result.
   I reproduce the drivepi in this repo and evaluate the open-loop result as :
   [2025-12-31 02:50:16,310][src.agent.drivepi0.eval][INFO] - Loaded model from /data/DriveMoE/log/train/drive-pi0/2025-12-25_22-24_42/checkpoint/step2640.pt
   [2025-12-31 02:50:17,130][src.agent.drivepi0.eval][INFO] - load_checkpoint took 61.20 seconds to execute.
   [2025-12-31 02:50:24,090][src.agent.drivepi0.eval][INFO] - Using cuda device: cuda:0 dtype: torch.float32
   [2025-12-31 02:50:24,090][src.agent.drivepi0.eval][INFO] - Allocated GPU memory after loading model: 13.24 GB
   init took 0.98 seconds to execute.
   Agent done
   Running evaluation ...
   100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 385/385 [38:31<00:00, 6.00s/it
   ]
   [2025-12-31 03:28:58,468][src.agent.drivepi0.eval][INFO] - Number of episodes: 385
   [2025-12-31 03:28:58,468][src.agent.drivepi0.eval][INFO] - Eval | l1 Loss: 0.224 | l2 Loss: 0.187 | acc thres 0.05: 0.341 | acc thres 0.1: 0.630 | acc thres 0.2: 0.832 | acc thres 0.3: 0.906 | acc thres 0.5: 0.96
   2
   I use the checkpoint with step2640, and the average L2 loss is 0.187 on 385 episodes. However, this result is much lower than that reported in the paper(0.56), I don't know where I'm wrong.

2. The sensor setting.
   I find in the training stage of Drivepi, only the front camera is used, while in the inference state of closed loop eval, I find in line 314 & 329 of drivepi0_carla_agent.py, it is default to use the additional back camera? I'm not sure why the setting is changed.

3. Closed-loop time.
   I use the checkpoint I reproduce(since there is no provided checkpoint in this repo) in closed-loop eval, and the game time is much longer than that reported in the repo. Such as route RouteScenario_1825_rep, and the report game time is 139.4, while my checkpoint has to last over 190s. I wonder how long the evaluation will take for the whole 220 route for DrivePi

[chaiyilin024]

Thank you for your interest in our work and for sharing the detailed reproduction results. Here are the responses to your questions:

(1) Open-loop result discrepancy
Thank you for pointing this out. For fair comparison with other methods, the L2 loss should be calculated using only the future 20 trajectory points, while closed-loop evaluation has no restrictions on the number of prediction points. There was a bug in our previous implementation, and this has been corrected in the latest version of the paper.

(2) Sensor setting
You are correct. DrivePI uses two front-view images as input. We will fix the problem with the agent code accordingly.

(3) Closed-loop evaluation time
We use H200 for inference, and the evaluation for all 220 routes takes approximately 10 hours using 4 H200 GPUs. If your inference is too slow, you can try reducing the number of tasks per GPU to improve efficiency.

[TabGuigui]

@chaiyilin024 Thanks for your quick reply.

For question1, how should I change the eval script in eval.py?

For question3, I added more memory, and the progress ratio can be maintained at this speed:
System time = 388.684 -- Game time = 16.300 -- Ratio = 0.042x
Is the speed normal for DrivePI?

Moreover, with my own checkpoint, I find that many routes that are successful in your log will be failures, even though I only use the 2 front cameras. I wish you could give some advice.

And are you going to share the Drivepi checkpoint in this repo?

[TabGuigui]

@chaiyilin024 Hello, I reproduced the drivepi0 and successful finish closed-loop eval with 220 routes. However, the result is far low than reported as follows:
"driving score": 46.92220729090908,
"success rate": 0.2,
"eval num": 220

I hope you can give some advice.

[chaiyilin024]

@TabGuigui Thanks for the feedback. As the author, I should note that results can vary with different random seeds, but such a large gap is unusual.

To help us debug this together, could you share your detailed test logs? Additionally, it would be great if you could provide specifics about your training setup.

We are also preparing to release a dedicated DrivePI checkpoint for closed-loop eval, along with an updated testing README for better clarity. Stay tuned.

[TabGuigui]

@chaiyilin024 here is the test log merge.json
The dependency version is torch2.6.0 cuda12.4
I use the train config in ./config/train/DrivePi0/base.yaml, and 8 A100 gpu. I fixed the sensor setting in eval.py to remove image_back.
In close-loop I set the config as TEAM_CONFIG=/data/DriveMoE/config/eval/DrivePi0/base.yaml+/data/DriveMoE/log/train/drive-pi0/2025-12-25_22-24_42/checkpoint/step2640.pt

Here is the loss curve:

[chaiyilin024]

@TabGuigui Your loss curve seems less smooth than ours. We are currently uploading the latest float16 and float32 checkpoints for Drivepi0. You are welcome to try them on your setup to see if they reproduce the results we have shared. If you still encounter issues with the evaluation, please feel free to provide more details, and we will look into it further.

[TabGuigui]

@chaiyilin024 https://huggingface.co/rethinklab/DrivePi0 This page is invalid, and I can't get the model ckpt.
Can you paste the corresponding loss curve in wandb?
And which checkpoint is corresponding to that in the released paper, the bf16 and f32?

[jayyoung0802]

Hi @TabGuigui, the repo has been updated. You can find it in the evaluation.

And our paper uses bf16; fp32 is a recent experimental discovery.

[TabGuigui]

@chaiyilin024 @jayyoung0802 Thanks for your reply.

However, I pull the latest commit and retrain the model, but still can not get the open-loop result in the paper.
Using the provided bf16 ckpt, open-loop result is Eval | l1 Loss: 1.051 | l2 Loss: 4.198 | acc thres 0.05: 0.218 | acc thres 0.1: 0.457 | acc thres 0.2: 0.625 | acc thres 0.3: 0.724 | acc thres 0.5: 0.850
Using the reproduced ckpt by myself(change the future time to 20), open-loop result is Eval | l1 Loss: 0.423 | l2 Loss: 0.941 | acc thres 0.05: 0.555 | acc thres 0.1: 0.705 | acc thres 0.2: 0.847 | acc thres 0.3: 0.912 | acc thres 0.5: 0.966.

Does this result match your open-loop eval result?

I will try the closed-loop result today.

[jayyoung0802]

Hi @TabGuigui, when training this model, we are predicting the next 10 steps(better than 20 steps in closed-loop evaluation), so you need to change the 10 steps in the config for open-loop result(other baselines in bench2drive predict 20 steps). The paper hasn't been updated yet; please refer to the current results.

[jayyoung0802]

And we look forward to your closed-loop results.