Question about the training time

[yanchi-3dv]

Hi, thank you for the excellent work and open-sourcing this project!

I’m trying to reproduce the training of VLM-3R, but I noticed that the training time seems much longer than expected.
Specifically, using 8×H200 GPUs, the training takes more than 70 hours to complete.

Could you please confirm if this training time is normal for your reported configuration?
If not, could you share any tips for speeding up the training?

Here are some details of my setup:
GPUs: 8×H200
PyTorch: 2.1.1 + CUDA 12.1
FlashAttention: 2.3.3
Training script: (default config)

[jzh15]

Hi, thanks for your interest in VLM-3R and for sharing the detailed setup!

The ~70h training time you observed is likely caused by two main factors:

1. Epoch setting vs. early stopping
   In the released training config, the number of epochs is set to 5, but in our actual experiments we stop training after epoch 1. As a result, if you estimate total training time based on the full epoch count, it can appear roughly 5× longer than what we actually run, which likely explains the inflated time you’re seeing.

2. Feature extraction strategy
   As noted in the code comments, pre-extracting visual features before training can significantly reduce training time. If features are extracted on the fly during training, it adds substantial overhead. In our experience, pre-extracting features can provide roughly a 2–4× speedup, depending on the hardware and I/O setup.

After accounting for these two points (early stopping at epoch 1 and pre-extracting features), the overall training time should fall into a normal and expected range for the reported configuration.

Hope this helps clarify the discrepancy. Let us know if you still see unexpected slowdowns after these adjustments!