How to perform dense reconstruction using DiffMVS?

[wsh1057]

First of all, I would like to sincerely thank you for your excellent work.

When performing dense reconstruction, I followed the instructions and used the pretrained weights provided in the README, and achieved very good results on my own captured data.

Due to hardware limitations, I decided to train DiffMVS first and then use the trained weights for dense reconstruction. However, I noticed that in the diffusion part, the network architecture is defined as CasDiffMVS, and I am wondering whether this is the reason for the loading error I encountered.

Could you please provide guidance or example code on how to perform dense reconstruction using DiffMVS-trained weights? My experience is limited, and I would greatly appreciate your help.

I am looking forward to your reply.

[wsh1057]

I have noticed that you have already implemented DiffMVS in the diffusion framework. However, since the network architecture is different, the hyperparameters used during dense reconstruction need to be adjusted.

Could you please provide guidance on how to modify the hyperparameters so that DiffMVS can run properly? For example, how should parameters such as stage_iters, CostNum, and other related settings be configured or adapted?

[wsh1057]

I was able to successfully perform dense reconstruction on my own data by directly using the parameters from script/test/test_dtu. However, I do not fully understand the specific roles of these hyperparameters or how they influence the reconstruction results.

Could you please help clarify the purpose of these hyperparameters and explain what impact they have on the dense reconstruction process?

[FangjinhuaWang]

Hi,

Thanks for your interest in our work! You can find the definition of diffmvs here:

diffmvs/models/diffusion.py

Line 36 in 451349e

if args.stage_iters[2] == 0:

About the parameters, the architecture mainly depends on these: --stage_iters, --cost_dim_stage, --CostNum, --hidden_dim, --context_dim, --unet_dim.
stage_iters: Three numbers correspond to the number of GRU updates on 1/8 (no GRU), 1/4, 1/2 resolution respectively. It is used to separate two models. If the last number of stage_iters is 0, it means that there is no refinement on 1/2 resolution, which means that the model is diffmvs (single-stage refinement on 1/4 resolution).
cost_dim_stage: it is used to define the dimension of group-wise correlation, we set is as 4 for both methods and all stages.
CostNum: this is the number of samples. As discussed in our paper, we set it as 6 for DiffMVS, i.e. 0 6 0, and 4 for two stages of CasDiffMVS, i.e., 0 4 4.
hidden_dim, context_dim, unet_dim: these are feature dimensions, which should be the same during training and testing.

[wsh0711]

thank you!