MCPDepth: Omnidirectional Depth Estimation via Stereo Matching from Multi-Cylindrical Panoramas

Feng Qiao, Zhexiao Xiong, Xinge Zhu, Yuexin Ma, Qiumeng He, Nathan Jacobs

 

Introduction

We introduce Multi-Cylindrical Panoramic Depth Estimation (MCPDepth), a novel two-stage framework designed to enhance omnidirectional depth estimation through stereo matching across multiple cylindrical panoramas. MCPDepth initially performs stereo matching using cylindrical panoramas, followed by a robust fusion of the resulting depth maps from different views. Unlike existing methods that rely on customized kernels to address distortions, MCPDepth utilizes standard network components, facilitating seamless deployment on embedded devices while delivering exceptional performance. To effectively address vertical distortions in cylindrical panoramas, MCPDepth incorporates a circular attention module, significantly expanding the receptive field beyond traditional convolutions. We provide a comprehensive theoretical and experimental analysis of common panoramic projections—spherical, cylindrical, and cubic—demonstrating the superior efficacy of cylindrical projection.

Cylindrical projection

A qualitative comaprison of Cassini (spherical) projection, cubic projection, and cylindircal projection for stereo matching:

An example of quantitative comparison for stereo matching of paired panoramas under spherical and cylindrical projection:

Circular Attention

The circular attention module is used to overcome the distortion along the vertical axis.

Dataset preparation

Deep360

Download: Deep360. More details can be founde in MODE.

cd scripts
# change the dataset name, root_path, and save_path in the file and then run
python spherical2cylindrical_disp.py  # convert disparity
python spherical2cylindrical.py  # convert RGB image

3D60 dataset preparation

Download: 3D60

# convert panorama to Cassini Projection
cd dataloader
python dataset3D60Loader.py
# convert Cassini Projection to cylindrical panorama
cd ../scripts
# change the dataset name, root_path, and save_path in the file and then run
python spherical2cylindrical_disp.py  # convert disparity
python spherical2cylindrical.py  # convert RGB image

Requirements

• gcc/g++ <=7.5.0 (to compile the sphere convolution operator)
• PyTorch >=1.5.0
• tensorboardX
• cv2
• numpy
• PIL
• numba
• prettytable (to show the error metrics)
• tqdm (to visualize the progress bar)

Usage

# trian stereo matching model
bash train_disparity.sh
# test stereo matching model
bash test_disparity.sh
# generate predicted disparity maps and confidence maps
bash save_output_disparity_stage.sh
# train fusion model
bash train_fusion.sh
# test fusion model
bash test_fusion.sh

Pretrained Models

Our pre-trained models can be found ./pretrained_model

Acknowledgements

Our project rely on some awesome repos : MODE, PSMNet. We thank the original authors for their excellent work.

Citation

If you find our work useful in your research, please consider citing our paper:

@article{qiao2024mcpdepth,
  title={MCPDepth: Omnidirectional Depth Estimation via Stereo Matching from Multi-Cylindrical Panoramas},
  author={Qiao, Feng and Xiong, Zhexiao and Zhu, Xinge and Ma, Yuexin and He, Qiumeng and Jacobs, Nathan},
  journal={arXiv preprint arXiv:2408.01653},
  year={2024}
}