ORStereo: Occlusion-Aware Recurrent Stereo Matching for 4K-Resolution Images

This is the project page of the IROS submission “ORStereo: Occlusion-Aware Recurrent Stereo Matching for 4K-Resolution Images”

Overview

Stereo reconstruction models trained on small images do not generalize well to high-resolution data. Training a model on high-resolution image size faces difficulties of data availability and is often infeasible due to limited computing resources. In this work, we present the Occlusion-aware Recurrent binocular Stereo matching (ORStereo), which deals with these issues by only training on available low disparity range stereo images. ORStereo generalizes to unseen high-resolution images with large disparity ranges by formulating the task as residual updates and refinements of an initial prediction. ORStereo is trained on images with disparity ranges limited to 256 pixels, yet it can operate 4K-resolution input with over 1000 disparities using limited GPU memory. We test the model’s capability on both synthetic and real-world high-resolution images. Experimental results demonstrate that ORStereo achieves comparable performance on 4K-resolution images compared to state-of-the-art methods trained on large disparity ranges. Compared to other methods that are only trained on low-resolution images, our method is 70% more accurate on 4K-resolution images.

4K-resolution stereo dataset

We collected a set of 4K-resolution stereo images for evaluating ORStereo’s performance on high-resolution data. These images and ground truth data will be made publicly available. The images are collected in various simulated environments rendered by the Unreal Engine and the AirSim plugin. 100 pairs of photo-realistic stereo images from 7 indoor and outdoor scenes are collected. We set up a virtual stereo camera with a baseline of 0.38m. In 3 of the 7 scenes, the cameras have a horizontal viewing angle of 46 degrees. For the remaining 4 scenes, the horizontal viewing angle is 60 degrees. The image size of the virtual camera is 4112x3008 pixels. These camera parameters are selected to match the real-world sensor that we are using for data collection. The AirSim plugin provides us with a depth image for every captured RGB image. We compute the true disparities and occlusion masks from the depth images.

The following figure shows another pair of the collected stereo images in detail. Note that all disparities and occlusion masks are associated with the left image.

Access the 4K-resolution dataset

All of the stereo images, together with the true disparity and occlusion mask are available here.

Concerning the large file size of a 4K-resolution floating point image, we save the disparity as compressed PNG files in RGBA format. We provide a simple Python function to read the floating point disparity back from those PNG files. Access the code here.

More results

Results on the Scene Flow dataset

When trained on the Scene Flow dastaset only, ORStereo achieves similar EPE value (0.74) among the state-of-the-art models trained with low-resolution data.

When working on low-resolution inputs such as the Scene Flow dataset, ORStereo only goes through the first phase. Two sample results are shown as follows.

In the previous image:

• a, b) Left and right images.
• c, d) True disparity and prediction after NLR.
• e, f) True disparity and prediction before NLR.
• g, h) True disparity and prediction by RRU.
• i, j) True disparity and prediction by BDE.
• k, l) True occlusion and prediction by BME.
• m, n) True occlusion and prediction by RRU.
• The numbers on the predicted disparities are the EPE and standard deviation.
• The true and predicted disparity values are all scaled according to the sizes of the feature levels. E.g., g) has a magnitude 1/2 of e) or c). The full-resolution versions of this figure can be found here.

Results on the Middlebury dataset

ORStereo acheives better accuracy on full resolution benchmark than the state-of-the-art models that trained on low-resolution data. Note that HSM is trained on high-resolution data.

We have submitted our results to the Middlebury evaluation page. Check out our results under the name ORStereo.

Results on 4K-resolution stereo images

ORStereo achieves the best EPE among all the related state-of-the-art models including the HSM, which is a model trained on high-resolution data.

The following figures are the sample cases shown in the submitted paper.

Results on 4K-resolution stereo images. a) the left image. b, c) the disparity and error of the first phase. d) the true disparity. e, f) the disparity and error of the second phase. Note that the error gets improved in the second phase.

The full resolution version of the previous two figures are available here.

Manuscript

Please refer to this arXiv link.

@misc{hu2021orstereo,
      title={ORStereo: Occlusion-Aware Recurrent Stereo Matching for 4K-Resolution Images}, 
      author={Yaoyu Hu and Wenshan Wang and Huai Yu and Weikun Zhen and Sebastian Scherer},
      year={2021},
      eprint={2103.07798},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Contact

• Yaoyu Hu: yaoyuh@andrew.cmu.edu
• Sebastian Scherer: (basti [at] cmu [dot] edu)

Acknowledgments

This work was supported by Shimizu Corporation.

References