A Statistical Inverse Ray Tracing Approach to Multi-View Stereo

Introduction  |  Datasets  |  Recent Results  |  Comprison with PMVS2  |  Convergence

Abstract:

This work addresses the problem of capturing 3D geometry and appearance from multiple 2D images taken from different views, which has wide applications in virtual reality, entertainment, human-computer interface, surveillance, navigation, and high-level vision tasks, etc. In this work, we present an inverse ray tracing approach based on statistical inference. Instead of matching image features/pixels across images, the inverse ray tracing approach models the image generation process directly and searches for the best 3D geometry and surface reflectance model to explain all the observations. It can better handle difficult problems in multi-view stereo, including large camera baseline, occlusion, matching ambiguities, etc. than traditional methods, without additional information and assumptions, such as initial surface estimate or simple background assumption. Here the image generation process is modeled through volumetric ray tracing, where the oclusion/visibility is accurately modeled. All the constraints (including ray constraints and prior knowledge about the geometry) are put into the Ray Markov Random Field (Ray MRF) formulation. This MRF model is unusual in the sense that the ray-clique, which models the ray-tracing process, consists of thousands of random variables, instead of two to dozens as in typical MRFs. This presents a big challenge to the inference algorithm, because of the combinatorial explosion of possible configurations. In this work an algorithm with linear computational complexity is developed to solve the estimation problem. More specifically, we show that a highly optimized belief propagation algorithm, deep belief propagation (DBP), can tackle the challenging problem effectively and efficiently, by exploring the deep factorization structure of the ray-clique energy. Then the DBP algorithm is also extended to solve the inference problem for a broader class of higher-order MRFs. The algorithm developed is capable of handling general and complex scenes of large varieties, general camera configurations (both small and large baselines), and can generate accurate and photo-realistic 3D models. These have been verified by extensive experiments on standard and home-grown challenging datasets.

Publication:

• Shubao Liu and David B. Cooper, "A Complete Statistical Inverse Ray Tracing Approach to Multi-view Stereo", to appear in Proceedings of CVPR'11, Colorado Springs, CO, 2011 [preprint pdf].