Directional light vector estimation from a virtual AR object and its 2D shadow mask

Abstract

Augmented Reality (AR) is a technology that enables merging real and virtual environments. Since AR works with 2D image frames, a coordinate system from the viewer’s perspective must be set, therefore allowing the 3D virtual elements to be merged to the 2D picture correctly. This is commonly achieved by using a fiducial marker, a known planar object that is set as the origin of the new coordinate system. However, simply finding that coordinate system is not enough to create a scene where virtual objects are seamlessly merged to the real world. There are issues such as correctly handling occlusion, field of view, shading, lighting, and others. In this work, we aim to solve the lighting problem. This work presents a novel method for inferring the main directional light in a 3D scene, given only 2D inputs, namely a camera image and a rough shadow mask. A two stage algorithm is proposed, in which the first stage handles the inputs and makes an initial estimation for the light source, and the second and main stage finds the most suitable vector corresponding to the directional light, given the system constraints. We sample a search space around the initial estimation, to which the proposed algorithm is constrained, to measure the accuracy of our method in finding the vector associated with the global maximum value of that space. The experiments are made both in virtual and real environments, in scenes with different levels of control and known data. Results show that our method is capable of finding the 3D light vector from the 2D scene and enhancing the initial rough shadow input.