ARRay-Tracing: a middleware to integrate real-time ray tracing and augmented reality

Abstract

In recent years we saw the increase and popularization of augmented reality applications. However, in these systems, it is common to perceive a visual discrepancy between real and virtual elements, which leads to a lack of realism, that is one of the reasons that discourage the use of this type of application. Physically-based algorithms, like ray tracing, can generate renderings with a high degree of photorealism and are becoming popular after the recent development of hardware accelerators. Some works in literature combine these two technologies, augmented reality and ray tracing, but rigidly, without modularization, making the solutions dependent on specific frameworks. In this work, we propose a middleware to integrate augmented reality and real-time ray tracing by mapping the camera’s coordinates into the ray tracing framework, and performing an image composition process. Besides, the middleware works in a modularized way, allowing to choose between existing libraries and frameworks those that better fulfill the user’s needs and expertise. We build an application using the ARRay-Tracing middleware to integrate two ray tracing frameworks, the Optix and the VKRay, to an augmented reality framework, the artoolkitX, obtaining in real-time more realistic reflections and refractions. After the evaluation, we concluded that our middleware enables the modular integration without adding processing overhead to the application, achieving the frame rate of 30 fps for rendering less complex scenes.