Towards Face-to-Face Telepresence Applications in Virtual Reality with Generative Neural Nets

VR promises great potential in a variety of applications. Especially low-cost VR that employs VR headsets is becoming more commonplace. However, authentic social interaction in VR is still an unsolved problem especially if faces are hidden behind a VR headset and it is only feasible to represent communication partners as stylized avatars. In this work, we present a working prototype that presents an easy and inexpensive way for 3D capturing and reconstruction of facial avatars for telepresence applications in VR using low-cost components such as a commodity RGBD camera and a VR headset. The system is able to preserve individual characteristics of facial expressions and, therefore, is able to convey important non-verbal information. Even though, the system is not finished yet, we have already achieved stunning reconstruction results of faces and it demonsrates the potential for using neural networks for social VR applications in the future.

Video of the presentation