With the help of mixed reality, users can achieve a more authentic feeling of being at another location. However, head-mounted displays (HMDs), meaning VR and AR glasses, cover parts of the face. Therefore, they are not applicable to classic applications such as Skype or Facetime. Our system aims to offer a solution to this problem by recording facial features and movements with cameras inside the HMD and sending image data to a neural network that is able to reconstruct the user’s three-dimensional face in real time. All that is required beforehand is a short recording process using color and depth cameras, during which the system memorizes the user’s face. This creates a database, which is used to reconstruct the respective face when the user wears an HMD and which is related to our project Face Tracking VR.


This project is sponsored by: German Federal Ministry of Education and Research (BMBF) under the project number 16SV8182 and 13FH022IX6. Project names: HIVE-Lab and Interactive body-near production technology 4.0 (German: Interaktive körpernahe Produktionstechnik 4.0 (iKPT4.0)).


Alexander Pech & Philipp Ladwig