Dr. Ralf Schäfer
Ralf Schäfer is Head of the Video Division at Fraunhofer Heinrich Hertz Institute (HHI) in Berlin. After studying Electrical Engineering at the Technincal University Berlin (TUB) and receiving his Dipl.-Ing. Degree, he joined HHI as a researcher. In 1984, he received his doctorate at TUB in the area of digital video coding of TV signals. He became head of department in 1989 and is currently responsible for 90 researchers and 50 undergraduate students. His research interests cover all areas related to images and video, from acquisition to display and from algorithm development to ASIC implementation. Beside his role as Head of Division, he is responsible for 3 technology centers, namely the CINQ Center, a forum for knowledge and technology transfer related to Smart Data (www.cinq.de), the Innovation Center for Immersive Imaging Technologies – 3IT (www.3it-berlin.de) and the TiME-Lab (www.timelab-hhi.com), where immersive technologies are developed and demonstrated.
VOLUMETRIC VIDEO – CREATING REALISTIC DYNAMIC 3D MODELS OF PERSONS
R. Schäfer, O. Schreer, I. Feldmann, P. Kauff
Fraunhofer Heinrich Hertz Institute, Germany
Thanks to the availability of new head mounted displays for Virtual Reality, such as Oculus Rift and HTC Vive, the creation of fully immersive environments has gained a tremendous push. In addition, new Augmented Reality glasses and mobile devices reach the market that allow for novel mixed reality experiences. In addition, mobile devices are now are capable to register their environment and put CGI objects at fixed positions in the viewing space. There are many application in different fields for these technologies, such as gaming, entertainment, virtual prototyping, planning, architecture, construction, chemistry, environmental studies, energy, e-learning and edutainment. In addition, there is also a high potential for the cultural sector. For many of these applications a realistic and lively representation of human beings is desired. This can be achieved with a new technology called Volumetric Video. This technology allows to capture real persons with a specially designed system and to create naturally moving dynamic 3D models, which can then be observed from arbitrary viewpoints in a virtual or augmented real scene.
The capture system developed by Fraunhofer HHI consists of an integrated multi-camera and lighting system for full 360 degree acquisition. The cylindrical capture studio uses 32 high resolution 20MPixel cameras and a large number (120-200) of LED panels that allow for arbitrary lit background. Hence, diffuse lighting from any direction and automatic keying is supported. The avoidance of green screen and provision of diffuse lighting offers best possible conditions for re-lighting of the dynamic 3D models afterwards at design stage of the VR experience. The 32 cameras are arranged in 16 stereo pairs, which are equally distributed in the cylinder. The stereo pairs create 3D information from all different viewpoints, similar as humans perceive with their two eyes. A 3D point cloud fusion algorithm creates a common and consistent dynamic 3D point cloud that is further transformed to a 3D mesh.
After the completion of the first prototype at Fraunhofer HHI in October 2017, a second, professional studio has been built up in Potsdam Babelsberg and is operated by Volucap GmbH, a joint venture between Studio Babelsberg, Arri, UFA, Interlake and Fraunhofer HHI.
This paper describes the construction of the volumetric studio, the capture and processing techniques and presents a number of production carried out for different field of application.