Events Calendar

PhD Defense- Koki Nagano

PhD Candidate: Koki Nagano

Committee: Paul Debevec (CS, chair), Hao Li (CS), Jernej Barbic (CS), Aiichiro Nakano (CS), Michelle Povinelli (EE)

Title: Multi-scale Dynamic Capture for High Quality Digital Humans

Time: April 19 (Wednesday) 10:30-12:30pm

Room: KAP 164

Abstract:

Digitally creating a virtual human indistinguishable from a real human has been one of the central goals of Computer Graphics, Human-Computer Interaction, and Artificial Intelligence. Such digital characters are not only the primary creative vessel for immersive storytellers and filmmakers, but also a key technology to understand the process of how humans think, see, and communicate in the social environment. In order for digital character creation techniques to be valuable in simulating and understanding humans, the hardest challenge is for them to appear believably realistic from any point of view in any environment, and to behave and interact in a convincing manner.

Creating a photorealistic rendering of a digital avatar is increasingly more accessible due to rapid advancement in sensing technologies and rendering techniques. However, generating realistic movement and dynamic details that are compatible with such a photorealistic appearance still relies on manual work from experts, which hinders the potential impact of digital avatar technologies in real world applications. Generating dynamic details is especially important for facial animation, as humans are extremely tuned to sense people's intentions from facial expressions.

In this dissertation, we propose systems and approaches for capturing the appearance and motion to reproduce high fidelity digital avatars that are rich in subtle motion and appearance details. We aim for a framework which can generate consistent dynamic detail and motion at the resolution of skin pores and fine wrinkles, and can provide extremely high resolution microstructure deformation for use in cinematic storytelling or immersive virtual reality environments.

This thesis presents three principal techniques for achieving multi-scale dynamic capture for digital humans. The first is a multi-view capture system and a stereo reconstruction technique which directly produces a complete high-fidelity head model with consistent facial mesh topology. Our method jointly solves for stereo constraints and consistent mesh parameterization from static scans or a dynamic performance, producing dense correspondences on an artist quality template. Additionally, we propose a technique to add dynamic per-frame high and middle frequency details from the flat-lit performance video. Second, we propose a technique to estimate high fidelity 3D scene flow from multiview video. The motion estimation fully respects high quality data from multiview input, and can be incorporated to any facial performance capture pipeline to improve the fidelity of the facial motion. Since the motion can be estimated without relying on any domain-specific priors or regularization, our method scales well to modern systems with many high-resolution cameras. Third, we present a technique to synthesize dynamic skin microstructure details to produce convincing facial animation. We measure and quantify how skin microstructure deformation contributes to dynamic skin appearance, and present an efficient way to simulate dynamic skin microstructure. When combined with the state-of-the art performance capture and face scanning techniques, it can significantly improve the realism of animated faces for virtual reality, video games, and visual effects.

Location: Kaprielian Hall (KAP) - 164

Audiences: Everyone Is Invited

Contact: Lizsl De Leon