Events Calendar

PhD Thesis Proposal - Zimo Li

PhD Candidate: Zimo Li

Title: Human Appearance and Performance Synthesis Using Deep Learning

Committee:
Stefanos Nikolaidis
Aiichiro Nakano
Andrew Nealen
Lauren Itti
Mike Zyda

Abstract:

Synthesis of human performances is a highly sought after technology in several industries. In this presentation, we will go over several new deep learning solutions which tackle the problems of human facial and body performance synthesis.

Facial performance synthesis is a complex multi-step graphics problem. First, the "target" performance to be modified must be tracked and captured accurately. Then, based on the desired modification (whether to change the identity, facial expressions, or both), a modified "source performance" must be synthesized and/or captured from a different actor. Finally, the original facial performance must be removed and replaced with the synthesized one. This multi-step process poses many unique challenges. Using conventional CG tracking and retargeting of expressions from the source to target using a 3D mesh and static texture will give an undesired "rubbery skin" effect. Furthermore, inaccuracies in the expression tracking of the source performance using a blendshape model will result in the "uncanny valley" effect in the output performance. It is often necessary to use costly capture methods, such as a Light Stage, to obtain highly accurate 3D captures and dynamic textures of a source performance in order to avoid these pitfalls. Even then, final modified performances are often uncanny.

When dealing with human body-motion synthesis, creating new motions often requires manual artist animations, tracking new motions on an actor, or stitching together subsequences of previous animations. These methods are limited by cost, or are not able to generate appreciably novel motions.

Over the last several years, the advancement of AI-based generation techniques have let us address many of these issues. In this presentation, we will go over several novel techniques which reduce the cost (time/money/ease-of-access), and/or improve the quality of facial re-enactment, as well as body motion synthesis, pipelines. The applications of these techniques allow us to tackle new problem settings in an efficient way, including visual dubbing (changing the lip motions of a facial performance), dynamic texture synthesis, 3D model generation, as well as extended human motion synthesis.

WebCast Link: https://us05web.zoom.us/j/81890781474?pwd=cjQ3YkVDT3drMlQ2VWtlbjU2YWxyZz09

Audiences: Everyone Is Invited

Contact: Lizsl De Leon