PhD Dissertation Defense - Jessie Hoegen

Title: Decoding Situational Perspective: Incorporating Contextual Influences into Facial Expression Perception Modeling
Location: GFS 104 / Zoom: https://usc.zoom.us/j/6871678093?pwd=VWdZZjJMaW9PK2cyRmMvNUJpbVV6UT09 
Date and time: July 1, 1pm - 3pm
Committee: Jonathan Gratch (chair), Emilio Ferrara, Giorgio Coricelli

 
Abstract:

The work that I performed during my doctoral studies relates to facial expressions within the field of Affective Computing. In this field, facial expressions are a popular subject of study. State-of-the-art facial expression models and emotion perception models have gone through leaps of improvements and nowadays match the accuracy of professionally trained expression coders. Because of these improvements, one might expect that automatic emotion recognition has become an indispensable tool for analyzing and predicting human social behavior.  Indeed, psychological theories argue that emotional expressions serve crucial social functions such as revealing intentions and shaping partner behavior. Yet these theoretical benefits have largely failed to materialize within the field of affective computing.  
 
There is now growing understanding that one of the obstacles to the advancement of affective computing is how the concept of emotion is typically represented within affective computing. Influenced by early theories from psychology, expressions are often treated as universal and context-independent signifiers of an underlying emotional state. This latent state is then assumed to shape subsequent human behavior. Yet more recent psychological theories argue that expressions should be seen more like words and function to coordinate social behavior.  My dissertation embraces this latter view and explores its consequences for affective computing.
 
Following recent ``pragmatic'' theories of emotional expressions, I adopt the perspective that expressions in social settings should be best treated like words.  Like words, the meaning of expressions must be seen as context dependent. Just as ``bank'' might refer to the side of a river or a financial institution, a smile might refer to pleasure or anger depending on the surrounding context.  And like words, expressions can be examined from multiple perspectives. We can consider the ``author's'' perspective (why did this person produce this expression? what was their intent? what does it signal they will do next?) but also the ``reader's'' perspective (how does this expression shape the observer's emotions, intentions and actions?).  
 
I illustrate the utility of this perspective for analyzing human social behavior.  Focusing on a series of social tasks such as social dilemmas and negotiations, I show how the interpretation of facial expressions is shaped by context, and that expressions, when combined with context, can usefully predict the author's intentions and consequences for the reader.  Together, this body of research makes several important contributions.  First, I add to the growing body of research that questions the utility of context-free methods for automatically recognizing emotional expressions. Second, from the perspective of the author, I show how both expressions and context are necessary for predicting an author's subsequent actions in a face-to-face negotiation from their expressions. Thirdly, from the perspective of the reader, I show how emotional expressions shape the readers actions in a social dilemma.  Finally, I show how these models could inform the behavior of interactive synthetic agents, for example allowing them to strategically select emotional expressions that will benefit a team task. More broadly, my dissertation illustrates the potential benefits of incorporating a pragmatic perspective on the meaning of emotional expressions into the field of affective computing. 

Location: Grace Ford Salvatori Hall Of Letters, Arts & Sciences (GFS) - 104

Audiences: Everyone Is Invited

Contact: Ellecia Williams