Logo: University of Southern California

A New Discipline, Behavioral Informatics, Takes Shape at the USC Viterbi School

If a picture is worth a thousand words, computational analysis of verbal and non-verbal activity may be worth trillions

October 17, 2011 —

In recent years, Professor Shrikanth S. Narayanan of the USC Viterbi School of Engineering has been energetically advancing a new field, behavioral signal processing and informatics, an emerging discipline that bridges engineering with behavioral sciences and aims to quantify and interpret human interaction and communication through the use of engineering and computing innovations. 

The discipline focuses on gathering, analyzing and modeling multimodal human behavior, both overly and covertly expressed.  The goal is to develop techniques and technologies that explain complex human behavior and support both research and practice of domain experts, such a clinicians and analysts, in their diagnostic and intervention design and evaluation.

Professor Narayanan.
Narayanan gave an overview of efforts ranging from understandings of the interactions of distressed couples in marital therapy to computing communication and social interaction behavior of children with autism spectrum disorders(ASD) as a plenary address at the recent International Conference on Multimedia and Expo held in Barcelona.

Narayanan started from a universally acknowledged understatement: “human behavior is exceedingly complex,” and moved on to note the new opportunities that advanced sensor and computing technology had opened up for understanding this complexity.

Behavior analysis is central to psychology and allied disciplines, but has been slow, manual and usually, at least in part, subjective. Narayanan sees another way, by treating "measuring and quantifying human behavior [as] a challenging engineering problem.”

Some work has already been done in this line. One is psychiatrists' long-used medical measurements of galvanic skin response, EKG and other inputs to monitor emotional response.

But much more is possible, Narayanan said, describing his lab's effort to bring other objective, automatic and non-manual tools to bear in these areas, as part of what he calls “multimodal Behavioral Signal Processing.”

The goal, increasingly in sight, is to create “technology and algorithms for quantitatively and objectively understanding typical, atypical and distressed human behavior—with a specific focus on communicative, affective and social behavior.”

This approach is to supplement human expertise, and he emphasized the role of behavioral signal processing and informatics is: "to support, not supplant."

ASD is a classic case, and is an area in which Narayanan and collaborators have been working intensively for some years, including in a collaborative multi-university effort through the National Science Foundation-supported "Expeditions in Computing."

There is considerable variability in the behavioral manifestations of ASD, making diagnosis particularly challenging. “We are trying to investigate how technology can assist in and enhance this difficult predominantly observation-based rating task, by using engineering techniques and computational tools to analyze each child's specific communication and social interaction patterns-- through their language use, intonation style, turn-taking trends and emotional expressions, processing by combining expert-inspired knowledge with data-driven techniques."

Early results are encouraging and several new research threads are currently underway.

Married couple interactions are another current target. In his book Blink, Malcolm Gladwell described long-term work by University of Washington psychologist John Gottman, who found that after observation of one hour of couple interaction, he could predict with 90 percent accuracy whether the couple would be married 15 years later. Narayanan and his team, in collaboration with colleagues in psychology, are finding ways to design new engineering techniques to support research and practice in this domain and other realms of mental health such as addiction and post-traumatic stress disorder with support from the National Science Foundation, National Institutes of Health and U.S. Department of Defense.

Narayanan, the Andrew J. Viterbi professor at the school bearing Viterbi’s name, holds professorships in the School’s departments of computer science and electrical engineering, and also in the USC Dornsife College of Letters, Arts and Sciences departments of linguistics and psychology. He is also director of the USC Signal Analysis and Interpretation Lab.

He is a recipient of numerous research awards, including a 2005 and 2009 Best Paper Award from the IEEE Signal Processing Society and was named an IEEE Signal Processing Society Distinguished Lecturer for 2010-2011.  Papers co-authored with his students have won awards at Interspeech 2011 Speaker State Challenge, ISSP 2011, Interspeech 2010, Interspeech 2009 Emotion Challenge, IEEE DCOSS 2009, IEEE MMSP 2007, IEEE MMSP 2006, ICASSP 2005 and ICSLP 2002.