A young married couple attends a counseling session together, sharing their feelings, frustrations and needs.
USC Viterbi's Shrikanth Narayanan and Panayiotis Georgiou are at the vanguard of a new field called behavioral signal processing (BSP) and informatics. BSP aims to capture, quantify and interpret human behavior through the use of engineering, computing and algorithms.
Illustration by Tiffanie Jan Lee
“You work too much. You’re never home,” the wife says. “I mean isn’t there life outside of work? I can’t be by myself all the time.”
Her husband leans back, sighs and briefly stares at the ceiling.
The woman calls her spouse a “workaholic” and complains that they are drifting apart. “I feel like I’m living with a roommate,” she says. “You’re a stranger to me, now. I wasn’t expecting this when we got married.”
Again — he pauses, leans back, sighs and briefly stares at the ceiling. “Are you done dumping on me?”
“I’m not dumping on you,” she says, her voice rising. “I’m calling it as it is.”
Clearly, this is a relationship in need of repair. However, the pair’s willingness to speak hard truths and confront problems head on would seem to give rise to some optimism.
Not necessarily. Beneath the surface, several largely hidden interactions are taking place, ones that cast their union in a much darker light. Thanks to the pioneering work of a team of USC engineers, marriage and family researchers and counselors might one day have access to a treasure trove of additional information that could both deepen their understanding of couple dynamics and increase their effectiveness. In the process, therapists just might save more troubled relationships.
Shrikanth Narayanan, the Andrew J. Viterbi Professor of Engineering, and USC Engineering Research Assistant Professor Panayiotis Georgiou are at the vanguard of a new field called behavioral signal processing (BSP) and informatics. BSP aims to capture, quantify and interpret human behavior, both overt and covert, through the use of engineering, computing and algorithms.
For the past five years, an interdisciplinary group led by Narayanan, Georgiou, USC Psychology Professor Gayla Margolin and researcher Brian Baucom (now an assistant professor of psychology at the University of Utah) has used behavioral informatics to gain a better understanding of distressed relationships. By sifting through massive amounts of data captured by videotape, audio recordings and physiological sensors, behavioral signal processing can consistently and objectively evaluate vocal synchronization, word choice, gestures and minute physiological changes – information that can provide clues about a relationship’s health that even the most attentive therapists might otherwise miss or just be unable to process.
The USC researchers’ work, funded by the National Science Foundation and other entities, has included videotaping Trojan students role-playing troubled couples and analyzing audio-visual recordings from a longitudinal UCLA-University of Washington study of married couples undergoing intensive marital counseling.
Along the way, USC scientists have refined their technology and algorithms to better detect and analyze both verbal and non-verbal behavior. This valuable information, when combined with a therapist’s professional observations, provides a richer and more nuanced picture of a couple’s relationship and perhaps future prospects.
“We want to support, not supplant, human expertise,” says Narayanan, whose ongoing BSP research includes autism diagnosis, learning quality, and measuring the effectiveness of addiction therapeutic practices.
“There are certain things computational methodologies can do really well,” he adds. “They can look at fine changes in patterns of human behavior. They can look at data records across time. They can look at multiple sources of information together in a way that complements human ways of analyzing data.”
How does this all work in practice? Let’s revisit our married couple.
In the above example, the husband and wife never shouted, swore or glared at one another. They also appeared to honestly communicate their emotions, no matter how painful. So far, so good. However, an audio-visual analysis of the session uncovered deeper fissures.
Turns out the couple used the word “you” repeatedly during their session. In counseling arguments, a preponderance of “you” is an indicator of blaming behavior. A computer evaluating their movements and gestures found that the couple continuously moved ever so slightly away from each another and that the husband often averted his wife’s gaze by looking at the ceiling. Not good.
Additionally, algorithms mapped a worrisome lack of vocal synchronization. When the wife spoke loudly, her husband spoke softly. When he spoke slowly, she often spoke quickly. They interrupted each other.
A lack of entrainment, or failure to subtly reflect one another’s speaking patterns, suggests a “high polarization and perhaps problems down the road,” says Jeremy Lee, a recent USC Ph.D. who has developed new algorithms that quantitatively measure vocal synchrony and offer new insights into behavioral dynamics.
Taken together, behavior informatics revealed the magnitude of the couple’s problems. Their marriage and family counselor — now armed with objective data about the couple’s lack of vocal entrainment, their frequent use of the blaming word “you,” and their avoidance behavior in moving away from one another — could use those new insights to modify the therapy for the betterment of all.
USC’s distressed couples’ research, though not yet in the translational phase, holds considerable promise. Professor Georgiou seeks additional funding for a study that would allow therapists to monitor interactions throughout the week instead of only during sessions.
For instance, couples in therapy might set aside time at home to talk about a particularly nettlesome issue and record their conversation on a mobile phone. The device could then remotely transmit the conversation securely to a computer for processing and analysis, giving therapists valuable information prior to the next counseling session.
And it’s not just couples that stand to benefit from Narayanan, Georgiou and their team’s behavioral signal processing studies.
“Computational tools can model not only couples who are interacting but also the psychologist who is judging,” Narayanan says. “Hopefully, we can offer insights about what works and what seems to work less well.”
