Can Computers Tell Stories? from USC Viterbi on Vimeo.
Computer scientist Andrew Gordon counts jazz legend Charlie Parker among his greatest inspirations. Gordon, a USC Viterbi research associate professor, is a serious student of improvisational jazz, but Parker’s influence also permeates Gordon’s advances in artificial intelligence research.
“When you listen to Charlie Parker play a saxophone solo for the first time, it sounds like an explosion of raw invention,” said Gordon, who leads the Narrative Group at USC’s Institute for Creative Technologies. “After analyzing dozens of his improvisational solos, however, I realized his brilliance came from creative reuse. Nearly everything he played was stitched together from something he or someone else played previously.”
In other words, Charlie Parker’s greatness was grounded in experiences.
Gordon’s goal is for computers also to learn from experiences—experiences people share when they narrate and interpret the events of their lives. Driven by a desire to develop machines that can think like people, he is identifying, collecting and studying stories in order to give computers knowledge they can apply in new situations.
His narrative research intersects with many of the multidisciplinary topics explored at the 15-year-old ICT, a research center established at USC by the U.S. Army to advance the state of the art in simulation and training. Story is a common thread throughout the institute, which specializes in the creation, study and use of believable characters and scenarios.
“Soldiers swap war stories for a reason,” said Gordon, whose current work is funded by the Army, Navy and Defense Advanced Research Projects Agency, also known as DARPA. “They help people explain why things happened and predict what will happen next. They serve to pass on knowledge of things that people didn’t go through themselves.”
USC, home of top art and communication schools, is a natural fit for someone interested in narrative. What is unusual is to find this focus in a department of computer science, where Gordon’s research ranges from basic science analyzing the structure of stories to the development of training video games that incorporate real-world lessons. He has turned the Internet into a living laboratory by collecting and analyzing millions of personal accounts posted on blogs. He even developed a documentary about the bloggers he studied.
Gordon’s overarching challenge is to understand the processes that produce stories and program computers with the same interpretive powers.
Take the following phrases: The birth of your child. The foreclosure of your home. The crashing of your Porsche.
“These experiences will have enormous significance in your life, evident in the stories you tell, but for today’s computers they are only syntactically similar noun phrases,” Gordon said.
In essence, Gordon, who earned his doctorate in computer science at Northwestern University, is trying to pass along the mental algorithms that make people creative. It is here where Gordon’s other lifelong pursuit, mastering improvisational jazz, has come in handy.
Though Gordon can’t duplicate Parker’s ability to draw upon a rich database of musical solutions (yet), he has given computers a vast library of source material to pull from in the form of other people’s stories.
He developed a pipeline for the automatic collection of tens of millions of personal stories from streams of online blogs. This database led Gordon’s former PhD student Reid Swanson to develop an interactive storytelling system called Say Anything, in which a human and a computer take turns adding new sentences to a story.
“The human’s contributions are limited by their own ingenuity, but the creative contributions of the computer come from the enormous body of source material in the nonfiction narratives posted to blogs,” Gordon explained. “It is an improvisational storytelling program, inspired by Charlie Parker.”
Gordon has learned that jazz improvisation requires the ability to seemingly automate the production of sounds on an instrument before any emotion can be layered in. A musician must master the basics so that he doesn’t have to think about the sounds he is going to make. It is a concept that Gordon brings to his latest interactive storytelling project, which aims to automatically generate narratives by interpreting the movements of geometric shapes.
“In this project, we train our software to recognize action verbs by giving them tens of thousands of examples,” he said. “We’re teaching these programs to effortlessly recognize actions like jumping, pursuing and hobbling, so that they can spend more of their computational effort thinking about the deeper-level intentions, motivations and emotions of the characters that they are observing.”
According to Gordon, for many listeners jazz improvisation seems unconstrained. In reality, jazz is quite structured, and the genre has a rich catalog of reoccurring structural idioms.
“I used to wonder how it was possible for great players to memorize around one thousand jazz standards, each with dozens of different chords,” Gordon said. “Now I hear each song as a collection of a handful of idioms, stitched together using one or two defining ideas.”
Gordon relates this revelation to his collaboration with USC’s Brain and Creativity Institute exploring the neurobiology of narrative framing, a project that aims to compare the structural differences in the ways that narratives are told and received across different cultures.
“Each language and culture has their own ways of describing past events, providing background information, introducing concepts, justifying behavior and making points,” Gordon explained. “The aim is to enable more effective storytelling across cultures by adhering to the structural conventions of specific audiences.”
No matter the language or the skill level, one thing Gordon says both jazz improvisation and storytelling have in common is that each is most effective when something personal is being shared. It is likely Charlie Parker would agree. When Parker’s friends once asked him why he liked country music, he said, “The stories, man. Listen to the stories!”
If Andrew Gordon has his way, future computer systems will be able to do just that.
