Five years ago, the Viterbi School’s Laurent Itti presented groundbreaking research on how humans see the world. Now, he is heading a $16-million Defense Advanced Research Project Agency effort to build machines that see the world in the way he discovered humans see.
But to do this, Itti and his fellow scientists first plan to learn more about how humans see, and then immediately use this more detailed knowledge to build new machines.
Artist's conception of neurovision sensor with custom machined gimbal head, custom printed circuit boards, and precision-machined unibody enclosure. The gimbal head is crucial to allow the 'eye' to move to actively see the scene based on feedback from the software system.
The “neuromorphic visual system for intelligent unmanned sensors” project builds on a previous effort called Neovision, which attempted the same goal, but relied on existing software systems that were not completely compatible with the neural systems that Itti had discovered were crucial.
The new project will try to build its eyes from the bottom up, from scratch, finding out more precisely how these neural systems function and then building new software and hardware into which the new understandings will be embedded, tying research directly to development in a continuing reciprocal process.
The goal is clear: “The modern battlefield requires that soldiers rapidly integrate information from a multitude of sensors," says Itti. "A bottleneck with existing sensors is that they lack intelligence to parse and summarize the data. This results in information overload … Our goal is to create intelligent general-purpose cognitive vision sensors inspired from the primate brain, to alleviate the limitations of such human-based analysis.“
To do this will require applying existing understandings as well as new fundamental research: “At the core of the present project is the belief that new basic science is crucial.”
This is because living creatures' vision involves much more than just images. Rather, it was honed by evolution to seek out the specific visual signals critical to a seeing creature's survival. This involves complex circuitry in the retina, where the outputs from light detector cells are processed to give rise to 12 different types of visual “images” of the world (as opposed to the red, green and blue images of a standard camera). These images are further processed by complex neural circuits in the visual cortex, and in deep-brain areas including the superior colliculus, with feedbacks driving eye movements.
Itti’s 2006 paper “Bayesian Surprise Attracts Human Attention,” co-authored by Pierre Baldi of UC Irvine, argued that a mathematical algorithm that analyzed incoming visual data looking for a precisely characterized element the authors called a ‘surprise’ seemed to fit experimental data of eye movement. The theory was the basis of the earlier Neovision work, and has since been extensively revised and developed.
The new plan is to go much farther than Neovision. The researchers will model the entire complex interactive system from the ground up to understand the exact messages transmitted from the retina to the cortex and further to the colliculus, and how the brain cells understand them. Then they will attempt to embed parallel transactions, using the same perception algorithms, into working silicon systems.
Board designed at USC that carries four Core2-Duo CPUs (8 cores total).
The execution of this strategy will be unusual. Typically, such projects aim at a finished system at the end of the project, reporting progress toward such a system at regular intervals.
Instead, Itti’s plan is to create a whole series of prototypes, complete with breadboard hardware at a rate of one every six months, as understandings emerge from the project's basic science side.
Researchers will regularly convene at USC's Catalina laboratories for intensive workshops, to learn the findings of a “core team of engineers at USC [interacting] with Ph.D. students and postdocs, performing the basic science directly in an academic setting. Because the engineering core will work directly with the researchers, we expect that the technology transition will be swift and efficient,” said Itti.
Itti’s proposal to use this system was directly backed by a letter from USC Viterbi School Dean Yannis C. Yortsos who noted that his school was “exceptionally well equipped to support the planned research with his current full state-of-the-art facilities for development of electronic and mechanical breadboards, brassboards and prototypes. This includes a machine shop that provides precision machining capabilities from a highly trained staff that Professor Itti has worked with in the past to build custom land-based and underwater robots.
"The computing labs at the Viterbi School Computer Science Department are also equipped with all the necessary CAD tools for custom circuit board design. Our high-performance computing center provides cluster computing capabilities with upward of 6000 computer cores. Finally the school’s Information Sciences Institute and the Institute for Creative Technologies are powerhouses of research and development in the areas of information science and technology. The Viterbi School strongly supports this proposal.”
Itti is an associate professor in the Viterbi School Department of Computer Science. The other institutions that will be part of the effort he is leading include UC Berkeley, Caltech, MIT, Queens University (Canada), Brown, Arizona State, and Penn State, along with a company, Imagize, that specializes in the field.
This document was cleared by DARPA on September 15, 2010 for Public Release, Distribution Unlimited.
