In 1948, a remarkable American researcher laid the foundation for computers, cell phones, compact discs, the Internet, interplanetary communications and most of the other aspects of what we now call the Information Age. Sixty-four years later the 2012 Andrew Viterbi Distinguished Lecturer in Communications offered a strikingly clear introduction to genius Claude Shannon’s work and its impact, with a presentation entitled “What is Information Theory?”
Sergio Verdú and Andrew J. Viterbi
Sergio Verdú is Eugene Higgins Professor of Electrical Engineering at Princeton University. He specializes in the field that Shannon created almost singlehandedly with his 1948 paper, “A Mathematical Theory of Communication” — a field in which USC's Viterbi School emerged as a major international center. The top honor in information theory (IT) is the Shannon Award. Verdú is a winner of it; so were four members of his audience, including lecture namesake Andrew Viterbi, Viterbi professors Sol Golomb and Lloyd R. Welch and visiting from Caltech, Robert J. McEliece. Another Viterbi faculty member, Irving Reed, is also an award holder. Only M.I.T. and Stanford also have four.
Two members of a Viterbi IT family were present as well. Verdú was introduced by his former Princeton student, Urbashi (Ubli) Mitra, professor of electrical en who was joined at the event by her father, Sanjit Mitra, also a Viterbi faculty member and a renowned specialist in the field. Another father-daughter group present: Verdú’s daughter Ariana is a USC senior, who came to the lecture.
Distinguished lecturer, distinguished listeners: from left: Sol Golomb, Lloyd Welch, William Lindsay, Sergio Verdú. Andrew Viterbi.
The USC Ming Hsieh Department of Electrical Engineering, which sponsored the event, has two major faculty divisions, one of which, Electrical Engineering - Systems, deals with IT; the EE chair of systems, Alexander “Sandy” Sawchuk was on hand as host. Faculty from the Viterbi School’s Information Sciences Institute – its name speaks to the IT connection – made the journey from Marina del Rey.
What they heard was a lecture remarkable for its clarity and lack of technical language. Verdú’s account contained – as he proudly noted – only one equation, and carefully defined its terms as he went along, beginning from the basic information nexus that Shannon proved for the first time was describable mathematically.
This was described by Verdú in a simple box diagram he called “The Coat of Arms.” On one side is an information source, on the opposite side an information destination. Between source and destination are two boxes; on the source side a transmitter, on the destination side a receiver. The transmitter box produces a signal; the receiver box receives a signal, but the signal transmitted was not identical to the one received because between is an input labeled ‘noise.’
The "Coat of Arms" : The unalterable structure of information transfer.
Shannon saw that if a message were in digital form, as ones and zeros, then information transmission could be seen as an interaction of two factors. The first, bandwidth, is the number of ones and zeros that a pathway (usually electronic or photonic, but alternatives have been used) can accommodate in a given time.
The second is the amount of noise, disorder or (to use Shannon’s expression) entropy that the pathway contains — that is, the likelihood that any given one received should actually have read as zero, and vice versa. Bandwidth is never infinite, and noise is never zero, but Shannon showed that mathematics can define and predict the relationship between the two and tailor rules and techniques to most efficiently work within these rules for different uses.
Fathers and daughters: Ariana Verdú, Sergio Verdú, Sanjit Mitra, Ubli Mitra
The insight, and its rigid mathematical foundation almost overnight turned electronic communication from what had been a mass of empirical guesswork and ad hoc blind design gadgets into a game with precisely defined rules. Finding the best solution within the rules was not obvious or easy – it depended on finding mathematics to at least sketch algorithmic descriptions of complex phenomena like images or sounds — but the rewards of finding ways were immense and built on each other. Many of the members of the audience had distinguished themselves precisely in creating such mathematical models.
If the important thing is to minimize bandwidth but still transmit a recognizable product — a picture, for example — one set of equations can perform the desired task. But if instead accuracy is critical, it can be achieved by adding redundancy (the mathematical equivalent of repeating on the phone, ‘that’s “b” as in barney, “r” as in rabbit…”) to the message, and Shannon theory specifies exactly how much redundancy to add to counteract any specified amount of bandwidth noise.
Verdú offered one universally familiar measure that combines the two factors. The bars displayed on a telephone do not indicate how much signal is available. They instead display how much communication is possible given both bandwidth and noise.
Once Shannon opened the door to these kinds of mathematical manipulations of information, defined in this new manner, new technologies came bursting out. Verdú went down a long list. He began with Viterbi algorithm applications, based on the idea that if a receiver system avoided trying to interpret a message until it had a larger message sample, then it could distinguish more separate messages, “and minimize the probability that the whole message is a mistake.”
Hsieh Department co-chair Sandy Sawchuk presents Verdú with non-digital mementos.
Verdú then answered questions about new directions Shannon theory might still lead. “Since we are still in the pre-history of biology,” Verdú began, going on to note that the intricacies of the information contained in genetic coding would occupy future generations of IT. His last question was: “can you say where we’d be without this?” and his clipped answer was “nowhere.” The entire structure was built on the new ideas. Someone else might have devised them, but for the Information Age, there is no substitute for the understanding.
Finally, Sawchuk presented a smiling Verdú with three non-digital artifacts to take back to Princeton: a plaque, a Trojan baseball hat and a T-shirt labeling its wearer a “Trojan Dad."
