October 08, 2007 — Researchers at the University of Southern California's Information Sciences Institute, one of the birthplaces of the Internet, have just completed and plotted a comprehensive census of all of the more 2.8 billion allocated addresses on the Internet -- the first complete effort of its kind in more than two decades, they believe.
Internet Census chart. Click on the image for a larger view and more details.
"An Internet census," explains John Heidemannn, an ISI project leader who also has an appointment in the USC Viterbi School of Engineering Department of Computer Science, "is just that: every single assigned address in the entire Internet was sent a probe."
The technical name for an Internet probe, more commonly called a "ping" is an "Internet Control Message Protocol (ICMP) echo request packet." It took some 62 days to send almost 3 billion of these from three machines, an effort carried out by Heidmann's ISI collaborator Yuri Pradkin.
Many (61 percent) of the pings received no response at all. Many others got a "do not disturb" or "no information available" response that many network administrators program into their routers and firewalls. Some of the non- replies were probably also due to firewalls intentionally blocking the pings. Still, as the census went on, millions of sites did respond, positively and negatively, and a unique Internet atlas took shape.
The atlas is not geographic, though geographic areas (North American, Europe, etc) show up on it. Instead, it is numerical, building on the mathematical structure of the Internet address system.
Each Internet address is a number between 0 and 2 to the 32nd power (4,294,967,295), usually written in "dotted- decimal notation" as four base-10 numbers separated by periods; for example 18.104.22.168. Each number represents one 8-bit part of the whole address.
These addresses appear in the chart as a grid of squares, each square representing all the addresses beginning with the same first number ("128," in the preceding example). The map is arranged in not in simple ascending numerical order, but instead in a looping pattern called a Hilbert curve, which keeps adjacent addresses physically near each other, on chart, but also makes it easier to zoom seamlessly in to show greater detail. "The idea of using a Hilbert curve actually came from a web comic, xkcd," said Heidemann.
The smallest feature the map shows is a singe pixel, which is records averaged responses from some 65,536 (2 to the 16th) addresses. The averaging is conveyed by color coding, with all-positive responses showing up as brilliant green, all- negative as brilliant red, equal numbers as brilliant yellow, with brilliance decreasing down to dim shades in areas where fewer addresses respond.
The map presents a novel census view of the visible Internet. "To our knowledge," said Heidemann," the only other census of the
Census takers: Pradkin, left and Heidemann, with ISI's location designated on wall chart.
Internet was in 1982," when the Intenet consisted of 315 allocated addresses.
Heidemannn and Pradkin have also plotted a second rendering where each pixel represents a single address. When printed out at laser-printer resolution, this map that literally shows every address in the Internet takes up a 9x9 foot space on a corridor wall in ISI's Marina del Rey campus.
The project is continuing. Heidemann hopes to continue censuses to create not just a snapshot -which is what the current map is - but a dynamic movie of Internet evolution, which can aid in detecting and monitoring trends. He and his collaborators are intensively studying the census results working toward this goal.
While the new census is the first they have visualized. ISI has been taking censuses since 2003, when Praydkin and Joseph Bannister (of ISI) and Ramesh Govindan (chair of the the Viterbi School 's Department of Computer Science) started collecting data. Their hopes were to study the growth of the Internet, and their group is still processing this data to look for trends.
"Internet census data is useful for several reasons", Heidemannn says. "As the Internet use becomes widespread, we are running out of Internet addresses—good predictions by Geoff Huston suggest all addresses may be allocated as soon as early 2010. The IETF (Internet Engineering Task Force, the technical body that manages the Internet) has anticipated this since the 1990s and designed a new protocol, IPv6, to solve this problem, but deployment has been slow. Our data can help illustrate the need to move forward."
It's hoped tha tthe census also can improve Internet security. In fact, the Department of Homeland Security "supported our work with the goal of improving network security," said Heidemann, pointing to the work of ISI researcher Jelena Mirkovic that is using this census data to study how worms spread in the Internet. Other researchers have plotted maps of where cyber-attacks originate.
"There’s also a sense of discovery in these maps", Heidemannn says. "We’ve built a huge Internet and use it every day. Like the far side of the moon, wouldn’t you like to know what it looks like?"
The census was undertaken by the Ant project, a research group, according to its web site,
" spanning USC/ISI, the USC and Colorado State University computer science departments, the USC Ming Hsieh Department of Electrical Engineering, and USC's Information Technology Services. "We're looking at novel ways to examine network traffic."
More details about the census project and the full-scale map are at http://www.isi.edu/ant/address/whole_internet/
ISI was one of the original nurseries of the Internet, playing a key role in the development of the domain name system and other features. ISI computer scientist Jon Postel (1943-1998) directed the Internet Assigned Numbers Authority for years.
The Department of Homeland Security and the National Science Foundation supported the research.