Sunday, October 18, 2009

The Internet: Our Telescope Into Human Society

Imagine that we've been developing telescopes to capture increasingly detailed snapshots of an alien civilization. At first, we could only detect their large structures such as buildings and roads. Based on how their cities are organized, we might infer that certain buildings or city regions are important. We might even be able to tease apart different communities just based on infrastructure information. From this, we start to formulate theories of their social behavior and organization.

As our telescopes grew more powerful, we became able to detect traces of activity, such as from transportation vehicles and eventually even from individual people. With this new wealth of information, we can further refine our understanding of this blossoming extraterrestrial society, and paint an amazingly vivid picture of their social dynamics.

Of course, we could just as easily tell the same story about our own society. Our telescope is the internet. In the early days, we relied on relatively the static hyperlink structure to infer the authoritative or trustworthy websites. We could even detect online communities by analyzing link density.

As the internet (and our dependence on it) expanded, much of its content became more dynamic (e.g., blogs, forums, Yahoo! Answers). Nowadays, we can even trace real-time online activities on sites such as Google, Twitter, Facebook, Amazon, and many others. Our numerous online activities all leave digital footprints which reflect the fine grained dynamics of our own society. Such information can be incredibly useful. For example, a Nature article published earlier this year showed how analyzing search behavior on Google can provide a faster turnaround time to detecting influenza outbreaks. As another example, a recently published PNAS article showed how mobile phone data can be used to infer friendships. That is the power of the digital medium.

We now also have an opportunity to introduce new levels of empirical rigor to many social science disciplines. In years past, acquiring social data was a very labor intensive task, often requiring months or years to collect a modestly sized dataset. Nowadays, sociologists can mine all of Livejournal or Facebook to study the global structure of things like social influence and gossip. Companies like Google constantly run auctions to determine which ads to show whenever someone issues a query -- this is an economist's dream.

Here at Cornell, Professor Jon Kleinberg is one of the leaders in studying the convergence of social and technological networks. As part of an undergraduate course he and Professor David Easley have been teaching the past few years, they have written a new textbook. From their website: "Drawing on ideas from economics, sociology, computing and information science, and applied mathematics, it describes the emerging field of study that is growing at the interface of all these areas, addressing fundamental questions about how the social, economic, and technological worlds are connected."

Incidentally, this was the topic I initially wanted to write about for the University Writing Competition hosted by Scientific Blogging. But seeing as how any meager offering of mine would pale in comparison to Jon Kleinberg's fantastic CACM article, I decided to change topics to something I'm more of an "expert" on: Self-Improving Systems that Learn Through Human Interaction.


Anonymous said...

If the Internet is truly a telescope into the world we live in, which it undoubtedly is, perhaps it's time to stop and consider who's looking in. Each one of our "digital footprints" is being recorded, tracked, analyzed and turned into a datapoint to be exploited by marketers and perhaps the govt. Google, Facebook and a host of other online entities are collecting data about us and our behavior in a manner that we'd never have considered allowing even a few years back.

Admittedly, the internet was a simpler place a decade ago, but it was also subversive and truly democratic. We seem to have handed over the keys to the castle to the Googles of the world.

Yisong said...

Yeah, I've also had such reactions in the past, but they've been largely gut reactions. I've had trouble pin-pointing specifically what would be so indefensibly terrible about collecting this type of information.

On the other hand, our progress thus far has been nothing short of amazing. Targeted advertising services have made it possible to connect countless small businesses with potential costumers from all over the world. Such businesses would not be viable in more traditional brick and mortar settings. This has made the market and our society more efficient in the purest sense of the word. Everyone has prospered from it.

I know that I'm being targeted for exploitation. I'm OK with that. The only way they can actually make such exploitation sustainable in the long run is if they continue to improve my user experience (by exploiting their collected knowledge of people's preferences), and also respect my privacy.