Friday, September 25, 2009

Turing Tests for the Internet Age

In a conventional Turing test, a human judge tries to discern which of two text chat conversations is with another human and which is with a computer program. Since it was first proposed in 1950, designing computer programs that can pass the Turing test has become a milestone target of sorts for artificial intelligence researchers.

Now that the World Wide Web has enabled us to spend significant time in online environments, web-based variants of the Turing test are quickly becoming research goals with immediate or near-term practical interest. For example, when we send emails to customer service and receive coherent (and hopefully helpful) responses, we typically assume we're being serviced by a human agent. As we all know, living in a information economy drives up the price of labor. A significantly cheaper alternative would have computer programs that can automatically understand and resolve each customer's specific issues while also providing a pleasant service experience. There would imply no more bitchy emails from disgruntled human workers, much faster turn-around time on service requests, and overall increased efficiency in this sector of society.

Currently, we are still quite far from having programs that are of practical value. One way to generate interest and spur innovation is through competitions, very much like how the X PRIZE Foundation awards prize money for demonstrating various feats of technological prowess. While we should, of course, also pursue other avenues of research, I echo John Langford's sentiment that competitions give people who know how to do things a chance to distinguish themselves. In fact, there already exists a Turing test competition for computer game bots within the gaming community.

Regardless of whether we promote this research direction through competitions or other means, it would certainly be useful to have automated online services with rich interactive components (such as the aforementioned customer support service). What might be some other interesting Turing test variants that we can/should focus on?

One idea is some kind of social networking bot. For example, one might have a profile on Facebook, MySpace, or OkCupid that is controlled automatically by a computer program. Consider the OkCupid scenario, which is reminiscent of the great Mark V. Shaney experiment but taken one step further. We can actually test two experimental settings in this space, male and female. In both settings, the learning component could, in principle, be controlled by some kind of reinforcement learning algorithm with a clever strategy exploration routine. I'd expect the male version to be more proactive since males are typically expected to initiate contact with females. One could start with the obvious reward function of maintaining message threads with other people and go from there. We even have some guidance on how we might craft a suitable strategy space.

A second idea, which takes the game bot idea to its logical extreme, is a bot in a sustained virtual world that tricks human users into thinking it is also human. For example, consider World of Warcraft (WoW), which has a sustained virtual universe that features many social components that we also find in the real world (such as a vibrant trading economy). Suppose we could design a bot player which can learn to operate autonomously in WoW and interact naturally with human players. Suppose the bot player could go on quests, team up with other players, even join a guild, all while under the pretense of being a human player. This is certainly an exciting setting to experiment in since the penalty of failure is limited (we can just create another bot character), and there already exists an enormous amount of data from usage logs of normal humans that we can use to bootstrap the learning algorithm.

In both the aforementioned scenarios, it should be noted that some kind of agreement with the service provider (e.g., OkCupid or Blizzard Entertainment) is required. Even if we ultimately weren't allowed to run live experiments, I'd imagine there is already sufficient data collected that can be used to generate interesting simulations. That data is just locked up behind corporate walls.


Jess said...

Along the lines of your social network bot idea - using a bot as a forum moderator. They'd be able to root out trolls and deal with reported posts.

Yisong said...

...or give parents another tool with which to spy on their kids? ;)