Sign in / Join

On Google, aggregation of information, and the wisdom of crowds

The phrase "madness of crowds" traces back to Charles Mackay's classic book on financial panics. James Surowiecki's new book, The Wisdom of Crowds: Why The Many Are Smarter Than The Few And How Collective Wisdom Shapes Business, Economies, Societies And Nations, looks at the other side - some might say the typical, non-pathological side - of collective decision making. Here's a taste of the analysis from Surowiecki's column in Forbes, applied to Google:

Google has succeeded for a simple reason: It regularly finds the Web pages that are most valuable and puts them at the top of the list. The heart of the technology that lets it do this is the PageRank algorithm (after cofounder Larry E. Page), which essentially asks Web page producers to vote on which other pages are most worthwhile. Each link to a page counts as a vote. Google is a republic, rather than a pure democracy; sites that have more links into them are effectively given more voting power. But the principle is fundamentally democratic--let the masses decide. Given the Wild West nature of the Web, you'd think that this would lead to chaos or irrationality. Instead, it leads to a remarkable order.

How does this work? What Google is relying on is something I call the wisdom of crowds: Under the right circumstances, groups are smarter, make better decisions and are better at solving problems than even the smartest people within them. On any one problem a few people may outperform the group. But over time collective wisdom is near-impossible to beat. No one, you might say, knows more than everyone....

The wisdom of crowds can be seen at the racetrack, where the odds on horses coincide very nicely with their probability of winning. (That is, if you look at a large collection of horses that went off at 4-1 odds, you find that 20% won.) And, of course, collective wisdom is also at work in markets, which is why it's so hard to outperform the market over time. Just as Google's PageRank encapsulates the knowledge of Web users, so does a market price embody, as the economist Friedrich Hayek suggested, all of the tacit knowledge and wisdom of investors and traders.

In the Journal of Economic Literature (JSTOR, subscription), I reviewed the economics of racetrack betting. There is a wealth of evidence at the track which sheds light on Hayek's theory of prices as information. (Here's a preliminary version if you can't access JSTOR). In the Journal of Finance (JSTOR, subscription), Bill Brown and I argue that models of financial prices are inherently limited relative to the complexity of the problems that markets assess. What we can't model (the "error term" in a regression equation) is often complexity that markets assess appropriately. Data on basketball point spreads (in contrast to stock prices) allow one to test the argument. And it works!

Surowiecki's book is reviewed here, (if the New York Times links are working when you read this). He may be taking these claims a bit far, but I believe the "irrational exuberance" argument has been oversold. His book might provide a useful antidote.