The thing about useful stuff is: Crowdswarm - circular popularity syndrome

Crowdswarm - circular popularity syndrome Monday, January 15, 2007

A fascinating study by Matthew Salganik of Columbia University in New York re-enforces a point I was making to Nic Brisbourne of Esprit Capital Partners. Nic had posted about the wisdom of crowds (Last.fm) v machine intelligence (e.g. Pandora).

I feel that there is an important sub-division in the “wisdom of crowds” category, between trust networks like LinkedIn and crowd sites which give you data on which you may or may not choose to rely like Toptable reviews. Clearly the former is much stronger since your reliance on the information is based on your assessment of the source. In the latter, popularity is your main guide, which doesn’t necessarily equate to wisdom. Sam Sethi (Vecosys and formerly techcrunchuk) and I also recently chatted about this and its impact on the long-term success of “social networks”.

As regards machine recommendations, the ability of computers to spot correlations in behaviour to help make “insightful” suggestions I believe is a very powerful force but only to the extent that the dataset and population on which it draws is sufficiently sizeable to allow meaningful conclusions to be drawn. Too often social networks have/use insufficient data points in making comparisons to allow them to identify genuine correlations e.g. male & works in capital markets as our shared points isn’t sufficient to guess our interest/tastes will overlap. Add on music, hobbies, education….. At which point we start to use something akin to “dating service” algorithms to identify people we “correspond” with and hence have increased likelihood that the machine recommendations will resonate.

The Columbia study created a "music market" and identified that when songs were ranked by how many times they'd been downloaded, people followed the crowd. When the songs were not ordered by rank, but displayed the number of downloads, there was a less significant "social influence". It wasn't wisdom, just popularity that drove behaviour - "Swarming" as they termed it.

As separate study by the Florida Institute of Technology is testing out "swarming" in supermarkets, by providing shoppers with a device that indicates to them how many people currently in the shop have bought the item (or from the shelf) they are walking past. The idea is supermarkets benefit from increased sales without giving discounts and consumer get the satisfaction of buying a "popular" product. This is obviously a vicious/virtuous circle depending on your view, with visibly unpopular products being further shunned.

Oddly enough, whilst reading about these studies last night, the TV was showing a BBC Nature programme on the migration of the wildebeest across Africa, with an annual migration of one million of them on a round trip of 3,000km. A large chunk of the programme was devoted to the "watering hole" inhabited by 300 or so crocodiles. Despite the crocs repeatedly grabbing victims as they drank, the crowd kept returning to drink every few minutes. Admittedly there was nowhere else to drink for 30km, but maybe it shows that sometimes the crowd never learns.

Labels: swarming, wisdom of crowds