Paper Reading #21, "Towards a Reputation-based Model of Social Web Search"

http://jimmymho.blogspot.com/2011/04/paper-reading-21-multimodal-labeling.html
http://vincehci.blogspot.com/2011/04/paper-reading-20-data-centric.html

Towards a Reputation-based Model of Social Web Search

Kevin McNally, Michael P. O’Mahony, Barry Smyth, Maurice Coyle, Peter Brigg
University College Dublin, Ireland
Presented at IUI '10, Feb 7-10 2010, Hong Kong

Summary
This paper is about the HeyStaks system for collaborative usage of search engines such as Google. The authors found a need for such a device due to extensive collaborative usage of such systems even without explicit software support in place.

HeyStaks works on a reputation model, carefully designed to have its incentives keep close to actual "production of usual shared search content". Other users can vote the information provided by any given searcher as useful or not useful, among other algorithms. This was found to be reasonably successful in preventing 'gaming' of the system to produce high reputation with producing content.

One interesting anomaly is that the users seemed to break down into searcher/follower clusters even without this being an explicitly coded or even intended outcome of the system. The results of the user study included in the paper (see image below) sort of reflect this, with five producers and twenty one consumers.

From the paper.

The authors currently have a 500 user beta underway, but no results from it were included in the paper.

Discussion
The idea here is interesting - certainly it is undeniable that a lot of collaborative searching happens, so anything that helps in that area would be useful - but I'm not sure about it's broad applicability. Since the users already have two terminals available to them, I don't see where there is a big prospect for improvement. After all, the example of current collaboration they gave was a user suggesting search terms over another's shoulder.

Let me put my concern this way: When searching the desert, a two seat plane is better than a one seat plane. But is it as good as or better than two one seat planes? I doubt it.

The public beta is a good next step, and I'd kind've like to see a Wizard of Oz type study mocking up the planned finished product.

The other thing they could have improved on was a clearer explanation of how the HeyStak worked. Not the mechanics of the algorithm, but how it interacted with the users. How were searches sent back and forth, for instance? The information may be there, but I can't seem to sift it out.