Thursday, 1 September 2011

Shirky's broken filter

Information overload doesn't exist. Or at least it doesn't in Clay Shirky's world. Shirky's famous solution to the firehose of digital information was to fit a better filter. By building networks of people we trust to deliver the news, tailored to our own unique interests, we make the abundance of data irrelevant.

It's a utopian vision of data gathering. But how many of us truly trust their own networks? Of course, they may be full of smart people with great connections of their own, people to whom we outsource the delivery of breaking news hours before the major news networks manage the same feat. But information must be broad as well as fast and smart. Does your network teach you things you never knew, or does it teach you things you could never know?

The great joy of the web is in serendipitous discovery; finding those nuggets of knowledge we never knew existed. Great projects to expand the sum of human knowledge are now, thanks to the web, well within our grasp, from efforts to digitise the previously unpublished words of Isaac Newton to attempts to document the Salem witch trials of the 17th century. Will your networks deliver information of this breadth? Unlikely. How successful can we ever be in finding experts on subjects in which we have no expertise? The filter is only effective if we can work out how to build it.

Filtering a firehose turns out to be just as challenging as drinking from one. What if the digital age's spikiest irony is that as information becomes less scarce, its very abundance becomes the biggest barrier to access? Consider Google. Its dominance of search has produced an unintended and damaging consequence in the organisation of the world's information. The highest ranked articles achieve their position as a result of the number of web links to that content. The objective is relevance, but the system often promotes the mundane and popular over the rare and treasured. As Nieman Journalism Lab's Maria Popover writes:

"An esoteric piece of content, however valuable and interesting, will remain confined to the niche community of scholars and hobbyists who have linked to it, ranking it low enough in Google's search results to prevent all but those actively seeking it out from accessing it and engaging with it. Instead, the trivial thrives and the remarkable remains rare."

Through Google's lens, all of the information we require is available, but how accessible is it? How many of us does it reach? It's one thing to organise the world's information, to claim it as "universally accessible", but it's quite another to propagate a system of accessibility that rewards popularity over value. In Google's glass, the trivial floats right to the top, an oily viscous of marginal data.

Shirky may have been right about the filter, but that filter is in need of a good clean. Google's attempt to socialise the processing of content is a good start. Google+ divides relevance into circles, making it easier for the right content to be promoted to the right people. Another way is to take the filter straight to the search engine. Bing's ambitions to become a "decision engine" are underpinned by its links with Facebook, which allow it to deliver search results that are punctuated with the recommendations of friends.

This so-called "socialisation of search" returns us to our original question: how much faith can we really have in our own networks? It's an apposite question for those of us obsessed with the numbers. If SEO is largely a popularity contest, what does that make our quest for social capital? While the ultimate goal is more friends, more clicks and more visibility, social search will always struggle for relevance. How do you decide what's important with 5,000 friends clamouring for attention?

But social search does have its own built-in editor. Perhaps over time, the links between our search queries and our friends' recommendations may come to influence the very make-up of our networks. After all, bad advice is an effective filter, too.

Pic credit: catspyjamasnz