Block Diagonal

tdl:

nique:

thedailywhat:

The Filter Bubble author Eli Pariser gives an illuminating TED talk on the dangers of online personalization algorithms that limit our worldview by attempting to predict our interests.

[ted / reddit.]

This is important. This is why I resent the internet, just a little bit.

This has actually been a real problem in my research. I am working with a rarer genus of bacteria called Burkholderia but I’m also taking a general microbiology class. I want to learn about bacterial movements in the most well-studied model systems (E. coli, for example)— but I keep getting preliminary findings in Burkholderia ranked higher. I’ve been pretty mad with Google for a while.

I hope Google is listening. 

This topic has come across my radar several times in my past software engineer life — how do you build a recommender that doesn’t suck? How do you make machines  balance the front-shelf blockbuster recommendations with the quirky video store clerk “there’s only one copy, and I keep it under the cash register” movie recommendations? I want both. Two years ago (that’s when I stopped actively following the field), nobody was working on that. 

The way everybody does it — the iTunes genius playlists, Netflix recommendations, Google News frontpages, and likely the facebook news feed (FB engineers are the only ones that were so tight-lipped that they wouldn’t even let me guess how they did it) — is to use a technique called “collaborative filtering”. It’s a pretty broad term, but basically, you take a look at what you have liked in the past, and you hope that there’s a bunch of other people that have liked similar things. The algorithm then hopes that those similar people have liked items that you haven’t tried yet. Those items that don’t overlap between you and your “taste neighbors” are the ones shuttled off to you as recommendations. There’s more nuance to it than that, but this is the gist.

Of course there’s several problems with this. The first is, what happens when people with otherwise similar taste disagree wildly and regularly about the quality of a particular item? A tied-in problem is, how do you make risky recommendations (aka “quirky”) without constantly pissing your client-base off? The closest research problem to providing quirky recommendations that’s been studied is termed the “Napolean Dynamite problem”. The issue, simply put, is some people love that movie to pieces, and others just hate that damn movie, but all other algorithmic signs point to them loving it. You liked Juno? Okay, you dig indie flicks. Oh, and Rushmore as well? Dude, you are going to eat this film up. Nope, not always. There’s some inscrutable, incompressible element of taste in there that’s just not captured by the current way of thinking about recommender systems. 

Last, how do you make recommendations for items that nobody has even tried? Implicit in this collaborative-filtering method of making recommendations is a dependence on a consistent cadre of taste-makers, forging ahead and trying new things, and giving reliable, even-keeled recommendations. For movies, restaurants, plays and the like, professional critics serve this role, and their output is often algorithmically digestible, because they provide out-of-five star ratings. But for news? How would you recommend that somebody go and check out the riots in Egypt? Where do you get the information to build a recommendation like that? Do you trust Twitter? Do you mine hashtags, and look for emotionally-charged words to scrape for valence? It’s not clear to me.

So what’s the solution? Algorithms aren’t going away. I’m aligned with Google here — the amount of information out there is much too vast to not even try to organize it. Using Twitter, I’m restricted to my little networks, although through various stochastic events, I’ve become connected to people across the world, and we still dialogue regularly.

Just like there was a Netflix competition to improve movie recommendations, I think a good idea would to be a Google News competition, to reward the development of an algorithm that simultaneously nailed people’s static tastes, but was also nimble enough to understand what was new and what was important. The Netflix prize was a million dollars, and Netflix got immensely more value out of the competition than that. Google should offer ten million for this, and they’d likely get an entire tranche of the computer science community working on it tirelessly.