An experiment: gaming Slashdot's moderation system

Or: why moderation systems are trickier than they look.

Understanding the dynamics of online communities is one of my pet interests, so as a regular reader of Hacker News, I took notice when Paul Graham started experimenting recently in an attempt to stave off the gradual but unmistakable decline (AKA redditization) of what used to be my favorite online community after TurnKey.

The discussion inspired me to write a blog post about an experiment I tried a while back at my other geek haunt - Slashdot. Just to throw in my two cents.

In theory comment moderation systems are democratic and promote a high signal to noise ratio. But I've long suspected them of promoting group-think and being easy to game once you understand a little bit about the dynamics at play.

Let me back up a bit and tell how it all started. Slashdot have a pretty sophisticated moderation system that scores comments from -1 to +5. I like reading comments (for some stories), but for hundreds of comments it's just not practical without some kind of filtering, and I figured asking Slashdot to filter everything below +5 would be a good way to get straight to the good stuff which I assumed would bubble naturally to the top.

But then this one time I put my heart into a comment (where I usually just lurk) and was a bit surprised that it was promptly buried by the moderation system. It wasn't censored so much as drowned out by the noise. Sort of a censorship by glut.

This got me thinking about the dynamics of moderation and motivated me to take a closer look at one of the most famous and supposedly sophisticated moderation systems online to see if I could spot any patterns.

I came up with a few ideas regarding potential influencing factors that would skew the system and then ran a brief ad-hoc experiment to test my hypothesis. With interesting results.

6 out of 7 test posts which I submitted according to "the rules" received a perfect +5. The 7th message received a +4. From start to finish I had boosted my karma up 5 levels from neutral to excellent (the highest possible score) in 48 hours. Having "Excellent" Karma gives you a bonus point so all your posts now start from +2. This makes it even easier to game the system. Too easy really. I ended the experiment.

Common artifacts that can be used to "game" a moderation system:

• Old boys club effect: On Slashdot it seemed pretty obvious that posts by users with low user ids have a much higher chance of getting moderated up. I think this is because of how Slashdot's moderation algorithms put most moderation points into the hands of old-timers which are more likely to moderate their own kind up.

• First posts: there seems to be a powerful snowball effect on Slashdot whereby posts that are submitted early are much more likely to make it to a high score compared with another post of equal merit submitted later.

  Especially on low traffic stories it is quite common for a high-quality comments to not get moderated up. This is meaningful because high-scoring posts are much more likely to be read, due to the sheer volume of low-scoring posts.

I also noticed that posts have a higher chance of being moderated up if they are:

• Promoting political viewpoints which align with popular sentiment in the community (e.g., libertarian).
• Brief: there seems to be an inverse correlation between moderation points and length of posts. Short, concise posts are more likely to be moderated up. Probably because they are more likely to be read and enjoyed in the first place.
• Have good grammar and no speling mistakes.
• Not "selling" anything. Slashdot moderators are instinctively hostile to marketing. A post that seems (even superficially) to promote someone's commercial interests is far less likely to receive moderation points.