Tag Archives: trolls

Trolling isn’t outlier behavior, and we can stop it

Large troll standing over a house

For years the picture painted of trolls was pretty straightforward: while most members of online communities are good people, there are a few horrible, unchangeable, malicious people called “trolls” who live to make everyone’s life terrible. Our job was to try to keep them out, ban them when they showed up, and sigh and accept that they were an inevitable part of any online community.

What has become clear is that we were wrong; most trolls are regular people.

Two recently released studies have shown that the majority of “troll” behavior is actually generated by normal people who have been triggered into acting negatively, usually through a combination of their own mental state (i.e. having a bad day) and social norms (e.g. seeing other people troll and get away with it).

  1. The famously toxic League of Legends found that only about 1% of their players were consistently toxic, and those produced only about 5% of the toxicity. “The vast majority was from the average person just having a bad day.”
  2. Scientists from Cornell and Stanford found that people are more likely to troll if they were in a negative mood, late at night, and if the first comment on a thread was a “troll comment”.

This is a game-changer for several reasons.

One, it means we may have been banning or punishing a large number of normal people who were just doing what they saw others doing. It’s likely that we only reinforced their negative behavior, rather than helping them adjust it.

Two, it means there’s a lot more we can do to prevent trolling. A recent experiment on Reddit found that rule posts stuck to the top of a thread increased rule following by 7.3 percentage points and increased newcomer participation by 38.1%. League of Legends found that some simple priming “reduced negative attitudes by 8.3%, verbal abuse by 6.2% and offensive language by 11%”. Some people are further down the rabbit hole of negativity, but even they may be saved. We are not helpless to decrease trolling, and continuing to act like we are is irresponsible.

(You can find my much longer post on ways to create positive online spaces here.)

Three, it means community managers are even more important in any organization that has an interactive online space. We are no longer just reactive janitors, apologizing for the mess. We can be proactive social designers. (Be sure to go seek out some behavioral psychology books and classes, folks.)

To me, this is extremely exciting. It means our online communities can become more positive, safe places. And it means that our work is far from done. Complacency happens in every industry. The community industry has finally started pushing through our complacency about ROI. Next, let’s tackle trolling.


It’s important that I note that these findings don’t mean there aren’t real, horrible people on the internet. It doesn’t mean we need to put up with harassment just because someone had a bad day. I’m not condoning bad behavior – I’m just optimistic that we can change much of it.


Troll photo courtesy of EE Shawn

Two Causes of Toxic Online Spaces (and some solutions)

This is adapted from my talk at Bridge Keepers.

speeding cars

Have you ever gone over the speed limit?

Probably. In an observational study, 7 out of 10 drivers sped in an urban area. At some point, you were likely part of that 7.

Have you heard of how toxic the community for the game League of Legends is? Legendary. (Pun intended.) And you would expect that this is mainly generated by a bunch of bad actors. But it turns out only about 5% of negativity came from trolls – the rest came from normal folks “having a bad day”.

So what is it that causes relatively normal people to behave so badly?

I think there are two major things at play: Normalization and Lack of Punishment.

Normalization

We like to think that we are individuals, unaffected by others. But it’s just not true.

The amazing researcher Dan Ariely (author of one of my favorite books) did a study in which he found that people would change their choice of beer at a brewery based on what the person before them ordered. If that person ordered what they were planning on ordering, they changed their order.

That’s kind of insane. If you want one type of beer, why order a different type? Well, because there’s a norm at play that we should achieve variety to better know what options we have, and our brain automatically kicks in. Even though, as found in the study, people who make these group-motivated choices are generally unhappier with the result.

What I’m saying is: Observing others’ behavior actually changes ours.

Which means if we see someone tweet something this horrible…

Tweet from Milo Yiannopoulos offensive tweet: There's a new feminist Barbie. When you pull the string it says "Math is hard, let's lie about rape."

…then we think this tweet must be acceptable, because at least it’s a joke! Right? Right??

Offensive tweet: A lesbian, an alcoholic, and a heavily sedated woman walks into a bar. The bartender asks, "What'll you have, Hillary?"

Types of Norms

There are two types of norms.

 Injunctive norms are basically rules.

 Descriptive norms are the norms we understand from our interactions with others or the actual reality we see in the world around us.

Going back to the speeding example: We all know that speeding is illegal. And we know what the rules are. But the descriptive norm is “eh, it’s ok to speed a little bit”. So we do it without thinking of ourselves as criminals.

ivy covering a building

The problem with this is that norms are like ivy. Once they’re firmly rooted, they’re very hard to change.

So, how do we create positive norms?

Clear Guidelines

Obvious, but bears repeating. Your community needs to have guidelines, and they should be:

1. Simple. If they’re complex, nobody will bother reading them.

2. In-line. Nobody is going out of their way to find your rules. Put them in-line where possible. Many subreddits do a great job of this:

Screen Shot 2017-01-17 at 8.29.09 PM

3. General. If your rules are too specific people won’t bother spending the time understanding what they can and can’t do. (And bad actors may try to find a way to technically obey the rules while causing trouble.)

Consistently Applied Guidelines

Studies show that people are much more likely to obey the rules if they’re consistently applied. This means:

1. Train, train, train your team. Being consistent is hard. You should drill, review past moderations together, do flashcards, whatever you need to do to get consistent across your whole org.

2. Create awareness of implicit bias. None of us wants to think we’re biased, but as shown in the beer experiment above, we often are without even realizing it. And when we are rule enforcers, this can be really problematic. There’s immense evidence that police are biased, and that’s something that has very serious consequences. Are these police racist? No, they’re just operating off biases they may not realize they have. There are plenty of great orgs that help teams work on realizing what implicit biases they have. (A few that have been recommended to me: Paradigm, Project Include, Women’s Leadership Institute.)

Self-Moderation

I think this is one of the most criminally underrated set of tools for driving specific behavior.

1. Display true norm rates. People adjust their habits when they see how others behave (like these college students that drank less once they saw the average number of drinks their peers were drinking).

2. Prime people. Putting people in the right mindset can be incredibly effective. League of Legends found that simply displaying messages before a game (like “Players perform better if you give them constructive feedback after a mistake”) they decreased bad behavior by 11%.

3. Add friction. We are obsessed in the tech world with “frictionless” experiences. But if someone’s action may negative affect dozens, hundreds, millions of people? A little friction can be good. Nextdoor added some additional steps you have to take before posting about suspicious people on their platform, and racial profiling dropped by 75%.

Bad news, though…Guidelines backfire if behavior doesn’t match.

coffee-987119_1920 (1)

The more litter we see? The more likely we are to litter.

The more we see rule breakers unpunished? The more likely we are to break rules.

Rules are not effective if they’re not enforced.

Especially if there’s a big benefit to breaking the rules. After all, if:

  • Attention is the goal

  • And negativity generates attention

  • And punishment is rare…

…why would you stop? Especially if you can get a book deal from being horrible?

This is the Tragedy of the Commons.

We all know the world would be better if we all obey the rules. But if you personally gain benefit from you personally breaking the rules, you may do it at the expense of others.

Certainty of Punishment

So how do you make these guidelines affective? Create certainty of punishment. If punishment is unlikely, you will continue offending.

The death penalty? Doesn’t look like it actually decreases crimes. Because even though it’s a severe punishment, you’re unlikely to get caught and unlikely to be given this sentence.

Blood alcohol checkpoints? Very effective at decreasing drinking and driving. Because you’re very likely to be caught and punished.

So, how do we create certainty of punishment?

Automation

This is the baseline stuff you should be doing.

1. Blacklist words. Anyone using unacceptable words should automatically be penalized.

2. Spot suspicious behavior. Multiple posts in a short period of time? Similar posts across sub-forums? Shut it down automatically.

Flagging

User flagging is a key tool in the fight against negative behavior. Your users are on the front line and they will always be faster than your team.

1. Make it prevalent. This functionality should be really easy to find, always.

2. Create specific flows. People often struggle with flagging because so many things get flagged that they get overwhelmed. Consider more complex flows depending on what flag was thrown. Marked as annoying? Deprioritize? Marked as racist? Prioritize. Marked as “bugs me”? Implement a clever self-resolution flow like Greater Good and Facebook did.

Reputation Systems

Treat repeat offenders differently. Go pick up Building Web Reputation Systems and figure out what’s best for you.

1. Create visibility thresholds. Have repeat offenders’ posts be less visible to others (which really hurts their desire for attention). Or require people to get special flags to show up to general audiences, like Kinja did.

2. Have reputation affect flag weight. If a repeat offender’s post gets a single flag, weigh that more heavily than a single flag on a good actor’s post.

The punchline: Investing in moderation now saves money later.

I know it’s hard to prioritize spending for moderation, especially when you’re starting out. But “we’ll deal with that issue if it comes up” and “I’m sure people will behave” clearly don’t bear out. And if you wait until you have truly toxic norms and lack of certainty of punishment, it’s going to be way more costly.

Let’s go back to that ivy example. Have you seen what walls look like after you laboriously remove the ivy?

ivy suckers left on wall

You have to sand those suckers off and repaint.

The equivalent for communities? A whole lot of messaging, a whole lot of banning, and a whole lot of complaints until a new norm is established. Just ask Reddit.


 

A few notes…

Much of the research here is taking from the hard-to-read but incredibly valuable “Building Successful Online Communities”.

Although I have 10 years of experience in the world of community, I am not an expert at moderation or trust & safety. I’m sure I missed things or mischaracterized things. I would love to hear your insights in the comments!