Some friends of mine on Facebook were talking about rape (as you do), and one of them brought up how anti-rape public awareness campaigns targeting men have been found to be effective.
Their evidence, just like the evidence of everyone else who makes this claim, was an article in the Vancouver Globe and Mail, which noted that the city’s anti-rape campaign – called “Don’t Be That Guy” and consisted of hanging posters with catchy slogans about how men really shouldn’t have sex with non-consenting women – had successfully decreased rapes in the city. They know this because the year before the campaign, sexual assault rates went up, but the first full year of the campaign, sexual assault rates went down.
I wish I lived in a world that worked like this. If hanging up pictures of distressed-looking pretty women that say “DON’T BE THAT GUY” can reverse rising sexual assault rates within a year, maybe we could end the drug war with pictures of unhappy-looking people smoking that say “WINNERS DON’T DO DRUGS”, or prevent teenage pregnancy with pictures of smiling couples that say “TRUE LOVE WAITS”, or decrease racism with pictures of different ethnicities holding hands and the caption “DIVERSITY IS OUR GREATEST ASSET”.
But I shouldn’t be so cynical. This initiative does claim to have evidence supporting it. So let’s analyze that evidence.
The evidence is that after the “Don’t Be That Guy” campaign in Vancouver, sex crimes decreased by 10%.
But Vancouver was not the only city to try the campaign. “Don’t Be That Guy” originated in Edmonton. After that city’s “Don’t Be That Guy” campaign, sex crimes increased by 14%, even as other categories of crime were decreasing.
Am I blaming Edmonton’s “Don’t Be That Guy” campaign for causing more rape? Probably not (although if you read the links above, you know that’s not as far-fetched as it sounds). I am saying that these kinds of small changes in crime are very common for cities and mean nothing.
Just as it would be silly to seize upon a +14% increase in rapes in Edmonton as suggesting anti-rape campaigns cause rape, so it is silly to seize upon a -10% decrease in rapes in Vancouver as suggesting anti-rape campaigns prevent rape.
It is generally bad to take a single number out of context and use it to claim a causative effect. If a hundred people take homeopathic remedies, one of them will have her blood pressure go down 10%, just because blood pressure goes down sometimes. If you are the person selling the homeopathic remedy, you can say “Look! Vanessa used homeopathy, and her blood pressure decreased by 10%! Clearly it’s working!”
Likewise, if you are a movement pushing this kind of anti-rape campaign, you can publicize the article in the Vancouver newspaper that says crime rates decreased 10% after your “remedy” was tried, and just fail to publicize anything about Edmonton . If you have a good enough campaign, everyone will hear about it from you, no one will bother to double-check, and it will become a well-known fact that anti-rape campaigns targeting men lowers the sexual offense rate.
The counterspell is moderate skepticism of articles in the popular media claiming amazing social science results when there is no peer-reviewed research corroborating the discovery. In the case of “Don’t Be That Guy”, as far as I know such peer-reviewed research does not exist.
This concludes the part of this essay that will not be lots of boring numbers.
Still with me? Good. There are actually some other interesting ways we can pick apart the Vancouver rape data, and for the sake of completeness we might as well try. Our source will be the Vancouver Police Department’s crime statistics reports.
These statistics confirm that in 2011, when the “Don’t Be That Guy” program was introduced, sex offenses decreased by -9.7%. But just from that number, we don’t know whether this is an effect of the program, or just the continuation of a long-term trend. Maybe sex crimes in Vancouver decrease by 9.7% every year. So the Globe and Mail article very correctly focuses on the change in the trend. They observe that in 2010, the year before the campaign, sex crimes rose +4.7%, and now in 2011 with the campaign they’re dropping -9.7%. That’s much more impressive. It’s a big switch in the trend.
The mathematical term for this kind of change in trends is “second derivatives”, and the second derivative of sex crimes from 2010 to 2011 was -14.4 (the difference between a +4.7% increase and a -9.7% decrease).
But from the same crime statistics, we see that breaking and entering was down -11.1% in 2010, but up by +5.4% in 2011, so its second derivative shifted 16.6. Possession of stolen goods shifted from -18.7% in 2010 to +3.8% in 2011, a second derivative of +22.5. Arson shifted from -6.5% in 2010 to +38% in 2011, for a second derivative of +44.5. It would be moderately unfair to include murder on this list since there are so few murders that large variations are easy to come by, but if you’ll humor me, change in murder rate went from -53.7% in 2010 to +48.1% in 2011, for a second derivative of 101.8.
My point with all these numbers is that second derivatives are volatile things, and the change in sexual offenses in 2010-2011 was equal to or less than the change in a whole bunch of other crimes, none of which had campaigns targeted against them. In other words, it’s very much the sort of thing you would expect to see by chance.
This is true across years as well as across crimes. In 2004, sex offenses were up by +21.9%, but in 2006, they were decreasing by -1.3, and the next year, in 2007, they were down -7.2%, almost as much as with the introduction of the “Don’t Be That Guy” campaign in 2011 (of course, there was no campaign in 2007) By 2009, the pendulum had swung wildly once again, and there was a huge +17.7% increase in sexual offenses. This tells the story of a city where it is common for sex offenses to go up by a large amount one year and then go down by a large amount in another year, or vice versa, whether people are putting posters on streetlights or not.
I’m not exactly sure how kosher this is, but we can try to quantify this with a third derivative – the change in the change in the change . We mentioned before that the second derivative of sex crimes from 2010 to 2011, the year of the “Don’t Be That Guy” campaign, was -14.4 – a movement from +4.7 to -9.7. But from 2009 to 2010, the year before the program was introduced, the second derivative was -13.0 – it moved from +17.7% to +4.7%.
In other words, the impressive change in the second derivative of sex offenses in 2011 was the near-exact predicted continuation of a trend that had started a year earlier. There was just as big a shift in 2010 – when the program had not started yet – as in 2011, when it had.
There’s something a little sneaky about this – it should always be possible to find some level of derivative that supports the results you want – but given that the Globe and Mail article was making a second-derivative based argument, I don’t think it’s too unfair for me to follow along. Readers who know more about statistics than I can tell me whether this is already a solved problem.
Of course, the proper way to do this would be to use all the different crime statistics to figure out what the average noise in changes in Vancouver crime rates are, and then see whether the change in rape in 2011 was significantly greater than the noise at a 5% level. I have neither the time nor the intelligence to do this formally, but it’s what I’ve been doing informally throughout this article and it should be clear by now that the answer is “no, it wasn’t”.
So now we just have some mopping up to do. For example, did you know that the majority of the “sexual offenses” reported as declining in the Globe and Mail article aren’t rape at all, but things like “groping, grabbing, or kissing”, and that as far as I know no statistics have ever been published about what happened to rape itself during the campaign period? Or that according to official Vancouver statistics, the large majority of sexual offences are committed by strangers – which is contrary to what we know to be true about sexual offences in general, which suggests there’s a large bias in terms of which sexual offences get reported to the police, and which are the category of sexual offence it’s least plausible that posters defining rape would help with? Or that police attribute much the previous rise in sexual offenses to offenses committed against prostitutes, who are totally ignored by the “smiling girl asks to have a drink with you but probably doesn’t want sex” focus of the “Don’t Be That Guy” campaign?
Of course, none of this proves that campaigns like “Don’t Be That Guy” don’t work. It just shows there’s not much evidence that they do. This brings us back to our prior that putting up some posters around town can arrest major social ills. Based on the examples above like drug use and abstinence-only sex education, my prior for this is very low.
Your mileage may vary, but before deciding I urge you to consider an alternative hypothesis. That claims that the posters lower rape – never very plausible on the face of it – are fake consequentialism (I don’t have a good link for this concept yet, but search “fake consequentialism” here). That the real reason people put up “Don’t Be That Guy” posters is the same reason they put RAPE FREE ZONE posters on college classroom doors and the same reason men’s rights activists have taken to putting up “Don’t Be That Girl” posters. It’s a way of signaling your membership in a specific tribe and demonstrating the power of that tribe by being able to take over public places with your symbols.
Or maybe I’m being totally paranoid. We’ll probably know one way or the other soon enough, as its “confirmed success” in Vancouver has led to everyone everywhere adapting the “Don’t Be That Guy” campaign and we will soon have oodles of useful data that a real social scientist can use to do a real analysis. I look forward to reading what it says.
In the meantime, please don’t say that there is already evidence in the program’s favor unless you’re willing to give many more caveats than people typically do when they make that statement.
: The only blogger I have ever seen mention Edmonton in regards to “Don’t Be That Guy” is Greta Christina, who very correctly talks about Edmonton as the birthplace and center of the campaign, says how effective it was in Edmonton and how happy the police there were with its effects, and cites as proof of this……that rape went down in Vancouver. Did you know that 100% of my uses of the phrase “what is this I don’t even” this year have been while reading Freethought Blogs?
: Third derivative can also be called “jerk”, which seems appropriate when we are using it to measure the number of rapists.