(see also parts 1, 2, and 3 of ∞)
Some friends of mine on Facebook were talking about rape (as you do), and one of them brought up how anti-rape public awareness campaigns targeting men have been found to be effective.
Their evidence, just like the evidence of everyone else who makes this claim, was an article in the Vancouver Globe and Mail, which noted that the city’s anti-rape campaign – called “Don’t Be That Guy” and consisted of hanging posters with catchy slogans about how men really shouldn’t have sex with non-consenting women – had successfully decreased rapes in the city. They know this because the year before the campaign, sexual assault rates went up, but the first full year of the campaign, sexual assault rates went down.
I wish I lived in a world that worked like this. If hanging up pictures of distressed-looking pretty women that say “DON’T BE THAT GUY” can reverse rising sexual assault rates within a year, maybe we could end the drug war with pictures of unhappy-looking people smoking that say “WINNERS DON’T DO DRUGS”, or prevent teenage pregnancy with pictures of smiling couples that say “TRUE LOVE WAITS”, or decrease racism with pictures of different ethnicities holding hands and the caption “DIVERSITY IS OUR GREATEST ASSET”.
But I shouldn’t be so cynical. This initiative does claim to have evidence supporting it. So let’s analyze that evidence.
The evidence is that after the “Don’t Be That Guy” campaign in Vancouver, sex crimes decreased by 10%.
But Vancouver was not the only city to try the campaign. “Don’t Be That Guy” originated in Edmonton. After that city’s “Don’t Be That Guy” campaign, sex crimes increased by 14%, even as other categories of crime were decreasing.
Am I blaming Edmonton’s “Don’t Be That Guy” campaign for causing more rape? Probably not (although if you read the links above, you know that’s not as far-fetched as it sounds). I am saying that these kinds of small changes in crime are very common for cities and mean nothing.
Just as it would be silly to seize upon a +14% increase in rapes in Edmonton as suggesting anti-rape campaigns cause rape, so it is silly to seize upon a -10% decrease in rapes in Vancouver as suggesting anti-rape campaigns prevent rape.
It is generally bad to take a single number out of context and use it to claim a causative effect. If a hundred people take homeopathic remedies, one of them will have her blood pressure go down 10%, just because blood pressure goes down sometimes. If you are the person selling the homeopathic remedy, you can say “Look! Vanessa used homeopathy, and her blood pressure decreased by 10%! Clearly it’s working!”
Likewise, if you are a movement pushing this kind of anti-rape campaign, you can publicize the article in the Vancouver newspaper that says crime rates decreased 10% after your “remedy” was tried, and just fail to publicize anything about Edmonton [1]. If you have a good enough campaign, everyone will hear about it from you, no one will bother to double-check, and it will become a well-known fact that anti-rape campaigns targeting men lowers the sexual offense rate.
The counterspell is moderate skepticism of articles in the popular media claiming amazing social science results when there is no peer-reviewed research corroborating the discovery. In the case of “Don’t Be That Guy”, as far as I know such peer-reviewed research does not exist.
This concludes the part of this essay that will not be lots of boring numbers.
Still with me? Good. There are actually some other interesting ways we can pick apart the Vancouver rape data, and for the sake of completeness we might as well try. Our source will be the Vancouver Police Department’s crime statistics reports.
These statistics confirm that in 2011, when the “Don’t Be That Guy” program was introduced, sex offenses decreased by -9.7%. But just from that number, we don’t know whether this is an effect of the program, or just the continuation of a long-term trend. Maybe sex crimes in Vancouver decrease by 9.7% every year. So the Globe and Mail article very correctly focuses on the change in the trend. They observe that in 2010, the year before the campaign, sex crimes rose +4.7%, and now in 2011 with the campaign they’re dropping -9.7%. That’s much more impressive. It’s a big switch in the trend.
The mathematical term for this kind of change in trends is “second derivatives”, and the second derivative of sex crimes from 2010 to 2011 was -14.4 (the difference between a +4.7% increase and a -9.7% decrease).
But from the same crime statistics, we see that breaking and entering was down -11.1% in 2010, but up by +5.4% in 2011, so its second derivative shifted 16.6. Possession of stolen goods shifted from -18.7% in 2010 to +3.8% in 2011, a second derivative of +22.5. Arson shifted from -6.5% in 2010 to +38% in 2011, for a second derivative of +44.5. It would be moderately unfair to include murder on this list since there are so few murders that large variations are easy to come by, but if you’ll humor me, change in murder rate went from -53.7% in 2010 to +48.1% in 2011, for a second derivative of 101.8.
My point with all these numbers is that second derivatives are volatile things, and the change in sexual offenses in 2010-2011 was equal to or less than the change in a whole bunch of other crimes, none of which had campaigns targeted against them. In other words, it’s very much the sort of thing you would expect to see by chance.
This is true across years as well as across crimes. In 2004, sex offenses were up by +21.9%, but in 2006, they were decreasing by -1.3, and the next year, in 2007, they were down -7.2%, almost as much as with the introduction of the “Don’t Be That Guy” campaign in 2011 (of course, there was no campaign in 2007) By 2009, the pendulum had swung wildly once again, and there was a huge +17.7% increase in sexual offenses. This tells the story of a city where it is common for sex offenses to go up by a large amount one year and then go down by a large amount in another year, or vice versa, whether people are putting posters on streetlights or not.
I’m not exactly sure how kosher this is, but we can try to quantify this with a third derivative – the change in the change in the change [2]. We mentioned before that the second derivative of sex crimes from 2010 to 2011, the year of the “Don’t Be That Guy” campaign, was -14.4 – a movement from +4.7 to -9.7. But from 2009 to 2010, the year before the program was introduced, the second derivative was -13.0 – it moved from +17.7% to +4.7%.
In other words, the impressive change in the second derivative of sex offenses in 2011 was the near-exact predicted continuation of a trend that had started a year earlier. There was just as big a shift in 2010 – when the program had not started yet – as in 2011, when it had.
There’s something a little sneaky about this – it should always be possible to find some level of derivative that supports the results you want – but given that the Globe and Mail article was making a second-derivative based argument, I don’t think it’s too unfair for me to follow along. Readers who know more about statistics than I can tell me whether this is already a solved problem.
Of course, the proper way to do this would be to use all the different crime statistics to figure out what the average noise in changes in Vancouver crime rates are, and then see whether the change in rape in 2011 was significantly greater than the noise at a 5% level. I have neither the time nor the intelligence to do this formally, but it’s what I’ve been doing informally throughout this article and it should be clear by now that the answer is “no, it wasn’t”.
So now we just have some mopping up to do. For example, did you know that the majority of the “sexual offenses” reported as declining in the Globe and Mail article aren’t rape at all, but things like “groping, grabbing, or kissing”, and that as far as I know no statistics have ever been published about what happened to rape itself during the campaign period? Or that according to official Vancouver statistics, the large majority of sexual offences are committed by strangers – which is contrary to what we know to be true about sexual offences in general, which suggests there’s a large bias in terms of which sexual offences get reported to the police, and which are the category of sexual offence it’s least plausible that posters defining rape would help with? Or that police attribute much the previous rise in sexual offenses to offenses committed against prostitutes, who are totally ignored by the “smiling girl asks to have a drink with you but probably doesn’t want sex” focus of the “Don’t Be That Guy” campaign?
Of course, none of this proves that campaigns like “Don’t Be That Guy” don’t work. It just shows there’s not much evidence that they do. This brings us back to our prior that putting up some posters around town can arrest major social ills. Based on the examples above like drug use and abstinence-only sex education, my prior for this is very low.
Your mileage may vary, but before deciding I urge you to consider an alternative hypothesis. That claims that the posters lower rape – never very plausible on the face of it – are fake consequentialism (I don’t have a good link for this concept yet, but search “fake consequentialism” here). That the real reason people put up “Don’t Be That Guy” posters is the same reason they put RAPE FREE ZONE posters on college classroom doors and the same reason men’s rights activists have taken to putting up “Don’t Be That Girl” posters. It’s a way of signaling your membership in a specific tribe and demonstrating the power of that tribe by being able to take over public places with your symbols.
Or maybe I’m being totally paranoid. We’ll probably know one way or the other soon enough, as its “confirmed success” in Vancouver has led to everyone everywhere adapting the “Don’t Be That Guy” campaign and we will soon have oodles of useful data that a real social scientist can use to do a real analysis. I look forward to reading what it says.
In the meantime, please don’t say that there is already evidence in the program’s favor unless you’re willing to give many more caveats than people typically do when they make that statement.
Footnotes
[1]: The only blogger I have ever seen mention Edmonton in regards to “Don’t Be That Guy” is Greta Christina, who very correctly talks about Edmonton as the birthplace and center of the campaign, says how effective it was in Edmonton and how happy the police there were with its effects, and cites as proof of this……that rape went down in Vancouver. Did you know that 100% of my uses of the phrase “what is this I don’t even” this year have been while reading Freethought Blogs?
[2]: Third derivative can also be called “jerk”, which seems appropriate when we are using it to measure the number of rapists.
Preventing rape is a really really hard problem and whenever people try to solve it they end up trying to solve neighboring easier problems like “porn exists” or “people don’t communicate enough with their sexual partners” or “there are not enough posters in this town explaining to people that rape is wrong.” I don’t know what to do about this, but it is depressing.
I’m nowhere near well informed enough to speak with confidence on positive strategies for preventing sexual assault, but I doubt it helps that it’s probably the most politicized family of crimes out there. Not just gender politics, either.
The lies, damned lies and facebook posts are definitely some of my favorites.
While I generally approve of seeing the social sciences getting a kicking, I have to harrumph a tiny bit about your swipe at homeopathy – not that I think homeopathy works beyond the placebo effect, but I rather feel that even a pharmaceutical company might go around with a “Tests show 10% improvement in underlying condition when you use our new Bibioxidotin!” campaign to doctors, even if digging down into the results reveals “Bibioxidotin is no more effective than changing your diet to drink more pomegranate juice”.
You just wrote about the third derivative being the ‘jerk’ in a casual blog post. Given your membership in a profession famous for recently rediscovering integration, you have now lost your ability to claim to be bad at math.
Some may wish to have it pointed out that “jerk” is specifically the third derivative of position (or displacement), not a generic term for the third derivative of any quantity. (In particular, it’s a physics term rather than a math term.)
It’s not uncommon for “acceleration” to be appropriated to describe the second derivatives of quantities other than position, so why not do the same for “jerk”?
You’re failing to differentiate two concepts, where one is something like “knows lots of interesting facts about pi” and the other is “can understand, work with, and prove equations”. I’ve never claimed to be bad at the first, am definitely bad at the second (compared to other people in my social circle).
Also, reinventing integration if you’re not already familiar with it is actually pretty darned impressive! It took frickin’ Isaac Newton to invent it the first time! If I were Czar, I’d have plucked that doctor out of medicine, sent her through some accelerated version of a postgrad program in mathematics, and seen whether she could perform at a Newton-level at genuinely new innovations.
Here is the paper, 202 citations according to google scholar; correspondence. Most doctors have taken calculus, so I don’t think you should be impressed by them reinventing it. Since the author is not an MD, I don’t know what classes she took.
Both of the “Don’t Be That [Sex-Specific Human]” posters state obvious truths, reminding us once again that the meanings of such communications are seldom the literal ones. (Cf. the reversal test.)
The true meaning of “Don’t Be That Guy” would be clarified by amending the slogan to
“Just Because She’s Drinking, Doesn’t Mean She Wants Sex With You“
This actually does suggest a pretty plausible mechanism whereby propaganda is effective. I mean, in the limit case, I bet cross burning is pretty effective, or used to be.
For Social Science causation, I tend to look for Granger Causality or Sugihara Convergent Cross Mapping tests, with a ‘long-term trend-discontinuity at the point of changed policy surprise’ statistically-significantly above the typical variance for at least a few periods.
My guess is that when you test the impact of a lot of our clever, novel attempts to affect one our classic stubbornly difficult social objectives, you’ll see a lot of support for null hypotheses. Significant impacts on behavior usually requires significant consequences for that behavior. Cheap and easy indoctrination is a pipedream.
Surely total Vancouver sexual assaults is the base number, assaults per year is the first derivative, change in assault rate is the second, so what you are talking about is a third derivative?
The change in the change in assault rate, yes? (i.e. the difference between two consecutive years’ changes in assault rates)
Right, the assault rate itself is a first derivative as the name suggests, so the change in the change is a third derivative.
Not every rate is a derivative. I think it is only reasonable to call it a derivative if it is on a short time scale. Maybe the weekly assault rate is fast enough to count as a derivative. The normally reported assault rate is annual not because the weekly numbers are too noisy, but because there are predictable seasonal trends.
So I think you should treat the assault rate as its own thing and call the change in assault rate a first derivative.
But that doesn’t mean that the best analysis is by taking annual numbers as their own thing. Instead of averaging out seasonal trends over a year, maybe we should seasonally correct the numbers. Or have a model that includes the weather. Comparing to other crimes is a pretty reasonable approach that doesn’t require much modeling.
I see where you’re coming from, but that also feels wrong to me. I’d never thought of it this way before, but possible what number you think of as a base and what number you think of as a derivative is subjective and relative? That is, if what we care about is the number of crimes per year, then change in number of crimes per year is the first derivative, but if we what we care about is how many crimes have ever occurred in Vancouver (which would be weird) then change in number of crimes per year is the second derivative.
This is correct.
Exactly.
Also, the standard rate is not crimes per year but crimes per person-year. I don’t think that is the time derivative of anything. One could consider crimes per year as the derivative of total crimes, but neither is very popular.
It is not weird to care about total crimes as a terminal value; it is just that the abstraction of crimes per person-year is a better as an abstraction, more amenable to policy. For a total utilitarian, total crimes is very close to the terminal value, whereas an average utilitarian cares about crimes per person-year or maybe crimes per lifetime.
In the meantime, please don’t say that there is already evidence in the program’s favor unless you’re willing to give many more caveats than people typically do when they make that statement.
No, it much better to give the one caveat that matters (“Throwing out data I don’t like”) than to drown it in many caveats.
One is more than none.
But I’m not sure it’s many more than one.
I’m mostly just happy that all those engineering calc classes finally paid off at one point in my life and I was able to finally see someone use second and third derivatives for actual analysis of actual things in the world other than hypothetical Newtonian physics.
Also, rape is bad, m’kay.
Eesh. Speaking of conceptual weapons – “No, seriously, I was raped.” “Oh man, people actually still say that?! Don’t you know that’s just what a regretful slut would say?”
(Hence the common tactic of attempting to destroy a rape accuser’s reputation in the courtroom. Though of course if every accused rapist does this, one shouldn’t update the belief of a false accusation from its prior somewhere around 8%.)
I’m actually surprised – like seriously, time-to-update-my-model surprised – by how much principle-of-charity/humanization being extended to rapists this whole debate is demonstrating.
(I’m usually the one arguing that no, “rapist” does not imply “unrepentant, irredeemable (psychopathic?) monster who deliberate set out to cause suffering for their own gain.” Just on general demonization as bias/fundamental attribution error principles.)
What are you saying? (We should not de-demonize this much?)
Not making a policy suggestion. Just noticing that I am confused, and probably need to update my model.
There certainly is a lot of feminist writing to the effect that most rapists are probably seemingly not awful people, to counter 1. the tendency to focus on uncommon depraved stranger rapes and 2. the tendency to disbelieve the accuser if the accused is even slightly sympathetic.
Also, there is some talking about how rapists often have no idea that what they are doing is wrong.
Heaven forbid we add 3. because it’s probably true.
I thought I might supplement your post with some actual examples of feminist writing:
Pro “rapists aren’t monsters, at least not openly”: http://pervocracy.blogspot.de/2011/11/slavering-beast-theory.html
and Contra “they don’t even know what they’re doing is wrong”: http://yesmeansyesblog.wordpress.com/2011/03/21/mythcommunication-its-not-that-they-dont-understand-they-just-dont-like-the-answer/
“In the fall of 1972 President Nixon announced that the rate of increase
of inflation was decreasing. This was the first time a sitting president
used the third derivative to advance his case for reelection.”
http://www.ams.org/notices/199610/page2.pdf
Sounds like my prior for ‘public education campaign can have the desired effect’ is higher than yours. (It’s still not very high.)
I think the drop in smoking in the West is overdetermined, but I suspect some of the campaigns have some detectable role. But the big ones to my mind are drink-driving and HIV/AIDS in the UK. “Everybody knows” that the rate of drink driving (as distinguished from the rate of people getting caught) has fallen, following a lot of public education campaigns (and some more punitive measures), because the social acceptability of drinking and driving changed. Similarly, “(nearly) everybody knows” that the UK infection rate for HIV never got as high as in similar countries because the UK had a hard-hitting public information campaign early.
I’ve presented these two examples as stylised scare-quoted factoids because I don’t have my hands on the evidence to back them up, I’m afraid. But I would be genuinely surprised to find the evidence to be so sketchy as you set out for the “Don’t be that guy” campaign.
On which, my understanding is that sexual assault statistics are strongly affect by reporting effects: awareness campaigns, for the public or for law enforcement, sometimes result in increased recorded crime through increasing the rate at which the crimes are reported, rather than the rate at which it happens. There may be something of that effect going on here.
Of course, disentangling that is going to be tricky. Tune in to next week’s episode of Social Mission: Impossible!
Oh, I came here to make this comment, (particularly about drink driving and seatbelts in the UK) and you’d made it already 🙂
Same in Belgium. The Bob campaign against drunk driving (http://en.wikipedia.org/wiki/Bob_campaign) changed public opinion and behavior, raised social status of designated drivers, and even created a word/meme for designated driver (“Bob”) that caught on.
When someone takes a higher derivative of a noisy time series, I’m prone to reach for my revolver; each derivative you take roughly halves the signal-to-noise ratio, and the whole point of the article is that the noise in the figures is already too large to work with.
I’m also quite inclined to believe that crime statistics are mechanistically too noisy to do much with even if they could be perfectly collected: fairly small number of criminals, each committing a number of crimes which is probably not Poisson-distributed and almost surely has a different mean per criminal, against victims with hard-to-model-at-all reporting strategies.
It’s not quite correct to calculate “second derivatives” (which might also better be called second-order differences) the way you do by adding percentages, because they are percentages of different baselines.
To illustrate, if you imagine that the baseline 2009 figure for sex crimes was 100 crimes, then in 2010 the number rose +4.7% to 104.7 crimes, and then in 2011 it fell -9.7% from 2010’s value of 104.7 to become 94.5. The “first derivatives” in actual numbers, not percentages, are +4.7 and -10.2, and the “second derivative” is the difference between those: -14.9, not -14.4. You could say that the second derivative is -14.9% with respect to the baseline 2009 figure. In this example, the difference is small, but in the wildly fluctuating murder rate example, the “second derivative” turns out to be ~76% rather than ~101%.
> Your mileage may vary, but before deciding I urge you to consider an alternative hypothesis. That claims that the posters lower rape – never very plausible on the face of it – are fake consequentialism (I don’t have a good link for this concept yet, but search “fake consequentialism” here). That the real reason people put up “Don’t Be That Guy” posters is the same reason they put RAPE FREE ZONE posters on college classroom doors and the same reason men’s rights activists have taken to putting up “Don’t Be That Girl” posters. It’s a way of signaling your membership in a specific tribe and demonstrating the power of that tribe by being able to take over public places with your symbols.
That last part was pure Reactionary. Don’t stare into the abyss for too long, Scott…
Neoreaction doesn’t have a monopoly on the signaling model of political behavior.
Writing about signaling isn’t about writing about signaling.
(Not actually sure that that sentence means anything. Just referencing Robin Hansen. I don’t think he’s considered a reactionary, am I wrong?)
No, and even if he was, that particular meme has spread through LW, and while LW is apparently a reactionary breading ground, it’s not itself reactionary. Nor is the meme itself particularly reactionary – it is fully general cynicism.
No, but the notion that politically correct demands, campaigns and “movements” are more about demonstrating the power of the ruling class* over the subjugated is a staple of reactionary thought.
*or, translated into Progressive: “minorities”.
Oooh! Minorities are the ruling class! That must be why black people are enormously more likely to be in prison for drugs than white people despite using drugs at the same rate, why LGB people are not allowed to marry the partner of their choice, and why trans kids occasionally get set on fire.
Shit, man, I’d hate to see what not being the ruling class looks like.
*shrugs* I never said it was a particularly *sane* staple of Reactionary thought.
(Scott had a great analysis o his old blog, of the different types of “power” involved here and how Reactionaries keep switching between them to make their point, but I can’t seem to find it again.)
If Donald Trump and Rebecca Black got in a bar fight, who would win?
It seems to me that identifying minorities as a politically privileged class does at least as much of the work there as the signaling model per se, and isn’t necessary to the point Scott was making.
That is, being able to generate visible symbols of political organization and involvement doesn’t necessarily imply any significant degree of political power.
As Ozy says, it’s not plausible that minorities are the ruling class, and intelligent reactionaries wouldn’t say they are; they would say that they’re clients of the progressive ruling class. (I would say that “the” ruling class, singular with no further context, is an anti-useful concept. Different kinds of power are differently concentrated.)
Pingback: Link blog: work, interviewing, productivity, business | Name and Nature
Taking 2nd or higher derivatives of discrete numbers usually produces too much noise to be useful.
The right way to do this is to develop a model for a typical progression in sexual assault crime. A simple model, for example, might be to do a quadratic fit over a ten year period. Then subtract the model from each datapoint to create residuals. The residual data should be evenly and randomly distributed around zero (except for, perhaps, the poster year). Then we compare the variation in the residuals to the residual of the poster year and check for statistical significance.
But the biggest problem is that nine data points probably aren’t enough either way.
oh man this is cool. For some reason it’s natural to me to look at other, similar things for an estimate of the mean given the null hypothesis, but getting an estimate of the variability, showing that a particular kind of statistic is a “volatile thing”, that’s something I never did before, or never noticed.