Beware The Man Of One Study

Aquinas famously said: beware the man of one book. I would add: beware the man of one study.

For example, take medical research. Suppose a certain drug is weakly effective against a certain disease. After a few years, a bunch of different research groups have gotten their hands on it and done all sorts of different studies. In the best case scenario the average study will find the true result – that it’s weakly effective.

But there will also be random noise caused by inevitable variation and by some of the experiments being better quality than others. In the end, we might expect something looking kind of like a bell curve. The peak will be at “weakly effective”, but there will be a few studies to either side. Something like this:

We see that the peak of the curve is somewhere to the right of neutral – ie weakly effective – and that there are about 15 studies that find this correct result.

But there are also about 5 studies that find that the drug is very good, and 5 studies missing the sign entirely and finding that the drug is actively bad. There’s even 1 study finding that the drug is very bad, maybe seriously dangerous.

This is before we get into fraud or statistical malpractice. I’m saying this is what’s going to happen just by normal variation in experimental design. As we increase experimental rigor, the bell curve might get squashed horizontally, but there will still be a bell curve.

In practice it’s worse than this, because this is assuming everyone is investigating exactly the same question.

Suppose that the graph is titled “Effectiveness Of This Drug In Treating Bipolar Disorder”.

But maybe the drug is more effective in bipolar i than in bipolar ii (Depakote, for example)

Or maybe the drug is very effective against bipolar mania, but much less effective against bipolar depression (Depakote again).

Or maybe the drug is a good acute antimanic agent, but less effective at maintenance treatment (let’s stick with Depakote).

If you have a graph titled “Effectiveness Of Depakote In Treating Bipolar Disorder” plotting studies from “Very Bad” to “Very Good” – and you stick all the studies – maintenence, manic, depressive, bipolar i, bipolar ii – on the graph, then you’re going to end running the gamut from “very bad” to “very good” even before you factor in noise and even before even before you factor in bias and poor experimental design.

So here’s why you should beware the man of one study.

If you go to your better class of alternative medicine websites, they don’t tell you “Studies are a logocentric phallocentric tool of Western medicine and the Big Pharma conspiracy.”

They tell you “medical science has proved that this drug is terrible, but ignorant doctors are pushing it on you anyway. Look, here’s a study by a reputable institution proving that the drug is not only ineffective, but harmful.”

And the study will exist, and the authors will be prestigious scientists, and it will probably be about as rigorous and well-done as any other study.

And then a lot of people raised on the idea that some things have Evidence and other things have No Evidence think holy s**t, they’re right!

On the other hand, your doctor isn’t going to a sketchy alternative medicine website. She’s examining the entire literature and extracting careful and well-informed conclusions from…

Haha, just kidding. She’s going to a luncheon at a really nice restaurant sponsored by a pharmaceutical company, which assures her that they would never take advantage of such an opportunity to shill their drug, they just want to raise awareness of the latest study. And the latest study shows that their drug is great! Super great! And your doctor nods along, because the authors of the study are prestigious scientists, and it’s about as rigorous and well-done as any other study.

But obviously the pharmaceutical company has selected one of the studies from the “very good” end of the bell curve.

And I called this “Beware The Man of One Study”, but it’s easy to see that in the little diagram there are like three or four studies showing that the drug is “very good”, so if your doctor is a little skeptical, the pharmaceutical company can say “You are right to be skeptical, one study doesn’t prove anything, but look – here’s another group that finds the same thing, here’s yet another group that finds the same thing, and here’s a replication that confirms both of them.”

And even though it looks like in our example the sketchy alternative medicine website only has one “very bad” study to go off of, they could easily supplement it with a bunch of merely “bad” studies. Or they could add all of those studies about slightly different things. Depakote is ineffective at treating bipolar depression. Depakote is ineffective at maintenance bipolar therapy. Depakote is ineffective at bipolar ii.

So just sum it up as “Smith et al 1987 found the drug ineffective, yet doctors continue to prescribe it anyway”. Even if you hunt down the original study (which no one does), Smith et al won’t say specifically “Do remember that this study is only looking at bipolar maintenance, which is a different topic from bipolar acute antimanic treatment, and we’re not saying anything about that.” It will just be titled something like “Depakote fails to separate from placebo in six month trial of 91 patients” and trust that the responsible professionals reading it are well aware of the difference between acute and maintenance treatments (hahahahaha).

So it’s not so much “beware the man of one study” as “beware the man of any number of studies less than a relatively complete and not-cherry-picked survey of the research”.

II.

I think medical science is still pretty healthy, and that the consensus of doctors and researchers is more-or-less right on most controversial medical issues.

(it’s the uncontroversial ones you have to worry about)

Politics doesn’t have this protection.

Like, take the minimum wage question (please). We all know about the Krueger and Card study in New Jersey that found no evidence that high minimum wages hurt the economy. We probably also know the counterclaims that it was completely debunked as despicable dishonest statistical malpractice. Maybe some of us know Card and Krueger wrote a pretty convincing rebuttal of those claims. Or that a bunch of large and methodologically advanced studies have come out since then, some finding no effect like Dube, others finding strong effects like Rubinstein and Wither. These are just examples; there are at least dozens and probably hundreds of studies on both sides.

But we can solve this with meta-analyses and systemtic reviews, right?

Depends which one you want. Do you go with this meta-analysis of fourteen studies that shows that any presumed negative effect of high minimum wages is likely publication bias? With this meta-analysis of sixty-four studies that finds the same thing and discovers no effect of minimum wage after correcting for the problem? Or how about this meta-analysis of fifty-five countries that does find effects in most of them? Maybe you prefer this systematic review of a hundred or so studies that finds strong and consistent effects?

Can we trust news sources, think tanks, econblogs, and other institutions to sum up the state of the evidence?

CNN claims that 85% of credible studies have shown the minimum wage causes job loss. But raisetheminimumwage.com declares that “two decades of rigorous economic research have found that raising the minimum wage does not result in job loss…researchers and businesses alike agree today that the weight of the evidence shows no reduction in employment resulting from minimum wage increases.” Modeled Behavior says “the majority of the new minimum wage research supports the hypothesis that the minimum wage increases unemployment.” The Center for Budget and Policy Priorities says “The common claim that raising the minimum wage reduces employment for low-wage workers is one of the most extensively studied issues in empirical economics. The weight of the evidence is that such impacts are small to none.”

Okay, fine. What about economists? They seem like experts. What do they think?

Well, five hundred economists signed a letter to policy makers saying that the science of economics shows increasing the minimum wage would be a bad idea. That sounds like a promising consensus…

..except that six hundred economists signed a letter to policy makers saying that the science of economics shows increasing the minimum wage would be a good idea. (h/t Greg Mankiw)

Fine then. Let’s do a formal survey of economists. Now what?

raisetheminimumwage.com, an unbiased source if ever there was one, confidently tells us that “indicative is a 2013 survey by the University of Chicago’s Booth School of Business in which leading economists agreed by a nearly 4 to 1 margin that the benefits of raising and indexing the minimum wage outweigh the costs.”

But the Employment Policies Institute, which sounds like it’s trying way too hard to sound like an unbiased source, tells us that “Over 73 percent of AEA labor economists believe that a significant increase will lead to employment losses and 68 percent think these employment losses fall disproportionately on the least skilled. Only 6 percent feel that minimum wage hikes are an efficient way to alleviate poverty.”

So the whole thing is fiendishly complicated. But unless you look very very hard, you will never know that.

If you are a conservative, what you will find on the sites you trust will be something like this:

Economic theory has always shown that minimum wage increases decrease employment, but the Left has never been willing to accept this basic fact. In 1992, they trumpeted a single study by Card and Krueger that purported to show no negative effects from a minimum wage increase. This study was immediately debunked and found to be based on statistical malpractice and “massaging the numbers”. Since then, dozens of studies have come out confirming what we knew all along – that a high minimum wage is economic suicide. Systematic reviews and meta-analyses (Neumark 2006, Boockman 2010) consistently show that an overwhelming majority of the research agrees on this fact – as do 73% of economists. That’s why five hundred top economists recently signed a letter urging policy makers not to buy into discredited liberal minimum wage theories. Instead of listening to starry-eyed liberal woo, listen to the empirical evidence and an overwhelming majority of economists and oppose a raise in the minimum wage.

And if you are a leftist, what you will find on the sites you trust will be something like this:

People used to believe that the minimum wage decreased unemployment. But Card and Krueger’s famous 1992 study exploded that conventional wisdom. Since then, the results have been replicated over fifty times, and further meta-analyses (Card and Krueger 1995, Dube 2010) have found no evidence of any effect. Leading economists agree by a 4 to 1 margin that the benefits of raising the minimum wage outweigh the costs, and that’s why more than 600 of them have signed a petition telling the government to do exactly that. Instead of listening to conservative scare tactics based on long-debunked theories, listen to the empirical evidence and the overwhelming majority of economists and support a raise in the minimum wage.

Go ahead. Google the issue and see what stuff comes up. If it doesn’t quite match what I said above, it’s usually because they can’t even muster that level of scholarship. Half the sites just cite Card and Krueger and call it a day!

These sites with their long lists of studies and experts are super convincing. And half of them are wrong.

At some point in their education, most smart people usually learn not to credit arguments from authority. If someone says “Believe me about the minimum wage because I seem like a trustworthy guy,” most of them will have at least one neuron in their head that says “I should ask for some evidence”. If they’re really smart, they’ll use the magic words “peer-reviewed experimental studies.”

But I worry that most smart people have not learned that a list of dozens of studies, several meta-analyses, hundreds of experts, and expert surveys showing almost all academics support your thesis – can still be bullshit.

Which is too bad, because that’s exactly what people who want to bamboozle an educated audience are going to use.

III.

I do not want to preach radical skepticism.

For example, on the minimum wage issue, I notice only one side has presented a funnel plot. A funnel plot is usually used to investigate publication bias, but it has another use as well – it’s pretty much an exact presentation of the “bell curve” we talked about above.

This is more of a needle curve than a bell curve, but the point still stands. We see it’s centered around 0, which means there’s some evidence that’s the real signal among all this noise. The bell skews more to left than to the right, which means more studies have found negative effects of the minimum wage than positive effects of the minimum wage. But since the bell curve is asymmetrical, we intepret that as probably publication bias. So all in all, I think there’s at least some evidence that the liberals are right on this one.

Unless, of course, someone has realized that I’ve wised up to the studies and meta-analyses and and expert surveys, and figured out a way to hack funnel plots, which I am totally not ruling out.

(okay, I kind of want to preach radical skepticism)

Also, I should probably mention that it’s much more complicated than one side being right, and that the minimum wage probably works differently depending on what industry you’re talking about, whether it’s state wage or federal wage, whether it’s a recession or a boom, whether we’re talking about increasing from $5 to $6 or from $20 to $30, etc, etc, etc. There are eleven studies on that plot showing an effect even worse than -5, and very possibly they are all accurate for whatever subproblem they have chosen to study – much like the example with Depakote where it might an effective antimanic but a terrible antidepressant.

(radical skepticism actually sounds a lot better than figuring this all out).

IV.

But the question remains: what happens when (like in most cases) you don’t have a funnel plot?

I don’t have a good positive answer. I do have several good negative answers.

Decrease your confidence about most things if you’re not sure that you’ve investigated every piece of evidence.

Do not trust websites which are obviously biased (eg Free Republic, Daily Kos, Dr. Oz) when they tell you they’re going to give you “the state of the evidence” on a certain issue, even if the evidence seems very stately indeed. This goes double for any site that contains a list of “myths and facts about X”, quadruple for any site that uses phrases like “ingroup member uses actual FACTS to DEMOLISH the outgroup’s lies about Y”, and octuple for RationalWiki.

Most important, even if someone gives you what seems like overwhelming evidence in favor of a certain point of view, don’t trust it until you’ve done a simple Google search to see if the opposite side has equally overwhelming evidence.

This entry was posted in Uncategorized and tagged , . Bookmark the permalink.

270 Responses to Beware The Man Of One Study

  1. This essay seems to be suggesting that no matter how many studies you have, you cannot use them to figure out much of anything. How does this jibe with your article on race and justice in which you try to use studies to determine whether the justice system mistreats blacks? Aren’t you saying here that what you tried to do a few weeks ago is a fool’s errand?

  2. Pingback: “Beware The Man of One Study” * The New World

  3. Pingback: Beware The Man Of One Study | Slate Star Codex « Economics Info

  4. Pingback: On examining evidences for points of view, etc | The Daily Pochemuchka

  5. Pingback: Links for December 2014 - foreXiv

  6. JayMan says:

    Scott, I meant to comment on this earlier, but I’m way behind on everything.

    First of all, let me say very good post. This:

    But I worry that most smart people have not learned that a list of dozens of studies, several meta-analyses, hundreds of experts, and expert surveys showing almost all academics support your thesis – can still be bullshit.

    …is solid gold!

    Other very good points in here, like:

    there will also be random noise caused by inevitable variation and by some of the experiments being better quality than others. In the end, we might expect something looking kind of like a bell curve. The peak will be at “weakly effective”, but there will be a few studies to either side.

    Now, those said, you do indeed seem to preach from a point of adamant skeptical neutrality – indeed, radical skeptical neutrality. That’s not the best approach to take, and frankly isn’t really true skepticism.

    Let me illustrate the problem. You say:

    Can we trust news sources, think tanks, econblogs, and other institutions to sum up the state of the evidence?

    Okay, fine. What about economists? They seem like experts. What do they think?

    Here’s the problem: if you don’t understand a matter well enough to look at the evidence and come to a conclusion yourself, you’re screwed. There is no strategy, no system, no method you can use to ascertain truth with contradictory results when you, ultimately, have to rely on “experts.” End of story. If I can’t look at a matter and come to my own conclusions, I don’t form an opinion.

    Proper skepticism isn’t just being critical of claims on a matter, but being able to methodologically examine the evidence to get towards the correct conclusion. Indeed, this is integral to good science.

    Those said, again, good post. That funnel plot was rather telling (I love funnel plots), if not definitive thanks to methodological issues. Nonetheless, still telling.

  7. Pingback: A week of links | EVOLVING ECONOMICS

  8. Pingback: The Liberty Herald – Potpourri

  9. Pingback: Potpourri

  10. Pingback: 10 Tuesday PM Reads | The Big Picture

  11. Floccina says:

    The margins on many studies are too small to show much of anything. Minimum wages at the currently proposed levels being a good example. So on minimum wage I think that we all would agree that raising it to $30/hour would increase unemployment, fuel an increase in the black market for low skilled labor and drive many very low margin businesses into bankruptcy. So does the dose make the poison or can we interpolate from that, that smaller changes in he minim wage probably increase unemployment?

  12. Luke Somers says:

    > These sites with their long lists of studies and experts are super convincing. And half of them are wrong.

    More than half of them are wrong, since many of them on the right side will have incorrect reasons for reaching their conclusions!

  13. Pingback: Quotes & Links #22 | Seeing Beyond The Absurd

  14. Two points on the minimum wage issue:

    1. One of the authors of Card and Krueger, in an interview, commented that the study was for a small increase and he wouldn’t expect the result to hold for a large one. That’s some evidence on one side, given what Obama is proposing. On the other hand, in the same interview, he described the hostility he got from colleagues for the article, which is some evidence for the other side, since it suggests that studies which go against the dominant view in the field are less likely to get published. (To reveal my own bias as one member of the field, I am confident that raising the minimum wage reduces unemployment for unskilled workers, although I have no strong view on the size of the effect. My guess, polls aside, is that most serious economists, people who view economics as a way of making sense of the world rather than a fun game, would agree. It comes down to expecting the demand curve for unskilled labor to slope the same way as all other demand curves for inputs.)

    2. A lot of the discussion confuses the issue by talking about the unemployment rate rather than the unemployment rate for unskilled workers—more precisely, workers who are now making less than the proposed minimum wage. Minimum wage workers are a very small part of the labor force, so even a substantial change in their unemployment rate has an almost unobservably small effect on the national unemployment rate. Anyone who tells you “state X raised its minimum wage and the unemployment rate went up instead of down” with the implication that that’s relevant evidence is either a fool or a rogue. Ditto in the other direction.

    • Matthew says:

      . (To reveal my own bias as one member of the field, I am confident that raising the minimum wage reduces unemployment for unskilled workers, although I have no strong view on the size of the effect.

      I’m almost certain you typed the opposite of what you meant there. (Should read employment, not unemployment. Or possibly increases instead of reduces.)

    • Oops. That should have been “reduces employment.”

  15. Roger says:

    When two sincere opponents get locked into such a stalemate I would suggest that they might just be asking the wrong question.

    The question is not “do small increases in the minimum wage harm employment for lower skilled workers?” It is “do small increases in the minimum wage distort the labor market?”

    As a business person I would be surprised if small changes had a measurable effect on unemployment for the following reasons:

    1) only a small percentage of workers are earning minimum wage and only a small percentage of the currently unemployed are seeking minimum wage jobs. This latter fact is critical as a higher minimum wage will change the distribution of unemployed toward the unskilled as higher skilled workers now find it easier to find a job. The net effect is to skew the employability toward higher skills.

    2) minimum wage does not directly translate into higher total employment costs. As profiled in the comments above, hourly wage is only a component of total compensation. There is also training, uniforms, breaks, benefits, discounts on purchases and so forth. A business will logically cut these to the detriment of the employee. Total compensation costs (and hence total employee rewards) therefore do not necessarily increase with minimum wage.

    3). Labor is a contract with two variables. The wage can change, but so too can expectations to get or keep the job. At a higher compensation cost (assuming no other compensation factors adjust as in the prior point) the employer can just change the demands placed upon the employee. Less absence or tardiness, more burgers flipped per hour, etc. In other words, the employer can respond by being more of a hard ass. The cost/benefit of higher job demands has changed again to the detriment of the employee and the employer.

    4). Minimum wage raises the price of an hour of employment, not the cost per employee. The logical short term response of a part-time employer is not to reduce workers. It is to reduce hours per worker. This still counts as employment, but does not lead to higher total wages. Note that the reason employers don’t have larger pools of part time workers is greatly because the workers are competing with each other for more hours. The increase in the minimum wage reduces this bargaining chip for employees. The net effect of a minimum wage hike would be fewer hours worked on average for a wider pool of part timers. You could also see relative shifts as better workers get more hours and lower skilled or problem employees get less.

    5). Employment costs are only one component of the production costs. As this component rises, there are other areas which an employer can adjust. For example, it may make sense now to reduce how often the building is painted, or how good of a landscaping service is used. Or it may make sense to increase the pace of roll out for the new automation which replaces burger flippers.

    6). If all the above fail, the employers can scrape by on lower margins. This reduces incentives for new employers, new branches, new competitors in the industry, but would take place over years. Possibly even a decade or more and would be completely opaque to any employment statistics.

    In summary, the effects of a higher minimum wage would be expected to be broad based and multi-faceted. The labor market would be clearly distorted, but only one small fraction and quite possibly none of this distortion would show up as less employment.

    For an economic study to focus only on the employment effects is perplexing to say the least. I would assume they have talked to businesses to see how they would actually react…no?*

    Yet for some reason the debate continues along this one dimension.

    *yes I am aware of the studies which track these other effects.

  16. Pingback: Weekly review: Week ending December 12, 2014 - sacha chua :: living an awesome life

  17. Esquire says:

    Not relevant to the argument, but: I think you are misinterpreting Aquinas. I always understood the quote to mean (and the Wikipedia page you link to backs me up) that somebody who has thoroughly mastered a single book is very formidable. (As opposed to superficial knowledge of many books.)

  18. I must admit I haven’t looked (this isn’t a subject that really bothers me greatly either way (even if that might be surprising to anyone who knows my political ideology; but it’s true)), but I wonder if there have been studies about minimum wage not affecting employment, but the wages of the ‘middle-class’.

    As in, the idea that people will not hire any less people, but they will take the money they’re paying to the jobs that they were previously paying less than minimum wage to from the people who were just above the minimum wage.

    Differently stated (numbers used have been pulled entirely out of thin air): If person A makes 1$/hour and person B makes 3$/hour and the minimum wage were instated/raised to 2$/hour, would person A and person B both make 2$/hour afterwards? Have there been studies on that? I imagine so, I just find myself curious, since it seems to be entirely absent from the narrative in the media that I’ve encountered, which keeps harping on the unemployment topic.

    (For the record, even if there were such an effect, I think arguments about such an effect can be made both ways.)

    • lupis42 says:

      I can’t give you studies directly on that, as I’m not aware of them.

      I can’t think of any economic theory that would lead to that though. The negotiating power of person B hasn’t changed, and their marginal value hasn’t fallen (unless they were just managing a group of As now being made redundant). There would be no reason to expect the market wage for their type of work to fall much, and if a company cut their wages anyway, it could expect them to up sticks for greener pastures.

      What I’d expect to see instead is companies that are structured around employing lots of person A type people struggling, and companies that are structured around employing lots of person B types taking advantage.

      For example, Costco pays something like $11/hr + benefits, employs people who have valuable skills (personable, self reliant, strong work-ethic, etc) and competes on customer experience and product quality wants the minimum wage to go up, while Wal-Mart pays something like $8/hr + all the out-of-work disrespect you can tolerate, employs people who couldn’t get jobs at Costco (introverted, physically handicapped, uneducated, criminal records, etc) and competes on price and accessibility (more locations open more hours) would suffer from the minimum wage.
      Unsurprisingly, Costco is a proponent of raising the minimum wage, because that makes Wal-Mart’s business model less efficient but doesn’t hurt Costco.

  19. Jared says:

    I think it’s interesting that after all of that you still manage to believe that liberals are probably right on this issue even though your reason is that “it’s probably publication bias”. If you were a conservative, would you find that convincing?

  20. tom says:

    Nice post.

    “But I worry that most smart people have not learned that a list of dozens of studies, several meta-analyses, hundreds of experts, and expert surveys showing almost all academics support your thesis – can still be bullshit.

    Which is too bad, because that’s exactly what people who want to bamboozle an educated audience are going to use.”

    You should read http://en.wikipedia.org/wiki/Propaganda:_The_Formation_of_Men%27s_Attitudes , among all the other insights it offers, the author rather thoroughly destroys the notion that education is of any use when it comes to resisting propaganda.

  21. Decius says:

    If I know that I haven’t done the research, should I decrease my confidence in “The minimum wage has a significant positive overall economic effect”, “… negative overall…”, or “has no significant overall…”?

  22. nr says:

    If you haven’t heard about p-curve, see:

    http://psycnet.apa.org/journals/xge/143/2/534/
    http://papers.ssrn.com/sol3/Papers.cfm?abstract_id=2377290

    When publication bias results from selective reporting of studies based on significance, funnel plot methods (trim and fill) aren’t adequate.

    • Anonymous says:

      Not everyone who uses funnel plots uses trim and fill. In particular, this post did not. Funnel plots subsume the p-curve.

      Your first link is broken. The correct link is here.

  23. Whateverfor says:

    I’m on the side of minimal effects of current US minimum wage, mainly due to my experience with sabermetrics in baseball. The rest of this post assumes basic baseball knowledge, but I think it’s a useful example of how some of this stuff works in practice.

    Probably the biggest gap between baseball intuition and the sabermetric consenus was DIPS (Defense Independent Pitching Stats). Voros McCracken had the bright idea to try and measure only the things pitchers had control over and the defense had no control over (Ks, BBs, and HRs), to help disentangle pitcher performance from fielder performance. He ended up discovering that this measure was not just more predictive of future performance than measuring the total runs allowed by a pitcher, but that the leftover result (BABIP, batting average on balls in play) was effectively random, and had no correlation on a year to year basis. Everyone who has ever played baseball knew that pitchers had control over how hard the batter hit the ball, so they generally totally rejected the findings despite the overwhelming empirical evidence.

    The solution to the paradox was mostly selection effects: first, pitchers that were not very good at preventing batters from hitting the ball well never make it to the majors. Second, when a Major League Baseball player gets a “bad” pitch in modern baseball, they tend to hit it out of the ballpark, so it’s no longer a ball in play. The current US minimum wage is similarly highly selected, given that it’s set below the prevailing wage for unskilled labor. In a world where nobody employed at Wal-Mart makes minimum wage, people who make the minimum wage are really weird. The exceptions are teenagers, and minimum wage increases affect teenage employment in the expected way. For people who are that weird, if it’s a result of antisocial behavior or total inability to follow directions, they aren’t worth employing for free (analagous to the HRs that end up out of the sample). So you end up with empirical results that contradict the general theory.

  24. Austin Parish says:

    Why are effect sizes in medicine often so small?

    Human body is massively complex system where any given change is overwhelmed by intricate, overlapping feedbacks?

    Different genetic/metabolic signatures between different humans would predict larger effect sizes for certain individuals – but studies don’t/can’t segregate people into these groups, so what would be a large effect size in a specific set of 100 people becomes a tiny effect in a randomly selected set of 100 people?

    Why, why, why?

    • theLaplaceDemon says:

      Could also be that it’s hard to find compounds that have large desired effects but not large undesired (side)effects.

      /not an expert

    • Scott Alexander says:

      How would you fix this?

      Like, suppose I know that of every 100 people with depression, 50 get better by chance/placebo per year.

      I get 100 people and I give them Paxil. 60 get better. I do my tests and find it’s statistically significant with a small effect size.

      Now what do I do?

      If we knew for sure what dimension responders vs. non-responders varied upon, that’d be one thing. Until then, as best I can tell all we can do is say “100% of all responders respond!”

      Chasing down the characteristics of responders vs. nonresponders is almost never a good use of your time. If you’re doing genetics, you’ll get a zillion false hits and no true ones. If you’re doing metabolomics, you’ll get a zillion things that make a tiny amount of difference and nothing useful.

  25. Don’t forget to distinguish employment levels from unemployment levels – those two measures can vary independently. Expect people to select whichever measure produces evidence they like.

  26. gwern says:

    This is more of a needle curve than a bell curve, but the point still stands. We see it’s centered around 0, which means there’s some evidence that’s the real signal among all this noise. The bell skews more to left than to the right, which means more studies have found negative effects of the minimum wage than positive effects of the minimum wage. But since the bell curve is asymmetrical, we intepret that as probably publication bias. So all in all, I think there’s at least some evidence that the liberals are right on this one.

    Besides the publication bias, another interesting thing this funnel plot shows is systematic vs random error, or to use a more LWy term, ‘confidence levels inside and outside an argument’: does anyone actually believe that the real causal effect of minimum wage might be an elasticity of -20? Or that it’s remotely correct to note that 3 studies or so have a precision of ~1/300 and interpret this as giving a correct measurement of the genuine correlation (much less causation)? Past 1/50 or so in precision and 1 in elasticity, I suspect random error is going to be negligible and all the results driven by systematic biases & errors in the underlying data & analyses.

    (I suspect a ‘credibility ceiling’ approach, as ad hoc as I find it, would produce better results; for an explanation of the idea, which is basically just specifying some estimates for systematic error and capping influence of individual studies at the point where they’re primarily systematic error, see “Synthesis of observational studies should consider credibility ceilings”, Salanti & Ioannidis 2009 and an example application, “Application of credibility ceilings probes the robustness of meta-analyses of biomarkers and cancer risks”, Papatheodorou et al 2014; eyeballing it, I would guess that reasonable credibility ceilings would drain the influence of the >1/50 studies and produce a point-estimate somewhere 0 to -1 – which is not a definitive answer since a small shift like that is ascribable to publication bias and other highly plausible issues (seriously, -20‽) but at least is a sensible answer.)

  27. Coscott says:

    I believe you made a sign error when you said that leftists would say:

    “People used to believe that the minimum wage decreased unemployment.”

    You probably wanted to say:

    “People used to believe that the minimum wage decreased employment.”

  28. Patri Friedman says:

    When the empirical evidence is so mixed, one can sometimes turn to theory. Minimum wage is about as close as you can get in economics to a simple question answered by the most basic microeconomic models which while not always right in more complex settings are almost certainly right here. Namely, if you increase the price of something (low end labor), people will buy less of it. There are rare exceptions (positional goods) but unskilled labor does not at all resemble them.

    So there is a very strong reason to expect a very small but very definitely negative effect on employment of the least skilled workers. Generally these are young, poor, and uneducated, exactly those most vulnerable. This is a short-term effect and you can argue there are beneficial long run effects, which is not ruled out by this model, although the evidence is for negative long run effects too

    Given the effect should be tiny, the studies should cluster around zero. The larger the hike and the more the study focuses on the tiny population of people who work for minimum wage,the more accurate it should be.

    • Anonymous says:

      Why does theory say that the effect should be tiny? Don’t the authors of these dozens of studies disagree with that expectation?

      • Glen Raphael says:

        The effect should be small and hard to tease out if the change in the wage is small and gradual, as it usually is. The main US counterexample I can think of is in 2007 when we tried to ramp up wages in American Samoa and the Mariana Islands to US levels. (That minimum wage change killed entire industries and had to be put on hold.) But usually each individual minwage change is small on a percentage basis and phased in gradually, so its effect is small too.

    • Patri,

      The controversy bears a certain resemblance to the marginalist controversy of the early-middle of the previous century. See https://www.princeton.edu/~tleonard/papers/minimum_wage.pdf. What can falsify an economic theory? Note the James Buchanan quote on page 21. In the case of the marginalists they were right to be skeptical of the prima facie interpretation of the survey evidence that firms are not profit-maximizers. Frank Knight taught of the inverse relationship between choice and competition, and in the end the marginalists were probably right to dismiss the opinions of the business owners as mostly irrelevant. Milton Friedman and Paul Samuelson found some common ground here.

      Let me push back against the idea that we must turn from evidence to theory. Perhaps the discussion is too parochial. The economic theory of the effects of price floors has far more evidence for it than specifically studies on the effects of the minimum wage. Given that, how mixed is the evidence really? It must be asked why we would think labor might be an exception, and if it is an exception, surely there must be others. Presumably welfare-improving opportunities for price floors must exist in many industries if we are to take this argument very seriously.

      The debate around the economic effects of the minimum wage is like cutting at a single scale on Leviathan’s back. Whether it is in fact loose enough that it may be torn off the main body is a question, but the reason for doubting it has to do with the strength of the great sea monster’s entire corpus. It would take a very sharp knife indeed to injure it, as we have seen time and time again. So we do not turn from evidence to theory at all but instead ask how strong this new evidence really is, that it should demand the attention of the great sea beast.

      Reasons for skepticism that do not seem sufficiently discussed to me are the questions of how predictable and periodic these minimum wage increases are, and what role the minimum wage itself might play in promoting monopsony. Part of the value of this sort of research is that it has tended to incline economists away from treating economics as the study of choice and instead as the study of the optimizing pressures that scarcity imposes on some given system. The resulting insights are some of the reasons Knight, Alchian, and Becker are so highly regarded, to name just a few. Yet that the minimum wage should make its own demands on the industry seems to be an underrated insight. Market forces will tend to minimize the negative impact of the minimum wage over time (internships are an obvious example, as are selection effects producing monopsonistic firms), and so the clearest study of “the” effects of the minimum wage would be a surprising and unprecedented minimum wage the industry is not prepared for. Then you might see some unemployment, I suspect. It will probably be difficult to find such a thing in a democratic society where policy is made slowly. I do not know much about the impositions or effects of minimum wages in more dictatorial countries, but I would be more persuaded if minimum wages in such countries also seemed to have little to no unemployment effect.

      While I am skeptical of the conclusions both the anti-marginalists and the skeptics of the unemployment effects of the minimum wage advocates wish to draw, it is unfortunate that no one seems to want to discuss the importance of the effort that goes in to these studies and the quality attitude they require. The anti-marginalists or those who became them had the brilliant idea of calling up businesses and asking them how they set prices. Card and Krueger’s criticisms of econometric malpractice have been largely ignored relative to the perhaps less interesting question of the effects of the minimum wage, although this may be because no one has any very major disagreements. I would say the same of real business cycle theory–it reflects an important effort and a sound attitude, regardless of the theory’s ultimate veracity.

      I do also see value in these sorts of efforts in that they probably do function to dispel the impressions economists form that may not be quite implied by their theories but nevertheless probably tend to manifest themselves in the human mind. That is, economists are forced to adjust to these data, and usually for the better. Economists learned from the marginalist controversy, and they will learn from the minimum wage controversy as well. These things have significant value, but people do tend to get a bit too excited about them.

      • 27chaos says:

        “Part of the value of this sort of research is that it has tended to incline economists away from treating economics as the study of choice and instead as the study of the optimizing pressures that scarcity imposes on some given system.”

        Thanks for the new crystalized concept!

  29. eqdw says:

    Something I’ve found to be an effective way of learning about these kinds of econ/policy questions is by stalking smart people on Reddit. There are a handful of people that commentate on reddit.com/r/badeconomics, as well as holding good economics discussion in the r/economics sub (one of them is a moderator). They routinely post papers, technical blog discussions, etc. Several of them are economics professors in universities (though, they like to keep their anonymity and won’t divulge where).

    On the question of minimum wage specifically, here is my (layman’s) attempt at summary of their consensus on the minimum wage discussion (source: discussion at reddit.com/r/economics and reddit.com/r/badeconomics)

    * The labour market has unique frictions that make a naieve supply and demand model inapplicable.

    * Specifically, the minimum wage labour market appears to function somewhat as a monopsony, with one buyer (the employer) buying many employees. This seems to largely stem from mobility issues amongst the workers; They are too poor to move somewhere else, commute farther for work, etc., and so their employers can exercise wage suppression power on them.

    * Minimum wage may be an acceptable way of addressing this problem, but it’s known to be a rather weak solution. In terms of an anti-poverty measure, minimum wage only helps the subset of the poor who can pull it together enough to pass a job interview; this largely locks out the worst off folks.

    * Especially given that the monopsony power of minimum wage employers comes from lack of options and lack of mobility, more effective long term solutions would involve things that break this hold directly. Increased access to inexpensive transportation, increased ability to move in search of new work, additional options for minimum wage employment. Either in-kind provision or subsudies, both would be effective in the long term.

    * In the short term, moderate minimum wage increases (+ pegged to inflation) are probably a good partial solution.

    * Small to moderate minimum wage increases likely do not have significant effects on employment or inflation. Large minimum wage increases likely do, and they will likely cause other disruptive shocks.

    * Minimum wage increases preferentially shift employment from small companies to large companies in the long term; McDonalds is more able to absorb a cost increase than the mom&pop down the street. This may or may not be a bad thing for overall welfare.

    • Samuel Skinner says:

      * Specifically, the minimum wage labour market appears to function somewhat as a monopsony, with one buyer (the employer) buying many employees. This seems to largely stem from mobility issues amongst the workers; They are too poor to move somewhere else, commute farther for work, etc., and so their employers can exercise wage suppression power on them. ”

      Monopsony? Really? So real wages don’t increase in the absence of the minimum wage?

      I guess that the minimum wage didn’t have to be continually raised because real wages were increasing then and that real wages were flat prior to the introduction of the minimum wage… oh wait, neither of those are true.

      Wages are only monopsony in company towns and isolated small towns (and for the latter only in the short run).

    • Salem says:

      Specifically, the minimum wage labour market appears to function somewhat as a monopsony

      This is absolutely false, and typical of the whack-a-mole that needs to be played. There are huge numbers of firms hiring at the minimum wage, and minimum wage employees are the most likely of any employees to quit their job to take a new one. There is a complete absence of data to suggest monopsony, and a huge amount of data to suggest a very competitive marketplace. Above, we even have a (pro minimum wage) commenter saying that of course no-one is so foolish as to argue for a monopsony model to justify the minimum wage… yet here we are.

    • 1. each reddit has a particular majority point of view (by self-association / upvote-seeking). it would be a mistake to characterize the consensus of a reddit (in favor of minimum wage increase) as the consensus of an academic field (which I assume is closer to the opposite)

      2. fakers on reddit are pretty common – wouldn’t trust “I’m an anonymous professor”

  30. onyomi says:

    This also leads me to a related conclusion: progressives and liberals love empirical data with counter-intuitive conclusions, because it gives them a chance to relive the smug superiority they feel with respect to intelligent design in the same way they love to discover a new oppressed group (or a new way of conceiving of oppression, such as “micro-aggression”), as it allows them to relive the glories of the civil rights era.

    Therefore, if you can say, “it may seem like minimum wage would increase unemployment, but actually many scientific studies have shown this to be false,” or even, “it may seem like minimum wage would increase unemployment, but in certain cases it’s been empirically shown to LOWER unemployment” (I have seen this claim, btw), then that is great on TWO counts, because, not only do you get empirical support for something you wanted to do anyway, but you also get to look down on all the neanderthals who refuse to accept the scientific consensus just because of their dumb intuitions about “common sense.”

    Hope the above didn’t come off as too snarky; I am just so sick of my Facebook feed being full of people proudly, “bravely” coming out against: hating gay people, raping, not vaccinating your children, intelligent design, and, of course, “stupid conservatives who just don’t understand the economy,” even though I’m pretty sure virtually nobody on my friend list, or even on most of my friends’ friend lists actually believes in intelligent design, hating gay people, raping, anti-vax, etc. and even though I’m sure 99% of the people complaining about the stupidity of conservatives, or, if they’re more sophisticated, “neo-liberal economics,” are not as smart as Milton Friedman, to put it charitably.

    • Hainish says:

      I am just so sick of my Facebook feed being full of people proudly, “bravely” coming out against: hating gay people, raping, not vaccinating your children, intelligent design, and, of course, “stupid conservatives who just don’t understand the economy,”

      It sounds like you are complaining about people using facebook for its intended function.

      • onyomi says:

        So the intended function is to provide a platform for people to signal to their friends how enlightened they are by attacking boo lights and coming out in favor of applause lights? I’m not saying you’re wrong, but… how depressing?

        In the interest of fairness, I should also note one of my own tribe’s biases–a bias I tend to share–which is to be contrarian:

        http://bleedingheartlibertarians.com/2014/10/the-contrarian-trap-the-source-of-the-liberty-movements-dark-side/

        We are much more likely than average, for example, to be into Crossfit or the Paleo diet, which are sort of contrarian views on fitness and nutrition, respectively.

        • Anonymous says:

          “So the intended function is to provide a platform for people to signal to their friends how enlightened they are by attacking boo lights and coming out in favor of applause lights?”

          Almost, or rather, too much:

          “So the intended function is to provide a platform for people to signal [things, mostly positive] to their friends”
          Certain people do it the way you mention, some do it by talking about their personal achievements, others like to talk about how cute their grandchildren/pets are.

          • onyomi says:

            I think you succeeded in depressing me further by making me realize that what I thought I hated about Facebook is actually something I hate about people in general: not that they like to signal positive things about themselves, but that it’s incredibly difficult to have a dispassionate discussion about controversial or consequential topics while maintaining amicability and without shifting into “arguments are soldiers” mode.

            In this respect Facebook is like a giant party: most of what is said is innocuous and pleasant; sometimes somebody brings up politics, religion, or philosophy and either is met with pats on the back because everyone agrees, or else tempers start to flare and someone adroitly changes the subject.

            What very, very rarely happens (in my experience, at least–in real life, and on Facebook), is that someone expresses a dissenting opinion and then people thoroughly and deeply explore various arguments without anyone’s feelings getting hurt.

            What usually seems to happen is either people get really pissed off, or, much more likely, just kind of clam up, change the subject, say, “oh, politics, you know…” etc. which is always the point when I was getting really interested.

            Then I can go back to looking at pictures of people’s grandkids and hearing about how anti-rape they are.

          • Hainish says:

            but that it’s incredibly difficult to have a dispassionate discussion about controversial or consequential topics while maintaining amicability

            That is is, but . . . with regards to facebook, most of the “friends” who I disagree with about anything substantial are not capable of any sort of discussion that involves critical thought . . . the best I can do is expose them to information they would not have come across on their own.

  31. LTP says:

    You know, it’s issues like these that have made me default to just ignoring most empirical social science work (as I don’t have the time nor inclination to investigate every claim I see), and being much more interested in theory and a priori reasoning that makes sense to me on these issues, even if in principle empirical evidence should be better.

  32. Tom Scharf says:

    A couple notes:

    1. Studies that attempt to tell you what will come (predictive) are in a different category than those that try to unscramble the eggs of what has occurred. Predictive studies are very dependent on their assumptions.

    2. Some things are simply not knowable or close to it. As above many studies attempt to root out causation where many confounding factors exist (unscrambling the eggs). Invariably they use models with assumptions and “truth” pops out of the math-e-magic. Many studies falling on both sides may simply mean nobody really knows.

    Some of these studies are likely best efforts and possibly without any bias. And they can still be pretty much useless in finding fact. The best and brightest trying their hardest doesn’t equate to useful outcomes.

    You will find activists to be the most prone to “single study syndrome”, invariably only quoting the most extreme outcomes that support their side. This seems to be counter-productive as it invariably hurts their credibility in the long term.

    This is all fine and business as usual, but what I find distressing is where there is a somewhat corrupt sector of science that feeds activists what they want to hear and attaches an aura of authority to it. The less ugly cousin is confirmation bias where outcomes are heavily dependent on data processing methods and the “right” answer can be found by tweaking methods and models.

  33. Of course, this leaves out any mention that the minimum wage is totally immoral, because it’s a form of coercion. If you believe it’s wrong to point a gun at someone who doesn’t obey your ideas of a utopia, then you have to disbelieve that the minimum wage is a good idea. Also, you’d need to analyze so many effects along such a long time curve that absolutely no study on the minimum wage, pro or con should be trusted, even slightly. Noone can see the perverse incentives associated with the tendency of more coercion to prevent innovation, and the opening of new marketplaces.

    This could be a result of the minimum wage, or it could be the result of the coercion and uncertainty of the political environment that allowed the minimum wage (who’s to say it won’t get hiked again, and again? this eliminates the ability to reliably plan, and that ability is assessed differently by every different business. What if your business or planning ability is robust and fault tolerant? Then you keep your business open, because “it can take it” no matter what the minimum wage is. So, do the studies really examine every single business, and determine 1–what the IQ of the business owner is 2–what the cash reserves at every stage of the business’s life have been, and currently are 3–how fault tolerant the business itself is 4–what the competition for the business is like 5–what other options there are for the business owners if they close 6–what surrounding areas are there, that a business could move to, if they viewed the minimum wage as unacceptable 7–would the business owner have invested the excess money now spent on wages on capital improvements to the business that would have employed more people, or allowed him to expand? This last question is totally unknowable, but one can assume the business owner would have spent the money in the way he saw was optimal. So, by forcing him to pay that money on minimum wage, you’ve threatened him into behaving in a way he previously had indicated he wouldn’t act, without the coercion.)

    The only thing we know is that the minimum wage requires coercion, and the market does not. That most businesses pay more than minimum wage indicates precisely what the market would pay. If people want to pay less, then inherently one can assume that, if they would pay much less, while wanting more work done, then there would be some people who go jobless because the employer cannot afford to pay.

    What if it’s only 1 person? Even so, if one person has their rights violated in one area and cannot employ as many low-skilled workers as he’d like, he may move to another area. Even if he doesn’t he still had the displeasure of having the state tell him how to run his business implying that his judgment wasn’t accurate.

    And what if it was? What if the min-wage person was happy enough to work for the min wage, because he didn’t need to pay for housing and wanted the easier job? Now that his employer is forced to have fewer workers, he may want a better quality worker, and fire or “lay off” the less hard-working. After all, the state can’t force him to remain as an employer.

    The fact that the state can’t force an employer to remain in business (that’s known as “slavery”), means that the state shouldn’t impose a minimum wage. What right do they have to tell anyone how to spend money that they didn’t even have to make? What right do they have to bully people who don’t want to be bullied?

    • suntzuanime says:

      It’s generally not useful to make comments assuming a specific morality that the vast majority of your interlocutors are not going to share.

      • Anonymous says:

        “useful”? There you go again, assuming consequentialism, when Jack is clearly a deontologist.

        • Susebron says:

          Jack is also arguing. The point of argument is generally to convince one’s opponents. Just because someone isn’t a consequentialist doesn’t mean that they never have goals. Sure, it’s possible that Jack is arguing for no reason, but unless you’re Jack and accidentally posted anonymously, I don’t think there’s any reason to believe that.

    • Scott Alexander says:

      Everyone seems to have missed this, but I’ll bother you specifically about it.

      This doesn’t “leave out” anything. I don’t care about the minimum wage. I was using it as an example of how there can be multiple studies on both sides of an issue. Whether or not it’s immoral doesn’t matter in that context.

    • Jared says:

      You’re right that the state uses coercion. The problem is that the same can be said about property rights.

  34. Zubon says:

    In the particular example shown, “the bell curve skews slightly left” is the consensus expert opinion: the minimum wage likely has small negative employment effects. If you’re on the right, this proves that the minimum wage increases unemployment, done. If you’re on the left, it proves that the policy has small costs relative to benefits, done. Obviously neither side can/will recognize offsetting arguments in the sound bite/headline version of the argument.

    (I am personally concerned that the harms of unemployment are understated by looking at “wages lost” instead of horrible psychological effects.)

    We also know from one study (woo! I can find a link when I’m at a real computer) that US labor economists skew left. Would you interpret that to mean that the research somewhat understates the harm of the minimum wage (policy preferences drive research results) or that research like this is good and pushes you leftward as you study labor economics (research results drive policy preferences)? (Feel free to shout “false dichotomy” here.)

  35. gattsuru says:

    What’s really fun is there’s not even fundamental agreement on terms. You can get together a lot of studies on “unemployment”, “wage”, and “negative impact”, none of which use the same terms to mean the same thing, and many of which will use the terms in a way no sane person would. Unemployment can reasonably mean workforce participation rate, or people who are looking for work and not currently working a certain number of hours a week, or people who have been looking for work for a certain period of time, or a count of individuals who are not employed for N hours a week, all of which are different things and look toward entirely different trends. Wage can range from the actual per-hour wages to including any of a wide variety of non-monetary benefits, may or may not be indexed for cost of living, and may or may not be the actually paid rate as opposed to the declared rate.

    That’s before we get to the unreasonable definitions, which are a dime a dozen, or the question of which well-recognized standard of measurement we use.

    And this is a field with well-recognized special terms, and government agency after government agency giving the exactly definitions and numbers for many of these things.

  36. onyomi says:

    Economics presents the problem of being much more politically charged than medicine, but the advantage of being more amenable to a priori reasoning (here revealing my Austrian tendencies).

    That is, there’s no logical, a priori reason to expect any substance to have any particular effect. We might say that “given that amphetamine is a stimulant, we wouldn’t expect methamphetamine to be a depressant,” but that still depends on empirical observation of the effects of amphetamine.

    Compared to this, you can say, logically, regardless of what data points show, that we would expect people to buy less of a thing the more it costs. This doesn’t require any particular observation, other than those involved in being a person living in a world of scarcity. We know it by simple reflection and have every reason to believe others act similarly.

    Of course, we can and should test the a priori theories against real life to some extent. There may be some weird case of a luxury good where people buy something expensive to signal something about their status, and so the high price itself becomes a “good” of a kind. We may also be able to figure out whether there is a good reason either to rethink our a priori conclusions, or else to believe that there are some empirical effects strongly ameliorating them–like an inelastic demand curve, say (and this is, tbh, the only reasonable way to argue for minimum wage, imo–not to say that it doesn’t cause unemployment, but that the degree to which it does cause unemployment is relatively small compared to the benefits).

    But we also have to be super careful about inferring causality, of course, since there are always so many potential confounding factors–say unemployment was about to go way down due to other factors, and then we raise the minimum wage and it still goes down, only less. One can start to intuit a general empirical trend if there is enough data available for say, many different increases of the minimum wage at different times and places, but no one thing can ever be isolated (though in this respect, it is kind of like medicine, where you’ve got so much going on inside any given body).

    Anyway, since basic logic tells us that raising the minimum wage should increase unemployment (though not by how much), I think one should err on the side of assuming it does, unless very strong empirical evidence can point you in the direction of why this would not be so.

    This is different from drug studies because the human mind cannot come to rational, a priori conclusions about the effect of some chemical substance on the human body. It can, however, come to conclusions about its own decision making and that of other minds, which is what economics is all about. These conclusions may be far from perfect due to “no typical mind” and the huge number of other factors at work, but they are still possible.

    I guess what I’m really getting at is that we shouldn’t start a debate about the effect of something like minimum wage from a stance of total agnosticism and only then make up our minds based on pure data, given that we have logical reasons to believe one thing rather than another. Empirical data could so strongly favor a conclusion opposite to our logical intuition that we are forced to reexamine the latter for bias or failure to take into account some factor, but we should start with the assumption that things are as they seem.

    Given that the empirical studies seems fairly inconclusive, I’m going to go with the seemingly obvious conclusion: minimum wage does cause unemployment. A little increase in the minimum wage increases unemployment a little, and a big increase in the minimum wage increases unemployment a lot.

  37. Drew Hardies says:

    The ‘minimum wage and unemployment’ argument is part of a really interesting meta-debate.

    The key thing to watch is how the conservatives are able to massively out-maneuver the progressives by exploiting two effects. First, they’re focusing on a narrow technical question that they know will be misunderstood. And second, they’re exploiting progressives’s treatment of equality as a sacred value.

    Both of these tricks are encapsulated in the AEA survey.

    “Over 73 percent of AEA labor economists believe that a significant increase will lead to employment losses and 68 percent think these employment losses fall disproportionately on the least skilled.”

    To see why these problems are misleading, consider a scenario where we could increase wages by $5/hour for 100,000 people at the cost of 2 jobs.

    Technically, the first question is asking if a LARGE increase wage will have ANY effect on employment. That invites the economists to answer yes. Even if they think the effects are as one-sided as in the hypothetical.

    The gimmick is that economists say ‘AN effect’ and the public hears ‘A LARGE, IMPORTANT effect’. And so they assume that the question is somehow relevant to policy making.

    Progressives tend to get lured into an ambush. It feels rhetorically important to refute the conservative subtext. But the literal economic claim “big price increases tend to reduce demand” is super-easy for conservatives to defend.

    The progressives should be willing to concede that policies aren’t one-sided. The wage increase will help the overwhelming majority of people. If the conservatives want to argue for a LARGE effect, they should do it — but the research doesn’t really bear that out.

    But that’s where the second question: “68% think these employment losses fall disproportionately on the least skilled” comes in. The factual question is trivial — minimum wage changes will mostly impact people making the minimum wage.

    I think it’s asked to invoke ‘equality’ as a sacred value. That $5/hour increase for 100,000 people at the cost of 2 jobs would be massively beneficial. But technically, it would increase inequality in that group.

    The result is that people are primed against acknowledging possible tradeoffs. So, instead of the policy important debate of “will unemployment increase enough that we should care” most people get stuck at “will unemployment increase at all”?

    • Alex Godofsky says:

      Conservative economists do not actually believe “haha, the minimum wage actually will have utterly insignificant negative effects in the aggregate, but we have this cute rhetorical trick to make it sound like they will”. They genuinely believe that the policy will be a net harm to the class of people it is intended to help, and a net harm overall. That is to say, they believe it is an instrumentally bad policy even under their opponents’ value system.

      • suntzuanime says:

        Yes, but their opponents don’t, and yet their cute rhetorical trick makes it sound like they do. This model permits the conservatives to be perfectly sincere in their beliefs; they’re just dishonest in their argumentation. They’re trying to make their honestly-held position seem like more of a consensus than it is by using leading questions that obscure the actual controversy.

        • Jos says:

          See below. I think most conservatives in that position honestly believe that they are responding to someone who is arguing for the proposition that there is either no loss or almost no loss of employment from minimum wage increases.

          This was my perception of unemployment extensions. There were a lot of people in 09 on the blogs arguing that under economic conditions at that time, extending unemployment from 13 to 99 months would have literally no effect on unemployment, because there weren’t jobs to be had, and conservatives saying “what are you talking about?” Eventually, it turned out that everyone who knew anything (about 25% of the debaters basically agreed that under those conditions, the best but imperfect estimate of the effect would be that unemployment would increase by about 0.5%, and at that point, we could talk about whether helping the other several percent made that worth it.

        • Caspian says:

          The dishonest argument is a meme that thrives in an environment of people arguing both for and against it, with the people arguing against it not understanding the trick. But the people arguing for the argument don’t have to understand the trick either. It could even be “better” if they don’t, so they can’t leak the understanding to their opponents.

      • Drew Hardies says:

        The economists were answering the question they were asked. I have no objection to that.

        What’s interesting is how partisans can ask experts “is the effect non-zero?” knowing that the public will hear “is the effect big enough to change policy?”

        Bonus points if the other side can get invested in “effect is definitely zero!”

        I’ve seen similar games played with questions like: Does public benefits fraud exist? Do vaccines risk permanent brain damage? Are there are natural causes of global warming?

        This pattern is interesting to me because it happens so frequently. Conservatives have a ton of true-but-trivial statements they can throw out. People rush towards the trap (“vaccines have no risks!”), and lose what should have been an easily winnable argument (“on balance, vaccines are overwhelmingly worthwhile.”).

        Thinking about it, this dynamic is the most common reason I see progressives lose otherwise-winnable arguments.

        • Anonymous says:

          What do you mean by “win” and “lose”? It sounds like you are using “lose” to mean make statements that you judge false. But that sounds like a lousy goal to me.

        • aesthete says:

          “Conservatives have a ton of true-but-trivial statements they can throw out.”

          The example statement you’ve provided does not seem trivial if large numbers of people are under the impression that the statement is false or debatable, to the point of “taking the rhetorical bait” (so to speak).

          You are right: it is important to evaluate the trade-off as a whole RE: benefits accrued to those who have their wages increase as opposed to those who see themselves without work. So long as many progressives do not acknowledge the latter drawback to a minimum wage increase, how can stating the fact that this drawback exists be “trivial” when the valid public policy question (as you yourself acknowledge) is something like, “are these disadvantages large and if so, do they outweigh the benefits of a wage increase for remaining workers?”

          To the degree that it is a “rhetorical trick”, it is because the discussion is not even close to arriving at what is already known and deduced in academic circles regarding the subject.

        • Cauê says:

          >’What’s interesting is how partisans can ask experts “is the effect non-zero?” knowing that the public will hear “is the effect big enough to change policy?”’

          I think this gives the public too much credit, by assuming people would react any differently to “non-zero” vs. “large enough”.

    • Jos says:

      Let me challenge your perception a bit.

      My perception is that when conservative economists argue there will be a reduction in employment as a result of minimum wage increases, they are responding directly people who have stated the position that there will in fact be no change in employment, or almost none. (And to be fair, Scott’s funnel graph suggests that’s a defensible position, althouth I think it’s intuitively not).

      My perception is that progressives often argue that their preferred proposal (high speed trains from point a to point b; Obamacare; minimum wage increase) has at least some benefit, but ignore the cost. Your perception is that conservatives often argue that the program has some cost and ignore the benefit — it might be that both sides are doing a cost benefit agreement but arguing with their mistaken perception of the other side’s position.

      • Drew Hardies says:

        I’m not critiquing economists, conservative or otherwise. They answered the question asked. They’re good at doing cost/benefit discussions.

        I’m saying that conservative policy advocates (who are generally not economists) use a clever rhetorical trick to bait otherwise intelligent progressives into taking dumb positions.

        Then I’m exploring the trick.

        I think I agree with your subtext that it’s not a particularly unfair trick. There totally are minimum-wage advocates who’ve invested themselves in there being absolutely NO effect. And it’s fair for conservatives to respond to that.

        I’m calling out the trick because I’d like people to stop falling for it.

        That said, it sounds like you lean conservative, so I have a question.

        Can you think of any examples where liberals use this sort of provocative technicality to out-maneuver conservatives?

        All of the examples that come to mind are cleverness by the ‘other’ tribe. That seems like a blindspot

        • lupis42 says:

          I generally feel like discussions of ‘needing’ X in gun control debates frequently take that general form, i.e. “Nobody needs (feature) for (Schelling point use case)” is deliberately used to try to get the argument onto what is and isn’t needed for what, and get the implicit concession that anything not ‘needed’ should be forbidden. It has, however, gotten less effective in recent years.

    • Tarrou says:

      Ahh yes, the cunning mastery of those devious conservatives of all academia, with their noted conservative bias!

  38. Liskantope says:

    Interesting that the main competing propositions being presented here are “negative effect” versus “no effect” on the economy. Maybe this has something to do with the fact that I’m mostly surrounded by blue-tribe people, but I often hear assertions that an increase in the minimum wage would positively affect the economy. The idea seems to be that not only would it not significantly increase unemployment, but it would directly benefit the working class, who would then spend more. I think I’ve even heard Obama make vague assertions about a rise in minimum wage strengthening the economy from the middle class outwards. Unfortunately, I don’t have a good enough understanding of macroeconomics to really evaluate the plausibility of this hypothesis.

  39. Anonymous says:

    Your second meta-analysis link is broken. Here is the correct link.

  40. Jos says:

    Minimum wage is a fascinating epistomological problem.

    On the one hand, it seems intuitively obvious to me that there should be some reduction in employment over the medium and longer term. First, there should be some jobs where it’s now cheaper to substitute other means of delivering the service. Second, and probably more importantly, you would expect an increase in labor costs to result in an increase in product cost, which should encourage consumers to substitute where they can.

    Now these effects may be pretty small if minimum wage labor is a small component of the overall costs, but I wouldn’t expect them to be zero.

    On the other hand, that’s a pretty convincing funnel plot (seriously). I’m honestly not sure what to do next.

    • Will says:

      I never understand why everyone gets this wrong- in econ 101 world, an increased production cost should not be associated with a price increase- the price that maximizes profit is independent of the production cost. Changing the price to cope with an increase in production cost will just hurt you twice.

      The employment effect happens because marginally profitable businesses are no longer profitable, and they close down. This result in fewer jobs. In an econ 101 world, where most minimum wage labor demand is highly profitable national chains we’d expect a very small effect.

      • Alex Godofsky says:

        I never understand why everyone gets this wrong- in econ 101 world, an increased production cost should not be associated with a price increase- the price that maximizes profit is independent of the production cost.

        … false?

        Profit = Quantity * (Price – Cost)

        In the econ 101 model with a downward sloping demand curve, the profit-maximizing firm will always increase Price in response to an increase in Cost, although not 1-for-1.

        And if you have perfect competition, then price = marginal cost, and so actually will increase 1-for-1.

      • 27chaos says:

        There’s a school of thought that says businesses are inefficient and need to be kicked into making improvements. During the recession, companies trimmed the fat and laid off workers, yet profits are record high anyway.

        • Samuel Skinner says:

          The government did provide financial incentives to invest in productivity increasing machinery.

          Although that probably isn’t the reason. What firms were making record profits and which ones where shedding workers?

    • anodognosic says:

      One hypothesis from Macroeconomics 101 is that increases in aggregate demand offset the increase in unemployment.

    • Tarrou says:

      Well, as someone old enough to have been working a shop job when the minimum wage was raised in the ’90s, the interesting thing to me was how mad it made people making just above the new minimum wage. I was the new kid, I’d been making $2-4 less per hour than the guys with more seniority. When the hike hit, we all were basically making the same. Those guys didn’t get a raise, and in effect, I took their raises for the next several years. Unamused were they.

      • Tom Womack says:

        That’s interesting, since one of the common anti-minimum-wage arguments appears to be ‘if we increase the wage for unskilled minions of the third and lower kinds, we will have to increase the wage for everyone else proportionally and that will be a much greater cost than just the minions’.

        I don’t have an opinion one way or another about wage compression, but I’ve had the good luck of a career in industries where wages are not so significant a cost that increasing them 3% annually is an issue – effectively support groups for really expensive bits of capital equipment, where the cost for the crystallography-software group or the CPU-design group is minimal compared to the cost of electricity and consumables for the synchrotron or the chip fab.

  41. Patrick says:

    FYI, the reason minimum wage discussions suck is, IMO, almost certainly because minimum wage changes have low to no short term negatives due to wage and employment stickiness effects. But conservative economists are lunatics who regularly deny that these exist, or else insist that these cannot possibly have meaningful effects. Acknowledging that they do would undermine conservative theory on unemployment.

    So they can’t explain their opponents error without committing heresy against conservative economic doctrine.

    • Jos says:

      I’d be interested to read a conservative economist who thinks that minimum wage would necessarily have a short term effect on employment levels – my guess is that most think that medium and long term effects are inevitable.

    • Alex Godofsky says:

      But conservative economists are lunatics who regularly deny that these exist

      I am having difficulty understanding how you could come by this belief.

      • Anonymous says:

        What other explanation is there for the huge publication bias?

        • 27chaos says:

          I imagine they generally set those issues aside because it’s so arbitrary how sticky you make things. It’s not really something that can be disproved or addressed outside of showing how other simpler models will explain the same thing.

          That said, I’m sure there are papers confronting it directly.

      • Patrick says:

        Because stickiness effects explain why wages do not adjust to economic downturns the way certain economic models predict. Because this leads to a chain of inferences that explain involuntary unemployment and justify stimulus bills. Because if you pay attention you can watch them do what I described.

        • Jos says:

          If you can, please send me some links to someone who denies short term stickiness in minimum wage effects, as opposed to arguing that there are medium or long term effects.

        • lupis42 says:

          Edit: sorry, that was nasty which I shouldn’t be.

        • Alex Godofsky says:

          You’ve made the error of identifying “conservatives” with “a particular school of macroeconomists, many of whom actually do accept nominal and/or real wage and/or price stickiness as a central feature of their models, but who tend to be misrepresented or weakmanned by Paul Krugman”.

          If you actually peruse the conservative econ blogosphere, for example, you will see widespread (near-universal?) use of stickiness as a prominent feature of the models where it is relevant.

      • Lambert says:

        I think one of the main points of this blog is that that is a bad argument.

    • Tarrou says:

      Perhaps you’d care to cite one of these “lunatics”?

  42. jjj says:

    This is why I put an unusual amount of weight on theoretical arguments that are simple enough for me to understand. The econ 101 argument against the minimum wage is intuitive to me. I can verify it against my personal experience: I’ve never bought anything without considering the price first. On the other hand, it’s hard for me to check whether XYZ study on the minimum wage was biased in some sophisticated way I can’t eyeball.

    • I follow the same heuristic. It seems to me to be correct: in the presence of conflicting evidence, or evidence which can be massaged or interpreted to support multiple contradictory conclusions, favor the simplest not-obviously-incorrect theory.

    • onyomi says:

      I agree. (Just wrote something similar, but more long-winded, myself).

    • Princess_Stargirl says:

      The econ101 argument doesn’t really tell you much about the magnitude of the decrease in demand. And if you don’t know the magnitude I am not sure you can estimate the policy implications very well at all. Though the econ101 argument should make one very skeptical of claims the minimum wage will boost employment.

      • aesthete says:

        The Econ 101 argument posits that aggregate utility between buyers and sellers is maximized when the market clearing price for a good has been established. Price floors and price ceilings therefore either reduce (or at least do not improve) aggregate utility.

        (There are problems with this view, including a person’s likelihood to specially value their own labor’s worth over that of a good not specifically associated with themselves, but so long as we are talking in an Econ 101 it is hard to argue positively for any sort of price floor/ceiling.

        • AspiringRationalist says:

          Large-ish nitpick: Econ 101 says that the total surplus (denominated in some currency) is maximized when the market clearing price is established. Because the buyer and seller often have different marginal utility of money, it does not necessarily follow that the total utility is maximized.

      • orangecat says:

        Though the econ101 argument should make one very skeptical of claims the minimum wage will boost employment.

        Right, and for me that’s a sufficient reason to oppose it. The best plausible result of raising the minimum wage is that it employers eat all the additional costs with no effect on employment or prices, in which case it’s exactly as good as raising taxes on employers and increasing the earned income tax credit. So why not just do that, especially when there are also plausible results that are significantly worse?

        • Matthew says:

          I think lots of liberal economists favor the EITC over the minimum wage, but there are two problems. One, the EITC is much harder to explain to a low information voter than a minimum wage is, so liberal politicians have an incentive to choose the latter. Two, conservative politicians have completely flipped on the issue, and largely oppose the EITC now, which intensifies #one, since an effective wedge issue needs to be easy to explain.

          Shorter:

          Liberal preference : EITC > MinWage > Nothing
          Conservative preference: Nothing … and that’s it.
          Uninformed voter preference: MinWage > EITC > Nothing.

          • orangecat says:

            Sigh, you’re probably right. My preference is basic income > EITC > Nothing > MinWage, so I expect to be disappointed, at least for the next several years.

    • Charlie says:

      There’s an interesting analogy, and also an interesting disanalogy, to approximation schemes in physics here.

      When one wants to calculate real-world things – say, the conductivity of copper – from basic physical laws, the calculations involved quickly become too difficult to do (“Argh, why won’t these stupid copper nuclei stop moving around?”). And so we make approximations – we replace questions we can’t solve with questions we can solve, and hope that the questions are similar enough to have similar answers. In the example of copper, one might just assume that the nuclei don’t move, and calculate the motion of electrons.

      There’s one technique that is particularly powerful, called perturbation theory, which is to start with an easy question, and change it into the hard question a little bit at a time. “Okay, now we know how copper behaves if the nuclei never moved, but what if we let them move just a little bit?” The reason this works is that if the changes are small, we can think of the small changes in terms of the easy answer – like the easy answer provides us a frame of reference that we can then do math in terms of. If we start out close, we can get even closer.

      So the analogy would be that the economics 101 argument is the question we can solve, and that higher course-numbers of economics argument can be thought of as corrections in the frame of reference of the original argument.

      And the disanalogy would be that this is not necessarily true, and that often even in physics this method doesn’t work, because sometimes the answer to the easy question is unstable – i.e. totally wrong – when you try to move towards the hard question.

  43. Ken Arromdee says:

    The motte: “Beware the man with one study” means to beware the man with one study.

    The bailey: “Beware the man with one study” means to beware the man who has dozens of studies..

    • Charlie says:

      I do not think that word means what you think it means.

    • Susebron says:

      “Beware the man with one study” is, in fact, an allusion to Aquinas (as mentioned) rather than Scott’s actual position. The point of this post was that everyone has dozens of studies, so “having dozens of studies” is not a good single way to determine truth.

  44. stillnotking says:

    Perhaps the economy is such a complex system as to be immune to prediction or directed manipulation, at least immune to anything that is not a hyperintelligent AI, and economists are just fooling themselves. (What I mostly learned from studying economics is that there’s surprisingly little evidence against this proposition.)

    This is why I don’t base my political decisions on economic policy. I base them on things that politicians can unambiguously affect, and to which (I believe) there are unambiguous answers, such as whether or not the government should torture people. Is that “radical skepticism”? Maybe. I think it’s possible in principle to definitively answer economic questions, just like I think it’s possible in principle to accurately model human cognition. I just don’t think we’re there yet, and I doubt we will be in my lifetime.

    • Jared says:

      “Should the government torture people” isn’t just a question of values. It’s also a question of facts. If the use of torture on 10 terrorists could stop 10 9/11s wouldn’t you be more willing to consider it’s use? The only political issue I can think of where everyone agrees on the facts is probably gay marriage, and even then I’m sure there are some conservatives who think it would destroy families or something. If your political views are going to be based off things where no one disputes the facts, then you might as well not have political views at all.

      Also, I think you underestimate economists. It’s true that avoiding recessions hasn’t been figured out yet but look at inflation. It has managed to keep it consistently low for the last 30 years. That’s pretty impressive compared to it’s volatility in the past.

      • Jared says:

        By “it” I mean the Fed.

      • stillnotking says:

        I’m not drawing a facts vs. values distinction, or relying on hypotheticals. I think it’s factually well-established that governments which torture people are not governments under which I want to live. I don’t think it’s factually established that right-wing or left-wing economic policies (read: within the context of US mainstream economic debate; I wouldn’t vote for a Leninist) have major, salient, predictable differences.

  45. Symmetry says:

    I posted this back in the discussion of the links thread but I thought I should bring it up again. Wages and employment are very sticky with respect to small pertubations. Looking at short term changes in employment shouldn’t show you much.

  46. Trevor says:

    Please do a book review of the (very) old but (very) good essay by Clifford, The Ethics of Belief.

    “But,” says one, “I am a busy man; I have no time for the long course of study which would be necessary to make me in any degree a competent judge of certain questions, or even able to understand the nature of the arguments.”

    Then he should have no time to believe.

    • US says:

      A ‘book review’ of an essay? Hmm…

      (On a slightly more serious note, Clifford’s essay is great and everybody reading a blog like this one should (/have) read it).

    • Tarrou says:

      The simple and glib criticism (which still carries weight) is that there are no questions of note which are capable of being really mastered by one human. Yes, someone may know 100 times as much on a certain subject as another person, but still know less than a millionth of a percent the relevant knowledge. In situations like this the “uneducated” person is guessing with a mathematically indistinguishable chance of being correct compared to the “educated”.

      • US says:

        @Tarrou

        I was debating with myself whether or not to engage because your comment seemed to indicate that it would be a waste of time to do so. I’m already regretting having written this comment, but now that I’ve written it I might as well post it.

        First, since when was complete mastery necessary for justification? This specific requirement was never defended in the recent epistemology text I read, and I don’t think that omission was accidental.

        Second, how often do you think the situation occurs where someone knows ‘100 times’ (how do you incidentally quantify that?) as much as another, yet still does not know enough to not have, say, a much better foundation for decision-making (or belief justification) than does the ignorant individual? Do you think it happens often? The way I’m reading you, you seem to think it *always* happens, no matter the context, but I find that proposition absurd so I ask these questions in order to clarify what you mean. Could you give some examples? I pose the latter question because I think it’s highly problematic for you to deal with – in the sense that if you can come up with examples, then your claim is implicitly wrong, and if you can’t present examples then the claim holds no interest (to me, at least). If you can come up with a specific example X, then you obviously know about the million other factors which are supposedly relevant for X and which make the expert knowledge irrelevant. But if you can come up with such an example, then *you* possess expert knowledge to properly evaluate the claim in question. If you can’t come up with an example your claim is just an untestable claim, which holds no interest to me regardless of whether it is ‘true’ or not, whatever one might mean by that.

        Third, how common do you think the situation is where knowledge is irrelevant to ‘a question of note’, compared to a situation where someone knows perhaps ‘twice as much’ (‘has spent twice as much time on topic X?’) and some of that extra knowledge somehow being relevant in terms of improving decision-making (evidence-evaluation)? Reading your comment the only reasonable conclusion to draw seems to be that you think the second scenario never happens. But then an obvious question seems to be what you mean by ‘a question of note’? Is the age of the Earth/Universe such a question? Whether humans think with their brains? How species evolve? What the climate was like in Europe 10.000 years ago? Whether people who smoke tend to die sooner than people who do not? Do you think people who grow up as hunter-gatherers in Kalahari have answers to such questions which are just as reasonable and well-justified in terms of the evidence as the answers which might be given by Harvard graduates? I ask because I don’t see how I can read your comment without concluding that your answer to this question would be yes, which I think is … the wrong answer.

        I’d argue that in most situations where a person knowing a lot doesn’t know enough to properly evaluate data he still knows more than the ignorant individual, because he’s more likely to know that he doesn’t know enough to evaluate the data, unlike the ignorant individual (“The trouble with the world is that the stupid are cocksure and the intelligent are full of doubt”). He knows more both about the things he knows, and about the things he does not know.

        But just to recap a central point: Arguing that people never (‘there are no questions of note…’) know enough about a topic/question of note to have enough knowledge to get ‘the right’ (‘evidence-based’) answer does not make sense to me, because even if that were true there’d be no way for you, or anyone else, to know it – this argument seems to me equivalent to just embracing radical skepticism. (Which is what motivated my ‘waste of time’ comment in the beginning – my life’s too short for those kinds of discussions).

        • Tarrou says:

          I merely point out that requiring a knowledge base to even give validity to another’s opinions (beliefs, w/e) is pretty hard to justify. Where do you draw that line? And do you really expect everyone to become an expert in everything they have an opinion on? And finally, as my point above attempted to illustrate by hyperbole, even the “expert” in a given field tend to not be particularly good at predicting events within it (See Phil Tetlock’s research on the limits of expertise, basically in a raft of social scientific endeavors, world class experts get real world predictions right about as often as your local drunk. The problems are too complex for analysis, marginally increasing the knowledge base does nothing).

          • US says:

            “I merely point out that requiring a knowledge base to even give validity to another’s opinions (beliefs, w/e) is pretty hard to justify.”

            It may have been what you tried to do, but it was not what you did, as my comment indicated. What you did do in your comment was pretty close to claiming that knowledge is always irrelevant and that people can just believe whatever the hell they want and shouldn’t worry about their positions being based on knowledge because nobody knows enough to have ‘enough’ relevant information about anything anyway. To find something remotely close to this view defended here made me angry also on account of the fact that I couldn’t sleep and felt lonely at the time I wrote my comment, a mind state not conducive to good commenting (if I was being rude that was the reason, but I apologize anyway).

            I find claims to the effect that the beliefs of an ignorant know-nothing are just as good as those of a knowledgeable person who’s spent a lot of time learning about a subject harder to justify than the alternative in the vast majority of contexts. In the cases where they’re not, they’re arguably not because the experts don’t know the relevant knowledge (that they don’t do better than people without extensive knowledge), and so do not really have the expertise they are claimed to have in the specific context. Again, even in such a situation it either boils down to knowledge we have or to an unprovable postulate.

            You didn’t address my criticism, which makes your counterarguments in the second comment more or less irrelevant; there are surely problems associated with demanding a knowledge base, or whatever you want to call it, and how to set up relevant criteria, but there are problems with not doing it as well, and you didn’t make a strong case for the alternative and did not address my criticism. There are situations where complexity is probably too high and social science stuff’s probably often like that (‘probably’ because in such situations we can’t know this to be true; if we know this to be true, we have relevant knowledge about the true state of affairs, and this makes the expert position better justified than the uninformed position, even if the expert position is to not have an opinion), but there’s a lot of stuff which isn’t social science. You don’t need to understand QM to understand evolutionary theory.

            You didn’t answer my questions so I see no compelling reason to answer yours, but on a personal note I should mention that I generally do not comment here without having read a relevant textbook on a related topic recently. I often refrain from having an opinion on topics about which I have not read a textbook. Other people use different criteria and that’s fine, though I personally find an ‘anything goes’ approach reprehensible. If you don’t know anything, the correct answer is ‘I don’t know’, not ‘whatever I want to believe’. This – the ‘I don’t know’-answer, that is – may or may not also be the correct answer to a question asked in a situation where you do happen to know a lot of stuff about a topic, but that’s besides the point.

      • Scott Alexander says:

        The other simple and glib criticism is Bayesian calibration.

  47. Jaskologist says:

    What happens when you don’t have a funnel plot? You go back to your basic model of the world, and derive the answer out from there.

    The right-winger basically says “Supply and demand is a law of the universe just as real as gravity. Raising the cost of (low-skilled) labor must decrease its demand, or we are in a contradiction. Therefore, the studies showing otherwise must be flawed in some way.”

    Somebody else can sketch out the left-wing version. I think this is actually how we approach all controversial questions. It’s not even clear that we’re wrong to do so. As you’ve shown time and time again, it is very easy to marshal piles of evidence for whichever position you wish, so empiricism is not really that big a help.

    • Princess_Stargirl says:

      “Supply and demand is a law of the universe just as real as gravity. Raising the cost of (low-skilled) labor must decrease its demand, or we are in a contradiction. ”

      Even if this “law” was true the “law” doesn’t tell you anything about the magnitude of the decrease in demand. Its entirely consitent with this law that demand would drop by a very tiny amount if the minimum wage is raised within reasoanble bounds. A tiny magnitude + noise could easily explain studies finding gains in employment.

      I actually think thats what probably happening. Minimum wagw increases, with in the range passed in the USA, probably negatively affect employment. But by a very small amount. (If this is true we should probably raise the minimum wage a little imo). This explains the data well and doesn’t require any exotic economic conditions.

    • Hainish says:

      “Supply and demand is a law of the universe just as real as gravity.”

      Except for Giffen goods.

    • Jaskologist says:

      You’re getting too hung up on the specific example of the minimum wage. The interesting question, and what the original post is actually about, is one of epistemology.

    • Scott Alexander says:

      Okay, but first principles aren’t that big a help either, given that left and right clearly have different common sense intuitions on this matter. The best that can be said about first principles is that they’re *no worse than* empirical evidence.

      But I’m pretty sure they’re also no better, and I would trust myself to sort through all the studies more than I would trust myself to have good intuitions in any area this complex.

      (this might be because I don’t understand economics well enough to have good intuitions, but I often feel the same way about medical studies too.)

      • Jaskologist says:

        Clearly, the answer is to pick the correct first principles.

        More seriously, I’m being descriptive rather than prescriptive above. I don’t have an actual solution. My suspicion is that there isn’t actually one weird trick you can use to divine the truth, just a melange of flawed ways of guessing and approximating it.

        (In the specific example, I don’t think most right-wingers would consider supply/demand an axiom so much as a thing humans have discovered and repeatedly confirmed over the years. They will question it in the wake of studies challenging it about the same way you would question evolution in the wake of a study showing irreducible complexity; obviously the study missed something. It’s not just the weight of opposing studies on that specific topic against it, it’s the weight of this whole other body of data, too.)

  48. Joel E says:

    I searched Google, and it said that peer-reviewed studies are totally trustworthy.

  49. kieran M says:

    The obvious (potential) problem with a funnel plot are the studies you choose to put on it. If all the zeroish results are based on bad studies then that will give you a misleading picture. Now the obvious thing you could do is to read the meta-analysis, and read their criteria for acceptance, and see if you agree. But of course even that may not be good enough, because sometimes this criteria can be pretty woolly, and you might disagree with their decisions if you took the time to look at every paper. And of course a funnel plot won’t necessarily beat publication bias, because if enough papers are biased you’ll get a consistent, incorrect effect.

    You didn’t mention that pharmaceutical companies are rather bad at releasing all the studies in which their drugs perform poorly, so a meta analysis will exaggerate the effect of their drug.

    It’s not… necessarily as bad as this makes out though. If I create a wonder drug that completely cures cancer in 99% of cases, then do some random studies with at least (say) 100 patients, then a study which shows my wonder drug is bad will turn up very rarely. So a strong effect, or even a weak effect with very low variation, will be relatively easy to demonstrate either way.

    The problem we have at the moment is that most new drugs are marginal improvements on previous drugs, and often for a very limited pool of patients. It becomes very difficult to identify which of those are actually effective, and of course the people making the drugs want the studies to show that their drugs are good. Note that all these drugs will probably be better than placebo, so it’s not like taking them will necessarily be harmful to their patients, but they will cost a lot more money.

  50. Tarrou says:

    Something that always triggers my skepticism is a failure to explain the costs associated with a certain policy, or acknowledge the potential downsides. If someone can’t even entertain the possibility that some policy will have some bad outcomes, they’re too biased to even be thinking properly about it. I see this a lot with my fellow drug-legalization/decriminalization folks. I agree with them, but we will not all be living in a post-scarcity world if pot is legal. “But, like, you can make EVERYTHING with hemp, man!”

    And as far as the minimum wage, the laws of supply and demand are as near total as you get in the human behavior arena. Anytime someone tells you they’ve found an exception to teh law of gravity, you should be very skeptical. It may well be the costs are lower than the benefits, but to suggest there are no costs associated with minimum wage is straight lunacy. And those costs will be born by the most marginalized, unemployed and poor in society, who incidentally are pretty hard to study for any effects at all, much less “prove the negative” stuff like “would B have had a job if the minimum wage were fifty cents lower?”.

    Barriers to work reduce work. This isn’t rocket science. Be this occupational licensing, general regulation or the minimum wage, we make those calls as a society. We could have a lot more doctors if there were no entrance requirements, but we think we’re better off with fewer and better educated ones. Maybe we’d be better off with fewer and better workers, but that’s a hard sell from the left, isn’t it? The question of where to draw those lines is the interesting, and horribly complicated one.

    • suntzuanime says:

      But this criterion can be gamed. You can still minimize the costs as you explain them, or pick out problems you don’t think your listeners will consider problems at all (like the guy who goes to the job interview and explains that his greatest weakness is that he works too hard). You can make deliberately weak arguments against your position while making arguments as strong as you can muster for it (cable news is a big fan of this one).

      And on the flip side some things just are really great. Antibiotics are really great. Yeah they can have side-effects or whatever but if I’m really excited that we don’t all have to die pointlessly from infection anymore, I might gloss over those side-effects while still giving you a pretty good explanation of the most relevant things about antibiotics, because not dying from infection is a big deal and the side-effects mostly aren’t.

      • Anonymous says:

        Being generous to Tarrou, I do something similar: If there is a serious, contentious issue that someone is writing about, I write it off as a propaganda piece if it doesn’t cover downsides and drawbacks at all. This is not to say that I automatically accept uncritically anything that *does* discuss downsides and drawbacks – merely that one of my more frequently invoked criterions for taking something seriously enough to even consider fact-checking it at all instead of writing it off as a bunch of nonsense not worth engaging with is some coverage of the opposition position.

        • Ken Arromdee says:

          I will ask you too what you think about creationism. I don’t know any downsides and drawbacks to rejecting it. Sure, theoretically I could say something like “refusing to teach it violates the rights of parents who want it taught to their children”, but that’s such a weak argument that it’s not really necessary for anti-creationism discussions to mention it.

          • Fadeway says:

            Rejecting creationism gave us the “just-so” evolutionary stories that are used to explain -everything-

          • Ken Arromdee says:

            I doubt that if you were writing an article against creationism, you would feel compelled to say “however, if we reject creationism, people will be more likely to use bogus evolutionary stories to explain things” and use that as a point in favor of creationism.

            That’s like saying that allowing green cars in your parking lot has a risk of people being run over by green cars, while this risk is absent if you don’t allow green cars in your parking lot. Being run over can happen with blue cars as well as green cars, and making up stuff can happen with creation as well as with evolution.

          • Anonymous says:

            I think the obvious downside to rejecting creationism is the risk of being wrong. If you reject it incorrectly you probably end up in hell. So, the major arguments are just whichever ones provide the best evidence for creationism being correct (gaps in the fossil record, whatever this is http://www.icr.org/earth-science).

            That said, I think you make a good point because I don’t think I’d expect an article about creationism and evolution to address these, and I suspect I would have agreed completely with the above poster’s heuristic if you hadn’t posted this.

      • Tarrou says:

        Of course it can be gamed, but that’s kind of the point. The care given to the opposition is a pretty good indication of the seriousness of the argument.

        And are antibiotics a seriously debated policy argument? The overwhelming majority are good with them, there’s a few religious nuts and tinfoil hats who disagree, but on the whole, who is against antibiotics? Same goes for evolution/creationism. Creationists are the BBGs of the rationalist community, despite the fact that they haven’t won a policy debate in eighty years. Even in Texas the courts struck down a local school board’s attempt to shoehorn it in. There’s just not much debate to be had there.

        That said, I was taught creationism exclusively as a child and I came out of it with a better understanding of evolution than the public school kids. The version of “evolution” commonly taught and referred to in pop culture hasn’t been the state of the science for many, many years. YMMV, but creationism (to me) made me understand evolution much better.

  51. Jack V says:

    Thank you for this! That makes a lot of sense, and I think it’s a really example about how to read citations of studies of controversial things in general, and probably about minimum wage specifically. Especially that it may be good in some situations and bad in others.

    About the minimum wage specifically, there’s something I said before, which once I saw, I couldn’t unsee, but when I tried to describe was met with stony silence or dismissal.

    1. There is a natural market value for mostly-unskilled labour, depending on the area and exact job. Surely there is — even if it’s not realised in reality, there WOULD be if the market was more fluid.

    2. But it’s not really a value, it’s a range. As with any deal it’s capped below by how little people are willing to work for, before it would be a better use of their time to scavenge on the streets (taking into account job skills, respectibility, etc). Left-wingers are right that if jobs somewhere in the US paid less than a tenth of cent an hour no-one would take them! And there’s a maximum value, of what companies are willing to pay, before it becomes cheaper to automate the job, or do without it, or hire labour off-the-books and risk the consequences. Right wingers are right that if jobs had to pay $1,000,000 per hour, no-one would hire for them. cf. http://negotiatewithchad.blogspot.co.uk/2011/03/batna-and-zopa-quick-introduction.html

    3. The actual wages aren’t ALWAYS in that range, because there’s overhead, hiring, firing, finding another job, finding a replacement worker. But they tend to be in that range by default. All this should apply in ANY economy, whether unskilled labour is over or undersupplied.

    4. WHERE in that range? Closer to the minimum, or closer to the maximum? This depends on the dark-side skills of negotiating. Which side is more desperate? Which side has better knowledge of the other side’s bargaining position? Which side is under the most time-pressure to reach an agreement quickly. And to a lesser extent, who is more practised at negotiating. A good society/economy is based on lots of deals which are positive-sum — they’re a very good thing to both sides, so no-one cares if the value shared isn’t exactly equally, it’s better to just get on with it. A bad one, is based on deals where you can only gain by screwing the other person over. Unskilled-labour market is probably somewhere in the middle.

    5. OK, this is where I choose a side in the argument. If all minimum wage workers in the country (working and not working) were in a union, and could strike at no notice, then yes, the companies and government and other people would be at their mercy (until they automated everything) and the wages for those jobs would be very close to the maximum value. If most minimum wage workers in the country are desperate and NEED a job, else they’re at real risk of not having healthcare, or food, then even if their labour is quite valuable, companies can easily say “here’s a bad deal that’s juuuuuuust worth it to you. Do you want to take it? Or starve and have some other unlucky schlub take it?” Which of those is closer to the truth? I think the second is a LOT closer to the truth. Hence, I think wages are (probably) below a natural market value in the middle of the possible economically feasible range.

    6. Hence, legislation restoring wages more to that natural market value is a useful fix that enables, not goes against, the market. Even if it has some unintended effects (eg. charity work which people DO want to do for below minimum wage, eg. being unresponsive to changes in the economy), and a different approach with the same result (eg. maybe guarenteed basic income??) would be better.

    7. THAT’s why I think a minimum wage should (probably) exist and be higher than it is.

    8. I ALSO think that if it’s a LITTLE bit too high, that’s probably ok, since it functions as a tax on companies to subside people with small incomes, which is probably on balance a good thing, but I agree that has downsides and disproportionately hurts small businesses, which I don’t think is a good thing.

    Now I’ve thought of that, I find it hard to see how it could NOT be the case. Why WOULD companies pay more than they need to to employ people? Even if many employers would like to, wouldn’t they be driven out competition which races to the bottom. If there’s a minimum wage job advertised, I hear a LOT more stories saying “I wish I could get that job but I couldn’t” or “I got that job but they screwed me on the money indirectly” than “I advertised a minimum wage job but no-one applied and I’m desperate, I’m going to pay more money or offer more perks”. To some extent that’s the political culture I come from, but are there really more of the second story? Who does that suggest has a stronger bargaining position (in terms of, manipulating the result, not the natural range of market value), and what does that suggest about wages?

    • Tarrou says:

      Now factor immigration into the deal. How many times do you hear about people “doing jobs Americans won’t do”? What they mean is “doing jobs Americans won’t do for five bucks an hour”. And yet, you find all sort of political wierdness, where the left wants more immigrants, which depresses wages, but more minimum wage increases, which supposedly increases them. And the right wants the opposite! No logic to it.

      • peterdjones says:

        I don’t see the contradiction. If you thought that immigration naturally increased wages, you wouldn’t be calling for am artificial mechanism.

      • There are conservatives who support minimum-wage laws in order to price immigrants out of the market.

        Using the algorithm “Find out the opinions of people you disagree with on other issues and disagree with them here” I’m against minimum-wage laws.

      • Leonard says:

        Of course is it logical. More immigration, at least post 1965, means more leftist voters. So the left wants it in spite of whether old-school economics asserts that it screws poor voters. This problem is one of perceptions, and is solvable. For one thing, call into existence Kreuger and Card. Economists, take that! As for your poor voters getting bad ideas about immigration, they can be managed by domination of the press and the schools. Everyone knows immigration plusgood! What are you, some kind of hateful white man???!! Meanwhile, when you raise minimum wages it is not like you are directly screwing your supporters who will be thrown out of jobs. They’ll get welfare of one kind or another. We don’t let people starve in the streets. People dependent on the state also vote left.

    • lupis42 says:

      The stories you describe at the end sound exactly like what you’d expect if the supply of labor is in excess of demand, and hence the minimum wage is *too high*. More people wanting employment than employers wanting to hire is a labor surplus, which would normally push prices down.

      As Harald K. mentioned above, it’s not really enough to have data, you need a model, something that explains how the proposed change will affect the existing system.

      The model that I’m working from says that, given the discrepancy in negotiating power you assert in (5), the net value of compensation is going to be slightly above the minimum needed to attract a sufficient workforce. Wages are one component of compensation, but far from the only component, so raising the minimum wage shifts more compensation into the form of wages, but doesn’t raise compensation until there’s no remaining slack in the compensation package (i.e. nothing else that the employer can save money on without losing more than they save).

      • Drew Hardies says:

        doesn’t raise compensation until there’s no remaining slack in the compensation package (i.e. nothing else that the employer can save money on without losing more than they save).

        What benefits could employers (legally) cut for minimum wage workers? I’d expect most minimum wage workers to be part-time and without healthcare or vacations.

        • Chris H says:

          There are several options actually. Worse management for instance. With less money to pay them attracting managers that can make the store run a profit and keep employees happy (beyond the bare minimum happiness needed for them to work) is harder and this model would favor focusing in more on the profit producers than the friendly to workers. Another option is being less flexible with working hours. Also any low skill labor employer who is above the government mandated minimum requirements for safety can reduce safety expenditures to the minimum. All these and more I probably can’t think of at the moment are potential options. Thinking of benefits solely as some form of formal compensation is a mistake.

          All that being said, while I’m highly uncertain of what the real effects of modest minimum wag increases are (or decreases for that matter, an area that seems to get FAR less attention, probably because few proposals to lower the nominal minimum wage even get seriously discussed), I do think wage subsidies are a better idea that’s more likely to be a way to set an income floor that does so by redistributing wealth from the rich to the poor rather than some redistribution that happens between different poor people or benefits and wages.

        • Glen Raphael says:

          What benefits could employers (legally) cut for minimum wage workers?

          Things they could cut? Free company-provided uniforms. Company-provided or company-subsidized training. Scheduling flexibility. A nice break room. Health benefits (yes, many minwage jobs have in the past provided some, and this tends to get cut as the minwage increases). A nice work environment in the sense of usable space, good lighting, a place to keep your stuff. Employee discounts on whatever the company sells. Subsidized meals. Subsidized parties. A 401k plan, possibly even with employer matching. Vacation/sick days.

          If you think of a job as a bundle of all the stuff that costs the employer money that is associated with an employee, raising the minimum wage initially has the primary effect of making the job more unpleasant, skewing that bundle in the direction of more money but less everything else than the employees themselves would have preferred.

          After a minimum wage increase you can divide the employees who were previously below the new minimum into two categories:

          (a) people who simply lose their job, because their productivity level doesn’t justify a higher wage
          (b) people who do gain salary but simultaneously lose benefits they valued more than the salary given up.

          How do we know the people in group (b) wanted those benefits more than a higher salary? Because employers in aggregate offer benefits in order to attract employees. If these benefits weren’t worth (to the average employee) more what they cost in foregone salary, the employers probably wouldn’t bother offering them

          BOTH groups are on-net made worse off by the change. The people who get fired AND the people who get a raise BOTH lose; their job situation has been made worse.

          So who benefits from minwage? Mainly people making a lot MORE than the minimum who are protected a bit from low-wage competition. (this was the original explicit purpose of setting a national minimum: to protect white workers in richer northern states from “unfair” competition by black workers in poorer southern states who were willing to offer lower wages in order to get the job.)

        • drethelin says:

          perks like free uniforms, coffee, office supplies, so on.

        • lupis42 says:

          Replace regularly scheduled hours with a model where schedules are determined a few days or a week in advance. Reduce or eliminate subsidies for commuting/daycare. Raise the price on the coffee/vending machine. Eliminate smoke breaks. Start enforcing rules about not taking home ‘perks’.
          Let the temperature on the shop floor go down to 50 in the winter, or up to 90 in the summer, instead of keeping it between 60 and 80. Require employees to buy more of their own safety equipment or uniforms. Fine employees for minor infractions of discipline. Require employees to get into uniform on their own time.

    • Anonymous says:

      Isn’t a minimum wage a tax on the people that shop at those companies, and thus (largely) a tax on low-income people? If Wal-Mart has to start paying it’s workers 15 dollars an hour, it seems like wishful thinking that they won’t just start selling things at a higher price.

      This is what I find so frustrating about the “companies costing taxpayers money by not paying their workers enough” rhetoric. That’s GOOD – taxpayers are RICH.

      • lupis42 says:

        The problem with that is the same as the problem with assuming that workers will get the extra money – it assumes that the negotiating power is mostly with Wal-Mart. Given that there are a lot of other retailers competing with them either directly or price, or on quality at a similar price point, it’s unwise to assume that Wal-Mart has that much pricing power over its customers. (They might, it just shouldn’t be assumed)
        Who pays – the workers (in the form of other compensation reduced), Wal-Mart (in the form of reduced margin), the customers (in the form of higher prices at Wal-Mart), or the taxpayers (as Wal-Mart substitutes more capital/skilled labor and lays off the existing workers, leaving them on taxpayer funded social insurance) depends on where the negotiating leverage lies.

    • Anonymous says:

      Your post is about employers vs employees. What about employed vs unemployed?

    • Menno says:

      Given your belief that it is a societal good for all jobs to pay a wage X or above, and that you consider it to be a form of a tax, why are you absolving the rest of society from paying the tax?

      If society at large benefits from wage X, society at large should fund wage X.

    • meh says:

      In point ‘5’, the first scenario is not a free market.

      The second scenario does seem to describe a natural market; why are the wages in this scenario described as not being the natural market value?

  52. chaosmage says:

    This makes it sound like the winning move (for voters, not active politicians) is to actively ignore political discussion as much as possible, except before elections.

    I use to rationalize my (habitual) interest as observer geekery – an attempt to learn something even about matters I cannot directly influence. But if I’m not learning actual world knowledge except maybe if I try really hard (or read Slate Star Codex), that rationalization goes poof.

    My system 1 reacts to the idea with relief and relaxation, but also worry I might be overlooking something. Too often have I heard of people who didn’t follow politics and then were plowed under by it.

    How can I find out if ignoring politics as much as possible (except before elections) is a safe thing to do?

    • The winning move is not to pay attention to national political discussions even during elections, and also not to vote.

      (I give an exemption for the discussion of broader political principles like we do here on SSC, and especially for Scott’s “More than you wanted to know” posts.)

      More broadly, I have stopped paying attention to articles that attempt to settle a political discussion with facts. The facts are always cherry-picked, and their actual information content is small, or zero.

      • Ken Arromdee says:

        What is your opinion on creationism?

        (And if your reply is “creationism doesn’t count as a political decision based on facts, because creationism is scientifically false”, you’re basically saying “let’s not settle political discussion with facts unless the facts are very relevant”, which is equivalent to “let’s settle political discussions with facts after all”.)

        • Princess_Stargirl says:

          I do not really see how you can settle the “creationism” debate with facts. The resolution revolves around alot of philosophical concepts. And much of this philosophy is far from obvious to someone who hasn’t seriously thought it over (part of the reason creationism was so popular for so long). The idea of evolution is pretty needed to explain how a world could “look designed” to so many people. But I don’t think many facts about evolution are needed, just the concept.

          Finally most of the time when people stry to debunk creationists with “facts” about evlution they will say at least a couple things that are not actually true.

        • First: “creationism” per se is not a political issue, but a scientific/theological issue, and the related political question “Should creationism be taught in schools?” is not straightforwardly answerable only by an appeal to facts.

          But more importantly, I’m not claiming that political questions should never be settled with facts. I’m claiming that any article or book claiming to settle a question with facts is automatically suspect, because the “facts” that it uses are always cherry-picked and unreliable. Which is basically the same as the point of Scott’s post.

        • jaimeastorga2000 says:

          What is your opinion on creationism?

          The theory of evolution is in the class of theories which have been so thoroughly corroborated by overwhelming mountains of evidence and decades to centuries of successful practical applications that I think they are not amenable to epistemological crises. Other theories in this class include classical mechanics, classical electrodynamics, relativity, quantum mechanics, the atomic theory, the cell theory, and the germ theory of disease.

          Stepping down to the object-level for a second, I think it is far more important to prevent the federal government from being able to dictate what local districts and homeschooling parents can teach their children than to prevent a few loony creationists from teaching their children a false theory which is unlikely to ever affect their lives.

          • Hainish says:

            I think it is far more important to prevent the federal government from being able to dictate what local districts and homeschooling parents can tell teach their children

            You cheated by lumping homeschooling parents in with local school districts, when the outweighing factor has only to do with homeschooling parents.

        • Scott Alexander says:

          I’m pretty sure there’s much broader agreement among biologists about creationism than there is economists about the minimum wage. In fact, even when the creationists trot out their list of however many scientists who support them, they rarely have biologists on them.

          I think there are subjects that there’s a strong and accessible consensus on, one just has to be careful to make sure to double-check when people tell you any particular subject is one of them.

  53. Alex Godofsky says:

    On the minimum wage you have the enormous, possibly (probably?) insurmountable problem that it’s really freaking hard to do a proper experiment (let alone a ‘natural experiment’). Like, even if you could randomly assign US states to control and treatment groups, and give the control group a $0 minimum wage and the treatment group a $10 minimum wage, the effect could still be somewhat muted by responses like migration. And in reality, you are looking at much, much smaller deltas.

    I am firmly convinced that the minimum wage “raises unemployment”* but I expect we will never have any clear, direct empirical evidence. And it’s because I don’t think we can ever convincingly resolve this with studies that I think the right answer is to go with the well-supported general principle that price floors reduce demand.

    *By which I actually mean that it inflicts harm on the lowest-paid workers via any number of mechanisms, e.g. reduced hours worked, more difficult labor, or worse non-pecuniary benefits.

    • Harald K says:

      Labour is no ordinary commodity. Even if a minimum wage is a bad idea, you really can’t tell that from such a simplistic model.

      It can’t be worked around: on issues like these, laypeople need to decide which experts are more credible. Actually finding the answers to such questions as whether increasing/setting a minimum wage is a good idea, requires more than just data. It requires a non-trivial model to interpret that data through. Unfortunately for us, it can be hard to see the difference between a model (of an unfamiliar domain) that is built on plausible assumptions and one which is carefully constructed to deceive. Sometimes there are even competing plausible models.

      But it also seems to me like the minimum wage question is so narrow and technical that it just isn’t productive for the public to get into it. Can’t we deal with such issues on a higher level, somehow? After all, there are countries that have no legal minimum wage that pay unskilled workers far more than some countries that do. Minimum wages don’t seem like they should have to be the linchpin on which increasing socioeconomic equality hangs.

      • Alex Godofsky says:

        Labour is no ordinary commodity. Even if a minimum wage is a bad idea, you really can’t tell that from such a simplistic model.

        You can add all kinds of bells and whistles to your model. You can pretend that employers have monopsony power. You can assume that low-wage labor and capital are highly substitutable. You might use an efficiency wage model. You might model non-pecuniary compensation.

        Some of these will make the minimum wage look less harmful, others will make it look more. And since there’s no clear evidence for which of these additional effects actually obtain*, you really should just default to the simple model, or at least keep it the median of your prior.

        In economics, “it’s complicated therefore you should totally just throw out supply and demand entirely” is a popular argument from the left, but basically never valid.

        *Actually IMO there is pretty clear evidence that the pro-minimum wage models are absurd, with the employer monopsony really taking the cake. But I’m making a point here.

        (In case anyone’s interested, wrt the examples listed I judge them laughably false, true-ish, true but in a way that hurts the case for the minimum wage, and true, respectively.)

        • Iskra says:

          Why bring up monopsony power instead of Nash Bargaining? If there are serious modern economists using monopsony power to talk about minimum wage issues, I will gladly join you in making fun of them and their schools. But, even if that turns out to be the case, I would suggest that it’s better practice to mention the stronger version of the argument, especially given that you hold an opposing viewpoint.

          • Alex Godofsky says:

            Why bring up monopsony power instead of Nash Bargaining?

            Because I rattled off the first two arguments that came to mind on each side? The list was not intended to be exhaustive.

            If there are serious modern economists using monopsony power to talk about minimum wage issues, I will gladly join you in making fun of them and their schools.

            The exclusive class of serious labor economists may not talk about it, but there’s plenty of conversation with and among the laity that cites the monopsony argument. This is partly because it’s a simple exercise to explain to non-economists, because it keys into the morality play of little guy vs. big business, and because it’s easily conflated with the false intuition a lot of people have that elasticities at the current margin are zero.

            It is worthwhile slaying arguments like this because they are believed by people who vote.

            As further evidence of the argument’s currency, I will note its prominent position in Wikipedia’s article on the minimum wage.

          • Iskra says:

            All right, I think I understand where you’re coming from, now, and I acknowledge that it’s a reasonable position. I apologize for mistaking the omission for a devious political tactic.

            I still encourage engaging with the strong version of an argument, regardless of how popular the weaker version is. I think it supports better decision making and avoids miscommunications like the one we experienced.

          • Alex Godofsky says:

            I feel so deliciously vindicated: from a later comment on this same post.

        • Will says:

          I do a lot of modeling of real world markets for a living(inventory managements models, stuff like that), and the first rule is always “throw out basically every textbook fact about supply and demand curves.”

          This isn’t quite the same as “throw out supply and demand” but its quite close. I think the criticism that econ 101 is too simple to capture the behavior of real world markets is probably valid, because I face it all the time at work, in non-political “how many widgets will we sell” type situations.

          And from your own post first post- you basically say that you’ll trust your model regardless of what evidence comes through the door, which seems like an obvious failure mode.

          • Nornagest says:

            I’m no economist, but I’d be a lot more inclined to trust one saying “the real-world supply-and-demand relationship is horribly screwed up relative to our models, as evidenced by these breakfast cereals and these widgets and, oh yeah, also minimum wage laws” than “we have to throw out 101 behavior in modeling the minimum wage, and only the minimum wage, for reasons”.

          • Will says:

            I read his comment of liberals wanting to throw out supply and demand because “its complicated” as a general argument liberals were using beyond the minimum wage.

          • Alex Godofsky says:

            And from your own post first post- you basically say that you’ll trust your model regardless of what evidence comes through the door, which seems like an obvious failure mode.

            I’ll trust my model of supply and demand regardless of what studies of the minimum wage come in the door, because I think those studies have almost 0 power (in the frequentist sense). In the Bayesian sense, I’m saying that I expect that even if my model is true, real-world studies of the minimum wage won’t be able to find an effect.

            Analogy: we would not update in either direction on the theory of cosmological inflation based on infrared scans of the sky using night-vision goggles.

          • Anonymous says:

            How did you compute the power?

          • Alex Godofsky says:

            See my original post at the top of this thread?

            Elaborating, real-world studies of the minimum wage have to work with small treatments, which will make the effects smaller and harder to discern. Additionally, it will often be very, very hard to provide sufficient controls, because policy is deeply endogenous to the very conditions it is supposed to affect. Additionally, the economic cost of a minimum wage can manifest itself in many different ways, some of which are impossible to measure. (To repeat an earlier example, non-pecuniary compensation could be reduced to offset the increased wage.) And even the ones that can be measured in principle, like “unemployment”, themselves have serious issues and can be measured in different ways that give different answers. Additionally, the natural rate of variation is high. Additionally, there are long-term secular trends in many of the variables of interest, and we don’t always know what those trends are.

            Start from there, and add on publication bias and all the other stuff people like to complain about and it seems clear that the project is doomed.

    • Salem says:

      I am firmly convinced that the minimum wage “raises unemployment”* but I expect we will never have any clear, direct empirical evidence. And it’s because I don’t think we can ever convincingly resolve this with studies that I think the right answer is to go with the well-supported general principle that price floors reduce demand.

      This is the exactly the point I was going to make. In addition, many minimum wage studies seem deliberately designed to find no effect, by only looking at immediate effects rather than medium or long-term effects. It’s highly disingenuous. See also here.

      Relatedly, this is why I hate the “Worst Argument In The World” post. In reality, our knowledge of the world is general, and slightly fuzzy. That’s the only way you can build up enough evidence, and a suitable theory, in reflective equilibrium. For example, “Be careful of large construction projects, because they tend to overrun in time and budget.” But no-one wants the rules to apply to them, so someone says:

      Your construction projects notion is a silly deontological rule. In my definition, building a huge new Parliament building in central Edinburgh isn’t a large construction project. You can define it as one if you like, but there’s no right or wrong answer. What really matters is utility. And there’s no evidence that this particular project will cause negative utility, or lead to overruns. After all, the best estimates for the cost and time taken for this project are exactly the ones we used in our estimates!

      Someone making this argument doesn’t care about delays and cost overruns. They just really want to build their building (naturally, it took twice as long in time and an order of magnitude more in cost to build). But they know they can’t say that, so they try to run away from the known general principles to take refuge in something very hard to prove either way, and then claim that the evidence isn’t certain, so we should do what they wanted to do all along. It’s fundamentally dishonest.

      The same applies to moral principles. We know that allowing people to get away with murder causes all kinds of problems, but the Let’s All Go Kill Bob Smith Foundation doesn’t define stabbing Bob Smith in the back as murder, and you can’t prove that the sorrow of his relatives outweighs the joy of his killers, so the judge should let them get away with it. Or at least, so argues the lawyer for the LAGKBSF.

      We know “price floors reduce demand”, and we see it across a range of circumstances. The minimum wage is a price floor by any reasonable definition. Some people really really want there to be a price floor for wages, and find it rhetorically inconvenient if that price floor reduces demand. Partially for good reasons and partially because those advocates like to muddy the waters, it’s been hard to prove the effect of the minimum wage on unemployment, but that doesn’t mean we don’t have knowledge, or that you somehow have a “free hand to do what you like.” It’s not what you’re “allowed to believe” it’s what you’re “forced to believe.”

      • Tom Womack says:

        What people want is an income floor for humans, not a price floor for wages. In the UK at least we have a fair number of processes to arrange that employers pay people according to their ability and the government tops up according to their and their dependents’ needs.

        Which means that lowering the minimum wage won’t do much to the amount of money people have at the end of the month, but will mean more of that money comes from the state and less from employers.

        • Salem says:

          I too am sympathetic to an “income floor,” but your conclusion doesn’t even begin to follow from your premises.

          Lowering/abolishing the minimum wage will most likely:

          1. In the short run, lead to higher employment, but almost no effect on the wages of the currently employed (as wages are sticky). This will stand to reduce benefit payments.
          2. In the medium-to-long run, lead to higher employment but slightly lower wages (ceteris paribus) at the bottom of the scale. Effect on benefit payments: ambiguous, but likely reduction.
          3. In the very long-run, shift norms and attitudes. Effects unknowable, but my prior is that encouraging work habits will be a beneficial shift.

          • Tom Womack says:

            Why would higher employment, in a world where low-paid employment incurs a cost to the State to make the wage up to the income floor, lead to reduced total payments from the State? Given that out-of-work benefits are pretty much required to pay less than the income floor to provide the incentive to work at all, moving someone from out-of-work benefits to being paid zero by an employer topped up to income-floor by the government ends up in more total government outlay.

          • lupis42 says:

            “moving someone from out-of-work benefits to being paid zero by an employer topped up to income-floor by the government ends up in more total government outlay.”

            That doesn’t follow. Suppose for a moment that the income floor is the out-of-work benefit. Then moving people from out-of-work to being paid zero by an employer means no change in the total outlay.
            It also seems unlikely that wages would be zero, even net of the financial costs of working.
            If the wages aren’t zero, but wages net of taxes/benefits/fees are zero, the government is capturing income from the workers that it would not capture from the nonworkers, and is paying smaller net outlays for the people who work than for the people who don’t, though gross outlay may be larger.
            If, on the other hand, the program is providing the incentive to work at no cost to the employer by setting the income floor above the out-of-work benefit, there is an incentive for the employer to replace skilled labor (some marginal cost) with free subsidized labor (no cost). I’m not sure why anyone would design a program that way, but that would be leading to higher costs from the government, because it was essentially employing people to do work for other people, according to some bureacratic process. That sounds like a process designed around funneling resources from taxpayers into private hands, but that’s plausible enough, lots of governments have one or more programs designed that way.

        • Mary says:

          “What people want is an income floor for humans,”

          Without their being on the hook for it, naturally.

          • JohannesD says:

            Northern European countries have what is basically an income floor (even though the implementation often leaves much to be desired with regard to the complexity of the system). The idea of a welfare state is well-supported across party lines and different income segments – it’s not like jobless socialists are using a mob rule to rob poor high-income people of their hard-earned wealth. Even though everyone and their grandmother sometimes likes to grumble about taxes, mostly people bite the bullet and accept them as a necessary evil to maintain a society that’s not completely morally repugnant.

          • Nornagest says:

            Must be nice to live in a world where your opponents are either confused about their own preferences or straight-up evil.

          • nydwracu says:

            Northern European countries are homogeneous. They are also actually foreign.

            If you think Jante law is a good idea, you like Scandinavia and you don’t like America. But who the hell thinks Jante law is a good idea?

        • RCF says:

          You seem to be asserting two things:

          1. In Britain, the term “minimum wage” refers to some sort of basic guaranteed income; the money doesn’t necessarily come from the employers but instead may come from the state.

          2. Given that a country different from country of the host, the majority of the readers, and the country that is quite obviously the intended context, uses a term differently, we should treat this other meaning as what the term means.

          I am not convinced of the first point, and the second is absurd. We should not have to preface every term with the phrase “the American meaning of the term”.

          • Anonymous says:

            No, he’s using the term “minimum wage” normally. He’s talking about the interaction of minimum wage and guaranteed income.

          • Tom Womack says:

            Britain has a minimum wage in the American style; this was introduced in 1998 as one of the early acts of an elected-by-landslide Labour government.

            Britain also has a substantial system of ‘working tax credits’, meaning that a large number of Britons on the minimum wage also get their income topped up by the state, and a substantial system of ‘housing benefit’ which means it’s not remotely true that the minimum wage somewhere must be greater than 3x the local market rent. I don’t know the shape of the equivalent system in the USA.

            The existence of these working tax credits, which have a terrifyingly high marginal withdrawal rate (get £1 more from your employer and lose 80p you would have got from the State) means that changes in minimum wage have a smaller effect on peoples’ post-benefits-income than you would expect. Similarly it defuses ‘but you can’t raise a family on minimum wage’ arguments somewhat, since you’re raising a family on minimum wage plus a set of state benefits designed precisely to make it possible to raise a family on minimum wage.

          • Jadagul says:

            The US has a similar system, with the moderately-horrifying property that it’s entirely possible to have a marginal withdrawal rate of more than 100%, and while I’m not sure whether it’s “common,” it’s not very hard to find people who actually can’t afford to accept raises and promotions.

      • Murphy says:

        >Some people really really want there to be a price floor for wages, and find it rhetorically inconvenient if that price floor reduces demand.

        Absolutely true but remember, there’s also people who really really really want to pay their employees less because they’ll personally benefit massively in the short term but it’s rhetorically inconvenient for them if there’s any negative effects to the rest of the economy and population.

        You’re committing the sin you describe in your first paragraph, only looking at the first effects.

        • Salem says:

          No, I’m not committing that sin. Sure, there are people who would like all kinds of things to be true, but there’s also a fact-of-the-matter. It so happens that reality favours the rhetorical convenience of people who oppose price floors.

          • Susebron says:

            …which is exactly what anyone on that side would say. Seriously, I can say “reality has a liberal bias”, and you can say “reality favors the rhetorical convenience of people who oppose price floors”, and it’s exactly the same argument and it won’t convince anyone who doesn’t already agree with you. Saying “my opponents find reality rhetorically inconvenient, and want to muddy the waters” is nonpartisan. It works from either side. And, as this post demonstrates, there will be lots of evidence for both sides, so reality’s actual bias is very, very hard to determine in this sort of case.

            Edit: Also, Jaskologist’s post here.

          • Markus Ramikin says:

            Hm, I’m trying to reply to Susebron’s reply below this post, but there’s no reply button on it, even though there is one one every parent post. Does anyone know why?

          • suntzuanime says:

            You’ve reached the nesting depth limit. All you can do is reply to the parent.

          • Salem says:

            …which is exactly what anyone on that side would say. Seriously, I can say “reality has a liberal bias”, and you can say “reality favors the rhetorical convenience of people who oppose price floors”, and it’s exactly the same argument and it won’t convince anyone who doesn’t already agree with you.

            I agree that my saying “reality favours the rhetorical convenience of people who oppose price floors” isn’t itself an argument for why price floors generally, or the minimum wage in particular, are a bad thing. If you want arguments for that, read Alex Godofsky’s excellent comment above, or the post I linked by Bryan Caplan.

            My point is a more general one. On this issue, one group is saying “Sure, evidence is hard to come by on this particular question, but look at the general rule.” And the other group is saying “Evidence is hard to come by on this particular question, so let’s just do what our agenda was anyway.” The second group is obviously in a state of epistemic sin. And I view this as par for the course, and aided and abetted by the kind of untethered utilitarianism and “category-dissolving” favoured by our host.

          • Markus Ramikin says:

            suntsuanime: thanks. I feel like an internet noob now.

            Oh well, Salem made pretty much the same response I would have.

          • Susebron says:

            One side is saying “Evidence is hard to come by on this particular question, so let’s just do what our agenda was anyway.” The other side is saying “Evidence is hard to come by on this particular question, so let’s just do what our agenda was anyway.” Both sides have general rules on their side: price floors cost money, yes, so companies will try to save money, by cutting jobs or by other means. On the other hand, people who get paid less than they need to live are obviously more likely to quit, and turnover costs money which companies will try to save by cutting jobs or by other means. The question is which effect is greater, which is a matter of hard-to-reliably-gather evidence.

          • Samuel Skinner says:

            ” On the other hand, people who get paid less than they need to live are obviously more likely to quit, and turnover costs money which companies will try to save by cutting jobs or by other means.”

            The liberal position is companies won’t lower wages because the current wages are effective? I think you need a better argument- both of your positions support the conservatives.

          • Susebron says:

            The second favors a minimum wage, and perhaps even a higher minimum wage, to reduce turnover. The first favors no minimum wage, to directly reduce costs.

            I don’t think that the liberal argument is that companies won’t lower wages. I think the argument is that companies will lower wages unless we prevent them from doing so with a minimum wage.

          • Samuel Skinner says:

            Well, if you have to toss out “companies like making money” to justify a position I’m not sure how good of an argument it is.

          • Susebron says:

            Which one? The turnover one, or the lowering wages one? The turnover one states that turnover costs money, which companies like to save. The lowering wages one says that people in general will, prefer a smaller short-term benefit to a larger long-term one. Lowering wages will (according to the argument) provide immediate savings, but eventually cost more money than they save. However, humans are not perfectly rational, and time discounting definitely exists.

          • Samuel Skinner says:

            Which is why of course companies never invest in things that don’t pay off immediately.

            Companies already pay employees more than the clearing price in order to insure loyalty and reduce turnover (depends on the field and time of course). Why would they suddenly stop doing this? Almost no one is currently paid at the minimum wage for exactly that reason.

        • Samuel Skinner says:

          The US had a time period were it didn’t have the minimum wage; shockingly enough real wages increased during that time period. Eliminating the minimum wage won’t give employers the ability to work their employees for nothing, anymore than it did during the Gilded Age (when “work your employees for nothing” was a lot more socially acceptable).

          • Deiseach says:

            Eliminating the minimum wage won’t give employers the ability to work their employees for nothing, anymore than it did during the Gilded Age (when “work your employees for nothing” was a lot more socially acceptable).

            Isn’t the socially acceptable way to do that nowadays “internships”? Hey, come work for us for no wages but we’ll say it’s the chance to get your foot in the door of the industry. Of course, we may not bother giving you any actual training or any skills, and as soon as your six month contract is up we’ll get another intern in, but if you want an entry-level position (which is the only way you’ll get a start), then you’d better take this!

          • Lambert says:

            Are the Guilds really gone?

          • Nornagest says:

            The medical and legal professions are probably the closest thing we have to guilds these days, but even without full-blown guild structure people are quite capable of working toward guild interests.

            Exhibit A: Uber vs. taxi drivers.

          • Julie K says:

            “Isn’t the socially acceptable way to do that nowadays “internships”?”

            Or “adjunct faculty” (non-tenure-track).

            I see it as yet another example of supply and demand- if you have a large supply of people who want to work in a particular profession, it’s going to drive down compensation.

      • Nornagest says:

        Shame they ruined that gorgeous landscaping by slapping those ugly-ass buildings on top of it.

      • ADifferentAnonymous says:

        But surely you have to be able to reject category-based arguments sometimes, right? Otherwise, how do you resist “The minimum wage is a law, and laws are essentially for the protection of private property, so the minimum wage is essential for the protection of private property! All you capitalists should love the minimum wage!” and other such nonsense? Or, for that matter, the examples in the WAitW post?

        • Salem says:

          But surely you have to be able to reject category-based arguments sometimes, right? Otherwise, how do you resist “The minimum wage is a law, and laws are essentially for the protection of private property, so the minimum wage is essential for the protection of private property! All you capitalists should love the minimum wage!” and other such nonsense?

          Obviously, any given category-based argument may be invalid. For example, “a fish is a mammal, therefore it must have udders” is based on categories, but it’s stupid reasoning, because a fish doesn’t fall into the mammal category, and not all mammals have udders. In your example, it’s not true that any imaginable law is essential to maintain private property, so your argument fails. Or do I misunderstand your categories?

          Or, for that matter, the examples in the WAitW post?

          1. “Martin Luther King was a criminal!”

          I am not American. Here, King is mostly known among people of my generation for “I Have A Dream.” His civil disobedience is not nearly as well-known, so this would probably be entering a new fact into the conversation. Perhaps this is different in America where King is a more central figure. I am therefore not in a position to judge this argument.

          2. “Taxation is theft!”

          Excellent argument. It points out that the government has no right to your money, and needs to overcome a heavy moral burden to be able to take it. That doesn’t mean all taxation is illegitimate – if it’s ok to steal a loaf of bread to feed a starving child, then surely there are cases where it’s ok to tax. But we are now viewing taxes from the correct moral and economic standpoint, rather than just undifferentiated “revenue” that springs from Zeus’ forehead into the Exchequer.

          3. “Abortion is murder!”

          Good argument against “we want to live in a world where women have reproductive control.” Bad (conclusory) argument against “abortion is self-defence.”

          4. “Capital punishment is murder!”

          Good argument against “we need to incentivise people not to kill.” Bad (conclusory) argument against “people who commit very severe crimes forfeit their rights.”

          5. “Euthanasia is murder”

          I don’t want to trigger our host.

          6. “Affirmative action is a form of racial discrimination!”

          I have never heard this argument in those words in the wild. But if slightly rewritten to “racial entitlements for blacks are bad for the same reasons as are racial entitlements for whites” then this becomes a very widespread, and in my view unanswerable, argument.

          7. “That’s racist!”

          Can be a good or bad argument. More context needed.

          This comment is getting super-long, so I’ll call it quits here, but I think you get the gist.

      • Glen Raphael says:

        besides, many minimum wage studies seem deliberately designed to find no effect

        It’s worse than that: most of the minimum wage increase laws are themselves deliberately designed to make the effect hard to spot, almost regardless of how one does the studies. So it’s no surprise that studies fail to find an effect.

        Basic economic analysis tells us raising the minimum wage will make things worse for people employed near the minimum but it doesn’t say how or when or by how much. Pretty much everybody – no matter how liberal they are – knows intuitively that if you raised the minimum wage to $100/hour and did this tomorrow with no prior warning it would immediately wreck the economy, drive most employment underground, and render half the country legally unemployable. So minimum wage advocates don’t suggest laws that do that. The ideal minimum wage law in terms of making the effect obvious and easy to spot in studies would have these attributes:

        – The wage increases by a large proportion
        – It takes effect instantly in one big jump (no stages)
        – The increase is not debated for years before it passes

        But people who want to claim credit for “raising the minimum wage” don’t want to be blamed for the obvious and predictable result of raising it too much or too fast, so the usual minimum wage law does the opposite of all these things. The politically ideal minimum wage law has these attributes:

        – Each increase is small as a percentage
        – It gets phased in gradually over years
        – After years of discussion.

        If the increase is large enough, fast enough, and surprising enough for the predictable negative results to be easily spotted, that means the legislators screwed up – the next one won’t be.

        The vast majority of minimum wage increases enacted in the US are sufficiently gentle by design that we shouldn’t expect even smart well-constructed studies to be able to tease out much of an effect size that rises above background noise.

        • 27chaos says:

          10/10 cynicism, you earned a black star: ★

          • Glen Raphael says:

            I did actually think of a counterexample after I posted. Which is: In 2007 congress tried to inflict the US minimum wage on American Samoa and the Mariana Islands. The increase was in stages – the plan was to increase only 50 cents a year – but this was sufficient to nearly destroy a few prominent local industries (tuna canning, garment manufacture, tourism) and the planned increases had to be put on hold after only 3 or 4 of the 50-cent bumps had taken effect. Total employment fell by 45 percent in the Mariana Islands from 2006-2012; total wages paid declined in both places over that period.

            If we actually listened to the people wanting to DOUBLE the minimum wage that’s what would happen here too. (Fortunately, the calls for that magnitude of change are a negotiating tactic rather than a plan; wiser heads generally prevail.)

            (here’s the relevant GAO report: http://www.gao.gov/products/GAO-14-381 )

          • E. Harding says:

            @Glen
            -Problem is, a recession also took place during that time, making the independent effects harder to spot.

        • Dude Man says:

          This seems to assume that anything that is bad in large doses is therefore bad in small doses as well. It is entirely plausible (though I cannot say whether or not it is true) that the benefits of raising the minimum wage from $7/hour to $8/hour next year outweighs the costs while acknowledging that raising the minimum wage to $100/hour tomorrow would be bad. After all, drinking 5 gallons of water in 30 seconds might kill you, but drinking a glass of water at some point is good.

          • Matthew says:

            One notable difference, aside from simple scale, is that raising the minimum wage by $85 would be a large increase in the real wage level. Increasing the minimum wage by a couple of dollars every decade or so, however, is an increase in the nominal wage level but doesn’t actually keep up with long-term inflation trends, so real wages are not actually rising.

          • Glen Raphael says:

            Dude Man: Sure, but if people are claiming this action is good in small doses but bad in large doses, they ought to be able to explain *why*. What model of the world produces this result? If raising the wage is good only up to a point, how do we know what that point is? How do we know that we’re not already past that point? Should we just keep raising it until the economy collapses, then back off a bit?

            I myself have not been able to come up with a model which says raising the mandatory minimum salary is beneficial for people who earn a little which wouldn’t just as strongly imply it’d be beneficial for people who earn a lot. I think if people really thought through the reasons that we all know a $50/hour minimum wage would hurt people who currently earn $45/hour, they’d have to conclude just as strongly that a $10/hour minimum wage hurts people who now earn $9.

          • llamathatducks says:

            Glen Raphael:

            I’m not an economist, but my intuitive understanding is that a $50/hour minimum wage would primarily hurt people who currently make $8/hour, not people who currently make $45/hour. I assume that the reason a $50/hour minimum wage would be bad is that employers of low-wage workers would suddenly have to severely increase their spending on their employees, probably more than they’re willing to. Whereas employers who already pay their workers $45/hour may find the newly required wage more palatable since it’s not as big a difference from what they’re already profitably doing.

            In this case the reason a small minimum wage increase could be good while a large minimum wage increase would be terrible is fairly clear: because a small minimum wage increase wouldn’t result in a huge increase in spending for any employers. (I mean proportional to their size & profit margin: Walmart would have to add a pretty substantial amount to their payroll budget, but it wouldn’t be a huge amount compared to their profits.)

        • pocketjacks says:

          This is in reply to Salem and Glen Raphael.

          If a “gentle” increase in the minimum wage doesn’t produce any noticeable effects, that sounds an awful like a lot like an argument for gentle increases. This sounds like such a simple question that you almost don’t want to ask, but if the effects are so hidden, how do you even know they exist? I concur with Will; as someone who works in finance, I can say that no one uses simplistic supply/demand models past Econ 101 for widgets or uber-liquid financial securities, let alone the labor market. If your theory doesn’t line up with the evidence, that’s a problem with the theory, not the evidence. Trying to couch your theory with excuses such that it never needs to be accountable to empirical evidence renders it a religion, not a theory or a “model”.

          This “unannounced” business is foolish. Announcing significant changes in policy, or being deliberative, are not somehow unique to minimum wage policy, be it for government or any other organization. Any other policy that ever has to be judged empirically goes through the same process. Having to stack the deck this much to finally be able to glean the results you want is a sign that one is running out of straws.

          Generally, yes, the economy is complex. And economics like all social sciences suffers from the drawback of having no feasible or humane ways to run clean experimental trials. That’s not an excuse to not try to find as much empirical evidence as you can and not just defer to a particular partisan theory.

          (1) Saying that these studies were “deliberately designed” to find no effect is an accusation of academic fraud. Do you have evidence of this besides that you don’t like the results? Or that their alleged bias is more than those of the studies that found the opposite results?

          (2) The best way I can explain why “if a big change is bad, then a little change in the same direction must also be bad, just the other side is tenting their fingers and being deceptive”, is…

          You want tighter monetary policy? Fine, raise the benchmark rate to 7% tomorrow with no announcement and no gradual escalations. If this ends up disastrous and spinning us into another recession, that proves that pump-priming for forever after is the correct policy.

          You want to privatize education? Fine, close schools down starting tomorrow, with no prior announcement. No transitioning, no public support of charter schools, no “vouchers”. And you have to own the results.

          You want to lower taxes to nothing but what your particular partisan sect considers the essentials? Fine, cut them down to that tomorrow with no announcement. If this ends up being a budgetary disaster, we become Sweden, no takebacks.

          You want a company to lay off some workers? Why not lay off ALL the workers? Should a company renegotiate pensions downward? Why not erase all accumulated all pensions overnight with no announcement? The former is obviously exactly the same thing, just with political deception.

          You want to privatize Social Security? Fine… you get the idea.

          Going back to the interest rate example, which I think is the most directly analogous, that raising rates to 7% would be bad is not an argument against raising by 25 basis points now. Not only can the latter be genuinely not as harmful as the former without invoking conspiracies of deception, the latter may be not as harmful as keeping it where it is now, or lower it.

          Or, just use the water example that the other guy used.

          (3) Similarly, “if we can agree that at some point along the spectrum of the application of this rule, we’ll start seeing bad effects, how would we know when to stop?” can be said for basically any rule in the history of rules. That’s not an excuse to err on your particular partisan side, let alone for not having the rule at all.

          How do know how much jail time is “enough” jail time for a particular crime? We can’t know that? The very question is kind of dumb, because the question should be effective vs. ineffective with regards to our policy goals and not some silly Platonic ideal? I don’t think this line of reasoning justifies not punishing that crime at all.

          (4) And finally, you’re not doing your side any favors by implicitly admitting that a minimum wage hike in the range of what is currently being discussed (up to $9-10 an hour for the federal rate) would have “hidden” effects that apparently can never be verified. You’d be much better off saying that it would have clearly empirically verifiable negative effects.

          • Glen Raphael says:

            pocketjacks: “unannounced” was shorthand; I should have made it clearer why making non-immediate changes after long debate screws up the accountability. To wit: A typical “designed to find no effect” study does something that seems to make intuitive sense: Examine employment levels and trends just prior to the new law taking effect and compare that with, say, a year or two after the law takes effect. Then tweak to “adjust for” various factors and assume whatever change remains might be caused by the legal change. With me so far?

            But now suppose this legal change is truly disruptive to some business model. Smart businesses can SEE the change coming down the pipeline. Knowing that a minwage hike is coming and that explicitly firing people is bad PR and produces legal liability, smart businesses are likely to shed workers via attrition, slowing down new hiring, and moving people from full-time to part-time well in advance of the law actually taking effect. They’ll start doing it while it’s being debated or after the law passes but before whichever January 1st the first step change kicks in.

            This means that the “before” point in our study already has job losses and negative employment trends baked in. We should then EXPECT to see little difference between the “before” state and the “after” state.

            I’m not actually saying you can’t spot the effect of the law. I’m saying you need a GOOD study, one that is SERIOUSLY LOOKING FOR a negative employment effect, in order to find one. Somebody who doesn’t really deep down EXPECT to see any job losses due to the law will do that naive comparison of a month before to a year or two later, see little difference, and declare their job done. In so doing, they are missing the effect minwage has on two categories of employer: especially smart ones that shed workers early (before our “before” point) and especially dumb ones that shed workers very late (after our “after” point, having lost a lot of money).

          • lupis42 says:

            “If a “gentle” increase in the minimum wage doesn’t produce any noticeable effects, that sounds an awful like a lot like an argument for gentle increases. This sounds like such a simple question that you almost don’t want to ask, but if the effects are so hidden, how do you even know they exist?”

            I think it’s safe to say that we know there are effects, but we don’t know what they are. Suppose you take a random pill from my medicine cabinet tomorrow morning. You don’t feel any different, but you wouldn’t assume that it had no effect, just that the effect was not large enough to be measured, given all of the other variables.

            The concern with the suddenness of the increase is related to trying to isolate the direct effects of the minimum wage from the long run behavior of an economy that has adapted to the minimum wage – because with months or years to plan ahead, firms have lots of time to do things like not replacing departing workers, shifting the workforce composition towards more skilled/productive workers, and gradually rolling out automation. Those changes probably wouldn’t show up against the background of turnover, but they all represent reduction in demand for the sort of low-skill labor that would have been employed by that company absent the minimum wage. A new business might rethink it’s approach before opening it’s doors at all.

            Let me give you a hypothetical: suppose that the US had never before had a minimum wage. Do you think the business models of both Costco and Wal-Mart would be the same, or do you think the mix of skilled and unskilled labor that each uses might change? Do you think the wages at either company would change?

            It seems pretty ridiculous to claim that there would be no effect, but there is no meaningful way to measure that effect. As a practical matter, there’s no way to know what other business models might have evolved, in the same way that, had the internet not come along, we wouldn’t have any idea what business models might have proved successful, but it seems safe to assume it would be something other than what we saw in a world with not internet.

            This is not an argument that the minimum wage is bad per se, but that whether the minimum wage is bad is something that cannot be settled empirically. We can make assumptions from models, but there are no long term experiments.

          • Alex Godofsky says:

            If a “gentle” increase in the minimum wage doesn’t produce any noticeable effects, that sounds an awful like a lot like an argument for gentle increases.

            Because the sum of a thousand unnoticeable effects can be very, very noticeable indeed.

            This sounds like such a simple question that you almost don’t want to ask, but if the effects are so hidden, how do you even know they exist?

            Because we have a theory

            I concur with Will; as someone who works in finance, I can say that no one uses simplistic supply/demand models past Econ 101 for widgets or uber-liquid financial securities, let alone the labor market.

            You are trying to make much more precise and quantitative predictions. We are trying to answer the question “what is the sign?”

            It’s also somewhat disingenuous to compare financial models to supply/demand, IMO.

            If your theory doesn’t line up with the evidence, that’s a problem with the theory, not the evidence. Trying to couch your theory with excuses such that it never needs to be accountable to empirical evidence renders it a religion, not a theory or a “model”.

            It’s fake evidence. My theory predicts their result. The theory is falsifiable – you could, for example, show imposed price floors causing increased demand in many diverse markets, and show that there were very few markets where such floors reduced demand – but these studies aren’t moving the needle.

        • Avi says:

          You could be less cynical and say that they actually don’t want to wreck the economy, rather than that they don’t want to be blamed for wrecking it.

  54. suntzuanime says:

    What counts as “obviously biased” is going to depend heavily on your priors (hence the joke “reality has a well known liberal bias”). Basically if you’re saying “do not trust websites which are obviously biased” what you’re really saying is “do not trust websites you don’t agree with” or “do not trust websites that give you new perspectives”. Which may be correct, but I’m not sure you’ve thought through the implicit existential horror of your recommendations.

    I guess the happiest way to read it is: don’t trust anyone who’s obviously biased; everyone is obviously biased; don’t trust anyone; constant vigilance.

    • Scott Alexander says:

      By “biased”, I mean something more like “politicized” or “pushing one side”.

      I think there’s much more agreement about whether Rush Limbaugh is a politicized figure than there is about whether Rush Limbaugh is correct about things.

      (I agree that there is not perfect agreement on what is or isn’t biased in this sense)

      • suntzuanime says:

        I think the people who would say Rush Limbaugh is correct about things would say that Rush Limbaugh is speaking common sense, as contrasted with the leftist political narrative pushed by the mainstream media.

        But I suppose there is a sense in which it is reasonable to say “don’t trust anyone who isn’t hanging out near the middle of the Overton Window, even if they seem to have evidence to back up their claims”.

        • gattsuru says:

          Perhaps it’s just the conservatives I talk to, since they may not be representative of the general populace, but most simultaneously believe that the mainstream media pushes a leftist political narrative and that the common sense Rush pushes it still laughably biased.

          There’s even an obvious explanation for this: very few conservatives match up to political practices of any high-profile conservative. They can’t treat him as a platonic form of common sense when they disagree with him on goldbuggery, or the role of police in a modern society, or same-sex marriage, or foreign policy.

        • Liskantope says:

          I think (at least part of) what Scott is referring to by “politicized” or “pushing one side” is the established context and origin of the voice pushing. Are they part of some sort of an organization or special interest group whose purpose is to support one side? Rush Limbaugh fits this description at least in some sense: Fox News operates on the premise that it is trying to shift an overall skewed-to-the-left media somewhat further right. This is agreed upon by those on the right who actually think that Fox’s spin on things is a common-sense corrective to the left-wing bias of the mainstream media, as well as those on the left who think the mainstream media’s presentations tend to reflect unbiased reality.

        • haishan says:

          But I suppose there is a sense in which it is reasonable to say “don’t trust anyone who isn’t hanging out near the middle of the Overton Window, even if they seem to have evidence to back up their claims”.

          This also seems horrifying to me, because what happens if “the truth” happens to lie well outside the Overton window? Hypothetically, let’s suppose that there is an actual genetic penalty to intelligence for people of African descent (and a smaller penalty to people of European descent, and no penalty to people of East Asian descent). What would happen to a broadly centrist political figure — let’s say Michael Bloomberg — if he starting espousing that position? All of a sudden he wouldn’t be a centrist, he’d be on the far right. This is the case even though (in our hypothetical!) everything he’s saying is true. And so everyone who isn’t already sympathetic to those views shakes their heads and thinks, “well, I guess old white men sometimes go a little crazy in the head,” and nobody gets convinced of anything they weren’t already.

          • suntzuanime says:

            Well sure, but like, what if that isn’t the truth? Then my policy of sticking safely within the Overton window pans out, and your policy of listening to Michael Bloomberg is really foolish. You can make any epistemic strategy seem bad by stipulating that the truth lies outside it, that’s not a real argument.

          • haishan says:

            I like to imagine there are epistemic strategies that actually tend to converge on the truth wherever it is. “Only pay attention to people in the center” very much does not. It pretty much only works if you believe society has evolved for the last 4000 years to develop towards the set of politically acceptable beliefs in 2014 America/Europe/whatever and that now that we’re there we’re gonna stay in that window forever.

          • suntzuanime says:

            Well, the center-left can be persuaded by the center-right or vice versa, and so the Overton window can creep one way or the other even if nobody pays any attention to the people on the far left or far right.

          • haishan says:

            I guess that’s true, but that’s still awful slow convergence, and the strategy could in theory — and likely will in practice — tell you to ignore valid and correct evidence for true political beliefs. Maybe you have to throw out a lot of correct evidence for true stuff to get rid of the wrong evidence for false stuff; I certainly hope that’s not the case.

            I’ve updated my position to: this strategy works, but it still seems terrible. It’s the bubblesort of politics.

          • Jaskologist says:

            We did it! We related this back to race in under 100 comments! I knew we could, but I really thought it would be something more along these lines.

          • Banananon says:

            FWIW my mathematical intuition is the strategey probably has better convergence in practice than in theory.

      • lupis42 says:

        I think the easiest version is “has a conclusion.” Whatever the conclusion might be, whether it’s stated or implicit, if the source is trying to convince you of something, it should be assumed that it is biased.

        Of course, this will lead us all to descend slowly into a writhing pit of self reinforcing priors.

        • suntzuanime says:

          A similar thought: https://twitter.com/admittedlyhuman/status/520897097965772800

          But I no longer endorse it, because it can be gamed, and honest people shouldn’t feel compelled to avoid concluding things in order to try to produce an easily-faked signal of honesty.

          • lupis42 says:

            I’m not sure if “doesn’t support any conclusion” can be faked while retaining value to the faker – after all, the whole point of faking is to drive people towards your preferred conclusion. Not *stating* your conclusion is easy. Not allowing your conclusion to be perceptible in your statement is much harder.

            I don’t actually endorse the position, because (as I tried to say pithily) it would follow that you should only update your priors in response to direct tests you perform, which is false.

          • FullMeta_Rationalist says:

            I think the issue is starting from a conclusion rather than starting with a question. I feel like the best analyses take the reader on a tour of the author’s thought processes as if they were happening in real time. The conclusion gathers at the end of the article as a natural product of reason, like how a lake just happens to exist where gravity naturally guides the river. But the question is the source, so that’s where the reader should begin. To lay out the conclusion up front is putting the cart before the horse, a solution in search of a problem. And I suspect the style the author chooses will often reflect the honesty of their inquiry.

    • Jaskologist says:

      I’ve long taken the opposite approach: go straight for the biased sources, just make sure to look at some on both sides. “Non-biased” sources either are putting up a front of false neutrality, or just don’t recognize their biases, which is the much worse problem of not knowing what they don’t know.

      The biased ones will engage the crap out of the other sides’ arguments and try their hardest to poke hole in them. Even if that doesn’t end up being more informative, it is at least more entertaining.

    • 27chaos says:

      A less demanding substitute for constant vigilance is finding extremely smart people whose opinions vary wildly and trying to adopt their opinions. Still a fair amount of effort, but easier.

    • Anonymous says:

      He said: “Yeah, Suz, you’re totes right”

      She said: “Suz, you’re an eedjit, you don’t know what your taking about”

      She was right.

  55. Why would you expect the funnel plot to be symmetrical? Especially considering that you’re mixing up different types of studies? Maybe the studies that look at unemployment statistics have on bell curve, the ones that look at employment statistics have another, the ones that look at new job postings have another, the ones that look at specific industries have another…. The overall mix will not, in general, be a new bell curve.

    (I think the main limitation of the original Card & Kruger study is not in the statistical details, but that they focused on a single industry, which may arguably benefit from minimum wage hikes affecting their competion more negatively then themselves: fast food, which while it relies on minimum wage labour, is more labour effective than its competion, cheap full-service restaurants.)

    • david says:

      Yes, it will. The sum of normal distributions is a normal distribution.

      There are of course reasons to expect an asymmetrical bell curve (the simple way to hack funnel plots is to game your choice of axis – here, for example, focusing on the elasticity of labour demand favours certain labour market models above others). But “adding up” different normal distributions is not one of them.

      • A mixture of normals is not normal.

        • g says:

          It may be worth making explicit what (I think) is going on here.

          Fact 1. If X and Y are normally distributed, then X+Y is also normally distributed.

          Fact 2. If X and Y are normally distributed, then “X with probability p, else Y” is usually not normally distributed.

          The sum of normal random variables is normal. A mixture of normal distributions is not normal.

          There isn’t really any such thing as a “sum of normal distributions” in the usual terminology, so I don’t know exactly what david had in mind — but I guess maybe he means a sum of normally distributed random things. Which is normal but doesn’t have anything to do with what happens when you have a bunch of different normally-distributed things and mix them all up, which is what Luis is talking about.

          • Decius says:

            How is p*X, where X is a normal distribution, not also a normal distribution?

            How is (1-p)*Y, where Y is a normal distribution, not also a normal distribution?

            “X with probability p, else Y” is the same distribution as pX+(1-p)y.

          • Anonymous says:

            > “X with probability p, else Y” is the same distribution as pX+(1-p)y.

            No: let X be normally distributed with mean -1000 and standard deviation 1, and Y be normally distributed with mean +1000 and standard deviation 1. Then “X with probability 0.5, else Y” is not a normal distribution but a bimodal one with two widely separated peaks.

          • Meh says:

            With probability p, flip a coin, and with 1-p roll a die.

            The result is a *mixture* of H,T,1,2,3,4,5,6 not a sum of any of them.

          • RCF says:

            More succinctly, if X and Y are normally distributed variables, the distribution of their sum is normal, but the sum of their distributions is not.

            P(x+y = t) is normal, but P(x = t) + P(y = t) is not.

            The problem is that given X and Y, there are two very different meanings of their “sum”: given a particular x and a particular y, one can first add x and y and the find the probability of that value, or one can first find the probability of x and the probability of y, and then add those two values. The former is normal, but the latter is not.

            One has to be careful in discussing generalization of operators, as there can be more than one generalization applicable in a particular situation. For instance, Laplace transforms are linear in the sense that L(f+g) = L(f)+L(g), but that does not mean that L(f(a+b)) = L(f(a)) + L(f(b)).

          • MTL says:

            Let X be normal. Let Y = -X. Then Y is also normal. Let Z = X+Y =0 (with probability 1). Do you still think the sum of normals has to be normal? Some one really needs to use the word independent in this discussion if they want any credibility.

          • Anonymous says:

            MTL, the constant random variable is normal. You can weaken sum of independent normals to: marginal of a multi-dimensional normal.

        • US says:

          (This is an interesting discussion, so I thought I should add a few details because I recently read a text on related topics.)

          “A mixture of normals is not normal.”

          One thing not mentioned yet here but which might be worth mentioning in this context is what whereas this is true, the *form* of a normal mixture may well be unaffected by mixing (I refer here to David Bartholomew’s text Unobserved Variables). This is actually important in the context of evaluating whether a statistical analysis is suffering from a missing variables problem because: “Any given normal distribution could have arisen naturally or be the result of normal mixing”. It’s obvious that mixing normals will sometimes lead to something that doesn’t look particularly normal, but the flip-side of that coin is that if you do have a normal distribution, you can’t know if that distribution is a mixture or not just from looking at it.

          Figuring out whether a distribution is a mixture or not is also really hard even if you don’t limit the analysis to ‘how the distribution looks like’: “even a two-component mixture of normals [has] 5 unknown parameters. As further components are added the estimation problems become formidable. If there are many components, separation may be difficult or impossible.” (Bartholomew refers to Titterington et al. (1985) for more on this topic).

          • Tom Womack says:

            It’s quite instructive to generate some data from a mixture of normals and then try to figure out the parameters from the data, particularly if you also add some other source of noise to the generation.

            Example: repeatedly: flip a coin; if true generate X from N(0,1); if false generate X from N(0.2,1). X is the actual brightness of the light; take a sample from Poisson with mean X to reflect the fact that you can’t measure light without shot noise.

            The naive ways to fit involve looking at very high moments of the data, which are impossibly noisy (in the very strong sense that the sample size required to get adequate power is larger than the population)

      • DanielLC says:

        Student’s T distribution is basically a sum of normal distributions with different standard deviations. It’s not skewed, but that’s because they all have the same mean. If you change that too, you can get any distribution you want by adding together normal distributions.

        • eggo says:

          I was going to say, this comment chain gave me a headache. I’ve SEEN sums of normal distributions that aren’t normal.
          It’s always weird to watch people question things like that, because you wonder if they’re right, and you’re actually crazy/stupid.

          • Setsize says:

            Terminological confusion is in play here. You don’t “add” distributions. You add random variables. If A and B are normally distributed independent random variables, then A+B is a normally distributed RV. If this were false, the central limit theorem would be false.

            There are ways you can combine RVs that work out to linearly combining the cumulative distribution functions or density functions of their distributions. But these operations are not “adding” the random variables.