I read the original article that tries to draw this conclusion, and I'm more than a little disappointed in the quality of the methodology. This was a pilot study at best. There are problems with sample selection, sample size, data collection, and data analysis -- basically all aspects of the study.

The coffee shops were admittedly not selected at random and there were only eight of them. I'm guessing that there were no power calculations conducted beforehand to determine adequate sample size, either of number of coffee shops or number of customers. (It certainly wasn't mentioned.) While customer selection was somewhat random, I would have liked to see that explored and verified a bit more as well.

There was no mention of inter-rater reliability being conducted between the enumerators. Also not included was the race or gender of the enumerators. It's entirely possible that they may have had a predetermined desired outcome in mind, and were unconsciously (or consciously) adding some lag time to the stopwatch times. Of course, we can't know this, because it was not reported or analyzed.

There's likely to be a common server behavior within a coffee shop, but an actual "which-coffee shop" variable was not included. An even better design would have been to model the data hierarchically within the coffee shops; clearly that wasn't done either. Also not reported was what sort of regression model was used to build the six(!) separate regression models. I presume a plain, old linear regression model, but there's no mention, also no mention if the appropriate assumptions were met. Also, why six models? Of course if you do enough models, some results will be significant at the alpha = 0.05 level purely by chance. Why not just a simple backwards regression method so you can get a nice parsimonious model, plus keep in any variables that the researcher feels to be 'clinically' relevant?

I'm deeply disappointed in this study. I'm even more deeply disappointed in Mr. Harford for taking it this seriously. It's an interesting pilot study at best, it could easily be used to design a much, much better study, but it in no way warrants this kind of attention.

The original research article is from the Middlebury College Working Paper Series. As an undergraduate research project in empirical studies, it is fine. As a pilot study, it is also interesting. But that is all it it, so no reason to be disappointed in it. For Harford to give it credence beyond that, though, is just silly.

Very well said. The method on this paper was shoddy. I'm disappointed it was taken so seriously. At most, it comments on a few coffee shops in Boston. In no way can this information be generalized to a broader population.

The author listed on the website is an associate Professor of Economics at Middlebury...

<link>

On the last few pages of the paper. There are a few outliers in the female arm, with disproportionately long wait times. If you throw these data out, the bar graph/curves are pretty similar, according to eyeball program version 1.0. Account for the outlying data--or use a different measure of comparison, like medians.

That's a good point about using the medians instead, Stop-truth-decay, although I wouldn't necessarily advocate tossing outliers without a closer look at the raw data. It's possible that those outliers might come from one particular coffee house or one particular enumerator and might be driving the whole alleged significant difference. (But again, we'll never know.)

Yes, I am aware that one of the authors is an associate professor. In fact, the paper clearly states that the "enumerators" were the professor and five students:

Six enumerators composed of five studets and an instructor visited eight coffee shops in the central Boston area over the course of two weekdays in January 2007.

The paper also states the data were collected as part of a class project.

Data for this project wre collected as part of an undergraduate seminar on empirical studies of discrimination.

My whole point is that given the source (an undergraduate seminar paper on empirical studies), little credence should be given to this paper. As an introduction to empirical methods it is fine. As a "working paper," it is fine. But that is all it is.

What I find mind boggling is that Harford actually took the time to write an article on this paper. What I find equally mind boggling is the people arguing about the merits (or lack thereof) of the original paper. Let me repeat: it is a undergraduate seminar paper!

For anyone that has ever done empirical research, published a peer review article, etc., doing "exercises" like these are quite valuable. Additionally, anyone that has ever attended college has quite likely written research papers, some far more shoddy than this. The problem is that someone actually took it seriously and Slate actually published an article on this research paper.

Well, exactly. I'm baffled, too, as to why Mr. Harford took such an interest in this particular paper. But just because it's an undergraduate piece of work does not necessarily mean that it has to be a half-baked piece of work. It frustrates me to read this because with not a lot more effort, this could have been a really neat study.

Not to mention that the lead author was an associate professor, not an undergrad, who presumably should have known better.

flora8...

Thank you so much for bringing some sanity to this discussion. So many of the other posts start with the assumption that the study is true and move on to complaining about how badly they are always treated and blah, blah, blah, whine, whine, whine.