...as did Kolata in her Times article.

As was long ago pointed out by, as I recall, Peter Medawar, in a a New York Review of Books article (I can't recall when, and I don't mean to drop names, but I want to give due credit here) this huge and impossible discrepancy between men's and women's responses calls into question the whole enterprise of self-reported sexual surveys.

It turns out that, typically, the reported number of partners is the only available check on the internal consistency of these surveys and, as Kolata only recently discovered (although it has been known for decades now), they all fail spectacularly.

How would you judge a so-called scientific methodology that consistently and obviously fails its only consistency test?

So, both Kolata and Ellenberg treat this as a
a quirk in search of an explanation, but fail to explore the deeper implication. If men and women are systematically misreporting on something as straightforward as their number of partners, why should we believe that they are telling the truth on all the other questions?

As an aside, I am not sure why Kolata wrote her article, since the "news" is many decades old and her analysis is very superficial. Nor am I sure why Ellenberg wrote this column, since the difference between the median and the mean turns out (unsurprisingly) to be unimportant. He acknowledges this fact at the very end of the column--I have heard the adage that in column writing you never give away the story in the lede, but this is ridiculous!

It may have been Richard Lewontin who wrote the book review. I met him when I was in grad school and asked him about a similar article. Of course any field that relies on truthfulness and self-reflection for its data is pure cloud-cuckoo.

"why should we believe that they are telling the truth on all the other questions?"

I'll give you an example that I believe demonstrates why we should believe SOME of the things they tell us.

Survey taker comes to YOUR door and asks you two questions.

"How long is your dick in inches"

"Do you own a computer"

Now, based on what you stated above, obviously we can't believe your response about your computer ownership. Now you can claim that the two questions are unrelated and therefore the survey is invalid - but that's only because you know more about the trivial survey than a normal person might in other situations faced with more complex questions.

This brings to light another point about the data. Yes, the data is suspect, but we don't know WHY the data is suspect. In fact we can't even really claim that it is, since we have no proof that it is - only our own unsubstantiated, but educated, suspicions. At no point can you go to the data and pick out the obvious errors. So I question why you are even attempting to do so. Bias might be one reason - you are biased against the data and/or the results and are looking for reasons to discredit the results. Experience - your own experiences lead you to alternate conclusions and this is at odds with what you are seeing in these results. But that doesn't give you the right to wholesale discard results, at most is gives you the right to demand more data or propose a better designed methodology - one that is backed up by your own data and published results.

Correct you are sir. It was Lewontin!

Thank you. (also for verifying the very existence of that long-ago article!)

The whole point is that we do know that the answers to the question on the number of sexual partners are wrong. The answers provided are simply mathematically impossible. This error is large, completely obvious, and has been widely known for decades.

The problem is that we have no such handy check on the other answers. But, we do on penis size, and I am prepared to bet that such surveys that ask about penis size generate overestimates!

It (almost, but apparently not entirely) goes without saying that such surveys do not ask questions about computers; they are, after all, studies of sexual behaviour, not computer ownership. But,, I suspect the same problem plagues surveys on height and weight, which I suspect generate, respectively, over- and underestimates. It wouldn't even surprise me if computer surveys generate overestimates of, say, the processor speed or hard disc capacity.

Of course, as a logical matter it is possible that people systematically lie about the number of sexual partners--which, to repeat for extra emphasis, we know that they do--but nonetheless tell the truth on all the other questions (condom use, say). To me, drawing such a conclusion involves, at a minimum, a leap of faith and, more realistically, wishful thinking.

I have no particular stake in the survey results themselves. Indeed, precisely because they missed the bigger issue, neither Kolata nor Ellenberg discussed any of the other urvey results. My point, to repeat for extra clarity, is that all such survey results are suspect because the surveys consistently and spectacularly fail the only real quantitative test they face.

To ensure the point is driven home to even the most obtuse (not, I hasten to add, necessarily Eigenvector) lets take a concrete example. Homosexual activity. I gather that such surveys find about 2 percent of men engage in homosexual activity. What are we to make of this number?

Well, we know for a fact that responses regarding the number of sexual partners are wholly unreliable, and in an unsurprising way: men overestimate and/or women underestimate, much as standard social norms and stereotypes would lead one to think. Even though the surveys are anonymous!

Now, knowing all this, do you think the responses on homosexual activity are underestimates, overestimates, or just about right?

I Don't think anyone really missed the point. Of course people lie on these surveys but then you just go on and try to figure out why they are lying. Humans are extremely difficult to study (ethically), but that doesn't mean we shouldn't try. Even when people lie on surveys they tend to lie for similar reasons so that can lead to more interesting, and maybe even useful, inquiry.

Cridge, I certainly agree that we should keep studying. But, I do think Kolata and Ellenberg (and, with respect, you) did miss the point.

The point is not that they lie on this question, although they do. The point is the implications for the other questions, on which we typically have no check.

Almost every day I read in the news that this or that study, based on survey results, shows this or that result. Maybe I read too much news, but it is also the case that social policy is being debated and made on the basis of such survey results.

And not just, or even typically, from surveys of sexual behaviour. Consider, for instance, surveys of drug use, or teen drinking (or, for that matter, adult drinking). Do you believe that people honestly answer questions about socially undesirable and highly illegal activity? I don't.

So, when Kolata and Ellenberg chose to treat this story as an issue of (trivial) statistics--including bringing out big guns like professors emeriti--I think it is fair to say that they fumbled the ball.

Ok,

Full disclosure

I work in sexual research and I routinely give both quantative and qualitative surveys to research participants that have questions asking for the number of sexual partners they've had as well as other highly charged questions such as drug use, condom use, etc.

To even begin to touch on ALL the major problems that can skew results even BEFORE the first question has been answered would take to much time, suffice it to say take a moment to think about how these scientists spending all their waking hours sequestered in a lab actually get people to take these tests and wonder if it's a well rounded, highly diverse sample of rich, poor, Black, White, etc. Probably not.

However setting that issue aside for a moment, any reseacher worth his or her salt knows to put "checks" into any survey asking for sensitive information from respondents to detect inconsistencies. It's one thing for a man to say "I've had 10 sexual partners in the past year" when infact he's had 6. It's another thing for a man to maintain that when strategically placed follow-up questions ask him about of partners number with which he used condoms fifteen questions later, and then how many he didn't use condoms with ten questions after that. Or swap those questions for "how many had you known for over three months" and "How many had you known for three months or less?" If the numbers are inconsistent, the data is suspect and should be thrown out.

It's all in study design.

I think we're also missing a more important point here: these surveys that ask about sexual behavior and produce mathematically impossible results about intercourse are still producing valuable and quantifiable data about societal and cultural expectations around intercourse. It would seem these studies that we're parsing over quantitatively prove a gender bias in the ways we report sexual activity. And while this may not come as a suprise, this data can be utilized to help fix an inherent skew in future results or taken as a jumping off point for other far more interesting studies. For instance since it seems that we have shown conclusively that woman reduce the number of sexual partners they claim to have on surveys while men increase them, perhaps they also lie to their sexual partners too. This could have a great impact on STD research. Why not create a questionaire that asks men to list the number of heterosexual partners their heterosexual partners have claimed to have (Woman one: Never mentioned it. Woman two: said three. Woman three: said she had slept with six guys this year etc.) And then ask women to do the same for their male partners.

First, I want to thank you, Stuckinboston, for your post.

It's curiously refreshing to hear from someone who (unlike me, in this case anyway) knows what he (or she, if that is the case) is talking about!

I have noticed that questionnaires turn to the same issue over and over again, and did realize that this was, in part anyway, to check for consistency.

As you say, if the data are "suspect", the results should be discarded. It seems that in these studies such checks were not sufficient to weed out mathematically very impossible results. This, it seems to me, renders the results very suspect, and not just on this one point.

So, my question to you is: Should the results of these studies be thrown out? If not, why not?

Your suggestion at the end, to match up the answers partner by partner, would seem to add additional checks, though I am no expert in survey design (well, what I mean is that I know nothing at all about survey design). Suppose the answers turn out to be inconsistent, as I suspect they will be. Do you throw out the study?

Finally, although I have carped at great length on this issue, I have to confess I have no real solution. After all, if you had external checks on the answers--you could somehow observe the number of sexual partners, for example--then you wouldn't need the survey.

This is the situation with some economic data. We do not have to ask people how much milk they bought and at what price, because we can observe both prices and quantities of milk transactions. This can provide checks, too. For example there is a discrepancy, in the obvious direction, of the number of cigarettes people claim on surveys to smoke and the number of cigarettes actually sold. It is not the case with other markets, like for illegal drugs, where there are no reliable market data (from trade groups or tax receipts, say).

Thank you lloyd667,

I will start this post by saying that while I heartily embrace the laurels you bestow I must make clear that I am familiar with but by no means an expert in the field of statistics or quantitative analysis. I currently coordinate a research study that is qualitative, whose questionnaires I helped write and currently implement. If we get someone who’s answers in the “check” questions are incredibly inconsistent it’s a red flag and we don’t use the data. There are three important things to keep in mind though: just because some people lied, doesn't mean everyone in the study lied; just because someone's answers are inconsistent doesn't mean the data overall is invalid; and just because someone’s data is inconsistent doesn’t mean they lied (knowing altered the truth). I looked at the CDC study and here is a potential explanation for the discrepancies: first it should be pointed out the CDC study claims sex and drug related questions were posed using ACASI, basically a computer alone in a room with the respondent. As the introduction to the study explains, “Prior to 1999, many of these same questions were asked in face-to-face private interviews. Studies have shown that the self-reporting of stigmatized behaviors vary by the mode of survey instrument, and that generally the more private methods of interviewing yield higher reporting of these behaviors. ACASI, which offers computer-based recorded playback features, has demonstrated participants’ willingness to report socially sensitive behaviors more often than with face-to-face interviews.” So we should expect a more accurate sampling. However when one looks at the Technical Terms section of the report we find less precision, while the question asking if a respondent has ever had sex and at what age he/she first had sex includes the clarifying statement, “by means of vaginal, oral, or anal sex.” The language in the number of sexual partners question is more open, “Participants asked about the number of people they had sex with in the past year and in their lifetime.” It’s possible (and to my mind likely) that without this inclusion statement as part of the question men were likely to cast a broader net over their sexual activity (namely including oral and anal sex) while women might have been more likely not to factor them in at all. In this case, no one is lying per se, they are just responding to a question --that is in my mind vaguely worded—in different ways. I think again it’s also important to keep in mind that EVERYTHING is useful data. If this same study is done in the future and the sexual partners question is uniform to the parameters of the first sexual encounters question and the numbers are even, then we know a whole lot more (empirically) about how men and women view sex than we did before.

As for how to correct for an inherent gender bias in sexual behavior reporting that can’t be fixed by replacing a human with a computer? Check out the Wiederman paper linked to at the end of the Slate piece.