enter the fray: our reader discussion forum
Search in:
Advanced
View:FlatThreaded
not quite scientific
by profcrabbe
+1 Reply

Inside of interesting anecdotes, the article has 2 main points:

1) Wines can have good or bad days, so a single blind tasting is misleading.

2) A blind tasting doesn't take into account future development of the wine (extrapolated from earlier examples).

Interesting points, but really mis-characterize how a proper scientific approach to the problem would proceed.

If wines can have good days and bad days, then you need to test the wines multiple times. No serious experiment in the last hundred years has been run that consisted of a single trial. The problem here isn't blind tasting, but drawing conclusions from a single blind tasting. What is needed is, say, 50 blind tastings, held in different regions by multiple overlapping groups of tasters over the course of a year (no taster sees the result of any tasting until all 50 tastings have been held). Then you run proper statistical tests on the data
to start drawing conclusions. I think this would satisfactorily address the quality variation issue.

As for the second point, I think there is a confusion in the function of a tasting within an overall review. To say that a blind tasting can't provide information about past aging performance is actually a benefit of blind tastings. They provide a single sample point on a single issue: the taste of the wine right now. To incorporate knowledge of external factors clouds that single issue, and is in fact a great example of poor experiment design. That doesn't mean such factors don't ever get incorporated, just later. I'd love to see a really honest review that said something like: "look, it just doesn't taste very good now compared to others, but if it's anything like it's older siblings, then in 5 years it will be great." To bump up its current taste evaluation based of a presumption of improvement is worse than sketchy. "Past performance is not an indication of future returns" and all that.

Finally, I have a problem with the statement that "But with or without knowing the name, a good critic ought to be able to deliver an honest and accurate assessment of a wine's quality." No, that's just not true. What the research shows is that humans are absolutely awful at delivering honest and accurate assessments of anything, and that our prior judgments and outside opinions completely overwhelm our evaluation systems. Recent work is starting to show that this happens on a deep sub-conscious level, probably making conscious control over it a physical impossibility. There was a nice article about cognitive dissonance in yesterday's NY Times.

Re: not quite scientific
by Sundown
The article made wine tasters look so bad I'm beginning to wonder if the author didn't intend it that way. (Of course that would be committing professional suicide, so I guess he just has no clue how silly he made himself sound.) With the exception of the tiny percentage of people who store wine long-term, aging performance is irrelevant. If you want a good wine with dinner it's stupid to pay a premium for the bottle that will hold up better over time.
Re: not quite scientific
by pwemail21

Thanks for summing up what I was feeling. You are absolutely right when you write about separating the taste test from the overall review.


This article just reinforces the notion that a wine critic is just a nicer name for a wine snob. It's only human nature for them to feel threatened by methods that encroach on their power to exalt or ridicule the wines they review.

Re: not quite scientific
by Sundown
Wasn't it ridiculous when the author said he'd never have agreed to a second blind tasting? As if it was some huge risk he'd just survived; like he'd just fought off a grizzly bear with his bare hands or something like that.
Re: not quite scientific
by profcrabbe

Sundown:
Wasn't it ridiculous when the author said he'd never have agreed to a second blind tasting? As if it was some huge risk he'd just survived; like he'd just fought off a grizzly bear with his bare hands or something like that.

I'm actually a bit sympathetic to that particular fear. A taster has a reputation at stake with little room for error. Like a field goal kicker, it doesn't matter that you have a career 95% accuracy, all that people remember is that you were wide right on the potential game winner in the Super Bowl. Imagine you're coming down with a cold that day and your nose is a little off. Getting things wrong leads to your name in articles all over the country. Personally, I would not want to take that risk. For the same reason that it's unreasonable to judge wines based on single trial blind tastings, it's unreasonable to judge critics on single trials too. To be truly fair, a critic's performance should only be reported in the aggregate, not in terms of individual tastings.

Re: not quite scientific
by TheRanger

The good day/bad day thing is rubbish. A single taster may have a good or bad day but if a panel is used, statistics eliminate the single taster with a bad day. The defense of bad day sounds like performance of an athlete which varies but a wine? This is excuse making by the manufacturer.

Future development in non-blind tastings is based on history of the wine maker. If it can't be done blind, then what exactly is being evaluated?

View as RSS news feed in XML