I’ve often been disappointed by a highly-rated wine. At the same time, I’ve found wines at the other end of the judging scale that I really liked. Over the years, it seems that my palate can vary from day to day, which can be really annoying when I open bottles of wine over a few weeks, and less impressed than before.
I thought it was just me, but when I was viewing the movie Somm last month, I noticed the discrepancy in taste between some of the candidates for the test. it turns out that some research is being done which calls wine tasting into question.
A winemaker named Robert Hodgson began tracking tasting panel results back in 2005, and, according to The Guardian, found that “…in most years, the results are surprisingly inconsistent: some whites rated as gold medallists in one contest do badly in another. Reds adored by some panels are dismissed by others. Over the decades Hodgson, a softly spoken retired oceanographer, became curious. Judging wines is by its nature subjective, but the awards appeared to be handed out at random.”
“Chance has a great deal to do with the awards that wines win.
These judges are not amateurs either. They read like a who’s who of the American wine industry from winemakers, sommeliers, critics and buyers to wine consultants and academics. In Hodgson’s tests, judges rated wines on a scale running from 50 to 100. In practice, most wines scored in the 70s, 80s and low 90s.
Results from the first four years of the experiment, published in the Journal of Wine Economics, showed a typical judge’s scores varied by plus or minus four points over the three blind tastings. A wine deemed to be a good 90 would be rated as an acceptable 86 by the same judge minutes later and then an excellent 94.”
This is a pretty interesting article, which discusses other aspects that seem to effect a tasters perceptions of the quality of a wine. The article goes on,
Hodgson isn’t alone in questioning the science of wine-tasting. French academic Frédéric Brochet tested the effect of labels in 2001. He presented the same Bordeaux superior wine to 57 volunteers a week apart and in two different bottles – one for a table wine, the other for a grand cru.
The tasters were fooled.
When tasting a supposedly superior wine, their language was more positive – describing it as complex, balanced, long and woody. When the same wine was presented as plonk, the critics were more likely to use negatives such as weak, light and flat.
In 2008, author Robin Goldstein co-wrote a paper called “Do More Expensive Wines Taste Better? Evidence From a Large Sample of Blind Tastings, which found that ‘wine experts’ tend to rate wines higher when told they are expensive.
Individuals who are unaware of the price do not derive more enjoyment from more expensive wine. In a sample of more than 6,000 blind tastings, we find that the correlation between price and overall rating is small and negative, suggesting that individuals on average enjoy more expensive wines slightly less. For individuals with wine training, however, we find indications of a positive relationship between price and enjoyment. Our results are robust to the inclusion of individual fixed effects, and are not driven by outliers: when omitting the top and bottom deciles of the price distribution, our qualitative results are strengthened, and the statistical significance is improved further. Our results indicate that both the prices of wines and wine recommendations by experts may be poor guides for non-expert wine consumers.
For those of you who have bought highly-rated wines and were disappointed, this is a pretty interesting article.
JR says
I suspect a look at wine scores at competitions in Australia might turn out differently. Rather than use buyers, critics, PR people, etc., the panels are primarily winemakers who’ve trained as judges. My experience judging there and in New Zealand was completely different from my experience judging in the U.S.
By the way, Dan Zwerdling covered some of the junk science around wine in his 2004 Gourmet magazine piece on the myths about wine glasses: http://www.gourmet.com/magazine/2000s/2004/08/shattered_myths
livetoeat says
Thanks for sharing this article. Fascinating! Especially the info on how much comes through the sense of smell. I have a highly sensitive sense of smell – which can be a real detriment sometimes, but it does make wine tasting interesting. I’m leaning more and more toward only buying wines that I’ve tasted. So happy we have some great tasting rooms/wine shops in PDX. That being said, I’m a sucker for some of those bargins that come through Garagiste e-mails. They can be hit or miss, but it gives me an excuse to visit Seattle.
brian comnes says
Oh come on – are MOVIE reviews junk science? – Look, its all personal – like movies you eventually find a wine critic/reviewer that you think has his/her head (tongue and nose too) screwed on straight and is not a corporate shill, and you follow them over time. Pretty soon you get to know where your likes and theirs line up and where they don’t and you make decisions accordingly. If you don’t trust a single voice you can subscribe to a group based wine newsletter or club.
There’s good research that shows that a consistent group of tasters, who taste together often can develop a reliable descriptive vocabulary and achieve some statistically significant agreement on relative rankings between/within a group of wines, i.e. these are better than those with some reliability. True it may be boring to read the same words about color and aroma over and over , but if you are serious collector trying to not mistakes in your cellar then its the best game in town.