Doing Your Own Research: The “Other Grandpas” Criterion

“My grandfather smoked a pack of cigarettes and drank four ounces of whisky every day, and he lived to be 94.”

If you don’t have this grandfather yourself, you know someone who does. Or, at the least, you’ve seen the perennial news stories about him. These people are common enough that you shouldn’t have to look beyond your hometown to find one.

Average life expectancy is a bit north of 70 years worldwide, more like 80 in wealthier nations. And health authorities inform us that heavy smoking reduces that life expectancy by ten or so years. That level of drinking, they say, cuts off another four or five. But Grandpa’s right there, still going strong, decades past what science says should have been his expiry date.

What are we to make of this?

There are two easy answers, and both are wrong. Some will simply say “the plural of anecdote isn’t anecdata” and “trust the science” and be done with it. Others will say “statistics are one thing, what I see with my own eyes is another,” and then print themselves a license to live as hard as they want.

We talked before about the dangers of blindly trusting experts, and also about the practical necessity of investing at least some degree of trust in these experts. So hopefully us critical thinkers are able to already see why “trust the science” isn’t an immediate game-winning trick here. But the problems with the other line of thinking are a little more subtle.

Part of what makes this hairy is how hugely psychologically rewarding it is to know something others don’t, and especially to be in possession of a truth that subverts the consensus. And how much better when it’s something that you already want to believe. There’s a reason pop science articles about the health benefits of chocolate or red wine or quitting your job are so widely shared whenever they appear. Not only does the reader get the immediate emotional reward of being in on a secret, they also get justification for doing the things they wanted to do anyway.

That said, information that challenges commonly held beliefs is, without a doubt, the most valuable kind of information. Science, done properly, is nothing more than systematically looking for evidence that your theories are wrong, or at least unlikely. So, as good critical thinkers, we should recoil in horror at the idea of disregarding a new piece of information just because it doesn’t fit the model. But we must also be vigilant against the impulse to latch onto contentious ideas or data as new truths, inherently superior to the old model simply by virtue of their controversy.

Instead, what me must do, is ask questions. In other words, we have to do our own research. The idea of “doing your own research,” of course, is much maligned today. But, in truth, it’s the only way any of us ever come to know anything. Yes, there’s a big difference between doing a Google search and conducting a double-blind study with an n in the thousands. Still, one way or another, you’ve got a theory and you’re seeking an answer.

But when you ask questions and go looking for new information, the way you do it is supremely important. A good critical thinker doesn’t go looking for evidence that a theory is right. They go looking for evidence that it’s wrong. And so, just as we must actively look for the information that questions the prevailing theories of the day, we must also then subject that new information to the same rigour when we find it.

For people like us—who have presumably never personally run a massive longitudinal research program on the effects of alcohol consumption on life expectancy—the paper in Nature arguing that booze kills you is really just one data point in our quest for well-supported understanding. And Grandpa’s glass of whisky is another. It’s up to us to look at these two things and figure out how to reconcile them. If they are truly mutually incoherent, then one of them needs to go. But one of the big questions we need to ask is just how incompatible they really are.

The scientific consensus on smoking obviously doesn’t say that literally every heavy smoker will die precisely a decade younger than the average life expectancy. The world is messy and unpredictable in the specific case, but it’s often surprisingly orderly in the general. That’s what gives us those nice bell-shaped curves. So, of course, Grandpa could be an outlier. This is, if we’re generous, what people are trying to tell us with that “anecdata” zinger. There’s room in the scientific consensus for outliers.

The plural of anecdote is indeed not anecdata. The plural of anecdote is just data. And if the data is incompatible with the scientific consensus, then there is a real problem. Otherwise, the consensus is not scientific at all.

Yes, there is room in the theory for outliers, but statistical models are extremely strict about how common outliers should be. And it’s not very. If your data set has too many outliers in it, the mean shifts and they’re no longer outliers at all.

Which brings us back to the curious ubiquity of chain-smoking, hard-drinking grandpas. If their longevity really is several standard deviations away from the norm, why are there so many of them? This is a very good question to ask.

In this case, though, there is an answer.

My grandfather—not a theoretical grandfather this time, my actual grandfather—smoked like a chimney and drank like a fish. He died in his late forties. If I tell this sad bit of family history to someone, the chances of them remembering it, and especially the chances of them repeating it, are low. Because my grandfather’s story fits the narrative, it’s not remarkable. But we do remember the story about the hard living 94-year-old. We talk about it to others. We write local newspaper stories about it.

Precisely because these stories are unexpected, they become salient. And precisely because of the psychological reward contentious knowledge gives us, we share these stories. We repeat them, and as a result, we make them seem more common than they are. You don’t have to look past your hometown to find the grandpa who beat the odds. But how many people are in your hometown? How many grandpas like mine did you never hear about? Or forget hearing about?

You may just find that these outliers are exactly as common as the scientific consensus would suggest. That won’t always be the case. Sometimes the most incendiary and controversial ideas turn out to be the right ones. But doing your own research, doing it well, means asking the same questions of the sexy radical theories that you ask of the boring consensus ones. Remember to count the grandpas you aren’t hearing about.