Extreme Age, Extreme Error

And the Problems with Peer Review

A few weeks ago I wrote about reading Galen, an ancient Roman author on medicine. The more I reflect on his texts, and the humility it should provoke in us, the more I begin to look for traces of similar problems in our own thought and practice today. I don’t mean I’ve started fact-checking every sentence I read, but rather, I’m just trying to think about how perverse incentives and errors might lead to bigger mistakes that are held by large numbers of people.

Recently, I came across an extraordinary piece of scientific scepticism. Of all places, it came in front of my eyes because it featured on Depths of Wikipedia’s venerable Instagram page. There it was:

Supercentenarians are an exciting idea. They are a glimmer of hope, a passport somewhere a little further from our own inevitable mortality. Somehow, there are special places in the world where the stars have aligned, maybe through diet and lifestyle, to make people live very long lives. There have been big documentaries on this phenomenon — the ‘Blue Zones’, and I’ve watched them and been entertained and interested by them. In hindsight, I’d like to say I was a bit confused by the suspiciously unhealthy diets and apparent lack of any special rigour with these places. In reality, I was completely convinced at the reality of the phenomenon, even if its explanation wasn’t that obvious.

Saul Newman’s finding suggests that — forget explanations — the observation of extreme longevity in certain regions is a statistical artefact. By this I mean it doesn’t exist. A statistical artefact is where the thing which is observed is actually a quirk of the way one is handling data, and not a real phenomenon.

Now, I’m not a statistician, and although I have done some very basic “descriptive statistics” in some of my work, this is always extremely simple. So I won’t even try to comment on the statistical detail of what Newman’s talking about. I will try and summarise it, though.

Basically, the special correlation of these ‘Blue Zones’ with extreme longevity can also be paired with special levels of poverty, pension fraud, low levels of literacy, and in fact, low average lifespan! The literacy is important here: the extreme ages are based on historical data. If you are considering a 115 year-old, you’re considering 115 year old documentation which may have been read and remembered by someone who actually couldn’t read. That seems outlandish, but we’re dealing with Sicily in around the year 1910 here. The pension fraud is another explanatory factor. Your aged relative dies, but you receive the pension cheques. You don’t tell the authorities, and the cheques keep coming. You’ve just created an extracentenarian. Because the numbers of these extremely agèd people are small in the overall population, it doesn’t take many slipups and fraudsters to bork the datasets. And borked the datasets seem to be.

An old census

Newman points to other indicative facts which drive further nails in the ‘blue zone’ coffin. For me, although they’re not covering whole data sets, are very compelling. For example, Newman has pointed out that an analysis of Japanese birth and death records showed that 80% of people ‘aged’ over 100 were in fact dead, but not recorded as such.

---

OK, you say, so there are compelling reasons to think that this ‘Blue Zones’ idea is not true. This isn’t a public health substack, you scream in rage. OK, fine, we’ll get on to the more abstract bit.

What interests me, and troubles me, more than the specific findings of Newman’s research is the reception of the research. This highly controversial, highly attention-grabbing work of his has been largely ignored until his Ig Nobel victory. Why? It all has to do with one of science’s most important mechanisms. That mechanism also happens to be one of its most fraught and problematic mechanisms. The mechanism is peer review.

There are several fundamental problems with peer review. One of them is that the reviewers are not paid. That’s bad. Another problem is that peer reviewers are often needlessly rhetorical, and sometimes just rude. That’s also bad. More profound problems, though, are available. For example, there’s the fact that peer reviewers can look at statistical methods and see if their data checks out, but in empirical science, they do not have the time or resources to see if results are reproducible. What this means, dear reader, is that they check the experiments themselves. This means it can take years to spot biased, erroneous, or fraudulent findings if they occur within the lab. Or they might never be found at all.

In the case of Newman, the problems are working the other way. It’s not that Newman has been unduly let through the peer review process. Rather, he hasn’t been let in. His article, which is thought-provoking, well-argued and robustly evidenced (as far as I can tell, admittedly with no expertise at all), has not actually been published yet. The paper sits on a pre-print server. It’s been sitting there (give or take a few edits) for five years. A pre-print server is not the same as a journal, even if it’s formatted to look prim and proper, and is not taken nearly as seriously by the academic community (this is normally sensible).

Newman’s work hasn’t been accepted by the community of anonymous peer reviewers that are recruited (unpaid) by journals to check academic work. Why? We actually don’t know these reviewers’ reasoning, because peer review reports themselves do not generally see the light of day, and because, well, the paper hasn’t been published so there hasn’t been any response to it.

Yet the reason, to Newman, and to any duly cynical onlooker, is probably obvious. It doesn’t matter what ‘fundamental’ problems have been found with Newman’s research. The biases on the part of the peer reviewers to rescue the field of ‘superlongevity zones’ are so powerful that there is no way they will let his work get through. The implicit biases, in other words, discredit the discrediting of his research. Because rejection is just a ‘no’, it doesn’t have a paper trail, and it isn’t itself open to any kind of scrutiny or even publicity.

Peer reviewers have to be selected for their expertise. But in simply being an expert in a field, you tend to believe that field should exist. Hence, we have the problem with a truly controversial piece of argument: it undermines the foundations of a whole field of study. This means all of the experts can become systematically biased against an argument, not just emotionally but also in the sense that their livelihood and career literally depends upon that field of study.

Editors’ decisions are final. In academia, the rejection based on peer review gives the scholar only two options. Either give up, or re-write and re-submit. If you take the former option, then the hours that you have spent (unpaid, of course) on the research will never see the light of day. If you take the latter option, you can only pray for a more sympathetic editor and a more sympathetic group of anonymous people. The problem is that the group of anonymous people are likely to have similar opinions, especially if they’re the very same people.

There is no mechanism in science against this kind of ‘groupthink’. The problem is a wicked one, because a paper which is just totally bogus and wrong will, of course, get the same response — but a deserved response. The difference here is that, when circumventing the system of peer review, a piece of research can be seen and seen to be valid. What’s needed is some system of handing peer review requests to people who are expert in a different but ‘adjacent’ field. They are knowledgeable enough to spot flaws in the writing, but not so ‘knowledgeable’ and interested in the field that it begins to warp their ideas of acceptable discourse.

Most fundamentally, I think, science and academia needs to think a lot more intelligently about what constitutes a ‘conflict of interest’. We like to think of conflicts of interest as cut-and-dry, as obvious fraud involving brown envelopes and favours. Yet having one’s academic position founded upon a particular idea of set of ideas being true is also a deeply, unavoidably motivating factor in the same way as cash. Indeed, a career is money, in the end.

The hyperspecialisation, the group-think, and the gatekeeping all make me nostalgic for just a moment for a more amateur period in academia. In those days, almost no-one was an expert in a ‘field’. They were a curious individual working in many fields at once. They were polymaths. They may, as a result, have been wrong on many occasions, but they were free to think at a much faster rate and without the strange incentives and the stranger entrenchment of ideas that too-often characterises modern academia.

On the other hand, of course, academia of this bygone age was also deeply iniquitous. It had the flavour of ‘amateurism’ because the people involved were generally patricians who didn’t need to be paid for anything. We need a middle way. Academia should be open to people, and it should pay people, but it also shouldn’t allow them to become so wedded to particular ideas that they forget the truth is more important and more interesting than being right.