Expert blog: In the wrong hands, vaccination statistics can prove deadly - Simpson’s Paradox shows why

Simpson's Paradox can create misleading conclusions

There has been much discussion of late about data published on 1 November, 2021, by the Office for National Statistics (ONS). It is titled ‘Deaths involving COVID-19 by vaccination status, England: deaths occurring between 2 January and 24 September 2021’.

The raw statistics show death rates in England for people aged 10 to 59, listing vaccination status separately.

Counter-intuitively, these statistics show that the death rates for the vaccinated in this age grouping were greater than for the unvaccinated. These numbers have since been heavily promoted and highlighted on social media by anti-vaccine advocates, who use them to argue that vaccination increases the risk of death.

The claim is strange, though, because we know from efficacy and effectiveness studies that COVID-19 vaccines offer strong protection against severe disease. For example, the efficacy and effectiveness of the Pfizer-BioNTech vaccine has been shown to be well over 90% in this regard in the most recent studies.

Vaccine efficacy of 90% means that you have a 90% reduced risk compared to an otherwise similar unvaccinated person, based on controlled randomised trials, while vaccine effectiveness refers to real-world outcomes. On either measure, vaccines work very well indeed.

So, what’s going on here?

Well, closer inspection of the ONS report reveals that over the period of the study, from January to September 2021, the age-adjusted risk of death involving COVID-19 was 32 times greater among unvaccinated people compared to fully vaccinated people. But hold on! How can we square this with the data from the table listing death rates of those aged 10 to 59 by vaccination status?

For the answer we turn to a classic statistical artefact known as Simpson’s Paradox, which seems to pop up and create misleading conclusions all over the place.

It is a consequence of the way that data is presented.

Essentially, Simpson’s Paradox can arise when observing a feature of a broad, widely drawn group, where there is an uneven distribution of the population within this group, for example by age or vaccination status. Ignorance of the implications of Simpson’s Paradox can generate misleading conclusions, which can be, and in this case are, very dangerous.

The paradox in these particular ONS statistics arises specifically because death rates increase dramatically with age, so that at the very top end of this age band, for example, mortality rates are about 80 times as high as at the very bottom end. A similar pattern is observed between vaccination rates and age. For example, in the 10 to 59 data set more than half of those vaccinated are over the age of 40.

Those who are in the upper ranges of the wide 10 to 59 age band are, therefore, both more likely to have been vaccinated and also more likely to die if infected with COVID-19 or for any other reason, and vice versa. Age is acting, in the terminology of statistics, as a confounding variable, being positively related to both vaccination rates and death rates.

Put another way, you are more likely to die in a given period if you are older and you are also more likely to be vaccinated if you are older. It is age that is driving up death rates not the vaccinations. Without the vaccinations, deaths would be hugely greater from COVID-19.

So, what if we divide the 10 to 59 group into smaller age groups?

If we break down the band into narrower age ranges, such as 10 to 19, 20 to 29, 30 to 39, 40 to 49, and 50 to 59, we find that the counter-intuitive headline finding immediately disappears. In each age band, the death rates of the vaccinated are vastly lower than those of the unvaccinated. This also applies in the higher age bands - 60 to 69, 70 to 79, and 80 plus.

Basically, unvaccinated people are much younger on average, and therefore less likely to die.

Yet there are those out there who are more than happy to use these statistics to mislead. The consequence is that many who would otherwise choose to be vaccinated might refuse to do so.

In truth, the age-adjusted risk of deaths involving coronavirus (COVID-19) over the first nine months of this year was in fact 32 times greater in the unvaccinated than the fully vaccinated.

This is a hugely important statistic, and we must not let statistical manipulation be used to obscure this critical information. The lives of countless people really do depend on us exposing this truth.

Leighton Vaughan Williams, Professor of Economics and Finance at Nottingham Business School. Read more in Leighton’s new publication, Probability, Choice, and Reason.