What is ‘Excess Deaths’?

The Excess Deaths statistic estimates the total mortality impact of a disease on a society. Excess Deaths tells us how many people died in excess of the number we would normally have expected over the same time period.

It’s like the ‘real’ metrics that remove the effect of inflation from the observed economic data. Real GDP growth measures the change in GDP independent of the general movement in prices. Excess Deaths tells us how many people died in excess of the general trends in mortality.

Excess Deaths sees through the administrative process of counting deaths to focus on the changes in the nation’s mortality. In doing so it avoids the inevitable errors and inconsistencies in the way Covid-19 deaths have officially been recorded.

At the early stages of an outbreak, an overwhelmed health service is more likely to make mistakes. It may not have the capacity to test for the pathogen, leaving our medics to judge whether it was responsible for a death. But even when we do have the testing capacity, deaths can go unrecorded. Some people may not know they have the virus until it’s too late. Others may be unaware it even exists.

In addition to the limitations on our ability to identify the disease, governments have different approaches to defining ‘death due to Covid-19’ in their official statistics. Some have clearly fudged their numbers, but even honest efforts will run into problems. When a patient with underlying conditions contracts Covid-19 and dies, can we say that Covid-19 caused the death?

The WHO have recommended counting all deaths with a confirmed or probable infection of Covid-19. This policy ensures that as many cases as possible are recorded in the dataset, and that no true cases go unrecorded. The WHO wants to have the best possible data on which to do testing and research. This conservative approach will bias the headline numbers higher, but we can refine our estimates later.

Ireland generally followed the WHO’s advice, as did Germany, Italy, Sweden, and, eventually, the UK. Poland went a step further, allowing doctors to include Covid-19 on the death certificate based on symptoms, with or without a test confirmation.

Advocates of Excess Deaths argue that it allows us to sidestep all of these thorny issues while also taking account of deaths indirectly caused by Covid-19, therefore providing a more reliable estimate of the overall mortality impact of the disease. It offers the additional benefit of providing a sense-check on the government’s official numbers.

 

How do We Calculate Excess Deaths?

Excess Deaths is the difference between officially recorded deaths and forecasted deaths.

To forecast the number of deaths we would have normally expected, we need a historical dataset and a forecast model. The data set is the last 5 – 10 years of the nation’s mortality numbers. It can take several months for a death to be officially recorded so the Health Information and Quality Authority (HIQA), used data from RIP.ie, as it was timelier. The forecast model is often described in the media as a historical average, but in practice, it is a far more complicated statistical process. I won’t discuss the specifics of the model, but I address the consequences below.

According to HIQA, there were between 851 and 1,290 excess deaths in Ireland between the 11th of March 2020, and the 16th of June 2020. The officially recorded number of deaths from Covid-19 over the same period was 1,709. HIQA’s analysis suggests that the official count overstates the true cost of Covid-19, and that a more accurate number is their central estimate of 1,072 deaths.

 

So Ireland Did Even Better Than We Thought?

No, nothing changed. We did the same. Excess Deaths is just another perspective; another yardstick by which to measure our results.

Excess Deaths is an estimate. This is not the ‘true’ number of additional deaths caused by Covid-19 (the calculation of which would require access to an otherwise identical Covid-free universe); this is just one estimate, based on one model. Another researcher could use a different model, or indeed the same model but with a different historical dataset. These approaches would all produce different estimates of Excess Deaths.

That means Excess Deaths is a model-dependent statistic; we can only compare countries by Excess Deaths if they have been calculated with the same model, which they usually aren’t. The Excess Deaths number you saw on the BBC isn’t comparable to HIQA’s, or anyone else’s. HIQA may have done an excellent piece of analysis, but the Excess Deaths number alone tells us little about how Ireland performed relative to other countries.

The Excess Deaths calculation is applied to a specific time period. In HIQA’s analysis, it was the 11th of March to the 16th of June. It doesn’t include any deaths that were recorded either side of that window, whether directly or indirectly due to Covid-19. There is a tendency to think of Excess Deaths as the ‘true’ number of deaths from Covid-19 but really, it is only an estimate based on a sample – one that loses relevance with each passing day.

Another flaw in the model is that Excess Deaths can’t account for deaths that were brought forward by Covid-19. If an elderly person died from Covid-19 but the model had ‘expected’ them to die within the time period anyway, their death won’t register in the final number. Only ‘excess’ deaths can be attributed to Covid-19. To argue that this is fair as the individual would have died anyway is, to me, both technically and morally inadequate.

The final problem with using Excess Deaths on Covid-19 is more theoretical than practical, but it is significant, and it should have been addressed in HIQA’s report.

A society in lockdown is a society with very little activity. Fewer cars on the road, fewer people in the pubs, fewer risky things being done. A lower level of activity means fewer opportunities for things to go wrong. You would naturally expect fewer deaths in that environment.

New Zealand for example, suffered 44 fewer road deaths in the first 6 months of 2020 than they had averaged over the previous 20 years. Maybe Ireland’s Excess Deaths is ‘only’ 1,000, but New Zealand’s Excess Deaths would probably be negative.

We can use Excess Deaths to estimate the effect of the influenza virus, because the ‘flu doesn’t require meaningful changes in society’s structure or function. One season is comparable to the last. Covid-19 however, required several months of restrictions and lockdowns. Any estimate of Covid-19 Excess Deaths will tell us about the effect of Covid-19 and the effect of the Covid-19 lockdown.

These are the kinds of inconvenient, messy situations you encounter when you decide to use complicated statistics. No model is perfect, but the more complex the model, the more complex the errors.

But, aren’t these exactly the kinds of problems Excess Deaths was supposed to solve? If we can’t make international comparisons because the numbers weren’t calculated the same way, aren’t we back to square one?

 

Is Covid-19 Excess Deaths a Waste of Time?

No, it’s not a waste of time. There are no ‘good’ or ‘bad’ statistics, it’s the interpretation that counts. Excess Deaths has value, but we need to understand how it works if we are to interpret it correctly.

In reality, Excess Deaths didn’t solve the problems in the official numbers; it just replaced one set of problems with another set. The argument its supporters will make is that the second set of problems are smaller, so the Excess Deaths estimates are more reliable than the official numbers.

I agree with that argument. A bias doesn’t necessarily invalidate a statistic. Statisticians are trained to account for bias anyway. Excess Deaths is more robust than Total Deaths, and it also adds a different perspective that can play counterpoint to Total Deaths. They are more valuable when used together.

Overall, Excess Deaths adds to our understanding of the problem, however as a researcher interested in a high-level comparison I wonder if Excess Deaths is like changing from one currency to another: the number on the price tag might be smaller, but the cost is the same.

 

Discussion

Based on HIQA’s Excess Deaths analysis, Ireland’s official count is probably too high by at least a few hundred deaths. That will be useful information for the public health experts and policymakers when they sit down to review Ireland’s performance.

That still leaves Ireland with well over 1,000 people dying in the space of 3 months from a preventable cause, and an unimpressive performance relative to our European peers (who were themselves among the worst in the world). HIQA’s Excess Deaths estimate doesn’t tell a significantly different story to my ‘back of the envelope’ Deaths per Million analysis. If anything, HIQA’s analysis has confirmed my prior position.

Think about Ireland and New Zealand. Based on the official deaths of 1,750 and 22, Ireland massively underperformed New Zealand. To me, the disparity in those numbers told us everything we needed to know. No translation from official Total Deaths to Excess Deaths was going to change the interpretation. The difference in these numbers, both in absolute and relative terms, is so big that it couldn’t be explained by different counting methods, errors, or randomness. It had to be policy.

The same is true for Australia, Singapore, Taiwan, Slovakia, South Korea etc. The variables that Excess Deaths accounts for cannot explain such large gaps between their performances and Ireland’s. There is bound to be some noise, but the vast majority of the difference is explained by the quality of their risk management.

HIQA’s Excess Deaths estimation was a necessary and useful exercise, but it was not the final word on Covid-19 deaths in Ireland. I look forward to researchers providing further analyses in the coming months so that we can fill in the gaps in our understanding.

In the meantime, I’ll continue to use Deaths per Million to compare national Covid-19 results. Deaths per Million has its flaws, but it is simple, updated daily, and, with minor adjustments, allows comparisons between countries. I prefer a simple analysis that is roughly right, over an advanced analysis that is more precisely wrong.

Most importantly, Deaths per Million is easy to understand. That means we – the people, the media, the politicians – can talk about it. Fancy statistics like Excess Deaths only shut down conversations. We need to be able to understand and talk about the numbers we see in the media. How else can we make the informed decisions upon which our democracies depend?

 

 


 

For an alternative view of what Excess Deaths can tell us about Covid-19, read Tom Prendeville