The ideal situation in any pandemic or catastrophe is that after the fact, the predictive models showing the worst-case scenario should look terrible. That’s actually a great outcome, because it means that either your worst fears were unfounded, or that your response to the crisis was so effective that the worst-case scenario (which could have happened) was avoided.
The problem, of course, is that if a worst case scenario is avoided, there will always be a suspicion that the worst case scenario was never going to happen anyway, and that the response designed to avoid it was a total over-reaction. We’re now entering that phase with Covid-19:
The Covid-19 modelling that sent Britain into lockdown, shutting the economy and leaving millions unemployed, has been slammed by a series of experts.
Professor Neil Ferguson’s computer coding was derided as “totally unreliable” by leading figures, who warned it was “something you wouldn’t stake your life on”.
The model, credited with forcing the Government to make a U-turn and introduce a nationwide lockdown, is a “buggy mess that looks more like a bowl of angel hair pasta than a finely tuned piece of programming”, says David Richards, co-founder of British data technology company WANdisco.
That story is from the Telegraph, which, being a UK paper, focuses heavily on the UK. But it is worth noting that the Imperial College model has been credited with influencing lockdown policies in countries across the western world, not just in the UK. For example, the model is the main driver of policy in the United States:
When updated data in the Imperial team’s model indicated that the United Kingdom’s health service would soon be overwhelmed with severe cases of COVID-19, and might face more than 500,000 deaths if the government took no action, Prime Minister Boris Johnson almost immediately announced stringent new restrictions on people’s movements. The same model suggested that, with no action, the United States might face 2.2 million deaths; it was shared with the White House and new guidance on social distancing quickly followed.
It’s unclear the extent to which the Imperial model influenced Irish thinking. We did, after all, issue some restrictions, such as schools closing, a few days before the UK, but it’s worth noting – and totally forgotten by most – that we actually entered full lockdown, closing businesses and restricting travel, five days after they did, which suggests Imperial College data played some part in that decision. In fact, it would be astonishing (and negligent) if the Irish Government hadn’t been looking at a major predictive study from the UK and been influenced by it to some extent.
In any case, we can now say with absolute certainty that the worst-case scenario outlined by the Imperial model has not come to pass. Which poses a question (why?) to which there are two possible answers: Either the response by Governments saved hundreds of thousands of lives, or the model itself was useless garbage which prompted a massive over-reaction.
Or, of course, there’s the third answer, which nobody will be particularly motivated to accept, which is that the model was garbage and the government response saved many lives.
Anyway, in the “garbage model” camp we now have convincing testimony, again in the Telegraph, from two experts:
Imperial’s model appears to be based on a programming language called Fortran, which was old news 20 years ago and, guess what, was the code used for Mariner 1. This outdated language contains inherent problems with its grammar and the way it assigns values, which can give way to multiple design flaws and numerical inaccuracies. One file alone in the Imperial model contained 15,000 lines of code….
….Ultimately, this is a computer science problem and where are the computer scientists in the room? Our leaders did not have the grounding in computer science to challenge the ideas and so were susceptible to the academics. I suspect the Government saw what was happening in Italy with its overwhelmed hospitals and panicked.
It chose a blunt instrument instead of a scalpel and now there is going to be a huge strain on society. Defenders of the Imperial model argue that because the problem – a global pandemic – is dynamic, then the solution should share the same stochastic, non-deterministic quality.
We disagree. Models must be capable of passing the basic scientific test of producing the same results given the same initial set of parameters. Otherwise, there is simply no way of knowing whether they will be reliable.
That last line in bold is the key bit: Nobody has been able to consistently reproduce the findings of the Imperial model using the same software and the same data Imperial College used to produce it in the first place. Imagine you entered 10 x 10 into a calculator five times. You’d expect to get five answers of “100”. But with the Imperial model, those who have tried to reproduce it have basically entered 10×10 hundreds of times and gotten answers ranging from 12 to 674. The calculator, they’re saying, is broken and useless.
Assuming they are right (and Imperial College hasn’t yet responded) then yes, lockdown policies, at least in the UK and the USA, and possibly in Ireland, were implemented on the basis of a useless model.
There’s also the question, of course, that the result you get from your calculator will depend entirely on what you ask it to calculate. 10 x 1 will give you a different answer to 10 x 9. And there is a big question about what Ferguson and his team put into their calculator:
Sir Nigel Shadbolt, Principal at Jesus College, said that “having a diverse variety of models, particularly those that enable policymakers to explore predictions under different assumptions, and with different interventions, is incredibly powerful”.
Like the Imperial code, a rival model by Professor Sunetra Gupta at Oxford University works on a so-called “SIR approach” in which the population is divided into those that are susceptible, infected and recorded. However, while Gupta made the assumption that 0.1pc of people infected with coronavirus would die, Ferguson placed that figure at 0.9pc.
One death per thousand, versus nine deaths per thousand, is going to give you a wildly different outcome, even if every other assumption both models make is identical. In fact, the Imperial Model’s “500,000 deaths” would become roughly 55,000 deaths if it had used the assumption made by the Oxford model. Current UK deaths, by the way? 34,000.
Of course, there’s one huge variable here: These models were designed to project what would happen with no lockdown at all, and if life just continued as normal. But life didn’t continue as normal – there was a massive effort to limit the spread of infection. So whatever model you use should look like a huge over-estimate at this point if you don’t take that into account. If it doesn’t, in fact, then you’d have to conclude that the model was wrong in the other direction – actually under-estimating deaths without a lockdown.
Anyway, what we know is this: Major policy decisions, which will lead to a historic recession and perhaps depression, were taken on the basis of this model by UK scientists. There are very serious questions about whether that model was actually correct, or totally flawed.
That deserves serious investigation, when all of this is over. But it does not, by itself, mean that the lockdown was a mistake or that it is safe to hold house parties. People are still getting this disease, and dying, and we do not need a model to tell us that. We might debate the precise measures taken by Governments when all of this is over, but for now, the smartest thing to do is to put your own health, and that of your family, first.