Coronavirus: Models predicting patient outcomes may be 'flawed' and 'based on weak evidence'

A group of European scientists worry models that predict how a patient may respond to the coronavirus are “flawed” and “based on weak evidence”.

With the new strain emerging at the end of last year, only the relatively few people who have encountered the virus have any immunity against it.

Amid concerns the pandemic could threaten health services all over the world, models have been created to predict everything from who is most likely to develop complications to the length of a patient’s hospital stay.

The scientists set out to uncover the validity of 31 of these models, rating all as “at high risk of bias”.

They concluded the models are “poorly reported” and “probably optimistic”, adding none should be used “in practice at this point”.

One expert stressed, however, models are often accepted as being “useful but not perfect”, with doctors still having to use their “judgement”.

Early research suggests the coronavirus is mild in four out of five cases, however, it can trigger a respiratory disease called COVID-19.

The coronavirus is thought to have emerged at a seafood and live animal market in the Chinese city Wuhan, capital of Hubei province.

It has since spread into more than 180 countries across every inhabited continent.

Latest coronavirus news, updates and advice

Live: Follow all the latest updates from the UK and around the world

Fact-checker: The number of COVID-19 cases in your local area

6 charts and maps that explain how COVID-19 is spreading

Since the outbreak was identified, more than 1.3 million cases have been confirmed, of whom more than 292,000 have “recovered”, according to Johns Hopkins University.

Incidences have been plateauing in China since the end of February, and the US and Europe are now the worst-hit areas.

The UK has had more than 52,000 confirmed cases and over 6,000 deaths.

Globally, the death toll has exceeded 76,000.

Coronavirus: Models at ‘high risk of bias’

In the review, three of the models predicted the risk a patient would be hospitalised with pneumonia or another coronavirus complication.

Eighteen looked at diagnostic models for detecting infection, 13 of which were via machine learning.

The remaining 10 models predicted a patient’s risk of death or severe disease, or how long they may be in hospital.

Note the models did not include the Imperial College London simulation that guided the government’s response to the coronavirus outbreak.

Of the 31 models, only one included patient data from outside China.

“There is the potential for different ethnic groups or healthcare systems to be different so any scorecard derived outside a UK population would need testing on a UK population for validity before being implemented”, said Dr Ray Sheridan from the University of Exeter.

The European scientists concluded all the models were at a “high risk of bias, mostly because of non-representative selection of control patients”.

“A high risk of bias implies that these models will probably perform worse in practice than the performance reported by the researchers,” they added.

With a “few exceptions”, the team felt most of the models had too small a sample size.

“This is a well known problem when building prediction models and increases the risk of overfitting the model,” they wrote in The BMJ.

The models that used machine learning for diagnosis “showed a high to almost perfect ability to identify COVID-19”.

As well as being at a “high risk of bias”, the scientists noted “poor reporting”, and “an artificial mix of patients with and without COVID-19”.

Coronavirus: Models may do ‘more harm than good’

The scientists worry the desire to understand how best to treat COVID-19 patients may “encourage clinicians to implement prediction models without sufficient documentation and validation”.

Although they stressed we “cannot let perfect be the enemy of good”, they are concerned following these models may “do more harm than good”.

“Therefore, we cannot recommend any model for use in practice at this point,” wrote the team.

The emergence of individual patient data should “validate and update currently available prediction models”.

Professor Derek Hill from University College London said: “Predictive computer models can be really helpful in both managing patients and also developing new treatments.

“The COVID-19 pandemic has led to some very rapid development of disease models.

“These could potentially help focus scarce healthcare resources on patients most likely to benefit from particular treatments and personalise treatment based on the symptoms with which people present at hospital.

“But any disease model used to make critical decisions does need to be carefully tested.

“To state the obvious, if the disease model is flawed, it could end up directing seriously ill patients at the wrong treatment, rather than the right treatment.”

Speaking of the new European research, Professor Hill added: “The overall conclusions are these models on the whole are not trustworthy and could be dangerous to use.

“Care must be taken to ensure doctors don't use disease models to make decisions without knowing these models have been properly tested.

“If a disease model is used to select treatment for a patient, then from a regulatory point of view, it is a medical device.

“You wouldn’t want to use an untested medical device on a critically-ill patient any more than use an untested drug”.

Dr Sheridan noted the CURB-65 Score is used to assess the severity of a patient arriving at hospital with bacterial pneumonia, while also directing doctors towards the most appropriate antibiotics.

“CURB-65 is accepted as useful but it is not perfect,” he said.

“It is still important to use the clinician’s common sense and experience, so if a scorecard gives a low score but you think a patient look sicker then the clinician should go with their judgement.

“This will also be the case for COVID-19.”

What is the coronavirus?

The coronavirus is one of seven strains of a virus class that are known to infect humans.

Others trigger everything from the common cold to severe acute respiratory syndrome (Sars), which killed 774 people during its 2002/3 outbreak.

The coronavirus tends to cause flu-like symptoms, including fever, cough and slight breathlessness.

It mainly spreads face-to-face via infected droplets expelled in a cough or sneeze.

There is also evidence it may be transmitted in faeces and can survive on surfaces.

In severe cases, pneumonia can come about if the infection spreads to the air sacs in the lungs.

This causes them to become inflamed and filled with fluid or pus.

The lungs then struggle to draw in air, resulting in reduced oxygen in the bloodstream and a build-up of carbon dioxide.

The coronavirus has no “set” treatment, with most patients naturally fighting off the infection.

Those requiring hospitalisation are offered “supportive care”, like ventilation, while their immune system gets to work.

Officials urge people ward off the coronavirus by washing their hands regularly and maintaining social distancing.