Generalizability

How can researchers improve the generalizability of their studies?

Researchers can improve generalizability by:

• Random sampling — randomly selecting individuals for the study to reduce the likelihood of bias
• Increasing sample size — increasing the number of participants, to build confidence that findings are significant for a larger group
• Using a diverse, representative population — so results are more readily applicable to many different types of people
• Training across multiple hospitals — using data from more than one setting, so results are more likely to hold up in different situations

Using a combination of these techniques from the outset of a project leads to more generalisable models that are also less likely to be biased.

What are the different levels of generalizability?

There are three levels:

• Internal — only generalisable within the context it was trained (i.e., the algorithm performs equally well on different segments of the training set within one hospital)
• Temporal — generalisable on new data created in the same setting in which it was trained (i.e., performs equally well on new patients at the same hospital)
• External — generalisable across settings (i.e., performs equally well in Hospital A and Hospital B)

In most cases, the aim is to achieve the highest possible level — external generalisability. However, this isn't always possible; for example, if an algorithm is used for a rare condition only treated at one hospital.

Why do algorithms fail to generalise externally?

If an algorithm that's supposed to be externally generalisable fails to generalise, it's usually the result of one of four issues: overfitting, underfitting, bias, or dataset drift.

OverfittingThis happens when an algorithm becomes so closely "fit" to its training data that it also learns noise and irrelevant details that may not appear in new data. A well-known example: an algorithm trained to distinguish huskies from wolves performed very well on its training set but missed obvious wolves during testing. It turned out every wolf photo in the training data had snow in the background — the algorithm had learned to associate wolves with snow, and failed when shown wolves in snow-free landscapes. Similar cases have occurred in healthcare, where an algorithm learned to classify a tumour type based on which scanner produced the image, rather than the tumour itself.

Underfitting
The opposite of overfitting — this is when an algorithm doesn't learn the features of its training data well enough, and so performs poorly on both the training data and new data.

Bias
For example, if an algorithm is trained to predict myocardial infarction (heart attack) risk on a dataset made up of 80% male and 20% female patients, then deployed in a hospital seeing an even 50/50 split, it will likely predict risk more accurately for male patients than female patients. This is because it was trained on a non-representative dataset, and myocardial infarction symptoms and risk factors are known to differ between men and women.

Dataset drift
This occurs when the demographic makeup of a population shifts over time — for example, a hospital's local population ages without younger patients moving in to balance it out, or a local area becomes more diverse. These "new" patients likely weren't represented in the original training data, even if that data was representative at the time, leaving the algorithm unable to generalise to the newer population.

Of these four causes, overfitting is the most common.

How can the risk of overfitting be mitigated?

Several methods can help reduce overfitting:

• Increasing the size and diversity of the training dataset — either by aggregating more datasets or via federated learning
• Data augmentation — adding new data, or noise, to the training dataset at pre-agreed intervals
• Feature selection — identifying the most important features in the training data and eliminating irrelevant or redundant ones (e.g., removing the "snow" feature in the wolves-vs-huskies example)
• Regularisation — identifying and reducing noise within the data when feature selection isn't possible (e.g., when it's not known in advance which features are most relevant), often used when an algorithm has become too complex

Is generalizability a "set it and forget it" problem?

No. These methods improve a model's generalizability upfront, but generalizability can still degrade over time — largely due to issues like dataset drift. This is why it's important to continuously monitor an algorithm's performance after deployment, so that generalizability errors can be identified and addressed as they arise.