Bias

What is bias in a healthcare context?

In healthcare, harmful bias typically stems from unjustified stereotypes tied to characteristics such as race, ethnicity, gender, religion, sexual orientation, socioeconomic background, or educational attainment. In a hospital setting, these biases can lead to people of specific genders, sexual orientations, or ethnicities receiving inaccurate diagnoses or poorer-quality treatment. 

Bias also shows up in medical research. If a drug trial's sample is largely white men in their twenties, there's likely to be a gap between the research results and how the drug performs across more diverse populations in real life — potentially causing harm to underrepresented groups.

‍

Why does bias matter in AI algorithms?

These biases matter in AI because they become embedded in the datasets algorithms are trained on. For example, algorithms in smartwatches designed to detect atrial fibrillation have been shown to work less well for users with darker skin than lighter skin — most likely because lighter-skinned patients were overrepresented in the training data.

Biased algorithms can deliver a poorer standard of care to the patients they're biased against. The good news: algorithms trained on more diverse population data, tested and validated independently, and monitored once in use are far less likely to be biased.

‍

What are examples of ways bias can arise in a healthcare context?

Bias in healthcare can arise in three main areas: clinical research, clinical consultation, and clinical statistics.

1. Clinical research bias

In research, bias is a systematic (reproducible, not random) distortion of the relationship between a treatment, risk factor, and clinical outcome. It can occur during planning, data collection, analysis, or publication.

Research bias, regardless of source, can produce false conclusions, loss of external validity, loss of generalisability, and patient harm. One of the most well-known examples is the retracted 1990s study falsely linking the MMR vaccine to autism — later shown to be biased due to a small, non-representative sample and inflated results.

2. Clinical consultation bias

In a consultation setting, bias usually refers to cognitive bias — the tendency for humans to rely on intuitive, "fast thinking" rather than slower analytical thinking. This is often an unconscious process: for example, assuming a woman presenting with weight gain, nausea, and tiredness is pregnant. Problems arise when this fast thinking is shaped by harmful stereotypes related to race, ethnicity, gender, sexual orientation, geography, education, or socioeconomic status, leading to diagnostic errors and poorer outcomes.

A related type is automation bias — where people assume computers are always right and accept AI-generated information without question. Clinicians should feel confident reviewing and challenging any AI output that doesn't look correct or feels "off."

3. Clinical statistics bias

In a statistical sense, bias is any systematic difference between the true parameters of a population and the statistics used to estimate them — a gap between "the results" and "the truth." This most often stems from incomplete or unbalanced datasets used to train statistical models, including AI/ML algorithms.

These datasets may be biased due to cognitive bias during consultations (e.g., overdiagnosis in one group, underdiagnosis in another), or bias baked into clinical research — both producing skewed training data. Algorithms trained on this data can lead to uneven distribution of resources, unwarranted variations in care, and discrimination.

‍

What is dataset drift?

Dataset drift is when the statistical properties of the data used to train a machine learning model change over time, due to shifts in underlying patterns, relationships, or distributions. This affects model performance and requires ongoing monitoring, evaluation, and techniques like retraining with new data to keep the model accurate.

‍

Can bias be eliminated from healthcare algorithms?

No — none of these sources or types of bias can be fully eradicated, and bias isn't a binary present/absent measure. Most medical algorithms will be biased to some extent, just as clinical research always carries some degree of bias. What's achievable is mitigation and reduction, not elimination. See how Owkin approaches this in K Pro's recommendations, for example, through diverse training data and ongoing monitoring

The "black box" nature of some medical algorithms can also make it difficult to identify and correct the source of bias. For this reason, everyone responsible for developing, deploying, and using AI algorithms in healthcare needs to stay alert to the risk of bias.