Quality vs quantity of data

Why is data quality harder to measure than data quantity?

Data quantity is relatively easy to measure. Data quality is much harder to measure, because what counts as "good" depends heavily on context.

This is why an important first step in any AI project is deciding ‘what good looks like’ for the model’s training data. For example: is the data up to date? Is any important information missing?

Here are some of the key dimensions that should be considered when assessing it*:

• Completeness: Does the dataset contain all the elements needed? Are any missing? Is it biased in any way?
• Uniqueness: Are there duplicates, for example, patients appearing more than once?
• Conformance: Is the dataset formatted to the required standard (e.g., are dates recorded in the expected format)?
• Timeliness: Is the data up to date, and can it be accessed easily when needed?
• Accuracy: Is the data correct according to an agreed source of truth?
• Consistency: Are specific metrics consistent across multiple sources (e.g., does the number of patient records from a GP practice match the number on the patient list)?
• Information loss and degradation: Will the quality of the data degrade over time?

What happens if data quality issues are identified?

Once an initial assessment is complete, data quality can be improved with methods including:

• Data curation: removing outliers, i.e., data points that are inaccurate, irrelevant, or so unusual they risk skewing the model's learning.
• Data harmonisation: aligning data collected from different sources or systems, such as different hospitals using different coding systems, so it can be consistently understood and combined.
• Data standardisation: converting data into a common format or scale, so values from different sources follow the same structure and can be compared fairly.
• Missing data interpolation: estimating an unknown value based on previous values, filling in gaps in a dataset using existing patterns, rather than leaving incomplete records that could undermine the model's accuracy.

‍

Can you train healthcare AI on smaller amounts of better quality data?

Yes, and in many cases, this is a preferable approach. This is particularly relevant in healthcare, where large volumes of perfectly labelled, representative data aren't always available or practical to collect. Here a smaller, carefully curated dataset, can produce a more reliable and trustworthy algorithm than larger, more scatter-gun datasets.

This is made possible through techniques like zero-shot learning.These techniques reduce the need for high data volume by allowing an algorithm to reuse what it has already learned. Instead of needing a labelled example of every possible scenario, the algorithm is pre-trained on one dataset, then given additional descriptive information (known as auxiliary information) so it can infer how to handle a new category it has never seen. For example, an algorithm trained on chest X-rays of patients with pneumonia and asthma can be adapted to recognise COVID-19, without being retrained on a large new dataset.

Other approaches work in a similar way. Data augmentation makes a small dataset go further, for example by flipping, rotating, or changing the resolution of images, so the algorithm learns which features matter most. Foundation models take this a step further: they are pre-trained on very large, broad datasets, then fine-tuned for a specific clinical task using a much smaller amount of high-quality, task-specific data. In each case, careful curation matters more than volume, because a small dataset leaves less room for errors and bias to be averaged out.

Why does this ultimately matter for AI in healthcare?

Developing robust, reliable, and safe AI algorithms depends on sufficiently high data quality. Poor quality data inputs can lead to dangerous, poorly performing algorithms, hence the expression "rubbish in, rubbish out." This is why it can sometimes be better to use less data of higher quality, rather than more data of poorer quality.

*Other dimensions of data quality:

• Concordance
• Plausibility
• Relevance
• Usability
• Security
• Flexibility
• Interpretability