Multimodal data

How does this apply to how humans experience the world?

Humans experience the world in an inherently multimodal way, using all five senses (sight, sound, smell, touch, and taste) to gather and interpret information. During a conversation with a friend, you look at their appearance and body language, listen to their words and tone of voice, notice whether they're wearing perfume, and perhaps hug them or make other physical contact. All of that information is taken in by the brain and used to reach a conclusion about how that person is feeling. If we relied on just one sense, we'd likely reach less accurate conclusions, for example, misinterpreting a text or email from a friend because we can't hear the tone of voice that would normally clarify how their words should be read.

How does this apply to AI?

The same logic applies to AI. Multimodal data means data captured in multiple different formats and joined together to reach a conclusion. The difference is that while the human brain takes in sights, smells, and sounds, AI algorithms typically take in data in the form of text, images, audio files, or videos.

For instance, every time you see a picture with a caption on social media, like Instagram, that's an example of multimodal data, it's image plus text.

In healthcare, multimodal data might include:

• Data from a person's electronic health record (EHR)
• An image captured by an X-ray
• The radiologist's written description of that X-ray
• Blood test results

Human clinicians typically draw on all of this data to reach a diagnosis. Relying on just one input, such as blood test results alone, could result in an inaccurate diagnosis. This is why AI algorithms designed for healthcare should also be multimodal, combining data from multiple sources and formats to produce more accurate results.

What is multimodal machine learning?

Multimodal machine learning is a sub-type of machine learning aimed at developing AI models that combine insights from multiple types of data. Multimodal data itself is simply data captured in different formats, such as image, text, EHR, video, audio, genetic, or self-reported data (like questionnaires). The UK Biobank is one example, containing patient-level genomic data, imaging data, and data from EHRs and questionnaires.

Multimodal models are thought to produce more accurate and robust predictions and outcomes, because no single type of healthcare data can provide all the information needed to make an accurate diagnosis, prediction, or treatment recommendation. An MRI scan, for example, might indicate the presence of a tumour, but it won't reveal what kind of tumour it is (a biopsy and pathology data would be needed for that), what medications the patient is already taking, or their age or ethnicity. Without this additional information, it wouldn't be possible to determine the exact type of cancer, predict a prognosis, or identify the most appropriate treatment.

What is data fusion?

At the core of any multimodal machine learning project is a process called data fusion, the mechanism by which different types of data (modalities) are linked, or fused, using machine learning or deep learning. There's no single agreed-upon technique for doing this; options range from relatively simple methods (like concatenation or weighted sum) to more complex ones (like attention-based recurrent neural networks or graph neural networks).

The choice of fusion strategy is critical and often determines whether a multimodal project succeeds. It depends on the specific modalities involved, the intended application of the model, and where in the development pipeline the fusion takes place: early, intermediate, or late.

• Early fusion (feature or data-level fusion) joins two or more input modalities into a single feature vector, which is then used as input for a single machine learning model. Techniques here are relatively simple, such as pooling or concatenation.
• Intermediate fusion (joint fusion) extracts features from different modalities in a stepwise process, with learned feature representations from each modality combined at different layers of a neural network to form a single input for the final model.
• Late fusion (decision-level fusion) involves training separate models, one per modality, then combining their outputs into a single prediction. This is the most complex approach, often involving ensemble methods.

What challenges does multimodal learning present?

The results of multimodal learning can be impressive, but it does raise several challenges:

• Multimodal models are more complex, more computationally expensive, and less interpretable than single-modality models.
• It can be difficult to simultaneously reduce complexity across modalities to make them work together harmoniously, without losing important detail contained within each one.
• The more information collected and combined about an individual, the easier it becomes to re-identify them, which makes multimodal models a particularly acute privacy challenge.

None of these challenges are insurmountable. Federated learning, for example, can help address the privacy challenge. It's simply important to be aware of these risks so they can be proactively mitigated.