Zero Shot Learning

What is auxiliary information?

Auxiliary information is additional information that describes an image without directly labelling it. For example, during training the algorithm might see an image of a fruit basket containing 6 apples, 2 bananas, and 5 oranges, described only as "this is a basket containing a number of different fruits, including 5 oranges." The algorithm is never explicitly told the oranges are oranges, but it can infer that the object appearing five times is an orange. As a result, the next time it sees an image of an orange tree, it's able to correctly classify it.

The same approach can be used to train algorithms to correctly classify medical images, for example, images of lung cancer, without needing a labelled training dataset that includes an example of every possible lung cancer tumour. This reduces the need for very large, accurately labelled datasets, which can be extremely difficult to produce, making algorithm training more efficient.

Why does zero-shot learning matter beyond image classification?

Zero-shot learning is a type of transfer learning that has become especially important in natural language processing and generative AI, since it allows a model to take on a new task it wasn't explicitly trained to do. Although it's traditionally been used for image classification, this capability reduces the need for large, perfectly labelled training datasets, reduces the need for a separate algorithm per task, and makes developing new, innovative algorithms more efficient.

How would this work in a real healthcare example?

Using zero-shot learning, an algorithm trained to classify chest X-ray images of patients with pneumonia and asthma could be adapted to also classify chest X-rays of patients with COVID-19, without needing to be retrained. Retraining would have been expensive, time-consuming, and potentially even impossible early in the pandemic, when there weren't yet enough COVID-19 X-ray images available.

The process would work like this:

1. Pre-training — the algorithm is first trained on a labelled dataset of "seen classes," a series of chest X-rays from patients with pneumonia or asthma.
2. Auxiliary information — the algorithm is then given additional information, such as a text prompt, describing a new class of data, in this case, a description of what COVID-19 chest X-rays look like and how they compare to the pneumonia and asthma images.
3. Inference — the algorithm is presented with a series of COVID-19 chest X-rays and uses the auxiliary information from the prompt to infer the correct classification.

This process could then be repeated for other types of chest X-ray images, including those showing lung cancer at different stages.

What's the key takeaway?

In essence, zero-shot learning is a method for making algorithms capable of "multitasking," classifying entirely new categories of data without needing to be retrained from scratch for each one.