Benchmarking

How do you benchmark an AI model?

Benchmarking an AI model typically involves testing it against one or more standardized tasks, using shared datasets. This allows different research groups to compare results fairly.

A model's performance is then measured using quantifiable metrics, for example, accuracy, the C-index, or AUC-PR (area under the precision-recall curve). Then it can be compared directly against the performance of other existing models.

This works well when the task itself is clearly defined and the correct answer is known in advance, for example, classifying whether a scan shows a tumour, or predicting whether a molecular target is associated with a drug that has reached a certain phase of clinical trials. In these cases, benchmarking is largely a matter of running the model on the same test set as its competitors and comparing the resulting scores. Here are a few well-known examples used in healthcare AI:

• CAMELYON - a structured benchmark with a defined task and public leaderboard, for detecting metastatic cancer in pathology slides.
• CheXpert - a structured benchmark for detecting abnormalities in chest X-rays, with a fixed dataset and expert-labelled test set.
• MIMIC-III/IV - large, publicly available databases of real ICU patient records, rather than a single defined benchmark. Researchers build their own standardized tasks on top of this data, such as predicting patient deterioration or length of hospital stay.
• BioMysteryBench - a bioinformatics benchmark, specifically designed to compare agentic systems’ performance, referenced later in this article.

Where does benchmarking become hard in healthcare AI?

Benchmarking becomes far more difficult when the AI being evaluated isn't just predicting a single, well-defined outcome, but is instead an agentic AI system (or AI scientist) designed to reason, design experiments, interpret results, and iterate on complex, real-world problems, much like a human scientist would.

Unlike a narrow model built for one task, an AI scientist agent's "job" is closer to an open-ended research process: proposing hypotheses, selecting experiments, interpreting noisy real-world feedback, and adapting its strategy over time. There's often no single "correct" next step to compare it against, several different experimental choices could all be reasonable, making it harder to define a clear, objective benchmark for success.

And there is a more fundamental problem: for open scientific questions, a "correct" signal isn't available to benchmark against. To benchmark an AI system’s ability to make real scientific discoveries, one would need to wait for the gradual accumulation of scientific validation across experiments, publications, and clinical results over years. 

Benchmarking AI scientists is hard, but we still try: what tools exist, and how do they work?

Despite these challenges, researchers haven't given up on benchmarking agentic AI systems, they've just had to build different kinds of tools. A few common approaches:

• Proxy tasks - rather than waiting years for a real-world outcome (like a drug passing clinical trials), researchers test an agent on a faster, related task that's believed to correlate with good scientific judgement, for example, whether an agent can correctly reproduce a finding from already-published research.
• Held-out ground truth benchmarks - datasets where the answer is known but deliberately withheld from the model, often built from anonymised biological data, so a system has to genuinely deduce the answer rather than recall it. BioMysteryBench, a bioinformatics benchmark published by Anthropic, is one example: each problem hands a system a pile of anonymised biological data and asks it to deduce one hidden fact, such as which organ a set of cells came from.
• Lab-in-the-loop evaluation - testing an agent's ability to propose an experiment, receive real (often messy) experimental results, and adapt its next step accordingly, closer to evaluating a process than a single output. You can read more about Owkin’s work here in this blog.
• Expert panels and human review - for tasks with no fast objective answer, having domain experts assess whether an agent's reasoning, hypothesis, or experimental design is sound, even if the ultimate outcome won't be known for years.

None of these fully solve the problem, each is a compromise, trading off speed, realism, or objectivity in different ways, but together they let researchers get a meaningful read on an agent's capabilities without waiting a decade for a definitive answer.

What do you actually benchmark for? Accuracy? Reliability? Creativity?

Even once a suitable benchmark exists, there's a further question: what quality is it actually trying to measure? A few common targets:

• Accuracy - does the system reach the correct answer? This is the most intuitive metric, but as discussed above, it's not always available quickly, or at all, for genuine scientific discovery tasks.
• Reliability - does the system reach the same answer consistently, when asked the same question more than once? Being able to rely on a system using the same approach each time for problem solving is important for interpretability and trust in the system.
  For example, in a benchmark comparing Owkin's K Pro with Claude on the BioMysteryBench benchmark, their raw reasoning performance was similar, but when asked the same question repeatedly, K Pro analysed the same patient cohort 90% of the time, compared with 58% for Claude.
• Creativity or novelty - can the system generate genuinely new hypotheses or experimental directions, rather than just reproducing patterns already well-represented in its training data? This is especially hard to benchmark objectively, since "creative" and "correct" aren't always the same thing.
• Efficiency - how much time, computation, or experimental resource does the system use to reach a given result?

Different benchmarks are designed to isolate one or more of these qualities, which is why a single benchmark score rarely tells the whole story about whether an AI scientist agent is actually good at its job. It’s important to bear some general limitations in mind. 

For example, proxy tasks often fail to account for the nuances of real world performance - as when dealing with messy clinical trial data. Benchmarks often rely on held-out ground truth tasks - but these truths can't be held out forever. As an example, the answers to BioMystery bench are now available for models to train against.

Lastly, when using expert panels, they are hugely time consuming - and simple human reviews can miss the details that experts would pick up. Even where expert panels can be used, real science isn't conducted by a small panel of experts but by the global community. Small panels of experts will often fail to see where the science is headed.

In summary, benchmarking AI scientists is complex. It needs to measure more than accuracy. In real scientific settings, agents work with ambiguous questions and noisy feedback, so consistency and reproducibility can matter just as much as getting the right answer.