C-Index

Why does performance testing matter?

Before an algorithm can be deployed in a clinical setting, it's essential to test how well it performs. This means asking questions like "is the algorithm capable of correctly recognising the presence of a tumour in a scan?" — and answering them using mathematical tests that measure accuracy, such as how often the algorithm correctly recognises a tumour versus how often it misses one.

Testing performance is the penultimate step in the algorithm development pipeline: evaluation. Robust evaluation must be completed before an algorithm can be safely deployed into a clinical setting.

Does testing performance in a lab guarantee real-world performance?

No. There's an important difference between testing an algorithm's performance in a "lab" setting and testing it in a real clinical environment. An algorithm that performs well on the C-index using clean, retrospective data might perform worse in an actual hospital, where IT equipment may be outdated or the patient population may differ from the one used to train and test it. This is why algorithms need to be tested both mathematically ("in the lab") and practically ("in the clinic") to confirm performance holds up.

What are the different types of performance metrics, and when are they used?

The list of available performance metrics is extensive, and each one is better suited to a different type of algorithm — classification, prediction, or regression.

• Classification algorithms have binary outputs (e.g., 1 = disease present, 0 = disease absent) and are typically evaluated using metrics like accuracy, sensitivity, specificity, precision, and the F1 score.
• Prediction algorithms may involve both binary and continuous variables (e.g., height), and sometimes censored variables — values not present for all patients, such as time-to-death. In these cases, classification metrics often aren't appropriate, so the C-index is used instead.
• Regression (multivariable) algorithms are evaluated by comparing predicted outcomes to actual observed outcomes and measuring the error rate, using metrics such as root mean squared error or mean absolute error.

Because prediction models are so common in survival analysis — where the goal is often to estimate when an event like disease progression or death might occur, rather than simply whether it will — the C-index has become one of the most widely used metrics in this space.

How does the C-index actually work?

The C-index estimates the probability of concordance (agreement) between predicted and observed outcomes. It does this by looking at pairs of patients with different outcomes and evaluating whether the model assigns the correct outcome to the correct patient.

For example, if an algorithm is developed to predict likelihood of hospital admission, and it assigns a higher likelihood to the patient who was actually admitted first than to the one admitted second, the prediction is considered concordant (in agreement) with the actual outcome, and a 1 is added to the count of concordant pairs.

• If there's a tie between the pair → 0.5 is added
• If the pair is not concordant → 0 is added

The denominator is the number of patients with the outcome of interest multiplied by the number of patients without it.

• A C-index of 1 indicates perfect separation between patients with different outcomes
• A C-index of 0.5 indicates the model is no better than a guess

Is the C-index enough on its own to judge an algorithm's performance?

No. Because every performance metric has different strengths and weaknesses, data scientists typically test an algorithm using multiple metrics alongside the C-index. It's also worth noting that in healthcare, a high performance score isn't the whole story — what matters most is whether the algorithm leads to the right outcome for patients and clinicians in practice, and whether that outcome is fair. Evaluating this requires additional performance metrics and clinical trials, not statistical testing alone.