What a Model Leaderboard Leaves Out

Executive summary

A leaderboard compresses a large evaluation into an ordering. That is useful when deciding what to investigate. It becomes misleading when the ordering is treated as an answer to a different question: which model should run this product?

The relevant gap is between the benchmark's task distribution and the application's. A system extracting invoice lines, answering customer questions or operating tools can fail in ways that a broad knowledge score barely touches.

Revised September 13, 2026. The earlier version overstated what a benchmark comparison could establish about contamination.

A lower score does not explain its own cause

MMLU measures performance across a broad collection of academic and professional subjects. Its breadth makes it a useful research instrument; it does not make its question mix representative of every application. 2

Microsoft's MMLU-CF work reports a GPT-4o score of 73.4% in its five-shot evaluation. Crucially, the authors describe the new benchmark as more challenging, as well as designed to reduce contamination. Comparing that score with a number from another dataset cannot isolate the effect of contaminated training data. Dataset construction, prompting and difficulty also matter. 1

That distinction is worth preserving even when the underlying concern is valid. An evaluation can provide evidence that rankings are sensitive to the test without quantifying why every difference occurred.

An average hides the expensive mistakes

Imagine two invoice extractors with similar aggregate accuracy. One occasionally omits a field and asks for review. The other fills the field with a plausible but incorrect amount. The second can create a more expensive downstream problem, even if the headline score looks slightly better.

The scoring rule needs to express that difference. Otherwise model selection rewards a behaviour the product should discourage.

The same issue appears in tool use. A valid tool name is different from a valid argument, which is different again from an authorised action. A product needs to measure the boundary it actually depends on.

Evaluation already has more than one dimension

It would be wrong to claim that public evaluations never examine cost, robustness or other practical dimensions. HELM explicitly argues for evaluating models across multiple scenarios and metrics. 3

The limitation is transfer. A public measurement may use a different provider, context length, concurrency level or acceptance rule. Even a carefully reported result needs to be translated into the conditions of a specific service.

Decision	Evidence to collect
Can the model do the work?	Task-level success, with failure categories
Can it do the work repeatedly?	Variation across runs and input cohorts
Can the service keep up?	Latency distribution under the expected load
Is it economical?	Cost per accepted result, including retries and review
Can we operate it?	Failure handling, observability and change control

The shortlist and the release gate

A practical organisation can use two different levels of evaluation. Broad external evidence creates the shortlist. A maintained set of product tasks decides whether a candidate is ready to replace the current model.

That second set should contain ordinary cases as well as memorable failures. Testing only hard examples gives a distorted estimate of everyday performance; testing only easy examples hides the reasons users lose trust. Keep development examples separate from the held-out cases used to judge a change.

When two candidates are close, inspect the disagreements rather than relying solely on their averages. Those cases often reveal whether the product needs a different prompt, better input data, an abstention path or a different model.

The result is a decision a team can explain: this candidate handles these tasks, within these limits, at this operating cost. That explanation remains useful after the leaderboard changes.