Every model launch comes with a bar chart, and the bars are almost always accurate. That is what makes them effective. The problem is not fabrication — it is that a benchmark measures one narrow thing under one set of conditions, and the chart invites you to generalise far beyond it.
The short version
- Ask what the benchmark actually measures before you look at who won.
- Contamination is the default assumption: if the test is public and old, it is probably in the training data.
- A number with no error bar and no described method is a claim, not a measurement.
- The only evaluation that reliably predicts your results is one built from your own tasks.
A benchmark is a proxy, and proxies drift
Benchmarks exist because the thing we care about — ‘is this model useful for my work’ — is expensive and subjective to measure. So we substitute something cheap and countable: multiple-choice exams, coding puzzles with unit tests, question-answering over a fixed corpus.
That substitution is reasonable right up until the proxy becomes a target. Once a leaderboard matters commercially, effort flows toward the leaderboard, and the correlation between the score and the underlying quality weakens. This is Goodhart’s law, and it applies to model evaluation with unusual force.
When a measure becomes a target, it ceases to be a good measure. Leaderboards are measures that have very much become targets.
Contamination: the failure that invalidates the number
Models are trained on enormous crawls of the public internet. Public benchmarks live on the public internet. If a test set was published before a model’s training cut-off, the sensible prior is that some of it was seen during training.
A contaminated benchmark does not measure reasoning. It measures recall of the answer key, and it reports a high score either way. This is why a model can post excellent results on a famous exam and then fail at a structurally identical problem written last week.
What to look for
- Was the evaluation set created after the model’s training cut-off, or held privately?
- Do the authors report a contamination check at all?
- Does performance collapse on rephrased versions of the same problems? That gap is the memorisation signal.
Read the conditions, not just the score
Two labs can evaluate the same model on the same benchmark and publish different numbers without either lying. The conditions do the work:
- Prompting. Zero-shot, few-shot, or chain-of-thought — each can move a score substantially.
- Attempts. One try, or best-of-many? ‘Best of 64’ is a meaningfully different claim from ‘first answer’.
- Scaffolding. Tool access, retrieval, code execution and self-checking are frequently included in the harness and omitted from the headline.
- Scoring. Exact match, human preference, or another model acting as judge — and if a model is judging, which one, and does it prefer its own style?
A difference of a few points between two systems evaluated under different harnesses tells you approximately nothing.
Averages hide the distribution you care about
Aggregate scores conceal variance. A model that is excellent on most inputs and catastrophic on a predictable minority may average out ahead of a model that is consistently good — and be far worse in production, because your users will find the failure cases and you will own the consequences.
This matters most where errors are expensive. For anything touching money, safety or law, the shape of the tail matters more than the mean, and almost no public leaderboard reports it.
Build the only benchmark that predicts your results
The evaluation that actually forecasts your outcome is one assembled from your own work. It does not need to be elaborate:
- Collect twenty to fifty real tasks from your actual workload, including the awkward ones.
- Write down what a good answer looks like before you run anything, so you are not grading on vibes afterwards.
- Run each candidate model under identical conditions — same prompt, same tools, same number of attempts.
- Score blind where you can, so you are not rewarding the output you expected to win.
- Keep the set and re-run it whenever you consider switching.
Fifty representative examples from your domain will tell you more than every public leaderboard combined, and the set keeps paying out every time a new model appears.
What benchmarks are still good for
Directionally, they are useful. Very large gaps are usually real. Trends across many benchmarks and several model generations carry signal. And a poor score on a well-constructed, uncontaminated test is meaningful negative evidence.
Treat them as a filter, not a verdict — the same way you would treat a processor’s peak clock speed when buying a machine, a habit we argue for in reading a laptop spec sheet.
Frequently asked questions
Why do models score well on exams but fail at my task?
Exams are multiple-choice, self-contained and well represented in training data. Your task probably has ambiguous requirements, implicit context and no single correct answer. Those are different skills, and the first does not imply the second.
Are human-preference leaderboards more trustworthy?
They measure something real but specific: which response people prefer at a glance. That rewards confident, well-formatted, agreeable answers, which is not identical to rewarding correct ones.
How often should I re-evaluate?
When you have a concrete reason — a new model in your price band, a change in your workload, or a rise in user complaints. Re-running a fixed set on a schedule with no decision attached is just expense.
