Benchmaxxing is a real incentive, but a high score isn't proof of cheating

Benchmaxxing is the habit of tuning a model to do especially well on the benchmarks everyone watches. The word sounds like an accusation, so it helps to separate two claims:

Developers have an incentive to spend effort on what the leaderboard measures.
A specific team secretly trained on the test answers or cheated.

The first is built into the setup. The second needs evidence about that team. A good score by itself proves neither.

What the evidence actually shows

In a controlled study, researchers compared older and newer language models on MMLU and GSM8K. At similar pretraining compute, the newer cohort scored about 7 percentage points higher on MMLU and 19 points higher on GSM8K. The researchers then fine-tuned models on the same task-relevant data before testing; the apparent gap largely disappeared. They also deliberately fine-tuned older models on those tasks and reproduced similar score gains (Dominguez-Olmedo, Dorner and Hardt).

That is good evidence that becoming better at the shape of a test can move scores and rankings, even without memorising exact test questions. Their analysis found that benchmark rankings could shift by an average of 7.7 places on GSM8K after adjustment, with a maximum shift of 21. This doesn’t prove any named lab was gaming a leaderboard. It does show that a leaderboard can partly measure how much a model has practised the task format.

A newer paper gives this an incentive model: when public test sets and rankings shape attention, developers can rationally spend post-training effort on benchmark-specific improvements. Its authors call this benchmaxxing, and they explicitly distinguish the incentive from cheating or wrongdoing (Leaderboard Incentives under Strategic Post-Training). That’s a formal analysis of the incentive, not an investigation into what any specific company did.

The useful question is transfer

If a model improves on the benchmark, does it also improve on fresh problems that test the same skill? That is the bit a single leaderboard number can’t answer. A strong evaluation should include a hidden or newly generated test set, show task-level results, and make the scoring setup reproducible. If the model wins only on a familiar public set, I would treat that as evidence about that set, not a general certificate of intelligence.

This is also why I don’t find one overall ranking very useful. What I want to know is whether the model can do my task reliably, at the cost and latency I can live with. The benchmark is a starting clue. The proof is whether the capability transfers.

The cited studies establish that test-task exposure and benchmark-specific effort can affect measured performance. They do not establish intentional misconduct by a particular model developer.