Why letting every model cram makes the leaderboards agree

TL;DR

• In Quanta, Moritz Hardt argues that LLM benchmark rankings have lost their reliability. His fix is to prepare every model the same way before testing it.
• His group fine-tunes 61 models on each of 24 benchmarks before scoring them. Average agreement between rankings (Kendall's $\tau$) rises from 0.52 to 0.76.
• If equal preparation makes rankings carry over between benchmarks, one well-run test tells you much more about which model is better.

What happened

On October 16, Quanta published John Pavlus's interview with Moritz Hardt, "a director at the Max Planck Institute for Intelligent Systems." The interview is about Hardt's new book, The Emerging Science of Machine Learning Benchmarks, where he calls benchmarks "the mistake that made machine learning." His argument has two halves. First, absolute scores mean little: "If you have 90% accuracy on one dataset and 73% on another, which is it? I have no idea how good this model is." Second, rankings used to be dependable. A model that beats another on one test set "will still do better on other datasets, even ones that are quite different than the test set."

LLMs break that second half, he says, because "We don't know what LLMs were trained on." A developer can therefore build training data aimed at particular benchmarks. His analogy is the 1,500 meters at the 1968 Mexico City Olympics. A heavily favored American lost to a Kenyan runner who had trained at high elevation. The sporting answer was to let everyone train at altitude, not to ban it: "If everyone optimizes, or 'altitude trains,' the absolute numbers will be inflated, but you can restore the validity of the rankings." In practice, that means a test set plus "some kind of protocol for preparing each of the models before you test them." These are Hardt's arguments in an interview. The article names no specific study. The evidence is in the paper below.

The paper in brief

Benchmarks that claim to measure the same skill often rank LLMs differently. Zhang, Dominguez-Olmedo and Hardt suggest this is because models arrive with very different amounts of preparation for each task. Their fix is called train-before-test. Before scoring a model on a benchmark, fine-tune it on that benchmark's training split using one fixed recipe:

• LoRA, which trains small low-rank update matrices instead of all the weights
• 5 epochs
• the best of three learning rates, chosen on a validation set

They ran this on 61 open models with up to 14B parameters, from six families (Llama, Qwen, Gemma, Pythia, GPT-2, Yi), across 24 benchmarks. That is 1,464 fine-tuned models. Ranking agreement went up for 274 of the 276 benchmark pairs. Perplexity became a good predictor of downstream rankings again. The model × benchmark score matrix became "essentially rank one." The paper is notable because it turns a diagnosis from the group's earlier work (models are "training on the test task") into a general evaluation protocol, and shows that the protocol makes rankings transfer between benchmarks.

The math

Setup. Write $S_{ij}$ for the accuracy of model $j$ on benchmark $i$. That gives a 24 × 61 matrix, built twice: once by direct evaluation and once after train-before-test. Fine-tuning is the only optimization step. Everything after it is statistics on $S$, asking two questions: do the rows put the models in the same order, and how many hidden factors explain the matrix?

Kendall's τ: count the pairs that flip

Take two benchmarks and $n$ models. For each of the $n(n-1)/2$ pairs of models, the pair is concordant if both benchmarks order the two models the same way and discordant if they disagree.

$$\tau = \frac{n_c - n_d}{n(n-1)/2}$$

Here $n_c$ and $n_d$ are the numbers of concordant and discordant pairs. $\tau = 1$ means identical rankings, $-1$ means one ranking is the reverse of the other, and 0 means no relationship. Like Spearman's rank correlation, $\tau$ looks only at order. It ignores scale, which matters because Hardt says scale is the meaningless part.

Worked example. Five models. Benchmark 1 ranks them A > B > C > D > E. Benchmark 2 ranks them B > A > C > E > D. Out of 10 pairs, two flip: (A, B) and (D, E). So $\tau = (8-2)/10 = 0.6$.

With no ties, the fraction of discordant pairs is $(1-\tau)/2$. With 61 models there are 1,830 pairs:

• At the paper's direct-evaluation average, $\tau = 0.52$, about 24% of pairs (roughly 440) swap order between two typical benchmarks.
• At $\tau = 0.76$, about 12% swap (roughly 220).

This is a back-of-envelope reading of averages, but it says train-before-test roughly halves the disagreements.

Some differences between models are within noise. Kendall's $\tau_b$ handles ties:

$$\tau_b = \frac{n_c - n_d}{\sqrt{(n_0 - n_1)(n_0 - n_2)}}$$

Here $n_0 = n(n-1)/2$, and $n_1, n_2$ count the tied pairs in each ranking. In the appendix, two models count as tied when a t-test on their standard errors finds no significant difference at the 95% level. So a hypothesis test feeds directly into a rank statistic. In the example, if A and B are indistinguishable on benchmark 2, then $n_2 = 1$, $n_c = 8$, $n_d = 1$, and $\tau_b = 7/\sqrt{10 \cdot 9} \approx 0.74$.

PCA: how many numbers describe a model?

The standard recipe is to center the score matrix, take the eigenvalues $\lambda_1 \ge \lambda_2 \ge \dots$ of its covariance (equivalently, the squared singular values), and report the share of variance explained by the first principal component (PC1):

$$\text{EVR}_1 = \frac{\lambda_1}{\sum_k \lambda_k}$$

If $\text{EVR}_1$ is close to 1, then $S_{ij} \approx \mu_i + a_i z_j$. Each model is summarized by one number $z_j$, which the paper calls "potential," and each benchmark by an offset $\mu_i$ and a loading $a_i$. If every $a_i > 0$, every benchmark sorts the models by $z_j$, so every pairwise $\tau$ equals 1. A rank-one score matrix and agreeing rankings are two views of the same fact.

A toy model makes this concrete. Suppose $s_{ij} = z_j + \sigma u_{ij}$, where $z_j$ and $u_{ij}$ are independent standard normals: a shared potential plus benchmark-specific preparation noise of relative size $\sigma$. Then:

• Any two benchmarks have correlation $\rho = 1/(1+\sigma^2)$.
• For bivariate normals, Kendall's $\tau = \tfrac{2}{\pi}\arcsin\rho$.
• With $k$ equally correlated benchmarks, PC1 of the correlation matrix explains $\rho + (1-\rho)/k$ of the variance.

Shared potential vs. benchmark-specific preparation (toy model)

interactive

• Kendall τ between two benchmarks
• PC1 share of variance
• paper average τ, direct (0.52)
• paper average τ, train-before-test (0.76)

Hover or drag across the chart to read values.

noise σ for readouts0.61number of benchmarks k24

Kendall τ at slider σ:

0.52

share of model pairs that flip:

24%

flipped pairs among 61 models:

439

PC1 share at slider σ:

74%

In this toy, going from τ = 0.52 to τ = 0.76 means cutting benchmark-specific noise from σ ≈ 0.61 to σ ≈ 0.27. PC1's share goes from about 74% to 93%. The paper reports 70% to 86%: the same direction, but the toy does not reproduce the data.

Where "preparation" first showed up

This paper builds on Dominguez-Olmedo, Dorner and Hardt, who fit this regression:

$$A = \alpha \max(0, \log C - c_e) + \theta N + r + \varepsilon$$

The symbols:

• $A$ is benchmark accuracy and $C$ is pretraining compute in FLOPs.
• The term $\max(0, \log C - c_e)$ is a ReLU-shaped hinge. Accuracy stays at chance $r$ until log-compute passes the threshold $c_e$, then rises with slope $\alpha$.
• $N = 1$ if the model was trained after November 2023, and 0 otherwise.
• $\theta$ is the newer models' advantage at equal compute, and $\varepsilon$ is noise.

This is ordinary regression with a dummy variable, followed by a significance test on $\theta$. Before adjustment, newer models beat older ones "by over 7 accuracy points in MMLU and 19 accuracy points in GSM8K." After every model was fine-tuned on the same task data, $\hat\theta$ was "both small and not statistically significant." The newer-is-better gap was preparation, not capability.

What the results say

• Cross-benchmark agreement: average $\tau$ rises from 0.52 to 0.76, and 274 of 276 benchmark pairs improve. NQ-Open, an outlier under direct evaluation, goes from 0.23 to 0.74 average agreement with the other benchmarks. Within the math category, agreement goes from 0.55 to 0.84. Treating non-significant differences as ties ($\tau_b$) gives 0.58 to 0.77.
• Perplexity: agreement between perplexity rankings and benchmark rankings rises from 0.48 to 0.74. Perplexity was measured on fresh 2025 text from Wikipedia, StackExchange and arXiv. For base models, perplexity measured before fine-tuning predicts benchmark rankings after fine-tuning ($\tau$ = 0.78). For instruction-tuned models it predicts much less well (0.51).
• Structure: PC1's share of variance goes from 70% to 86% overall, and to 93% within the Qwen family. PC1 scores line up with pretraining compute.

Limitations. The authors flag three:

• Fine-tuning makes evaluation more expensive.
• Many benchmarks no longer ship training data.
• Some commercial providers don't allow fine-tuning.

A careful reader should also notice:

• Scope. Every model is open and at most 14B parameters. Most benchmarks are multiple choice, and only two (GSM8K, NQ-Open) require generated answers.
• Benchmark selection. Six benchmarks were dropped for fine-tuning anomalies or odd behavior. One of them, Paws-EN, ranked models in a way that disagreed with 23 of the other 24 benchmarks under direct evaluation.
• "Potential" depends on the recipe. It means potential under one specific protocol: LoRA rank 8, 5 epochs, 3 learning rates.
• A wide compute range. The model pool runs from Pythia-70M up to Qwen2.5-14B, so one dominant factor is less surprising than it sounds.
• Not deployment readiness. The authors say train-before-test complements direct evaluation and doesn't replace it.

Connecting the dots

Quanta doesn't name this paper, so the link is our inference. Hardt's "protocol for preparing each of the models before you test them" describes train-before-test exactly. He is the paper's senior author, and the paper cites his benchmarks book manuscript. In the paper, "restoring validity" has a specific meaning: rankings carry over from one benchmark to others (external validity) and agree with perplexity. It does not show that the rankings match real-world usefulness.

The interview reaches further than the paper. It covers the history of benchmarks from ImageNet onward, the claim that frontier models are "benchmaxxed," and a future with "no training set" at all. The paper tests none of the closed frontier models those concerns are mostly about.

The interview also touches on reusing holdout data. Classical statistics warns that a reused public test set leads to adaptive overfitting: every decision made by looking at test scores leaks a little information about that particular test set. Hardt says the field ignored the warning and rankings survived anyway, and the paper cites earlier work showing that rankings on classic benchmarks transfer well. The LLM problem is different. It isn't the test set leaking. Each model was trained on a different, undisclosed data mix, and that mix confounds the comparison. Train-before-test addresses that confound, not adaptive overfitting.

Why it matters

For practitioners picking a base model to fine-tune, this is a recipe that may avoid picking the wrong model: compare models after equal preparation. A cheaper option for base models is to rank them by perplexity on fresh text, which tracked post-fine-tuning rankings well. That shortcut did not hold for instruction-tuned models.

For researchers, the authors ask benchmark creators to ship training splits again, and they argue that closed models that can't be fine-tuned are the providers' problem. For anyone relying on leaderboards, including procurement and policy decisions, the paper is a reminder that out-of-the-box scores measure readiness, which is not the same thing as potential.

Two things to watch: whether a major leaderboard adopts a shared preparation protocol, and whether the rank-one result holds for frontier-scale models and for generative or agentic benchmarks.

Go deeper

• Train-before-Test Harmonizes Language Model Rankings (arXiv), with code on GitHub
• Quanta: He Wants AI Benchmarking to Be a Science, Not a Dark Art
• Training on the Test Task Confounds Evaluation and Emergence by Dominguez-Olmedo, Dorner and Hardt (ICLR 2025), the precursor with the $\theta$ regression
• The Emerging Science of Machine Learning Benchmarks, Hardt's book manuscript as cited in the paper