Elo, Pass@1, and MMLU: A Field Guide to Benchmark Literacy

Scroll through enough model announcements and you'll see three kinds of numbers treated as if they belong to the same family: an Elo rating in the 1300s, a pass@1 score in the 80s, an MMLU score in the 90s. They get plotted on the same kind of chart, discussed in the same kind of sentence, and absorbed by readers as roughly interchangeable signals of "how good" a model is. They are not interchangeable. They are not even measuring the same kind of thing. Comparing them directly is closer to comparing a batting average to a credit score than to comparing two versions of the same statistic.

None of that makes any of the three useless — each was built to answer a specific, narrow question well, and each does that job better than almost any alternative. The trouble starts when a number designed to answer one narrow question gets read as an answer to a much bigger one. This is a field guide to what each of these measurements actually is, what it's genuinely good at, and where it quietly stops telling you anything at all.

Elo Was Built for Chess, Not for Chatbots

Elo ratings come from competitive chess, where they solve a clean problem: given a long history of head-to-head wins and losses between players of unknown skill, produce a single number for each player that predicts future outcomes. Arena-style model evaluations borrow the same math for a different kind of match. Two anonymous models answer the same prompt, a person looks at both responses without knowing which model produced which, and picks the one they prefer. Do that across a large enough population of prompts and voters, and each model accumulates a rating the same way a chess player would, climbing when it wins a comparison and dropping when it loses one.

What this setup is genuinely excellent at measuring is aggregate human preference under blind, pairwise conditions — which response a broad population of people would rather receive, on average, across a wide mix of everyday prompts. That is a real and useful thing to know. It is also a completely different thing from correctness. A confident, well-formatted, pleasantly worded answer that is subtly wrong will often beat a correct answer that reads as terse or hedged, because the voter is judging the experience of reading the response, not independently verifying its content line by line. Elo also inherits whatever biases live in its voting population and its prompt distribution — it tells you what a particular pool of raters, asking a particular mix of questions, tended to prefer, and it generalizes only as far as your own use case resembles that pool and that mix.

The rating itself also takes time to settle, in the same way a new chess player's Elo is noisy until they've played enough games for the number to reflect real skill rather than a lucky streak. A model with relatively few comparisons logged can show a volatile, misleadingly high or low rating simply because it hasn't accumulated enough head-to-head matchups yet. And because raters are typically skimming rather than working through a problem in depth, formatting has an outsized effect on the outcome — a response broken into a tidy list with bolded headers tends to out-poll a plainer response of equal or even superior substance, simply because it looks more thorough at a glance.

Pass@k Measures the One Thing Code Can Prove

Code benchmarks solve a different problem, and they solve it by borrowing something software engineering already has: an objective, mechanical way to check whether an answer is right. Give a model a programming problem, have it generate a solution, run that solution against a hidden suite of unit tests, and you get a binary result with no human judgment involved anywhere in the loop. Pass@k formalizes this across multiple attempts — it's the probability that at least one of k independently sampled solutions passes every test. Pass@1 is the strictest common version: one attempt, no do-overs, pass or fail.

This is, of the three, the closest thing to a genuinely objective measurement, and that objectivity is exactly why it's confined to a narrow slice of tasks. Code either compiles and passes its tests or it doesn't; there's no ambiguity for a rater to resolve and no preference to aggregate. But a binary pass/fail hides an enormous amount of what actually matters about code in the real world. A solution can pass every test in the suite while being unreadable, insecure, or built in a way that would make a human reviewer reject it on sight. It also says very little about the much harder and more common real task of working inside an existing codebase with its own conventions, dependencies, and half-documented constraints, rather than solving a self-contained problem from a blank file — which is exactly why a newer generation of coding benchmarks has tried to move closer to real repositories instead of isolated puzzles.

Pass@k has a second wrinkle worth understanding on its own: the value of k isn't a footnote, it's a different question entirely. Pass@1 asks whether a model's single best attempt succeeds, which maps closely onto how most people actually use a coding assistant in production — you ask once, you expect a working answer, you don't want to silently discard nine failed attempts to get a good one. Pass@10 or pass@100 asks something closer to "if this model got several tries, how often would at least one of them work," which is a reasonable research question about a model's ceiling but a much weaker guide to what a single interaction will actually deliver. A model that looks strong at pass@100 and comparatively unremarkable at pass@1 is telling you it has real capability that isn't yet reliably accessible on the first try — a distinction that matters enormously if you're shipping a product around it.

Multiple-Choice Tests Are Measuring Recall, Not Understanding

Then there's the MMLU family: thousands of multiple-choice questions spanning law, medicine, history, physics, professional exams, and dozens of other domains, each with four answer options and exactly one correct choice. Scoring is trivial to automate — check whether the selected letter matches the answer key — which is precisely why this style of benchmark became so widely adopted early on. It's cheap to run, easy to reproduce, and covers an enormous breadth of subject matter in a single pass.

That breadth is also the honest limit of what it measures. A four-option multiple-choice format has a guessing floor built into its foundations, and elimination heuristics — ruling out the two obviously wrong answers and picking between the remaining two — can inflate scores without requiring the kind of understanding the question was meant to probe. The format never asks a model to generate anything, only to select from options someone else wrote, which makes it a poor proxy for the open-ended, multi-step reasoning that most real work actually requires. And because frontier models have converged so close to the ceiling on the original MMLU, the benchmark has largely stopped discriminating between them at all — when everyone is scoring within a couple of points of the maximum, the remaining gap is as likely to reflect formatting sensitivity as any real difference in underlying knowledge.

This is exactly the saturation problem that has pushed the field toward harder successor tests built the same way but with tougher questions, more answer options, or domains chosen specifically to resist elimination heuristics. That response is worth noticing in its own right: when a multiple-choice benchmark stops separating good models from great ones, the honest fix is a harder version of the same format, not an abandonment of multiple choice as a category. The format still isn't measuring generation or multi-step reasoning directly. It's just measuring recall at a difficulty level that once again has room to distinguish one model from another.

Three Different Questions Wearing the Same Costume

Line these three up and the mismatch becomes obvious. Elo is asking "which response would people rather read." Pass@k is asking "does this code actually work." MMLU is asking "did the model recall and correctly apply the right fact." Those are not three ways of measuring the same underlying quantity at different resolutions — they're three different questions that happen to get reported as numbers on similar-looking charts. A model can lead comfortably on one of these axes and lag on another without any contradiction at all, because nothing about winning a popularity contest guarantees functional correctness, and nothing about acing a multiple-choice exam guarantees people will enjoy the model's writing.

This is precisely the gap that trips people up when a benchmark-topping model disappoints in actual use, or when a model that undersells itself on knowledge tests turns out to be the one a team actually prefers working with day to day. The scores weren't lying to anyone. They were just answering questions the reader never explicitly asked.

Reading a Benchmark Suite Like a Portfolio

The fix isn't picking a favorite among these three; it's matching the question to the job you actually need answered. If you're evaluating a model for open-ended writing, support, or conversational quality, an arena-style Elo number is a reasonable starting signal, with the caveat that it's measuring first impressions and stylistic preference more than depth. If you're evaluating a model for a coding-heavy workflow, pass@k on a benchmark that resembles your actual codebase tells you far more than a generalist knowledge score ever will. If you're trying to gauge broad factual reliability across many domains, a multiple-choice suite is a reasonable coarse filter, as long as you remember it rewards recall and elimination skill more than it rewards reasoning.

Treat a benchmark suite the way you'd treat an investment portfolio rather than a single stock price: no individual line tells the whole story, and the value is in reading them together, weighted by which ones actually resemble the work you're going to ask the model to do. The question worth asking before you trust any single number isn't "is this score high." It's "what exact question was this number built to answer, and is that my question too."