Cost-Per-Token Is the Benchmark Nobody Talks About Enough

Every capability leaderboard gets a chart, a headline, and a round of screenshots. The economics of actually running these models gets a spreadsheet nobody outside finance opens. That imbalance is backwards for most teams building real products, because for the overwhelming majority of deployments, the bill decides whether the project survives contact with a budget review long before the marginal capability gap between two models decides anything at all.

We'd go further: cost-per-token, and more precisely cost-per-completed-task at your actual usage pattern, is the single most under-discussed number in the entire industry relative to how much it actually determines. A team that picks its model purely off a capability chart and only discovers the real economics after launch is making the decision in the wrong order.

Capability Got All the Headlines, Economics Got None

Part of this is just narrative gravity. A chart showing one model edging out another on a reasoning benchmark is a clean, shareable story — line goes up, new champion, easy headline. A pricing page full of per-million-token rates for input and output, cached and uncached, across a half-dozen model tiers, is not a shareable story. It's homework. So the industry's public conversation skews hard toward capability, while the conversation that actually determines whether a deployment is sustainable happens quietly, after the fact, when someone on the finance side asks why last month's inference bill doubled.

This isn't a minor oversight. For a huge share of real-world applications, the difference between the top capability score and the fifth-best one is a few percentage points on a benchmark most users will never notice. The difference in cost to run those same two models at production volume can be several multiples. Optimizing hard for the first number while ignoring the second is optimizing for the metric that makes for a good tweet over the one that makes for a sustainable product.

It doesn't help that capability and price live in completely different parts of a lab's own communications. A new model's reasoning score gets the launch post and the comparison chart. Its pricing gets a line in a documentation page, updated quietly, sometimes changed again a few months later without much fanfare either way. Nobody is hiding the number — it's just never the number anyone was invited to get excited about, which is exactly why so many teams can recite a model's benchmark position from memory and have to go look up what it actually costs them per month.

The Sticker Price Isn't the Price You'll Pay

The headline rate on a pricing page is only the starting point, and treating it as the answer is where most back-of-envelope cost estimates go wrong. Input and output tokens are typically priced differently, usually with output costing several times more than input, which means a task that involves reading a lot and writing a little has a completely different cost profile than one that involves a short prompt and a long generated response — even at identical token counts on paper. A model that looks cheaper on a blended headline rate can end up costing more for your specific mix once you separate the two.

Context length compounds this in a way that's easy to miss until the bill arrives. Every additional token of conversation history, retrieved document, or system instruction gets billed again on every single call in that session, not once. A workflow that keeps stuffing more context into each turn is paying a quietly growing tax with every exchange. And reasoning-oriented models introduce another wrinkle worth budgeting for explicitly: much of their internal reasoning happens in tokens you never see in the final response but still pay for, which means a low-looking output length in the visible answer can hide a much larger real generation cost sitting underneath it.

Caching and batching add a further layer that a flat headline rate doesn't capture at all. Providers increasingly discount repeated or predictable portions of a prompt — a stable system instruction, a document that gets referenced across many calls — and separately discount work that can be queued and processed without an immediate response. A workload that's structured to take advantage of either one can end up paying a meaningfully different effective rate than the sticker price implies, in either direction, depending entirely on whether the traffic pattern was designed with that in mind or bolted on afterward.

Why a 'Worse' Model Can Be the Better Purchase

Here's the part that feels counterintuitive until you've actually run the math once: for a large share of real tasks, a less capable, meaningfully cheaper model is the better purchase, not a compromise you settle for. Capability gains near the frontier tend to show diminishing returns for routine tasks while cost differences between tiers stay roughly proportional or worse, which means the incremental accuracy you're buying at the top of the market often costs far more per point than the same accuracy gain did lower down the stack.

Picture a mid-size team running a mix of routine classification, simple drafting, and the occasional genuinely hard analytical request. Routing the routine ninety percent of that volume to a smaller, cheaper model and reserving the frontier-tier model for the hard tail will frequently outperform sending everything to the frontier model by default — not just on cost, but often on speed too, since smaller models tend to respond faster. The frontier model isn't wasted in this setup. It's aimed at exactly the slice of work where its extra capability actually changes the outcome, instead of being spent uniformly on requests that a cheaper model would have handled identically.

This is also the logic behind routing requests across tiers automatically rather than picking one model for an entire product. A simple classifier or even a smaller model can make a reasonable first pass at judging whether a given request is routine or genuinely hard, and only escalate the second category to the expensive tier. Done well, this kind of tiered setup captures most of the frontier model's value on the requests that actually need it while paying the cheap-model rate on everything else — which is a fundamentally different economic shape than a single flat rate applied uniformly across every request regardless of difficulty.

The Real Cost Equation Depends on Your Traffic Shape

There's no single right answer here, because the correct model choice depends entirely on the shape of your actual traffic, not on an abstract price-performance ranking. A high-volume, latency-sensitive workload with simple, repetitive requests has a completely different optimal answer than a low-volume workload with rare but complex requests where a slow, expensive response is perfectly acceptable because it happens twice a day instead of two thousand times.

Verbosity is a variable that's easy to underweight here. Two models priced identically per token can produce very different real bills if one of them tends to write twice as much to answer the same question — padding explanations, restating the question, adding caveats nobody asked for. A model with a slightly higher per-token rate but a genuinely terser style can easily come out cheaper in practice than a nominally cheaper model that talks twice as much to say the same thing. This is exactly the kind of gap a pricing page will never show you and only your own usage logs will.

A Framework for Actually Running the Math

The fix isn't complicated, but it does require actually doing it rather than eyeballing a pricing page. Start by profiling your real prompts and responses, not a hypothetical average case — pull a representative sample of your actual traffic and measure the real input-to-output token ratio, because that ratio, not the headline blended rate, is what determines your real cost per call. From there, compute cost per successfully completed task rather than cost per token or cost per API call, since a cheap model that needs two retries to get a usable answer can easily cost more per successful outcome than a pricier model that gets it right the first time.

Then actually test a cheaper tier against your real workload before assuming you need the top of the market — the gap between "this model can't handle my task" and "I never actually tried" is larger than most teams expect. And revisit the calculation periodically rather than treating it as a decision made once. Prices shift, new tiers launch, and a model that made economic sense six months ago may no longer be the efficient choice, in either direction.

It's worth building this review into a recurring habit rather than a one-time audit, precisely because both sides of the equation keep moving. Capability at a given price point tends to improve steadily, which means the cheaper tier that failed your task evaluation last quarter may pass it comfortably this quarter. Waiting for a dramatic price change to trigger a second look means missing most of the gradual shifts that, added together, are usually where the real savings live.

Treat cost-per-token the way a serious operator treats any other unit economics problem, not as a constraint to minimize after the real decision is made, but as part of the decision itself. The model that wins on a capability chart and the model that actually makes your product's economics work are sometimes the same model. Often they aren't. The only way to know which situation you're in is to run the number that never makes it onto the leaderboard: what it actually costs, at your actual volume, to get the actual outcome you need.