A benchmark inevitably gets “benchmaxxed” when people building the model can optimize for a public eval score.1 You don’t have to train directly on a benchmark to benchmaxx it; you just have to train on similar data or try a hundred experimental settings, then compare how your model performs. The benchmark selects the model even if nobody intended to game it. Things were not great when benchmarks were an academic pursuit, but now that they are tied to enormous attention and funding, negative incentives to game them abound.

Illustration of the Streetlight Effect - source

The field wants to measure general intelligence, but can only optimize what it can see. Public evals are the streetlight. Post-training therefore pushes capability fastest in the illuminated areas, while reliability outside them can stay flat or even get worse. My claim is that many of the spikes in “jagged intelligence” are the benchmarks.

Annotated version from source

Everyone is cheating

Models chase benchmarks.

Meta’s Llama 4 allegedly ranked near the top of LMArena, but it turned out that not only was it not the public model, Meta was also caught testing 27 private variants before picking the best one, which had the unusually verbose and emoji-heavy style that Arena users rewarded. The ordinary version later landed far lower. Even Claude (normally the goody two-shoes), on a benchmark to simulate a vending machine and end with the most money, formed price cartels, lied to suppliers and promised customer refunds it never sent.

Benchmarks chase users.

When GPT-6 Astra launched, Artificial Analysis’s Intelligence Index had it tied with GPT-5.6 Sol, which clashed with the widespread belief that Astra was a major leap. The next day, a revised index put Astra four points ahead. Three days later, another revision tied it with Claude Fable 5.1 for first. I do not think Artificial Analysis rigged the index. The changes are defensible. But the sequence shows the feedback loop: people’s intuitions about which model is better help determine whether an eval looks valid. The point of external evals should’ve been to inform the population, not the other way around.

People cherry-pick for their favorite narrative.

When the Chinese open-weight model GLM-5.2 beat Fable 5 on one web-design leaderboard, it became evidence that Chinese models had caught the frontier. Design Arena’s own analysis was narrower: GLM used more repeatable templates, generated 25% more code, took twice as long, and still lost to Fable on several design categories.

Benchmarks have been broken for a while.

Unspecialized humans scored 34.5% on MMLU in the original paper, worse than many small old models. The OpenAI coup was partially a result of safety researchers seeing models surpass PhD level at Google-Proof Question Answering (GPQA - an extremely hard dataset) in 2023. Yet humans still do most of the world’s useful work.

source

Everyone in the field knows they’re broken, but they still get quoted because everyone is competing for attention.

What benchmarks should be

The crux of the problem is (1) intelligence is hard to measure, (2) a (good actor) lab wants to say “we made the model smarter,” and (3) for it to be believed (to whatever extent is accurate).

Benchmarks are a shortcut to get around the hard problem of trust. They seem like a magnifier of trust, but instead are a loan: bad actors can then exploit the trust projected on the benchmark.

The way out is to be trustworthy in the first place. We need to stop outsourcing credibility to a leaderboard and just be honest.

For labs, that means publishing caveats, disclosing cherry-picking, including the evidence that makes you look bad, and de-emphasizing benchmarks even when you’re ahead.

Users have a responsibility too: run your own private evals, treat public ones with a grain of salt, and don’t amplify every number or plot you see.

At TypeSafe, we’re making a new type of model, which means existing benchmarks don’t apply. We can start the race to the bottom with a wall of evals showing that we beat everyone else, or start from a clean maximally honest slate. We are choosing the clean slate: no standard benchmark table in our model releases. New evals will be dated snapshots and immediately retired once posted rather than hill-climbed. We will also publish our evolving internal evals as our current best guesses, alongside the caveats, any cherry-picking, and evidence that looks bad for us.

Thanks to Ke Deng, Erik Gafni, and Sasha Sheng for feedback.

Rich Sutton’s bitter lesson states that compute beats algorithms.

Researchers want to encode their clever ideas about intelligence into machines. Yet over and over, the approaches that win are the general ones which leverage more computation. Search beat hand-built chess knowledge. Neural networks beat hand-built vision features. The lesson is bitter because researchers love algorithms, yet cleverness matters less than scale.

My experience is that Sutton’s bitter lesson is the tip of an iceberg of bitterer lessons: beyond compute and algorithms, there’s data and even picking the right task to do ML on.

The bitterest lesson in ML is that doing the right task > data > compute > algorithms.

Sutton’s bitter lesson is easiest to see in games because there are two major differences to the real world: the right task is obvious (follow the rules to win or maximize score), and data can be endlessly generated through self-play (through compute). It makes sense that the next most important thing is compute.

At the end of the day, machine learning makes reward go up or loss go down. Someone still has to decide the objective to optimize though. Getting this right requires understanding the external system in which the model will operate. Without the right task, everything can work perfectly, with the most beautiful loss and scaling curves, but the model may still be useless!

Unfortunately, ML research tends to attack these problems in the opposite order. Researchers love inventing algorithms. More recently, we have learned to love scaling curves. Meanwhile, data is messy. Choosing the right task often requires leaving the ML problem entirely to study users, products, organizations, or whatever part of the world is supposed to benefit.

This is not an argument against scale. Once the task and data are right, scale is incredible. It is an argument against treating scale as the be-all and end-all.

Bitter LLMs

When life gives you LLMs...

We learned this lesson at OpenAI when making InstructGPT/RLHF: GPT-3 was an incredible model trained to predict the next token on internet text, but people wanted something that followed instructions more than they wanted a super-powered autocomplete. GPT-2-sized models (>100x smaller than GPT-3) trained on the right task,1 even with the dumbest algorithm2 and barely any compute, destroyed GPT-3.

Annotated from figure 31 from the InstructGPT paper

Scaling pre-training would need to reach roughly GPT-7 level to beat even that baseline, and GPT-9 to beat InstructGPT built on GPT-3.3

You get what you optimize for and the bitterest lesson in ML is that the most important part of it isn’t ML at all.

Thanks to Samuel Sorenson, Ke Deng, Alex Warren, and Sasha Sheng for feedback.

Pop quiz: what’s the most important cost when running an agent?

Total amount billed broken down by %. Assuming Fable 5 + 100% cache hit rate. Only the additional cache write cost is separated from input cost.

If you’re like most AI engineers, your first two guesses are wrong. Output tokens? Per token outputs are the expensive ones, but they end up being a small slice of your final bill. Input tokens? Kinda, but not the ones you write in your prompts + messages. The answer is cache reads costs developers the most (or cache writes, if your cache hit rate isn’t great). Technically both are input tokens, just at very different prices.

An agent is a loop: the model reads the context then emits a tool call, the tool returns some tokens, and the whole thing goes back into the model. Every turn re-reads everything that came before it. With a context of N tokens and T tool calls, you process roughly N × T total input tokens (cached and not) to produce a comparatively tiny number of output tokens. And since N grows with every turn, the cost of a session grows quadratically with its length and even more with increasing tool calls.

The cost of an agent

Non-infra-engineer summary

• When generating an output token, the entire past context (KV cache) needs to be read.

• You aren’t charged for the KV cache, because… it is already cached.

• When an agent stops to call a tool then resumes, the entire past context needs to be read again.

• You are charged for the KV cache this time, because… fuck you.

• Also that’s the largest part of your bill now because agents have a LOT of tool calls.

Cache Cash

When Martin Alderson modeled a 100-turn session from 60k tokens of context, cache reads made up ~76% of his Opus 5 bill. I wanted real-world trajectories, so I wrote a small simulator that replays Nvidia’s Open-SWE-Traces (tokens counted with the gpt-oss tokenizer, cache writes at 1.25x input price) with Fable 5 pricing:

Output tokens, which most people associate with the big price tag, are just 9-18% of the total bill. The rest is the model re-reading its own history.

Here is the same dataset priced across various models, assuming a 100% cache hit rate:

Those are the costs with perfect caching. At Claude Code’s reported 89% hit rate, the same workload costs $1.97M instead of $1.07M: 84% more!!

So small changes in cache hit rate can make a big difference to the total bill. Here are the rates measured by the OpenCode team:

Note that others have measured Claude Code at 95%, so take the exact rates with a grain of salt. But even going from 95% to 98% saves roughly 17% of the bill on this workload.

Why cache reads are (almost) free (but not for devs)

Ingredient 1: Input tokens - prefill costs compute.

Prefill is the step where the model reads the prompt. All of the input tokens go through the forward pass at once, which is very efficient on a GPU, and the cost is proportional to the number of tokens. This is what you’re paying for with “input tokens”. It’s also the only step a cache can save: a cache write is a prefill where the provider keeps a preprocessed representation of the input tokens (“the activations”): this is what the KV cache is.

Cache reads are input tokens that do not need compute because the results were saved.

Ingredient 2: Output tokens - decode costs memory bandwidth.

Decode is the step where the model generates tokens, one at a time. To generate each token, the model has to attend over the entire KV cache, which means reading the whole context out of high bandwidth memory (HBM) for every single output token. There is very little compute involved but an enormous amount of memory traffic. This is why output tokens cost 5x more than input tokens. Decode is the real bottleneck of inference.

Every single output token needs to read the entirety of the input, but luckily they are already cached.

Ingredient 3: A tool call is just a pause.

When an agent stops to run a tool, the KV cache for the whole context is kept in memory. Nothing has to be recomputed. The only cost is keeping those bytes around while the tool runs, and keeping bytes around uses neither compute nor HBM bandwidth (the two things inference is actually constrained by).

So my claim is that the cost of cache reads is primarily already included in the price of output tokens, at the decoding step.

This is consistent with provider actions:

• Cache reads don’t count towards input token rate limits on the Claude API. Rate limits exist to protect capacity, so that’s Anthropic telling you what a cache read costs them.

• OpenAI ran automatic prefix caching for over a year without charging anything extra for cache writes; you can’t serve LLMs at scale without proper caching The 1.25x cache write premium only arrived with GPT-5.6, to match Anthropic.

How is inference so profitable all of a sudden?

Frontier models are more of a commodity than ever, open-weight models are on the rise, and yet the labs’ reported inference margins keep climbing. How?

Share of tokens to open-weight models from Vercel’s AI gateway

The answer of course is the KV cache (+ the agentic change of workload): more tool calls means more cache reads, which means more profit.

It also explains the “subsidized” subscription plans. SemiAnalysis found that the $200/month Claude plan can yield up to $8,000/month of API-equivalent tokens, and OpenAI’s up to $14,000/month. At list price that’s a 40-70x subsidy, which sounds insane, until you remember that 90%+ of a coding agent’s tokens are cache reads and the true cost of serving them is a small fraction of the sticker price.

Which is also why, if you run agents at scale, self-hosting makes way more sense than it does for chat. Self-hosting an open weight model is the only way to keep the cheapness of the KV cache for yourself rather than handing it to the lab. At least until cache read prices fall further. Fable 5.1 cut them from $1.00 to $0.25 per million tokens, which suggests they will keep dropping.

Why routing doesn’t work (for agents)

The pitch for routing is: use a cheap model for the easy steps and the expensive model for the hard ones, and save on output tokens. But output is the small slice of the bill. The moment a second model touches the context, you end up paying for it from scratch, since a KV cache isn’t shared between models.

Assumptions: 100k tokens in an Opus 5 session, next step generates 1k tokens and gets 3k back from a tool. Compare Opus[1] doing it all or routing to Sonnet then back.

The tiny amount saved on output tokens was dominated by cache costs, so the “easy” step cost almost 3x more, and it was done by a dumber model. 🙃

If a router picks a model per request, it pays for this cache thrash every time, especially on the longest priciest contexts. The only way to amortize the handoff is to leave the cheap model in charge for a while, which is what subagents do, but…

Subagents mostly don’t work either (if you want them to share the full context[2]). Because output tokens are a minority of the cost and you end up paying to write the whole context into both models’ cache, it’s rarely worth it.

Conclusion

The economics of agents come down to one question: how many times does the context get read, and who pays what for each read.

If you build agents, measure your cache hit rate and the number of tool calls. The number of tool calls matters way more than the number of tokens per call. And if you run agents at scale, consider self-hosting to keep those cache savings for yourself.

And if breaking free from the tyranny of the KV cache sounds interesting to you (if only there was a new kind of foundation model that could help 🤫), reach out! Would love to jam on designs, ideas, and sci-fi.

Big thanks to Ke Deng, Kevin Zhang, and Sasha Sheng for helping me write/review this post.

[1] AFAICT APIs don’t currently have a way to say “also cache my output” - this is perhaps a blindspot for agentic workloads

[2] The “mostly” is because subagents work great when they don’t need the parent’s context. If the parent says “search the codebase for X and report back in 200 tokens”, the brief is tiny, and more importantly the 50k tokens of grep output never enter the parent’s context, which makes N smaller for every turn after.

Everyone is arguing whether Kimi K3 distilled Anthropic. This whole conversation involves a horrible abuse of the term “distillation” which has come to mean almost anything the speaker wants it to mean.

I’m an AI researcher who’s ex-OpenAI, I was a co-author on the original RLHF paper that led to ChatGPT. I know what I’m talking about. So here’s how to actually understand the distillation debate, because it’s way more complicated than the government is trying to make it seem.

What do people mean by distillation?

There actually are multiple techniques here that often get mixed up for each other. But here are the three types of distillation, in order of how powerful they are.

1: Logit Distillation

This is what was originally meant by distillation. Most people (not Schmidhuber) would consider the original distillation paper as Hinton et al. There they show that models can be trained much more efficiently given a teacher’s raw outputs (“logits” - basically probabilities of each next token), which started the trend of using the technique to more efficiently train mostly smaller models given large teachers.

When people say that all the labs distill models, this is what they mean. Anthropic for example likely distilled Opus from training on Mythos logits, trying to get Opus’ logits to match Mythos’ logits as closely as possible.

But when talking about cross-lab distillation, no one refers to logit distillation. This is because logits aren’t returned via frontier APIs anymore (they used to be!), although it would be possible to do logit distillation on an open weight model. Obviously, if we’re accusing the Chinese of doing distillation, it’s not classic distillation, because they don’t have access to logits.

2: Behavior Cloning

Instead of learning the raw logits, one can instead learn to replicate output tokens from a teacher. It’s not as informative as distilling from logits, but is similar in expectation and most importantly, you can do it from the outputs of an API.

Supervised Fine-tuning (SFT), the first phase of most post-training algorithms, is equivalent to this, and it’s also commonly referred to as Imitation Learning.

When referring to cross-lab distillation, people almost always refer to behavior cloning, but importantly, this is not possible with today’s frontier reasoning models because reasoning models don’t return their full outputs! They show abridged reasoning traces which are just summaries, specifically to make behavior cloning less effective. The thinking goes, if you can’t see the actual steps the model took to reason and instead just see some “chicken scratch” and the final answer, you won’t be able to re-derive that yourself.

Distillation had been a big issue long before today, and the big labs have known this for a while, which is why when reasoning models were created, they chose not to share the reasoning traces with users and instead showed a summary. This allows for the model to do hidden work before outputting a final response, and this was supposed to prevent distillation - hence all the discussion about how distillation may not be possible (including from myself).

3: Behavior Parroting

Instead, I want to talk about what I call “behavior parroting” - mimicking the outputs of privileged work - and whether or not this can be used to train frontier models.

Note: past work (Chen et al.) has called the no-chain-of-thought version of this “Only Answer” training, though it was mostly negative results on small models.

Can Behavior Parroting work?

The case against behavior parroting: Hallucination

Intuitively, it doesn’t look like it should work. If you make a smart model do a lot of work in secret, then copy its outputs, the student doesn’t learn to mimic that work, it just learns to make plausible looking outputs. Hence, hallucination.

Case for behavior parroting #1: Style Transfer

Point 1.1: most of instruction following is style

This was a very bitter pill to swallow when first teaching models to follow instructions, but it’s unfortunately the simplest best explanation for a lot of the phenomena we’ve observed. One of the most surprising results here was in Hewitt et al. where training on only outputs without instructions closed most of the gap with fully trained RLHFed models. (Sounds familiar?)

Point 1.2: most of reasoning is style too

There is fairly strong evidence that this applies to reasoning models as well.

Reasoning models work almost as well when trained on reasoning traces that lead to incorrect answers. [source1] [source2]

Point 1.3: there seems to be transfer between reasoning and response

This one is more of a guess, but the final assumption to pull our argument together is that there is some style transfer between the reasoning and the final response (the text returned from an API). I don’t know of any papers on this, but some points in favor are that model providers have found a need to separate non-reasoning and reasoning models because they interface with each other.

The Hewitt et al. paper seems to also imply that a large amount of post-training is simple token-level output bias, which would facilitate that transfer.

Case for behavior parroting #2: Pre-training works

Fun fact: You can interpret large scale pre-training as behavior parroting on the Internet! Most would likely consider this Behavior Cloning, but when writing the internet, humans do all sorts of intermediate steps + have additional information not present in the tokens.

It’s hard to argue against the unreasonable effectiveness of pre-training!

Summary

• When talking about distillation in the future, we, as an industry, should be more clear about what type we’re talking about.

  • behavior parroting - learning to imitate token outputs after reasoning, but without reasoning traces (because they’re hidden)

  • behavior cloning - learning to imitate tokens

  • logit distillation - learning to imitate probabilities

• Overall, there is a pretty compelling case that distillation can occur from commercial APIs, but it is an approximation (behavior parroting) of an approximation (behavior cloning) of (logit) distillation.

• This is obviously an important problem, and I’d love to see scaling laws on behavior parroting of reasoning models!

Big thanks to Ke Deng, Sasha Sheng, Erik Gafni, and Haseeb Qureshi for helping me write/review this post.

Scaling laws were one of OpenAI’s most important results, both technically and philosophically (so much so that being scaling-pilled became a thing). They allow us to predict results for ever larger language model runs, and also allow for debugging models as we use exponentially more resources. All of this led to the era of LLMs we’re in today, but the craziest part was… the original Kaplan et al scaling laws were wrong.

Recently, Lilian Weng posted another awesome (and highly recommended) blog post on scaling laws. I was extra excited about the section “Reconciling Kaplan and Chinchilla”, the former being OpenAI’s original scaling laws and the latter being DeepMind’s follow-up with completely different scaling laws.

Figure 1 from Chinchilla. The black dotted line shows the original scaling laws, and the cyan star shows that significantly smaller models should be used.

Lilian’s article goes into the mainstream explanation of the difference between them from follow-up research (namely that it’s about how they counted the total number of parameters). That follow-up research unfortunately is inaccurate, though not due to any fault of the authors.

The reality of the difference between the original scaling laws and Chinchilla’s is that the former had a bug!

The bug: 3 ingredients

Non-researcher summary

• The 2 scaling laws (original and Chinchilla) give different “scaling recipes” for how to efficiently train large language models

• The former was incorrect because they:

  • Did not train on enough data (Step 1)

  • Gradually decreased the impact of data to make it look like more data wasn’t needed (Step 2)

  • Claimed that the gradual decrease was unimportant (Step 3)

• Thus, for a few years, people trained models that were much too large on too little data

Clue: Data scales with size.

It’s easier to identify this when working backwards: both scaling laws predict that data should scale with model size. The handwavy explanation is that bigger models have more capacity to soak up that data. Thus the amount of data is a very important parameter.

Step 1: Use a fixed amount of data.

The Chinchilla paper points out the root issue stating the original Kaplan et al paper authors “use a fixed number of training tokens and learning rate schedule for all models”. When every model is trained on the same fixed amount of data, the tiny model trained on ~130B tokens is getting way more training relative to its size than a giant model trained on the same ~130B tokens.

Relevant quote from Chinchilla’s related work section.

Figure 2 from Kaplan et al. showing all model sizes trained to the same ~130B tokens.

Figure 2 from Chinchilla with a pink arrow added to show roughly where the training curve would have been cut off if only trained to 130B tokens. It would have been obvious that training ended before reaching the scaling laws’ pareto frontier.

Keeping the amount of data fixed would be sufficient to get incorrect scaling laws, but if that was the only mistake, the results would look obviously incorrect. Except if you also…

Step 2: Use a cosine decayed learning rate schedule to zero.

This learning rate schedule caused learning to slow as training approached the target number of tokens. Performance naturally plateaued, appearing as if training is saturated. We now know that large models would have kept improving with more data and a different learning rate schedule, but the learning rate schedule artificially constrained results, making it appear that more data would not help.

Visualization of a cosine learning rate decay with a warmup (source) - you can see a smooth decay to lr=0, where learning stops entirely

The fixed amount of data and the learning rate schedule lead to both incorrect and hard to debug scaling laws, and it becomes even harder to debug if you…

Step 3: Claim that results were “largely independent of learning rate schedule”.

Given a maximum number of tokens, their conclusion is entirely accurate, but doesn’t apply to the true infinite data limit that scaling laws aim to model.

Aside: I too worked on LLM optimization at OpenAI at the time and missed the bug as well. 😅 The learning rate schedule seemed so obviously an important hyperparameter that it looked intentionally set.

Section 2.2 of Kaplan et al., describing how it was trained. Green box shows calculation for a constant number of tokens with model size. Red box shows the learning rate schedule.

Result: Models were undertrained and too large.

You can see how the difference of learning rate shows up: Chinchilla ended up with a model less than half the size of GPT-3, trained on over 4x more tokens. They could not have achieved this result if the learning rate decayed to 0 at just 300B tokens. 🙃

Table 1 from Chinchilla: showing how GPT-3 was both undertrained and oversized.

Conclusion

Eventually, the bug was discovered but not explicitly acknowledged (that I know of). By now, every big AI lab has long known this.

For future non-big-lab researchers: don’t waste your time on this question. Chinchilla’s scaling laws are the correct ones.

For whoever can amend the original scaling laws paper, it would be great to add a note that there was a bug.

Big thanks to Ke Deng, Sasha Sheng, Erik Gafni, David Dohan, and Sander Dieleman for helping me write/review this post.