TL;DR: Compute drives progress in AI, but what good is progress if you are not doing the right task!
Rich Sutton’s bitter lesson states that compute beats algorithms.
Researchers want to encode their clever ideas about intelligence into machines. Yet over and over, the approaches that win are the general ones which leverage more computation. Search beat hand-built chess knowledge. Neural networks beat hand-built vision features. The lesson is bitter because researchers love algorithms, yet cleverness matters less than scale.
My experience is that Sutton’s bitter lesson is the tip of an iceberg of bitterer lessons: beyond compute and algorithms, there’s data and even picking the right task to do ML on.
The bitterest lesson in ML is that doing the right task > data > compute > algorithms.
Sutton’s bitter lesson is easiest to see in games because there are two major differences to the real world: the right task is obvious (follow the rules to win or maximize score), and data can be endlessly generated through self-play (through compute). It makes sense that the next most important thing is compute.
At the end of the day, machine learning makes reward go up or loss go down. Someone still has to decide the objective to optimize though. Getting this right requires understanding the external system in which the model will operate. Without the right task, everything can work perfectly, with the most beautiful loss and scaling curves, but the model may still be useless!
Unfortunately, ML research tends to attack these problems in the opposite order. Researchers love inventing algorithms. More recently, we have learned to love scaling curves. Meanwhile, data is messy. Choosing the right task often requires leaving the ML problem entirely to study users, products, organizations, or whatever part of the world is supposed to benefit.
This is not an argument against scale. Once the task and data are right, scale is incredible. It is an argument against treating scale as the be-all and end-all.
Bitter LLMs
When life gives you LLMs...
We learned this lesson at OpenAI when making InstructGPT/RLHF: GPT-3 was an incredible model trained to predict the next token on internet text, but people wanted something that followed instructions more than they wanted a super-powered autocomplete. GPT-2-sized models (>100x smaller than GPT-3) trained on the right task,1 even with the dumbest algorithm2 and barely any compute, destroyed GPT-3.
Annotated from figure 31 from the InstructGPT paper
Scaling pre-training would need to reach roughly GPT-7 level to beat even that baseline, and GPT-9 to beat InstructGPT built on GPT-3.3
You get what you optimize for and the bitterest lesson in ML is that the most important part of it isn’t ML at all.
Thanks to Samuel Sorenson, Ke Deng, Alex Warren, and Sasha Sheng for feedback.
LLM history also shows where data without the right task hurts performance: FLAN, the most popular (and fairly large) fine-tuning dataset at the time, actually decreased performance at instruction following.
Data is useless without an algorithm capable of learning from it. A genuinely new task may require inventing a new algorithm first. But you only need the dumbest viable method that makes the task possible.
This might be interpreted as evidence that algorithms matter: you get two whole GPTs' worth! But even here, InstructGPT uses additional comparison data beyond the supervised baseline.