I'm going to mostly dodge the question because it would be pure speculation, though the case I make in the article is that a non-insignificant part of it is SFT on API responses. 🙃
Instead, I will say that I would seriously doubt that off-policy RL is the final stage of training (it is very hard to get it to have the right properties for text output), and that my sense is that every LLM (including openai's, anthropic's, open) has this trade-off between optimizing for human preference (RLHF) and single-minded task completion (RLVR) where the exact mixture is model-dependent.
Having worked on logit distillation (or what we called knowledge distillation and reading the original paper by Rich Caruana), I was always super confused when people talked about distillation. Especially since the word has entered the mainstream. This article provided a much needed clarification for me. If I were to phrase it more bluntly, it's fine tuning a "student" with synthetic data generated by the "teacher". I guess the distinction is because this is generative modelling. Classically, logit distillation was applied on classification tasks where fine-tuning on an unlabelled set using a teacher's prediction was not that successful
I'm going to mostly dodge the question because it would be pure speculation, though the case I make in the article is that a non-insignificant part of it is SFT on API responses. 🙃
Instead, I will say that I would seriously doubt that off-policy RL is the final stage of training (it is very hard to get it to have the right properties for text output), and that my sense is that every LLM (including openai's, anthropic's, open) has this trade-off between optimizing for human preference (RLHF) and single-minded task completion (RLVR) where the exact mixture is model-dependent.
US labs are distilling Chinese models. So it looks possible
How do you think the Chinese labs are actually using API outputs? Likely SFT on API traces and then do RL on top of that?
Or do you think there's some sort of off-policy RL going on with the API outputs too?
Having worked on logit distillation (or what we called knowledge distillation and reading the original paper by Rich Caruana), I was always super confused when people talked about distillation. Especially since the word has entered the mainstream. This article provided a much needed clarification for me. If I were to phrase it more bluntly, it's fine tuning a "student" with synthetic data generated by the "teacher". I guess the distinction is because this is generative modelling. Classically, logit distillation was applied on classification tasks where fine-tuning on an unlabelled set using a teacher's prediction was not that successful