Self-distillation isn’t new, but in 2026 it matters in a way it didn’t before. Models have reached a point where they can judge and rationalize solutions sometimes better than they can generate them from scratch. And the training landscape is also changing.
Earlier, in many cases we’ve relied on knowledge distillation – transferring knowledge and capabilities from large models to smaller ones – as well as RL-based post-training loops. But both approaches are quite expensive and compute-heavy. So we should move on and use what current models now can do well – and they are powerful enough to increasingly improve by comparing its own reasoning against a privileged, better version of itself and they are powerful in in-context learning. At the same time, scaling costs, continual updates, and synthetic data pipelines have made efficient post-training and test-time deployment a central task.
In light of all this, today we’re talking about self-distillation as a scalable way to refine reasoning trajectories and upgrade model behavior using its own judgments, offering a middle path between the two main corners: supervised fine-tuning (SFT) and reinforcement learning (RL): on-policy updates, dense feedback, no explicit reward model.
Fortunately, we have three really interesting works on how to make on-policy distillation shine:
Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models which shows how one more and explicitly self-critique on the full reasoning paths.
Self-Distillation Enables Continual Learning exploring how continual learning can gain from self-distillation.
Reinforcement Learning via Self-Distillation which focuses on the power of feedback.
It’s a really interesting journey into what is happening right now in self-distillation, so let’s move on to see the main focus areas.
Note: Thanks to Kevin Murphy for coining the phrase “On-Policy Distillation Zeitgeist” in a tweet.
In today’s episode, we will cover:
Broadening your knowledge of distillation
What is on-policy self-distillation?
Self-distillation in the field of continual learning
Self-distillation policy optimization, or how to get the maximum out of feedback
Conclusion: What it all teaches us
Sources and further reading
Knowledge Distillation vs On-Policy Distillation: What's the Difference
First of all, let’s look at what main options we usually have when it comes to model training.
The classic one is knowledge distillation – a way to train a smaller or weaker language model (the student) through imitating a stronger one (the teacher). This works well for the student model because it doesn’t learn just what is correct – it also learns from the teacher’s probabilities over words, exploring how confident the teacher is in different options. That extra information helps the student learn better patterns. Overall, training comes down to making the student’s predictions match the teacher’s predictions at every step.
Knowledge distillation is typically implemented as off-policy training. The form we know best is supervised fine-tuning (SFT), where the model learns to imitate expert outputs from a fixed dataset.
But this setup becomes fragile when the model must generate long reasoning chains at inference time. Classic distillation is trained on a fixed dataset of idealized reasoning paths, while at deployment the model generates its own imperfect trajectories. Small early mistakes can snowball, creating the well-known gap between training and inference – often referred to as distribution mismatch or exposure bias. That’s why there is another option – on-policy distillation.
On-policy distillation is much closer to realistic learning process: you try → your teacher corrects you → you improve. It lets the student generate its own answers during training. Then, the teacher evaluates these sequences, and the student gets token-by-token feedback on what it should have done instead. It is a combination of the realism of reinforcement learning and supervised learning with clear, dense feedback.
And another thing that is gaining most of traction right now is Reinforcement Learning with Verifiable Rewards (RLVR) – all those methods like GRPO (Group Relative Policy Optimization) or PPO-style (Proximal Policy Optimization) algorithms, where models’ answers are automatically checked if they are correct or incorrect and rewarded based on this. But the problem comes with this approach as well, because most methods evaluate only the final answer, ignoring mistakes in the middle steps. Rewards are binary (right/wrong) in general, and there is a risk to waste compute in case of all the answers are all correct or incorrect, because there is simply to learning signal there.
Yes, there is also an option to give step-by-step rewards during reasoning, for example, using Process reward models (PRMs), but they require manual labeling of reasoning steps, which is expensive and hard to scale.
So, that’s why we need to change something in our approach to knowledge distillation, and the many shifts are now coming in self-distillation particularly – it gives the opportunity to look inside the reasoning steps and give the most detailed feedback.
What is On-Policy Self-Distillation?
Basically, for models the best combination would be to get all the good parts at once:
on-policy learning
dense, token-level feedback
no external additional models like a teacher model or a reward model
Let’s turn again to how we learn. For example, if you solve a problem wrong, then look at the correct solution, it is often easier to explain why the solution works than to come up with it from scratch. One may call this chittering but if it helps you to get better at early stages why not to use this easy path? So, similarly, LLM may gain more from explaining a correct solution than generating a new one blindly.
Based on this idea, the researchers from UCLA, HKU and Meta Superintelligence Labs introduced On-Policy Self-Distillation (OPSD). Here, one single model plays two roles:
As a student, it sees only the problem and generates an answer. It matches inference-time conditions. This is the model we care about at test time.
As a teacher, it sees the question plus extra information, like the correct answer or a verified reasoning trace. It is used only during training. This model is prompted to read and understand the reference solution, generate its own reasoning and then naturally evaluate the student’s output.
It is important that these models have the same parameters and only the prompt/context is different.
The overall training process of OPSD works like this:
It starts from the student. Given a problem 𝑥, the student generates a full solution autoregressively: 𝑦^∼𝑝𝑆(⋅ ∣ 𝑥). The main thing is that the trajectory comes entirely from the student policy, so the model is trained on what it would actually produce at inference time.
Image Credit: “On-Policy Self-Distillation for Large Language Models” paper
Then, at every decoding step 𝑛:
The student produces its next-token distribution 𝑝𝑆(𝑦𝑛 ∣ 𝑥, 𝑦^<𝑛)
The teacher produces its next-token distribution 𝑝𝑇(𝑦𝑛 ∣ 𝑥, 𝑦∗, 𝑦^<𝑛)
Both policies condition on the same prefix – the student’s own partial generation 𝑦^<𝑛. The only difference is that the teacher has access to the verified solution 𝑦∗.
The teacher’s distribution reflects which tokens would keep reasoning on track and which would lead toward a correct solution. It evaluates the student’s solutions token by token, so the model is trained to make the student’s reasoning closer to what the teacher would prefer. Concretely, OPSD minimizes a token-wise divergence between 𝑝𝑇 and 𝑝𝑆. This comparison is performed for every token in the student’s generated trajectory, averaged across the full sequence and then averaged across the dataset.
Image Credit: “On-Policy Self-Distillation for Large Language Models” paper
Instead of typical KL divergence, the researchers use Jensen–Shannon divergence (JSD), which is much smoother. It’s symmetric (doesn’t overly privilege one distribution), numerically stable and introduces a mixing parameter 𝛽, which controls how strongly the teacher vs. student shapes the target distribution. With JSD, the teacher acts as a fixed, privileged reference distribution and the gradients flow only through the student.
There is also the lighter variant that avoids matching full vocabulary distributions and instead compares only the probabilities of the tokens the student actually generated. The learning signal becomes simply log 𝑝𝑇 − log 𝑝𝑆, used as a fixed advantage in a policy-gradient update that adjusts only the student. It is cheaper and simpler, but indeed have weaker signal than JSD variant, where full-vocabulary OPSD teaches the student how it should think at every step.
And since the rollout 𝑦^ is sampled from 𝑝𝑆, the entire procedure remains strictly on-policy.
In simple words, the model compares “What would I predict if I didn’t know the answer?” vs. “What would I predict if I did know the answer?” Here the student is corrected on the exact reasoning paths it actually explores, avoiding idealized expert traces it would never generate on its own.
And the advantages of OPSD are quite substantial:
Image Credit: “On-Policy Self-Distillation for Large Language Models” paper
First of all, you don’t need a separate teacher model, which indeed saves compute.
The model learns from its own outputs (on-policy), getting dense feedback at every reasoning step and over the entire vocabulary. You also get a profit from using ground-truth solutions directly.
Speaking about the efficiency, OPSD needs 4–8× fewer tokens than GRPO for similar or better results. And as for the sample efficiency, it needs only 1 rollout per problem, while GRPO needs 8.
It also beats traditional off-policy distillation and SFT across all model sizes and improves over the base model.
Image Credit: “On-Policy Self-Distillation for Large Language Models” paper
Larger models benefit more from OPSD and longer generation also helps. For example, 2k–4k token rollouts outperform 1k, because more teacher signal improves reasoning.
OPSD Limitations
But the hard part in this approach – the model needs to be reasonably strong to begin with for this self-distillation to work. What also matters a lot is the high quality of ground-truth and divergence design. If you are OK with higher memory cost, then full-vocabulary distillation (JSD) would be a better choice than sampled-token policy-gradient style training (it gives +2–3% on key benchmarks).
Another moment: OPSD can limit discovery of novel reasoning strategies, because it aligns to a privileged distribution and does not optimize directly for reward like in RL. Too hard problems also stay a limitation, where curriculum learning (gradual increasing of problem difficulty during training) may help.
Plus, it is still unclear whether good scaling effects continue at much larger model sizes, say, 70B+ models.
While OPSD is a general approach for training LLMs, the next idea that we’re going to discuss goes deeper into the concrete problem of continual learning – how we can help models to acquire new skills and knowledge without overwriting existing capabilities. The solution is also self-distillation but with some other interesting tricks.
Self-distillation in the field of continual learning
Researchers from MIT and ETH Zurich exactly named their recent paper “Self-Distillation Enables Continual Learning,” giving us a strong hit of the new self-distillation potential. Here’s what they did, basically, to help models learn continually: they connected off-policy SFT and on-policy RL, turning demonstrations into an on-policy learning signal using self-distillation, and called this approach Self-Distillation Fine-Tuning (SDFT). Here is why this on-policy learning from demonstrations works in reality.
Interestingly, SDFT uses one model for two roles as OPSD does. But the mechanics and interpretation are different.
Again, in simple words: If OPSD asks “What would I predict if I didn’t know the answer vs. if I did know the answer?”, SDFT main question is: “How does my behavior shift when I see a good example – and can I internalize that shift?”
Like OPSD, SDFT uses a single model with identical parameters, and the only difference is the context:
Image Credit: “Self-Distillation Enables Continual Learning” paper
The student sees only the query: P= 𝜋 (·∣𝑥)
The teacher sees the prompt plus a demonstration c: Q= 𝜋 (·∣𝑥, c). The demonstration is not necessarily the exact solution to the problem, it is a working example of similar structure. So SDFT relies more heavily on in-context generalization, because it is prompted to generalize from the demonstration. The teacher model is just the same model as the student behaving better because it saw one good example.
As OPSD, SDFT is strictly on-policy and starts from the student, generating the full trajectory.
The objective is to make the student distribution closer to the teacher distribution, but unlike OPSD, SDFT minimizes reverse KL divergence for this, decomposed token-by-token: DKL =(πθ(⋅∣x) ∥ π(⋅∣x, c)). Here, the teacher distribution is treated as fixed, and gradients flow only through the student.
Why the reverse KL is used? It pushes the student to concentrate where the teacher assigns probability and penalizes probability mass where the teacher is low. So thanks to this SDFT is slightly more aggressive in pulling the student toward teacher-preferred regions.
In practice, the teacher parameters are often an EMA (exponential moving average) of the student to stabilize training.
SDFT works better for continual learning because it stays close to the base policy via reverse KL (remember, that it is important not to forget the previous knowledge), the teacher remains near the student distribution and the distribution shift is small.
So what do we gain from this approach?
SDFT outperforms SFT on new tasks thanks to strictly on-policy training with better generalization.
It integrates new knowledge better. On newly injected 2025 disaster facts, SDFT reaches 89% strict accuracy vs. 80% for SFT, and 98% OOD accuracy, nearly matching oracle RAG.
Image Credit: “Self-Distillation Enables Continual Learning” paper
SDFT benefits from scaling. For example, at 7B it gets +4 points over SFT, and at 14B +7 points. This happens because larger models are stronger in-context teachers that give bigger gains.
On answer-only supervision, SDFT improves accuracy (43.7% vs. 23.5% for SFT) and preserves long reasoning chains.
And even with the same teacher, offline distillation generally underperforms SDFT.
This all shows that continual learning gets really better with SDFT, because new-task accuracy improves without catastrophic forgetting, and in sequential multi-task training, skills accumulate instead of collapsing.
But again…
SDFT Limitations
To make SDFT work for continual learning, the model must already have strong in-context learning abilities. If the base model cannot improve its behavior when shown a demonstration, then the teacher signal is weak. That’s why in small models, SDFT underperforms SFT.
SDFT is not a replacement for on-policy RL. It works when you have demonstrations but no explicit reward function. And if you need exploration or optimization toward a reward, RL is still necessary. In fact, SDFT is best seen as an initialization before RL fine-tuning.
Actually, there is also extra compute cost. SDFT requires ~2.5× more FLOPs and ~4× longer wall-clock time than SFT, because it generates on-policy rollouts during training.
A fascinating linguistic problem also appears – since the teacher is conditioned on demonstrations, it may produce phrases like “Based on the example…”. Unfortunately, the student sometimes learns to reproduce these markers.
Overall, SDFT doesn’t appear to eliminate catastrophic forgetting completely, but there is real promise in combining it with RL or other continual learning techniques.
We hope, you’ve noticed that SDFT and OPSD both focus on comparing of the solution of the better model with the less “educated” one. They got rid of the separated teacher model, giving this role to the single model that does everything. But the real gold mine in any education remains feedback. And the next method explicitly focuses on this aspect and what it can give to self-distillation.
Self-Distillation Policy Optimization (SDPO)
How can we turn this rich text feedback into better learning, without using a stronger external teacher model? – this is the third question for today and we got the answer from TH Zurich, Max Planck Institute for Intelligent Systems, MIT and Stanford researchers who explored how reinforcement learning can win from via self-distillation.
In many real tasks (for example, coding), we actually get much richer feedback than just right or wrong, like: runtime error: division by zero; failed test case: expected 42, got 39; wrong format: answer must be in Newton-seconds. That feedback contains a lot of useful information for further improvement. And the researchers even call this setting Reinforcement Learning with Rich Feedback (RLRF).
Image Credit: “Reinforcement Learning via Self-Distillation” paper
They noticed a simple shift they could make: take the rich feedback and feed the model with it so it gains the necessary learning insights.
Their method is called SDPO (Self-Distillation Policy Optimization), and here is how the model teaches itself step-by-step:
The model answers a question.
The environment gives feedback (error message, judge explanation, etc.). Feedback can include pass/fail, runtime errors, failed unit tests, judge explanations or even a previously successful solution from the same rollout group.
Then we give the same model: the original question, its answer and the feedback.
Image Credit: “Reinforcement Learning via Self-Distillation” paper
The model re-evaluates its own answer with the feedback in context. Since LLMs are good at learning from context, they can often recognize: “Oh, I divided by zero,” “There are wrong units,” “This test case fails because I didn’t handle edge cases.”
This creates a stronger version of itself – a “self-teacher”.
Finally, the original model is trained to match what this self-teacher would predict.
This process of self-distillation shows the model what is behind “wrong” exactly. Comparing to OPSD, for example, in SDPO you don’t need a full correct solution; just knowing what failed already helps the model rethink the reasoning.
In simple words: if OPSD compares “uninformed vs informed reasoning,” and SDFT asks “how does my behavior shift when I see a good example?”, then SDPO asks something even more practical: “What exactly went wrong, and can I use that information to correct myself next time?”
And what about the performance?
The first good news – SDPO works even in standard RLVR settings with only scalar rewards, like 0/1. SDPO uses successful attempts in the batch as feedback for failed ones. So without changing the environment, it gets this dense learning signal.
Even greater news – it learns much faster than GRPO. On Chemistry (Olmo3-7B), SDPO reaches GRPO’s 5-hour accuracy in 30 minutes – it is 10× speedup! In many tasks, 1 hour of SDPO ≈ 5 hours of GRPO. And in general, SDPO reaches GRPO’s final accuracy in 4× fewer generations.
Image Credit: “Reinforcement Learning via Self-Distillation” paper
SDPO’s responses are 3× shorter on average. On Chemistry, up to 7× shorter, while being more accurate. It also avoids loops, filler (“Hmm”, “Wait”), and circular reasoning common in GRPO, showing better efficiency as well.
Larger models gain more from SDPO. The self-teacher’s ability to detect mistakes appears to be an emergent property of stronger in-context learners.
It succeeds in coding: 53.2% discovery@2750, best-of-k: 41.5%, multi-turn: 35.6%. To reach 22% discovery probability, SDPO needs 3× fewer attempts.
Additionally, logit-level SDPO (top-K tokens per position) clearly outperforms token-level and sequence-level variants, but even sequence-level SDPO still beats GRPO.
Image Credit: “Reinforcement Learning via Self-Distillation” paper
What is even more important, SDPO continues to improve models: the self-teacher improves during training, and the final student surpasses the initial teacher.
But let’s also analyze the issues that SDPO may have.
SDPO Limitations
First of all, SDPO depends heavily on the model’s in-context learning ability. Weak model means weak adjustments from the feedback.
Logically, SDPO is only as good as the feedback it receives. Bad feedback can distort learning instead of helping it. In purely binary-reward settings without informative feedback, it cannot improve until some useful signal exists.
SDPO requires an extra forward pass to compute the teacher’s log-probabilities. While the overhead is modest (~+5–17% per training step), it can matter more for smaller models or short-generation tasks where generation itself is cheap.
SDPO’s advantages are derived from the self-teacher, so they are lower-variance but biased. This is not necessarily bad, but it may require care when combining with standard RL.
Overall, SDPO challenges us to focus more on the quality of the feedback.
OPSD vs SDFT vs SDPO: Key Takeaways
To sum up, here is what we’ve learnt about these three solutions, improving on-policy self-distillation:
OPSD = Explicit self-critique, where the teacher is explicitly privileged. It compares uninformed reasoning vs. informed reasoning with the true solution.
SDFT = Internalizing in-context improvement. The teacher is just better conditioned that the student. It compares baseline reasoning vs. reasoning after seeing an example.
SDPO lets the model read the detailed feedback and teach itself what it did wrong.
The message from these studies is very interesting: focus on and strengthen the powerful sides of the models we already have, like in-context learning and the growing ability to self-critique.
On top of this, we gain the insight that RL itself is not always the fundamental limitation. In many cases, the bottleneck lies in how feedback is constructed and utilized. If informative signals are discarded or compressed into overly sparse rewards, even strong optimization methods cannot extract their full value. When we preserve useful data and reinforce existing strengths, we open new possibilities for AI. The answer is becoming clearer in 2026 – models can increasingly improve by leveraging their own evaluations, if we choose the right focus for their “education.”
Sources and further reading
Resources from Turing Post
FAQ
What is self-distillation?
Self-distillation is a training method where a model improves by learning from a better-conditioned version of itself. Instead of relying on a separate teacher model, the same model plays both roles: one version generates an answer, while another version sees extra context, feedback, a correct solution, or a demonstration and provides a stronger learning signal.
What is a distilled AI?
A distilled AI is a model trained to absorb useful behavior from another model, dataset, or better version of itself. In classic knowledge distillation, a smaller student model learns from a stronger teacher model. In self-distillation, the teacher and student can be the same model under different conditions, which makes training more efficient.
What is the difference between off-policy and on-policy distillation?
Off-policy distillation trains a model on fixed examples or expert outputs that were generated separately from the model’s own behavior. On-policy distillation trains the model on trajectories it actually produces during training. This matters for reasoning tasks because on-policy methods reduce the gap between training and inference: the model learns from the kinds of mistakes it really makes.
Why does self-distillation matter for LLM reasoning?
Self-distillation matters because reasoning failures often happen inside the chain of thought, not only at the final answer. Methods like OPSD, SDFT, and SDPO give models richer feedback on their own reasoning paths, helping them correct intermediate mistakes, use demonstrations better, or learn from detailed error messages.
Is self-distillation better than reinforcement learning?
Not always. Self-distillation is useful when the model can learn from its own improved behavior, demonstrations, or rich feedback without needing a separate reward model. Reinforcement learning is still important when the goal requires exploration, direct reward optimization, or verifiable outcomes. In practice, self-distillation can complement RL rather than replace it.
