On Repulsive and Attractive Teachers

If you followed LLM post-training this year, one technique kept showing up: on-policy distillation (OPD).

The central promise of OPD is to transfer knowledge and skills from a teacher model to a student via the dense, token-level supervision of conventional knowledge distillation. Instead of imitating expert-level teacher responses (responses the student is unlikely to come up with on its own), the student first generates its own answer. The teacher then rescores this answer token by token, providing a dense training signal along trajectories the student actually follows, including errors the student is likely to make.11On-Policy Distillation (Lu et al., Thinking Machines Lab, 2025)

Self-distillation22SDPO 3, SDFT 4, and OPSD 5 independently introduced closely related approaches to on-policy self-distillation around the same time.3Reinforcement Learning via Self-Distillation (Hübotter et al., 2026)4Self-Distillation Enables Continual Learning (Shenfeld et al., 2026)5Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models (Zhao et al., 2026) takes this idea one step further by using the same model as both student and teacher. Rather than relying on a stronger external teacher, it creates a better-informed teacher distribution by conditioning the same model on privileged information, such as feedback or successful attempts. This distribution is then distilled into the student by scoring the student’s rollouts token by token.

This rich learning signal makes self-distillation an attractive ingredient in modern post-training recipes. For instance, Cursor uses an SDPO-style objective for targeted textual feedback in Composer 2.5.66Introducing Composer 2.5 (Cursor Team, 2026)7Aligning Language Models from User Interactions (Kleine Buening et al., 2026)8OpenClaw-RL: Train Any Agent Simply by Talking (Wang et al., 2026) Recent work also uses self-distillation to learn alignment and personalization from raw user interactions,7 while related feedback-conditioned distillation enables online agent adaptation from user replies, tool outputs, and environment state changes.8

However, conditioning an LLM on privileged information can also lead to information leakage and a preference for concise, confident responses. When the privileged context reveals a complete solution, the resulting teacher can suppress verbalized uncertainty, checking, and exploration, which may lead to overconfidence.99Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs? (Kim et al., 2026)10Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation (Harne et al., 2026) How strongly the teacher’s behavior shifts depends on the privileged information it receives.10 SDPO in its original form remains well suited to settings such as coding feedback, where the context identifies local errors without revealing the full solution and therefore induces a smaller behavioral shift. Yet, when the teacher sees a correct solution (e.g., from an expert or from the batch's rollouts), the behavioral shift to concise and confident responses can be very pronounced.

To prevent this suppression of exploration, AntiSD1111Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information (Shen et al., 2026)12Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVR (Kim et al., 2026) and Rebellious Student12 reverse the sign of the distillation signal from a teacher conditioned on a correct solution, that is, they ascend the KL divergence. Flipping the sign of the objective is surprisingly effective in settings where SDPO's conciseness is harmful to performance. Yet these approaches rely on an additional verification signal through a GRPO loss to reinforce only those explorations that succeeded, and require additional mechanisms that stabilize training.

We study what this sign-reversed distillation signal actually does by isolating it from GRPO.1313In the initial experiment (Figure 4), the repulsive signal moves away from a teacher conditioned on a correct answer, matching AntiSD and Rebellious Student. In all subsequent experiments, Repulsive moves away from a teacher conditioned on an incorrect answer.14By hybrid models, we mean models that can operate in either thinking or non-thinking mode. We find that one-sided repulsion induces a strong unintended behavioral shift regardless of whether the teacher is conditioned on a correct or incorrect answer. In hybrid models, it pushes the model toward its latent, pre-existing thinking mode.14 In models without an explicit thinking mode, one-sided repulsion instead mainly inflates response length and quickly leads to collapse.

Based on these insights, we use repulsion differently in a broader contrastive framework: we balance attraction to a teacher conditioned on a correct solution against repulsion from a teacher conditioned on an incorrect one. We find that the two opposing behavioral shifts roughly cancel, leaving primarily a correctness signal, as shown schematically in Figure 1.

Self-distillation starts with the student attempting a task using only the information available at inference time. Given a prompt $x$, the student generates a response $y \sim \pi_\theta(\cdot\mid x)$. This response is then evaluated in the task's environment, which may provide detailed and informative feedback ${\color{#A7C66B}\boldsymbol{c}}$, for example, error messages and failing unit tests in coding tasks.

The same model is then reused as a self-teacher by conditioning it on this privileged context ${\color{#A7C66B}\boldsymbol{c}}$. Keeping the student’s response $y$ fixed, we then rescore the student’s response under the self-teacher, evaluating its next-token distribution at each position along the same trajectory.1515Note that we do not generate a new response from the self-teacher. We only compute the log-probability that the self-teacher assigns to each token of the fixed response generated by the student. We train the student by minimizing the reverse KL divergence between its next-token distribution and that of the self-teacher:

where $\pi_\theta(\cdot|x, y_{<t})$ is the student's next-token distribution given the prompt and previous tokens, and $\pi_\theta(\cdot|x, {\color{#A7C66B}\boldsymbol{c}}, y_{<t})$ is the teacher's distribution when additionally conditioned on the context ${\color{#A7C66B}\boldsymbol{c}}$.

This objective has two equivalent perspectives: it moves the student's distribution toward the teacher through on-policy distillation, or, from an on-policy RL perspective, uses the following teacher-to-student log-probability ratio as a token-level advantage:

Intuitively, this loss provides a learning signal for every token: tokens that are more likely under the context-conditioned self-teacher than under the student, $\pi_\theta(y_t\mid x,{\color{#A7C66B}\boldsymbol{c}},y_{<t}) > \pi_\theta(y_t\mid x,y_{<t})$, receive a positive advantage and are reinforced, while tokens that are less likely are suppressed. The same update can therefore be read either as moving the student toward the teacher or as nudging the policy to increase its expected token-level reward $A_t$. See SDPO 3 for the full derivations and further details.

Conditioning the teacher on privileged information changes not only what the teacher knows, but also what behavior it prefers. When conditioned on a complete solution, the teacher can become more confident and concise, assigning less probability to uncertainty, checking, and alternative approaches.9 Distillation transfers this behavior to the student. The resulting shorter responses can benefit knowledge-based tasks, but may harm reasoning tasks where exploration and self-correction are useful.

We can see this directly in the token-level signal. Scoring Qwen3-4B1616Qwen3 Technical Report (Yang et al., 2025)17DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning (He et al., 2025) rollouts on DeepMath17 against the expert demonstration, the positive distillation advantage is most negative exactly on the tokens that verbalize uncertainty or trigger a check.

AntiSD11 and Rebellious Student12 counteract the suppressive effect by reversing the sign of the teacher signal: tokens that are unlikely under the privileged teacher receive positive advantages.

At first glance, this seems unintuitive: instead of learning from a correct, better-informed teacher, the student is explicitly repelled away from it. Neither method relies on this repulsion alone. In both cases, a verified sequence-level reward grounds the deviations encouraged by the sign-reversed teacher signal.1818AntiSD adds the sign-reversed distillation signal to a sequence-level GRPO advantage. Rebellious Student uses the distillation signal to reweight GRPO on verified correct rollouts.

Both papers motivate their approaches as a form of behavioral steering: reversing the sign of the signal from a teacher conditioned on a correct solution assigns positive advantages to uncertainty verbalization, checking, and alternative branches, which ultimately encourages exploration. GRPO then reinforces the explored solutions that led to success.1112

The effect is easiest to see on the sentence from Figure 2. Reversing the sign of the objective flips the token-level advantages, reinforcing the same uncertainty and self-correction tokens that positive distillation suppresses.

To understand the behavioral shift induced by the sign-reversed objective, we isolate its distillation signal and we remove both the GRPO component and the additional stabilizing mechanisms (i.e., entropy-triggered gate1919When the privileged teacher's median token entropy falls below a calibrated threshold, it disables the distillation term and falls back entirely on GRPO until the entropy recovers11.20Repulsion from a teacher that is conditioned on a positive solution21We train Qwen3-4B in non-thinking mode on a hard subset of DeepMath, and evaluate across six olympiad-level math benchmarks.). We investigate the training dynamics of this distillation signal,20 which we refer to as Repulsive.21

We observe that the run passes through three phases:

We can see this unintended mode switch directly in the rollouts. Because the model is initialized in non-thinking mode, Qwen3-4B's chat template pre-fills an empty <think></think> and the model answers directly. Yet midway through training, the model begins generating a reasoning trace and emits another closing </think> before answering, indicating that it no longer preserves the configured non-thinking behavior.

This mode-switch-and-collapse pattern raises two questions. Does sign-reversed distillation teach a new capability, or does it primarily change the model's operating mode? More broadly, can we learn from privileged information through self-distillation without inheriting the strong behavioral shifts induced by attractive and repulsive teachers? The second question naturally leads to a contrastive framing.

We have seen in the previous sections that minimizing the reverse KL to a policy conditioned on a correct solution, $\pi_\theta(\cdot\mid x,{\color{#A7C66B}\boldsymbol{c_+}},y_{<t})$, can teach the model new information and skills. At the same time, it elicits more direct, overconfident answers, which can reduce reasoning performance.

We also observed that ascending the same objective increases exploration, but at the cost of pushing the student away from a correct solution. If Repulsive functions mainly as behavioral guidance, it makes sense to instead condition the teacher on an incorrect solution, ${\color{#E47B71}\boldsymbol{c_-}}$.

This suggests a simple construction: create two conditional policies, one conditioned on a correct solution, ${\color{#A7C66B}\boldsymbol{c_+}}$, and one on an incorrect solution, ${\color{#E47B71}\boldsymbol{c_-}}$. When both are conditioned on behaviorally similar content, subtracting the second teacher from the first should cancel the behavioral shift they share while preserving the signal associated with correctness. Combining these two conditional policies yields a contrastive self-distillation signal, as explored in RLCSD2222RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation (Pan et al., 2026)23CEPO: RLVR Self-Distillation using Contrastive Evidence Policy Optimization (Heakl et al., 2026) and CEPO23. Both of these methods use pairs of oppositely conditioned teachers to modulate GRPO advantages. Below, we express this signal in terms of SDPO advantages.

Let ${\color{#A7C66B}\boldsymbol{c_+}}$ and ${\color{#E47B71}\boldsymbol{c_-}}$ denote positive and negative privileged contexts. We can create two self-teachers by conditioning on those:

Here $p_S^t$ denotes the student's distribution, and ${\color{#A7C66B}\boldsymbol{p_+^t}}$ and ${\color{#E47B71}\boldsymbol{p_-^t}}$ denote the distributions of the positive and negative self-teachers, respectively.

Contrastive self-distillation extends SDPO (attraction toward a positive teacher) by adding a term that pushes the model away from a negative teacher:2424In practice, both teacher inputs can be batched in a single vLLM call, keeping the added wall-clock overhead modest.

The same objective can be written as a token-level policy-gradient signal. For a teacher conditioned on context $c$, the SDPO advantage is

Positive values reinforce tokens that become more likely under the context-conditioned teacher. In our notation, the corresponding token-level contrastive signal is the difference between two SDPO advantages:

For the balanced objective, $\lambda=\tfrac12$, the shared student term cancels:

When this ratio is greater than one, $y_t$ is more likely under the positive teacher than under the negative teacher, so the token receives a positive advantage and is reinforced. When it is less than one, the token $y_t$ is suppressed.

The contrastive signal is the difference between the two SDPO signals and therefore reflects how the teacher's token probabilities change when conditioned on the positive rather than the negative context.

Many self-distillation variants can be expressed within the same framework by varying the mixing coefficient. We focus on three objectives:

The correct solution can be either an expert solution or the model's own verified rollout. Figure 6 visualizes the effect of all three objectives on the same rollout.

How do different self-distillation objectives behave when a hybrid model1427As positive context we use the dataset’s expert solution, with its thinking trace removed, while the negative context is another incorrect rollout from the same on-policy generation group. We skip generation groups for which only one side is available. starts in non-thinking mode? We initialize Qwen3-4B in non-thinking mode and train the three self-distillation objectives alongside a GRPO baseline on a hard subset of DeepMath.27

These results suggest that Repulsive primarily changes the model's operating mode. The model switches from non-thinking mode into its pre-existing thinking mode, but it does not outperform the thinking-mode baseline, and continued response growth eventually causes training to collapse. Attractive has the opposite effect, keeping responses short but producing little improvement on evaluation. Contrastive balances these behavioral shifts, leading to steady improvement without suppressing reasoning or causing a response-length explosion.

Does contrastive self-distillation still help when the model has no explicit thinking mode to switch into? We repeat the same DeepMath experiment from Qwen3-4B-Instruct-2507.2828The three objectives, privileged information, response budget, and six evaluation benchmarks are unchanged.

Response length makes the behavioral shift visible in both experiments. The Attractive objective keeps responses short, while Repulsive pushes them toward the token limit and eventually collapses.2929However, no run produces <think> tags, so changes in response length do not represent a switch into an explicit thinking mode. Contrastive improves both models without either extreme, showing that its gains cannot be explained by a switch into a pre-existing thinking mode.

To test whether Contrastive's gains persist when mode switching cannot explain them, we next consider a model whose thinking mode is already active. Math is a difficult testbed for this question because Qwen models are already close to saturation on many standard math benchmarks. We therefore use Group Anagrams3030Group Anagrams asks the model to sort words that are made from the same letters. For example, [tea, bat, eat, tab] becomes [[tea, eat], [bat, tab]].31REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewards (Stojanovski et al., 2025)32Here, the privileged contexts are the model's own correct and incorrect rollouts, including their thinking traces. from Reasoning Gym31, initializing Qwen3-4B in thinking mode.32

The dynamics mirror the math experiment:

We started by exploring the effects of self-distillation on model behavior. Both one-sided methods we studied inherit a behavioral shift from their privileged context: while SDPO can lead to overconfidence, repulsion from a correct rollout-conditioned teacher can push the model out of its configured non-thinking mode without outperforming the thinking-mode baseline, before response growth saturates the response-length cap. We use these observations to motivate a contrastive approach, separating correctness from behavior by subtracting a negative rollout-conditioned teacher from a positive one. Across three settings, we find that this combined objective successfully extends self-distillation to reasoning tasks, and stabilizes the exploratory tendencies of the KL-ascent objective. Improvements on already-thinking and instruct-only models show that the contrastive objective’s gains cannot be explained by unintended mode switching.

Self-distillation methods have the ability to improve the efficiency of RLVR methods through dense credit assignment. Mitigating the shift to overconfident responses and lack of exploration has shown to be a roadblock to their more widespread adoption, and contrastive objectives offer a practical solution to this problem. Prior contrastive methods have all used this signal to modulate GRPO advantages; we show it can serve as a standalone alternative. We see investigating different ways of balancing teachers, designing new sources of positive and negative context, and scaling these methods to larger models as promising directions for future work.

Please cite this work as:

Or use the BibTeX citation: