Beyond Distribution Matching:
Selective Margin Preservation for Supervised Fine-Tuning

Yuxuan Wang1,3* Guangxiang Zhao2* Xinyu Tan1 Yahui Han1 Xinxiang Yin4 Zhonghai Wu1 Xiangzheng Zhang2 Lin Sun2 Tong Yang1†

1Peking University    2Qiyuan Tech    3Xidian University    4The University of Hong Kong

* Equal contribution    † Corresponding author

ACL submission · arXiv:2610.19685v1

Paper PDF Code
Horizontal comparison of cross-entropy, LP-SFT, and SRP-SFT showing the quantities each objective preserves.
CE fits the target, LP-SFT matches local probabilities, and SRP-SFT keeps teacher-preferred margins above a lower bound.

Abstract

Supervised fine-tuning changes logit margins among non-target tokens at teacher-forced prefixes. Matching a local reference distribution keeps those probabilities and also penalizes margin expansions. SRP-SFT asks whether a soft auxiliary can preserve selected local non-target margin relations without freezing that distribution. A frozen reference selects top-K non-target pairs; a hinge penalty activates only when the student falls below a reference-scaled lower bound. Once the bound is met, the auxiliary stops acting.

Method

The target token remains the job of cross-entropy. SRP-SFT only repairs selected teacher-preferred alternative pairs when their margin becomes too small. CE supervises the observed target, while distribution matching can over-constrain useful adaptation.

Reference-guided set

Select the frozen model's top-K non-target alternatives at the same teacher-forced prefix.

Teacher-preferred pairs

Keep one orientation for every strictly ordered pair, up to K(K−1)/2 constraints.

One-sided hinge

Penalize only violations of the scaled lower bound; expansion beyond the bound is free.

SRP-SFT method overview at one response position: the frozen reference selects non-target pairs and a one-sided hinge repairs undersized or reversed margins.
At each response position, the reference chooses plausible alternatives and orients their pairs. CE supervises the excluded target; SRP-SFT applies a one-sided margin hinge.

Results

Qwen3-4B-Base on UltraFeedback, seed 1234. Reasoning and code are reported under the paper's common one-epoch screen.

54.56%MATH-500 pass@1SRP-SFT · +4.78 vs. CE
69.50%Code meanHumanEval+ / MBPP+ · +14.40 vs. CE
0.110Margin contractionHeld-out diagnostics · ↓ from 0.268 LP-SFT
SRP-SFT aggregate benchmark comparison Reasoning means are 36.20 for CE, 42.52 for LP-SFT, and 45.11 for SRP-SFT. Code means are 55.10, 65.38, and 69.50. 0 20 40 60 80 36.20 55.10 42.52 65.38 45.11 69.50 CE LP-SFT SRP-SFT reasoning · code reasoning · code reasoning · code Reasoning Code
Reasoning mean averages the paper's twelve displayed metrics; code mean averages HumanEval+ and MBPP+ pass@1/pass@4. Higher is better.
Selected seed-1234 results from the common one-epoch screen
Method MATH p@1 MATH p@4 AIME p@1 HE+ p@1 MBPP+ p@1 Mean
CE34.0859.960.8243.3843.3636.20
LP-SFT49.7877.062.8657.5350.2942.52
SRP-SFT54.5680.504.7662.8456.2445.11

Independent seed checks keep the same ordering on reasoning: SRP-SFT reaches 44.05 / 44.35 across seeds 42 / 3407, while its code mean is 65.78 / 69.48.

Mechanism

On 998 held-out examples, SRP-SFT reduces contraction and rank inversions while allowing more margin expansion. Preservation is about geometry, not only KL.

Mechanism diagnostics comparing CE, LP-SFT, and SRP-SFT on local KL, margin contraction, and margin expansion.
SRP-SFT has local KL 0.075, 5.65% inverted teacher-preferred pairs, 0.110 contraction, and 0.593 expansion.
CE LP-SFT SRP-SFT Length is relative within each metric

Local KL

Lower stays closer to the reference

CE0.359
LP0.049
SRP0.075

Flip rate

Inverted teacher-preferred pairs

CE15.40%
LP7.41%
SRP5.65%

Contraction

Lower means less margin collapse

CE0.600
LP0.268
SRP0.110

Expansion

Room left once the bound is met

CE0.543
LP0.183
SRP0.593

Design choices

Exclude the target

Cross-entropy learns the observed response token; the auxiliary only acts on non-target alternatives.

Keep the direction

The frozen reference orients each pair, so the student is asked to avoid losing a preferred relation.

Stop at the bound

Once the scaled margin is met, expansion is allowed. The auxiliary does not force exact distribution matching.

BibTeX

@article{wang2026srpsft,
  title={Beyond Distribution Matching: Selective Margin Preservation for Supervised Fine-Tuning},
  author={Wang, Yuxuan and Zhao, Guangxiang and Tan, Xinyu and Han, Yahui and Yin, Xinxiang and Wu, Zhonghai and Zhang, Xiangzheng and Sun, Lin and Yang, Tong},
  journal={arXiv preprint arXiv:2610.19685},
  year={2026}
}