基本信息

要点解读

这是什么

Ambient SFT 是一种监督微调方法:让微调后的模型在高温条件下逼近冻结的预训练模型分布,以利用不完美预测扩充训练信号,降低过拟合和数据分布尾部遗忘。

用在哪里

适用于大语言模型后训练场景,包括领域适配、指令微调和推理任务,尤其适合高质量后训练数据有限且希望改善数据利用效率的团队。

可以推断的

  • 推测:高温匹配可在保留预训练模型已有能力与学习新任务之间提供连续调节。
  • 推测:若方法能跨领域适配、指令微调和推理保持优势,它可能具有较广的后训练应用潜力。

来源摘要/节选

Collecting large-scale, diverse, and high-quality post-training datasets for autoregressive language models is challenging. Consequently, Supervised Fine-Tuning (SFT) is typically applied to datasets significantly smaller than those used for pre-training, which leads to overfitting and the catastrophic forgetting of the tails of the data distribution. To prevent this collapse, we introduce Ambient SFT, a principled framework for leveraging imperfect predictions to augment the fine-tuning set. In our method, a “frozen” version of the pre-trained model serves as a noisy teacher that the fine-tuned model must match at a coarse level. In particular, we require that the high-temperature versions of the conditional distributions produced by the post-training model should match the high-temperature versions of the pre-training ones. The temperature hyperparameter controls the level of trust in the pre-trained model, with the limit $T\to \infty$ reverting back to regular SFT and $T=1$ to knowledge distillation. We demonstrate that our approach outperforms standard SFT across domain adaptation, instruction-tuning, and reasoning, leading for up to $21%$ average relative improvement to standard SFT.

来源说明

当前保存的是来源摘要,不代表论文全文。请以原始来源为准。

「要点解读」由 AI Stack 依据上方已保存内容整理,不代表来源的完整表述;标注「推测:」的判断来自编辑,不是来源陈述。