基本信息
- 来源: arxiv
- 原始来源: http://arxiv.org/abs/2609.04172v1
- 发布域名: arxiv.org
- 分类: cs.AI
- 作者: Zixuan Fu、Bingxiang He、Yuxin Zuo 等
要点解读
这是什么
研究在大型语言模型的同策略蒸馏中，仅使用一条查询进行训练的效果。实验发现单查询在数百步后已能接近全数据蒸馏的收益，并用状态覆盖率解释了这一现象。
用在哪里
适用于资源受限但希望快速微调模型的研究团队，以及需要评估蒸馏所需数据量的工程实践者。
可以推断的
推测：在多数任务上，少量查询已能提供足够的监督信号，进一步增加查询的收益可能呈递减趋势。
推测：当前蒸馏过程在后期学习效率下降，未来的改进方向可能聚焦于提升每步的更新效率或引入更丰富的监督信号。
来源摘要/节选
On-policy distillation (OPD) combines student-generated rollouts with dense token-level supervision from a teacher. Existing work has mainly studied its algorithmic behavior, leaving the role of training data unclear. We examine this role at the data-minimal limit by training on a single query. One-shot OPD keeps improving for hundreds of steps and recovers most of full-data OPD’s gain across task domains and model families. We explain this result through the states visited during training and the rate at which the student aligns with the teacher. We measure \emph{state coverage}, the fraction of the states full-data OPD visits that a query set’s rollouts reach. A single query already reaches (71.5%), most of it within the first 100 steps. Adding semantically distinct queries raises coverage and validation accuracy together, until 16 queries reach (98.9%) and match full-data training. Yet alignment slows at a similar pace whether OPD trains on one query or the whole dataset, and even a fixed set of states takes hundreds of steps to absorb. OPD is therefore data-overfed but algorithm-starved. Its rollouts quickly expose broad supervision, while the student absorbs that supervision increasingly slowly. The state-coverage result extends to multi-teacher OPD, where 16 semantically diverse queries per domain match full-data MOPD. As a further stress test, content-light templates and off-domain WildChat queries also approach the real-query baseline. Task content and induced state coverage can therefore come apart. We hope these findings direct future work toward the step efficiency of OPD, and prompt a re-examination of the data and the mechanisms behind its recent successes in frontier post-training.
来源说明
当前保存的是来源摘要，不代表论文全文。请以原始来源为准。
「要点解读」由 AI Stack 依据上方已保存内容整理，不代表来源的完整表述；标注「推测：」的判断来自编辑，不是来源陈述。