基本信息

要点解读

这是什么

该研究提出在强化学习可验证奖励（RLVR）阶段，让目标大语言模型基于弱模型的推理前缀生成答案，以打破模型的过度自信并保持推理的多样性，从而缓解熵坍缩现象。

用在哪里

适用于大语言模型在数学推理等需要多路径探索的任务中进行强化学习微调的研究者和工程师，帮助他们在不额外进行监督微调或复杂奖励设计的情况下提升模型的推理覆盖范围。

可以推断的

推测：在其他需要保持生成多样性的生成任务（如代码生成或对话系统）中，类似的弱模型前缀引导策略可能同样有效。
推测：该方法对弱模型与目标模型之间的语言风格差异有一定敏感性，若差异过大可能导致引导效果受限。

来源摘要/节选

Reinforcement Learning with Verifiable Rewards (RLVR) significantly improves LLM reasoning but often causes a drop in policy entropy, leading to narrowed reasoning coverage and degraded pass@$k$ for large $k$. While existing methods mitigate this entropy collapse through algorithmic regularizations, cross-model non-parametric perturbation is also neglected. In this work, we propose a simple yet effective approach to preserve the generative diversity of LLMs during RLVR. Instead of relying solely on internal exploration, we force the target model to generate answers based on partial reasoning trajectories generated by a smaller, weaker language models. These unfamiliar prefixes effectively disrupt over-confidence and encourage the exploration of distinct reasoning paths. We empirically study the potential of outer prefixes, revealing the mechanism of the impact of distributional discrepancy to the exploration dynamics in RLVR training. Experiments across multiple mathematical benchmarks show that our method consistently outperforms vanilla RLVR. Notably, the performance gain becomes increasingly pronounced as $k$ scales up, demonstrating a substantial expansion of reasoning coverage. Furthermore, our approach efficiently mitigates entropy collapse without requiring additional SFT, intricate reward designs, or complex prompting.

来源说明

当前保存的是来源摘要，不代表论文全文。请以原始来源为准。

「要点解读」由 AI Stack 依据上方已保存内容整理，不代表来源的完整表述；标注「推测：」的判断来自编辑，不是来源陈述。