基本信息
- 来源: arxiv
- 原始来源: http://arxiv.org/abs/2610.02190v1
- 发布域名: arxiv.org
- 分类: cs.LG
- 作者: Cristian McGee、El Houcine Bergou、Aritra Dutta
要点解读
这是什么
该工作提出一种轻量框架，将方向选择与步长选取解耦：先用一阶梯度确定搜索方向，再仅在该方向的一维子空间进行零阶函数评估，依据局部曲率模型自适应确定步长，整体开销低于完整线搜索。
用在哪里
适用于大语言模型微调等大规模神经网络优化场景，帮助研究者和工程师在不进行昂贵线搜索的情况下提升收敛速度和最终性能。
可以推断的
推测：在梯度曲率变化剧烈的训练阶段，该方法的自适应步长可能带来更明显的收益。
推测：对已有可靠一阶优化器的项目，引入该框架的额外计算成本较小，可作为一种即插即用的步长调节手段。
来源摘要/节选
Step-size selection remains a central challenge in large-scale neural network optimization; conservative steps slow convergence, while aggressive steps can destabilize it. We combine \textbf{Z}ero-and-\textbf{F}irst-\textbf{O}rder optimization~(ZFO) and propose a lightweight framework that decouples direction selection from step-size. ZFO uses a trusted first-order optimizer to determine the direction and performs zeroth-order evaluations only along this one-dimensional subspace to choose how far to move. Using the current {gradient information} and two additional objective function evaluations, ZFO instances construct a local model of the objective function along the proposed direction and select a curvature-aware step within a bounded search interval. This yields an adaptive step-selection mechanism that costs less than a full line search. We provide theoretical guarantees to show that shared-sample evaluations produce reliable finite-difference curvature estimates, that the induced local model selects a near-optimal step along the search interval, and that ZFO converges to a neighborhood of a stationary point. Across the evaluated settings, language models and datasets, ZFO frequently improves optimization and final performance relative to fixed-step first-order baselines, with the magnitude and preferred local model depending on the objective. Our code is publicly available at: https://github.com/nizswan/Zeroth-First-Order-Framework .
来源说明
当前保存的是来源摘要，不代表论文全文。请以原始来源为准。
「要点解读」由 AI Stack 依据上方已保存内容整理，不代表来源的完整表述；标注「推测：」的判断来自编辑，不是来源陈述。