基本信息
- 来源: arxiv
- 原始来源: http://arxiv.org/abs/2608.11191v1
- 发布域名: arxiv.org
- 分类: cs.CV
- 作者: Shiyu Xuan、Zechao Li
要点解读
这是什么
该研究提出一种测试时自进化框架，使 GUI 视觉定位模型在部署后能够通过探索、评估、反思和内部化的闭环过程自行改进。框架中引入基于多模态大模型的评估器提供推理反馈，并通过有条件的自教师把高层反思转化为细粒度的 token 级监督，同时使用对比校准防止错误前缀破坏监督信号。
用在哪里
适用于需要模型在真实环境中不断适配新界面或新布局的场景，例如自动化 UI 测试、虚拟助手或跨平台界面理解等任务，尤其在缺乏人工标注的情况下能发挥作用。
可以推断的
- 推测：该方法在测试时进行自蒸馏和策略采样，可能会带来额外的计算和内存开销，因而更适用于算力充足的服务器端部署，而非资源受限的移动端。
- 推测：其核心的反思机制可迁移到其他需要视觉定位且环境频繁变化的领域，如机器人视觉或增强现实中的目标定位。
来源摘要/节选
GUI Visual Grounding is a fundamental capability for GUI agents. Existing models typically freeze their parameters after deployment, limiting their ability to adapt to unseen interfaces. Although recent methods attempt to adapt models via test-time reinforcement learning, they cannot reflect upon failed exploration. To overcome this, we propose a Test-Time Self-Evolving framework that enables models to improve after deployment without human-annotated ground truth. It constructs a closed-loop of Exploration, Evaluation, Reflection, and Internalization. Specifically, the agent first explores unseen interfaces by predicting grounding coordinates for given instructions. To evaluate these explorations, we introduce an MLLM-based Reflector to assess the generated results and provide the corresponding reasoning reflections. To internalize reflection knowledge into the model weights, we propose Reflection-Guided On-Policy Self-Distillation, which translates high-level reasoning into dense token-level supervision via a conditioned self-teacher. Furthermore, we design a Contrastive Calibration method to prevent incorrect auto-regressive prefixes from corrupting the supervisory signals during failed explorations. Extensive experiments across six benchmarks demonstrate our framework’s effectiveness, achieving an average accuracy improvement of 7.4% over the base model. To the best of our knowledge, this is the first work to successfully exploit on-policy self-distillation for test-time adaptation in GUI visual grounding. By filling the gap in post-deployment adaptation, our framework completes the self-evolving capability of GUI agents. The code will be released.
来源说明
当前保存的是来源摘要，不代表论文全文。请以原始来源为准。
「要点解读」由 AI Stack 依据上方已保存内容整理，不代表来源的完整表述；标注「推测：」的判断来自编辑，不是来源陈述。