基本信息
- 来源: arxiv
- 原始来源: http://arxiv.org/abs/2607.29626v1
- 发布域名: arxiv.org
- 分类: cs.AI
- 作者: Tianyu Huai、Tingshuo Fan、Xinchi Chen 等
要点解读
这是什么
该基准用于评估大语言模型在连续超参数优化任务中的实验决策能力，包含30个可执行的机器学习任务，每个任务要求模型在已有实验结果的基础上提出下一步的配置。
用在哪里
适用于在构建或对比基于大语言模型的自动化实验平台时的技术选型和性能评估，尤其在需要模型能够依据实验反馈进行迭代改进的场景。
可以推断的
推测：该基准能够揭示模型在长期迭代优化过程中的薄弱环节。
推测：评测结果可能帮助社区定位哪些实验干预对模型最具挑战性。
来源摘要/节选
As LLMs evolve from code completion systems into autonomous scientific agents, evaluating their ability to conduct experiments has become increasingly important. Existing benchmarks typically focus on static code generation, paper replication, or final answer correctness, but do not directly assess whether agents can interpret experimental evidence and use it to guide subsequent hyperparameter decisions. To address this gap, we introduce AgentHPOBench, a sequential benchmark comprising 30 executable machine learning tasks across seven research categories. Each task begins with a validated baseline run, after which an agent performs several sequential interventions. At each step, the agent observes the accumulated configurations, metrics, and logs before proposing the next valid configuration. We evaluate 12 widely used agents and conventional HPO baselines under a unified protocol. The results show that current agents exhibit measurable experimental optimization ability across domains, but still face clear limitations in sustained iterative refinement, complex log diagnosis, and consistent progress toward reported reference performance.
来源说明
当前保存的是来源摘要，不代表论文全文。请以原始来源为准。
「要点解读」由 AI Stack 依据上方已保存内容整理，不代表来源的完整表述；标注「推测：」的判断来自编辑，不是来源陈述。