基本信息
- 来源: arxiv
- 原始来源: http://arxiv.org/abs/2609.01603v1
- 发布域名: arxiv.org
- 分类: cs.SE
- 作者: Kefeng Duan、Dewu Zheng、Yanlin Wang 等
来源摘要/节选
Evaluating software engineering agents on realistic benchmarks is costly, since each task may require multi-step code exploration, modification, and test execution. Existing efficient evaluation methods select representative subsets to estimate full-benchmark performance, but are largely result-only: they fit historical pass/fail response matrices or static task semantics, discarding how agents solve problems. We propose PTA-IRT, a Privileged Trajectory-Aware Item Response Theory framework that fuses process and outcome signals. Historical execution trajectories supply process-level evidence beyond pass/fail, such as explored context, attempted edits, and solving paths, which PTA-IRT uses as privileged information for calibration subset selection and ability estimation. Under low calibration budgets, PTA-IRT consistently outperforms prior IRT baselines on score and ranking recovery across four SWE benchmarks. Code and data are publicly available at https://github.com/DeepSoftwareAnalytics/PTA-IRT .
来源说明
当前保存的是来源摘要，不代表论文全文。请以原始来源为准。
本页只呈现已保存的来源证据，不包含基于缺失正文的扩展推断。