基本信息
- 来源: arxiv
- 原始来源: http://arxiv.org/abs/2609.04198v1
- 发布域名: arxiv.org
- 分类: cs.AI
- 作者: Haoyaun Zhu、Jie Zhang
要点解读
这是什么
本文通过预先注册的大规模实验，验证“黑盒语言模型评判器在共享端点上对同一请求会产生相同结果”这一假设，发现实际排名一致性远低于预设阈值，表明该评判器缺乏仪器般的稳定性。
用在哪里
适用于依赖外部共享端点进行自动评分、排序或构建排行榜的场景，尤其适合研究者和工程师在设计语言模型评价流程时参考。
可以推断的
推测：在实际应用中，若未对评判工具本身的重复性和噪声水平进行测量就直接用于决定训练数据或模型排名，可能导致系统性偏差。
推测：未来的评估框架需要在实验设计阶段先对评判器进行可靠性检验，以确保后续结论的有效性。
来源摘要/节选
Language-model judges now gate training data, score generations, and drive leaderboards. The judge is then a measurement instrument, resting on one rarely stated assumption: the same request, sent to the same model name, reads the same tomorrow. We audited that assumption in two preregistered campaigns with every threshold fixed in advance; neither got past validating its instrument. Across 52,988 audited request attempts, same-window repeat rankings agreed at Spearman 0.400 against a required 0.90, and byte-identical next-day replays agreed at 0.78 against a required 0.99, each time with the execution record at ceiling. Three mechanisms explain the gap: a label-to-meaning mapping that biased readouts as strongly as the signal; candidate gaps seven orders of magnitude below the instrument’s own noise floor; and byte-identical inputs returning different rankings, a noise that exact-permutation readouts compound. Neither metric substitution nor sampling repaired it on the tested grid. Preregistered follow-ups bound the problem: waiting did not help on the days sampled (0.805 versus 0.800, replicated over five further days); switching providers did not help (four providers share the floor, medians 0.74 to 0.88, predicted by none of the metadata fields they expose); self-hosting on batch-invariant kernels helped only while the server was quiet; and on constructed errors with known gaps, the readout’s separation tracks error type, not size. We distill the evidence into a three-level snapshot-identity ladder, eight design rules, and a reporting checklist; a pilot at roughly 2% of the study’s call volume would have exposed both unreachable gates in advance. All results concern externally measured behaviour on shared serving infrastructure. On a shared endpoint, a model name is not a frozen instrument; a preregistered evaluation must measure its instrument before freezing any gate on it.
来源说明
当前保存的是来源摘要，不代表论文全文。请以原始来源为准。
「要点解读」由 AI Stack 依据上方已保存内容整理，不代表来源的完整表述；标注「推测：」的判断来自编辑，不是来源陈述。