基本信息

要点解读

这是什么

该内容报告了对前沿语言模型在分子属性回归基准上是否会逐字检索已发布数值的研究。作者对多个模型在多个基准上进行审计，比较模型在相同分子和相同提示下的输出与已有文献数值的匹配程度，并分析了推理层次对检索行为的影响。

用在哪里

适用于分子性质预测任务的模型评估与基准设计，尤其是需要区分模型真实学习能力与记忆已有数据的研究者和工程师；也可供关注大模型训练数据泄露问题的 AI 安全与公平性研究参考。

可以推断的

推测：在使用更高级的推理策略时，模型更容易触发对已有文献数据的检索，使得相同输入在低层次推理下不出现匹配而在高层次推理下出现匹配。
推测：若在评估或部署阶段抑制直接检索，模型的预测误差在不同模型之间可能趋于接近，从而更真实地反映其学习到的泛化能力。

来源摘要/节选

Large language models (LLMs) are increasingly evaluated on molecular property benchmarks, but accuracy cannot distinguish a model that predicts a property from one that retrieves a published number. We audit 22 frontier models on 12 regression benchmarks for verbatim retrieval and find that it is widespread but relatively benchmark-specific: on five datasets more than $50%$ of the LLMs show verbatim retrieval, while on the remaining datasets it appears only in isolated cells. We run our experiments at two reasoning levels and find that reasoning changes retrieval. The same experiments, on the same molecules and with the same prompt, are flagged $89%$ more often at the higher reasoning level than at the lowest one. Finally, we test a way to interrupt retrieval in our most contaminated cases, and find that the strongest models in some cases still recognise a combination of transformed SMILES strings and original labels. Furthermore, suppressing retrieval moves the prediction errors of the different models closer together in relative terms, while their differing use of verbatim retrieval spreads them apart. This indicates that the general predictive capability of an LLM is not determined solely by the amount of memorised values. This work provides an overview of the amount and depth of verbatim retrieval in molecular regression benchmarks using LLMs.

来源说明

当前保存的是来源摘要，不代表论文全文。请以原始来源为准。

「要点解读」由 AI Stack 依据上方已保存内容整理，不代表来源的完整表述；标注「推测：」的判断来自编辑，不是来源陈述。