基本信息
- 来源: arxiv
- 原始来源: http://arxiv.org/abs/2608.07435v1
- 发布域名: arxiv.org
- 分类: cs.CV
- 作者: Zixuan Lan、Luzhe Sun、Matthew R. Walter 等
要点解读
这是什么
SABRE 是一个可扩展的自动化流水线，将结构化的任务说明转化为图像与问答对，并通过过滤和人工审查剔除模型轻易答对的样本，以构建针对视觉‑语言模型的抗压基准。
用在哪里
适用于需要系统化评估视觉‑语言模型在违背常规先验条件下表现的科研团队和基准制定者；也可用于在模型迭代期间快速生成新测试样例。
可以推断的
推测：随着模型规模提升，研究者会更频繁地依赖此类自动化工具来保持基准的时效性。
推测：在实际部署中，模型若无法抵御先验误导，可能导致在真实场景中出现系统性错误。
来源摘要/节选
Vision-language models (VLMs) are improving rapidly, but benchmark development lags behind, making weaknesses hard to identify. Building stress tests is costly: samples must satisfy controlled conditions, remain answerable, and challenge current models. We present SABRE, a scalable, automated pipeline that converts a Test Primer (a Markdown Task Design with Data Schema) into structured specifications, generated or edited images, and question-answer pairs. Automated filtering removes candidates solved by a Filtering VLM, while human review verifies candidate validity and supports annotation correction and localized image repair. We instantiate SABRE-Prior to test whether VLMs follow visual evidence instead of relying on world priors – learned expectations about familiar objects and scenes. Its 600 images and 1,000 questions span Context (unexpected entities in familiar scenes), Texture (counterfactual materials), Attribute (noncanonical component counts), and Language Elicitation (answers suggested by language but unsupported by the image). Across six VLMs, macro-average accuracy ranges from 17.8% to 31.3% (22.6% mean). A real-image Attribute control is comparably difficult for the Filtering VLM. SABRE-Counting and SABRE-Spatial pilots show that the workflow supports other stress-test settings. These results establish SABRE as a reusable framework for constructing and refreshing VLM stress tests rather than a single fixed benchmark.
来源说明
当前保存的是来源摘要，不代表论文全文。请以原始来源为准。
「要点解读」由 AI Stack 依据上方已保存内容整理，不代表来源的完整表述；标注「推测：」的判断来自编辑，不是来源陈述。