基本信息
- 来源: arxiv
- 原始来源: http://arxiv.org/abs/2611.17334v1
- 发布域名: arxiv.org
- 分类: cs.CV
- 作者: Ellis Brown、Chris Hoang、Shengyi Qian 等
要点解读
这是什么
这项研究将多模态模型的视觉编码拆解为视觉先验、独立视觉容量,以及视觉模块在融合前的位置,并在统一数据、训练条件和计算预算下进行比较。结果表明,独立视觉容量更关键,而视觉模块不必单独放在多模态融合之前。
用在哪里
适用于设计或评估多模态语言模型的视觉输入方案,以及规划视觉编码器与语言模型的联合训练流程。研究还涉及视觉问答、语言基准测试和首 token 延迟等评估场景。
可以推断的
- 推测:在资源受限或强调响应速度的场景中,将视觉模块放入早期骨干层可能比采用独立预融合编码器更值得考虑。
- 推测:选择视觉编码方案时,可能需要优先保证模型拥有足够的专用视觉处理能力,而不是单独依赖视觉编码器预训练。
来源摘要/节选
How multimodal language models should encode visual input remains up for debate. Most models use a pretrained vision encoder, while some recent recipes train the encoder from scratch with the language model or remove it entirely. Vision encoders bundle together three design choices: a visual prior from upstream pretraining, dedicated vision capacity, and pre-fusion placement of that capacity (i.e., before the multimodal backbone). Because published models also differ in data, architecture, and compute, comparing them cannot isolate the effect of each choice. We disentangle these choices via a controlled framework where we pretrain all models from scratch on common data and compare them on visual question answering and language benchmarks at matched lifetime compute. We find that the visual prior is not necessary; an encoder trained jointly from scratch is on par with a pretrained one, without a separate training stage. What matters instead is dedicated vision capacity: an encoder-free model trails on visual understanding even when its backbone is enlarged by the encoder’s parameter count. Yet that capacity need not precede multimodal fusion, as vision-only modules added in-fusion to early backbone layers perform comparably to a pre-fusion encoder and can lower time to first token. These results also hold across backbone sizes, when encoder pretraining is treated as free, and when starting from a pretrained language model. Overall, our findings favor dedicated vision capacity trained jointly with the backbone and placed early within it.
来源说明
当前保存的是来源摘要,不代表论文全文。请以原始来源为准。
「要点解读」由 AI Stack 依据上方已保存内容整理,不代表来源的完整表述;标注「推测:」的判断来自编辑,不是来源陈述。