基本信息

要点解读

这是什么

该工作提出一种多模态大语言模型，通过在学习阶段为每张图像加入可学习的汇总标记，将其解码为紧凑的三维高斯喷洒形式，并以此三维表示作为答案的生成条件，从而提升模型在多视角场景下的空间推理能力。

用在哪里

适用于需要从多张视角图像推断整体三维布局的任务，如视觉问答、机器人环境感知或增强现实的场景理解；也可供研究跨视角特征一致性和多模态语言模型空间推理的学者参考。

可以推断的

推测：在训练中让模型同时完成 token 预测和三维表征重建，可能促使其内部图像特征在帧间形成更强的对应关系。
推测：如果该方法在更多基准上表现稳健，未来有望与现有的三维重建流程结合，为语言模型提供更直观的场景结构信息。

来源摘要/节选

Reasoning about the 3D world from multi-view images remains a fundamental challenge for Multimodal Large Language Models (MLLMs). While modern MLLMs handle single-image inputs effectively, they struggle to integrate evidence across viewpoints into a coherent 3D understanding. A growing body of work attempts to close this gap by injecting 3D awareness into MLLMs, either by boosting fine-grained pixel-level cross-view correspondence or by fusing features from 3D geometry foundation models, yet a substantial gap to human reasoning persists. In this work, we revisit human spatial reasoning, which suggests that rather than relying on fine-grained geometry cues, humans roughly identify common objects across views, infer the relative geometry between viewpoints, and assemble a coarse 3D layout of the scene. Inspired by this process, we introduce Imagine3D-LLM, an MLLM that learns to assemble a similar compact 3D representation of the scene and conditions its answer on this representation. Concretely, we append a small set of learnable summary tokens after the image tokens, decode them into a compact 3D Gaussian Splatting representation supervised by a photometric reconstruction loss, and train jointly with the standard next-token prediction objective. Notably, although only the summary tokens receive direct reconstruction supervision, this objective also induces stronger cross-frame correspondence within the LLM’s underlying image features, suggesting that learning to reconstruct propagates 3D-aware signals throughout the model. As a result, Imagine3D-LLM consistently outperforms prior approaches across multiple spatial reasoning and 3D understanding benchmarks, suggesting that imagining the scene can be more effective than being told its pixel-wise geometry.

来源说明

当前保存的是来源摘要，不代表论文全文。请以原始来源为准。

「要点解读」由 AI Stack 依据上方已保存内容整理，不代表来源的完整表述；标注「推测：」的判断来自编辑，不是来源陈述。