基本信息
- 来源: arxiv
- 原始来源: http://arxiv.org/abs/2608.26095v1
- 发布域名: arxiv.org
- 分类: cs.CV
- 作者: Kaichen Li、Zhilin Zhu、Jianhao Huang 等
要点解读
这是什么
该研究提出一种在多模态大模型持续后训练中，利用视觉依赖信息来平衡旧任务保持与新任务学习的框架，包含视觉约束的最优传输和视觉调制的适应两部分。
用在哪里
适用于需要从无标签流数据中不断更新多模态模型的部署场景，例如实时视频分析、跨模态检索或持续学习的移动端应用。对关注模型长期可维护性和跨模态灾难性遗忘的研发团队有参考价值。
可以推断的
推测：该方法对视觉注意的量化较为敏感，实现时可能需要额外的可视化或特征提取工具来捕捉 token 级的视觉依赖。
推测：在资源受限的终端上运行时，视觉约束的最优传输计算开销可能导致延迟，需要在精度和效率之间做权衡。
来源摘要/节选
In this paper, we explore a novel task of Multimodal Unsupervised Continual Post-Training (MU-CPT), enabling deployed MLLMs to continually evolve from streaming unlabeled data. Existing unsupervised post-training methods for MLLMs typically optimize target tokens uniformly, overlooking their heterogeneous visual dependence (VD). However, we reveal that token-level VD is crucial for MU-CPT. Specifically, its structural distortion serves as an indicator of cross-modal catastrophic forgetting, and its inherent heterogeneity acts as a compass to guide new-task learning. Leveraging this property, we propose a Visual Dependence-Aware (VDA) framework with two main components. First, Visually Constrained Optimal Transport (VC-OT) formulates the VD structural distortion of old-task VD during new-task learning as an optimal transport problem to mitigate cross-modal forgetting. By designing a region-aware ground cost and a dependence-stratified transport penalty, it prevents global shifts in visual focus while strictly prohibiting visual reliance from degenerating into language bias. Second, Visually Modulated Adaptation (VMA) exploits VD heterogeneity to emphasize visually grounded new-task learning, promoting new-task plasticity. Together, our method simultaneously maintains old-task stability and new-task plasticity during challenging MU-CPT. Extensive experiments under our MU-CPT setting validate the effectiveness of VDA.
来源说明
当前保存的是来源摘要，不代表论文全文。请以原始来源为准。
「要点解读」由 AI Stack 依据上方已保存内容整理，不代表来源的完整表述；标注「推测：」的判断来自编辑，不是来源陈述。