基本信息

要点解读

这是什么

Spool 是一种免训练的低延迟流式推理框架,面向结合注意力层与线性循环层的混合视觉语言模型。它缓存各帧对应的状态变换并高效组合,使滑动窗口前移时无需重跑整个视频窗口即可移除过期帧。

用在哪里

适用于视觉无障碍代理、机器人等需要围绕近期视频内容快速问答的应用,也适合研究实时视频推理与服务优化的工程团队。

可以推断的

  • 推测:在 20 组模型与基准测试中均实现最低的首 token 延迟,并优于两种近期 vLLM 基线,说明其工程优化具有较强的实际吸引力。
  • 推测:逐层精确移除与跨层残余影响之间的取舍,可能使 Spool 更适合强调响应速度的实时场景,而非追求历史帧影响完全清零。

来源摘要/节选

Applications such as visual accessibility agents and robotics require answering questions about recent video content promptly, over a bounded window of past frames (e.g., the last two minutes). Hybrid vision-language models (VLMs), which interleave attention with linear recurrent layers such as Gated DeltaNet (GDN), are attractive for this setting, but maintaining a sliding window is expensive. The reason is that recurrent layers combine all past frames into a shared state, so expiring old frames normally requires re-running the entire window through the model. We observe that the recurrent update is affine in the state with input-dependent coefficients, so the effect of each frame can be captured as a transformation that composes associatively. We present Spool, a training-free framework that caches per-frame state transformations and composes them efficiently as the window advances, removing expired frames from each recurrent layer’s state without replaying frames through the backbone. Removal is exact within each layer; as with key-value eviction in attention, expired frames can retain residual influence through the cached inputs computed by earlier layers. We design a dedicated data structure and GPU kernels to achieve low caching and composition overhead. Across five open-source hybrid VLMs and four video reasoning benchmarks, Spool achieves the lowest p95 time-to-first-token in all 20 model-benchmark tests, reducing it by a geometric mean of $6.6\times$ over the faster of two recent vLLM baselines, while improving accuracy by 2.3 percentage points on average.

来源说明

当前保存的是来源摘要,不代表论文全文。请以原始来源为准。

「要点解读」由 AI Stack 依据上方已保存内容整理,不代表来源的完整表述;标注「推测:」的判断来自编辑,不是来源陈述。