CoLT: Teaching Multi-Modal Models to Think with Chain of Latent Thoughts

📄 arXiv: 2606.31986v1 📥 PDF

作者: Lianyu Hu, Shengqian Qin, Zeqin Liao, Qing Guo, Liang Wan, Wei Feng, Yang Liu

分类: cs.CV

发布日期: 2026-06-30

备注: Accepted by ECCV2026. Code is available at https://github.com/hulianyuyy/CoLT

🔗 代码/项目: GITHUB

💡 一句话要点

提出CoLT框架以提升多模态模型的推理效率

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 多模态模型 链式思维 潜在思维 推理效率 外部解码器 内部监督 视觉推理 深度学习

📋 核心要点

  1. 现有的链式思维推理方法在推理效率和表达能力上存在不足，导致推理过程缓慢且不稳定。
  2. CoLT框架通过引入潜在思维表示链，结合轻量级外部解码器和内部监督，提升了推理的效率和稳定性。
  3. 实验结果显示，CoLT在八个基准测试中表现优异，推理时间减少了10.1倍，文本解码时间减少了22.6倍。

📝 摘要（中文）

链式思维（CoT）推理使多模态大语言模型（MLLMs）能够通过生成明确的中间推理步骤来处理复杂的视觉推理任务。然而，这种基于文本的推理在推理时速度较慢，并受到自然语言表达能力的限制。本文提出了CoLT（链式潜在思维），一个新颖的框架，教会多模态模型通过潜在思维表示链进行推理，且仅需3个步骤。为有效调节潜在推理过程，本文引入了轻量级外部解码器，提供逐步监督，并在推理时去除解码器以保持高效性。大量实验表明，CoLT在多个基准测试中超越了现有的潜在推理方法，并显著减少了推理时间。

🔬 方法详解

问题定义：本文旨在解决现有多模态模型在复杂视觉推理任务中推理效率低下和表达能力不足的问题。现有的链式思维推理方法在推理时需要处理大量文本令其速度缓慢且不稳定。

核心思路：CoLT框架的核心思路是通过潜在思维表示链进行推理，减少对文本的依赖，从而提高推理效率。引入轻量级外部解码器为每个潜在推理步骤提供监督，确保推理过程的有效性。

技术框架：CoLT的整体架构包括潜在思维表示链、外部解码器和内部监督机制。外部解码器分为前向模式和后向模式，前者将潜在思维解码为文本推理，后者则对齐解码器隐藏状态与模型的潜在思维。

关键创新：CoLT的主要创新在于引入潜在思维表示链和轻量级外部解码器，显著提升了推理效率和稳定性。这一设计与传统的文本推理方法有本质区别，后者依赖于大量文本令推理过程缓慢。

关键设计：在设计中，外部解码器提供逐步监督，确保潜在思维的连贯性。此外，内部监督机制鼓励潜在思维的顺畅过渡，确保推理过程的有效性。

🖼️ 关键图片

fig_0
fig_1

📊 实验亮点

在八个基准测试中，CoLT框架的表现超越了现有的潜在推理方法，如CODI和SIM-CoT，推理时间减少了10.1倍，文本解码时间减少了22.6倍，显示出显著的效率提升。

🎯 应用场景

CoLT框架在多模态推理任务中具有广泛的应用潜力，特别是在需要快速响应的实时视觉分析、自动驾驶、智能监控等领域。其高效的推理能力将推动相关技术的进步，提升实际应用的价值。

📄 摘要（原文）

Chain-of-thought (CoT) reasoning has enabled multi-modal large language models (MLLMs) to tackle complex visual reasoning tasks by generating explicit intermediate reasoning steps in natural language. However, this text-based reasoning paradigm is inherently slow at inference time with even thousands of tokens and fundamentally constrained by the expressiveness of natural language. In this paper, we propose CoLT, (Chain of Latent Thoughts), a novel framework that teaches multi-modal models to reason through a chain of latent thought representations instead of verbose text tokens, which can perform thinking with as few as 3 steps. Naively forcing the model to think with latent states easily produces meaningless semantics and makes training unstable. To effectively regulate the latent reasoning process, we introduce a lightweight external decoder that provides step-level supervision for each latent reasoning step in two complementary directions: a forward mode that decodes latent thoughts into the textual reasoning of the next step, and a backward mode that aligns decoder hidden states with the model's latent thoughts given preceding textual context. We further incorporate internal supervision that encourages coherent step-by-step latent transitions. The decoder and internal supervision are removed during inference to maintain high efficiency of latent reasoning. Extensive experiments on eight benchmarks demonstrate that CoLT not only outperforms existing latent reasoning methods such as CODI and SIM-CoT, but also surpasses latent visual reasoning approaches that rely on auxiliary images with costly annotation requirements. Compared to text CoT methods, CoLT can notably reduce the inference time by 10.1$\times$ and text decoding time by 22.6$\times$. Code is released at https://github.com/hulianyuyy/CoLT.