基本信息
- 来源: arxiv
- 原始来源: http://arxiv.org/abs/2608.02602v1
- 发布域名: arxiv.org
- 分类: cs.CL
- 作者: Jiajun Liang、Yucheng Liao、Yukang Cao 等
要点解读
这是什么
AURORA-LM 是一种在连续潜空间使用扩散技术进行文本生成的语言模型，通过查询式编码器‑解码器将文本映射为高容量可解码的潜码，并利用块因果扩散Transformer学习该潜码的分布。
用在哪里
适用于对生成质量要求较高的自由文本生成或摘要任务，尤其是想探索连续潜空间扩散语言模型的研发人员和工程师。
可以推断的
推测：扩散过程在连续潜码上通常比离散 token 模型需要更多的计算和迭代步数。
推测：在需要保持语言细节或提升生成逼真度的场景，这类模型可能更具优势。
来源摘要/节选
Language remains an outlier in generative modeling: while images, video, and audio are increasingly modeled in continuous latent spaces, text generation still relies predominantly on discrete tokens. Existing continuous language models either inherit embedding spaces not designed for joint generation and decoding, or compress autoencoded latents to ease diffusion, sacrificing token-level fidelity. Instead of simplifying the representation to suit the generative model, we preserve a high-capacity, decodable text latent and design the diffusion model to learn its distribution directly. We introduce AURORA-LM, a continuous-latent diffusion language model that separates the construction of a decodable text representation from the modeling of its distribution. A Query-based Encoder-Decoder organizes text into a high-capacity, prefix-aligned latent sequence, and a Block-causal Diffusion Transformer learns its distribution through flow matching, generating blocks left to right while denoising positions within each block in parallel. Because such a latent is harder for diffusion to model, AURORA-LM restricts only the noisy-input pathway while retaining the full clean-latent prediction target, accommodating full-width latents without reducing decoder-facing capacity. We further calibrate the noise-level distribution to the latent width, and introduce self-trajectory consistency to bridge independently sampled training noise and iterative denoising at inference. AURORA-LM achieves the strongest performance among evaluated continuous and diffusion-based language models on OpenWebText free generation and XSum summarization. Scaling to 1B parameters with about 1500 EFLOPs of total compute yields further gains, surpassing a larger publicly released latent-diffusion language model under a matched evaluation protocol. All experiments are conducted on Ascend NPUs.
来源说明
当前保存的是来源摘要，不代表论文全文。请以原始来源为准。
「要点解读」由 AI Stack 依据上方已保存内容整理，不代表来源的完整表述；标注「推测：」的判断来自编辑，不是来源陈述。