基本信息
- 来源: arxiv
- 原始来源: https://arxiv.org/abs/2605.22821v1
- 作者: Jan Tempus, Philip Whittington, Craig W. Schmidt, Dennis Komm, Tiago Pimentel
- 分类: cs.CL
- 论文时间: 2026-05-21T17:59:56Z
- 论文 PDF: https://arxiv.org/pdf/2605.22821v1.pdf
来源摘要/节选
Tokenisation is an integral part of the current NLP pipeline. Current tokenisation algorithms such as BPE and Unigram are greedy algorithms – they make locally optimal decisions without considering the resulting vocabulary as a whole. We instead formulate tokeniser construction as a linear program and solve it using convex optimisation tools, yielding a new algorithm we call ConvexTok. We find ConvexTok consistently improves intrinsic tokenisation metrics and the bits-per-byte (BpB) achieved by language models; it also improves downstream task performance, but less consistently. Furthermore, ConvexTok allows the user to certify how far their tokeniser is from optimal, with respect to a certain objective, via a lower bound, and we empirically find it to be within 1\% of optimal at common vocabulary sizes.
来源说明
当前只保存了官方论文摘要，不代表论文全文。请以原始来源为准。
本页只呈现已做哈希绑定的来源证据，不包含基于旧正文或缺失原文的扩展推断。