基本信息
- 来源: arxiv
- 原始来源: https://arxiv.org/abs/2601.18791v1
- 作者: Iaroslav Chelombitko, Mika Hämäläinen, Aleksey Komissarov
- 分类: cs.CL
- 论文时间: 2026-01-26T18:55:28Z
- 论文 PDF: https://arxiv.org/pdf/2601.18791v1.pdf
来源摘要/节选
We present a large-scale comparative study of 242 Latin and Cyrillic-script languages using subword-based methodologies. By constructing ‘glottosets’ from Wikipedia lexicons, we introduce a framework for simultaneous cross-linguistic comparison via Byte-Pair Encoding (BPE). Our approach utilizes rank-based subword vectors to analyze vocabulary overlap, lexical divergence, and language similarity at scale. Evaluations demonstrate that BPE segmentation aligns with morpheme boundaries 95% better than random baseline across 15 languages (F1 = 0.34 vs 0.15). BPE vocabulary similarity correlates significantly with genetic language relatedness (Mantel r = 0.329, p < 0.001), with Romance languages forming the tightest cluster (mean distance 0.51) and cross-family pairs showing clear separation (0.82). Analysis of 26,939 cross-linguistic homographs reveals that 48.7% receive different segmentations across related languages, with variation correlating to phylogenetic distance. Our results provide quantitative macro-linguistic insights into lexical patterns across typologically diverse languages within a unified analytical framework.
来源说明
当前只保存了官方论文摘要，不代表论文全文。请以原始来源为准。
本页只呈现已做哈希绑定的来源证据，不包含基于旧正文或缺失原文的扩展推断。