LangMAP: A Language-Adaptive Approach to Tokenization

Forthcoming work on letting the tokeniser bend to the language rather than forcing every language through one fixed scheme. Currently under review.

This is a short note on a paper that is still under review, so I will keep it to the shape of the idea rather than any numbers.1Under review at EMNLP 2026 (Main Conference); no public page yet. See my Google Scholar profile for updates. LangMAP is about a tension anyone who has trained multilingual models runs into quickly: a single tokenisation scheme, fixed once and applied everywhere, rarely serves every language equally well.

The default recipe learns one vocabulary over a pooled multilingual corpus and applies it uniformly. That is convenient, but it quietly encodes the statistics of whichever languages dominated the training mix. Languages with rich morphology, non-Latin scripts, or simply less data in the pool tend to get shredded into far more tokens per word than English does. The tokeniser is doing the same thing to everyone, which is precisely the problem, because the same thing is not equally appropriate for every language.

The premise

LangMAP starts from the position that tokenisation should be a property that adapts to the language in front of it rather than a single global commitment. The name gestures at mapping the tokenisation to the language, so that the segmentation a model sees for one language can reflect that language's structure instead of a compromise struck across the whole corpus.

Framed this way, the question the paper takes on is whether adapting tokenisation per language buys you something worth the added machinery, and how to do that adaptation without giving up the shared vocabulary that makes a multilingual model coherent in the first place. Those two pulls, adapt to each language yet stay unified, are the interesting part.

I will write this up properly once the paper is public and I can point to the actual method and results. For now it sits alongside my other tokenisation work, including ByteSpan, as part of a broader interest in treating the tokeniser as a design choice with real consequences rather than a fixed preprocessing step you inherit and forget about.