ByteSpan: Information-Driven Subword Tokenisation

A tokeniser that decides where a subword begins by watching a byte-level language model hesitate, then freezes those groupings into a fixed vocabulary.

Most subword tokenisers, including the ubiquitous Byte Pair Encoding, build a vocabulary from surface statistics: they merge the byte or character pairs that co-occur most often until they hit a target vocabulary size. That works, but it has little to say about where one meaningful unit ends and the next begins. ByteSpan1Goriely, Salhan, Lesci, Cheng & Buttery. ByteSpan: Information-Driven Subword Tokenisation, ICML 2025 Tokenization Workshop. starts from a different intuition, borrowed from cognitive models of how humans might segment a continuous speech or text stream.

The idea: boundaries live where prediction breaks down

There is a line of work in computational modelling of word segmentation that places lexical boundaries at spikes in prediction error. When an autoregressive model is reading through a stream and suddenly becomes much less certain about the next symbol, that surprise is a decent signal that a unit has just ended and a new one is starting. Inside a familiar chunk the next byte is easy to guess; at the seam between chunks it is not.

ByteSpan turns that observation into a tokenisation procedure. During training it runs an external byte-level language model over the corpus and uses the model's predictability to identify contiguous runs of bytes that hang together, then groups each run into a single subword. The result is not a dynamic, per-example segmentation but a fixed subword vocabulary you can reuse, in the same slot where you would otherwise drop in BPE.

How it differs from dynamic byte models

This is worth separating from a nearby family of methods. Recent dynamic tokenisation approaches also operate directly on bytes, but they pool the latent byte representations into patches inside the model. ByteSpan does something more conservative: it groups the bytes themselves into reusable subword units rather than pooling their representations. You keep a discrete vocabulary and all the tooling that assumes one, while choosing the vocabulary on information-theoretic grounds instead of raw frequency.

What the paper reports

For English, the authors report that ByteSpan produces efficient vocabularies whose subwords align better with morphology than BPE's do, measured by morphological alignment scores. They then extend the evaluation to a multilingual setting spanning 25 languages, where ByteSpan achieves compression and Rényi efficiency comparable to existing tokenisers. So the claim is not that predictability-driven grouping blows past BPE on every axis, but that it recovers more morphologically sensible units in English while staying competitive on standard efficiency measures across languages.

I like this direction because it treats the tokeniser as a place where a small amount of linguistic structure can be recovered cheaply, without changing the downstream model architecture. The segmentation signal comes from a model that already knows how to predict bytes; ByteSpan just listens for where that model gets surprised and writes the answer down as a vocabulary.