What's the Best Sequence Length for BabyLM?

A design choice that BabyLM submissions tend to make quietly: how many tokens of context to train on. We treated it as a variable worth studying rather than a default to inherit.

Transformer language models train on fixed-length context windows, and those windows have grown as pretraining datasets have grown. BabyLM submissions, though, often use much shorter sequences than frontier models do. That mismatch is what prompted the question here.1Salhan, Diehl Martinez, Goriely, Buttery. What is the Best Sequence Length for BabyLM?, BabyLM Workshop, EMNLP 2025. When you are limited to a small corpus, is a short context a sensible economy or a quiet handicap?

The setup

We compared two 125M-parameter models, a Mamba model and an OPT model, trained on 100M words under fixed computational budgets.2The 100M-word track is the standard low-resource setting of the BabyLM Challenge, which asks how much can be learned from roughly the amount of language a child hears. Holding compute fixed matters, because sequence length trades off against how many sequences a model sees. Pairing a state-space architecture with a Transformer also lets the question be asked across two quite different ways of handling context, rather than reading a single model's quirks as a general rule.

What we found

The short answer is that longer is often better, but the best length depends on both the task and the architecture. That qualification is the point. Shorter sequences were enough for grammatical generalisation tasks, where the relevant structure sits within a fairly local window. Longer contexts helped on morphological analogical reasoning, where the model benefits from seeing more material at once.

So there is no single number to reach for. The right context length follows from what you are asking the model to do and from how the architecture uses the tokens it is given. A grammatical judgement and a morphological analogy make different demands on context, and the sequence length that suits one need not suit the other.

Why it matters for small models

For BabyLM-scale work the practical lesson is that sequence length is a real lever, not an afterthought to be set by convention. Under a fixed budget it interacts with everything else, and the sensible move is to pick it against the tasks you care about rather than copying a default from a much larger model. Reporting it as a deliberate choice, and matching it to the evaluation, seems worth doing.