BabyLM: Sample-Efficient Pretraining on a Developmentally Plausible Corpus

The shared task and workshop returns for a fourth year, this time with a new multilingual track, and I was again on the organising side.

The BabyLM Challenge asks researchers to train a good language model on the amount of language a child could plausibly hear.1Event site. At EMNLP 2026 in Budapest it runs for its fourth year, and the premise has held up: instead of competing on who can afford the most data, participants work inside a fixed, human-scale budget and compete on what they can do with it. The rules cap training on datasets under 100 million words, with a Strict track at that ceiling and a Strict-Small track at 10 million words. The smaller budget is deliberately uncomfortable, and it is where a lot of the interesting ideas about curricula, tokenisation, and objectives have tended to show up.

What is new this year

The headline change is a MultiLingual track built on BabyBabelLM, covering English, Dutch, and Chinese at 100 million tokens. This matters because a corpus that looks developmentally plausible for an English-speaking child is not automatically plausible, or even available, for children learning other languages, and a method that only works on English tells you less than it seems to. Some previously standalone tracks, for multimodal and interaction settings, have been folded into the main tracks this year, which tidies the structure and keeps the core comparison clean.

Why the constraint is the point

Fixing the data budget turns pretraining into a science rather than a spending contest. Two submissions on the same track saw the same words, so a difference in results is a difference in method, and the leaderboard becomes something you can actually reason about. It also keeps the work close to a real cognitive question: children are demonstrably sample-efficient learners, and it is worth knowing which of the tricks that help a small model, curriculum ordering, better segmentation, auxiliary objectives, correspond to anything a child plausibly does.

My role

I served as a co-organiser, working with a team spread across IBM Research, ETH Zurich, Cornell, NYU, Boston University, and other institutions. The practical side of the challenge is less glamorous than the leaderboard suggests: agreeing on the corpora, keeping the evaluation suite honest and reproducible, and making the rules tight enough that the comparison means something. Submissions closed on 20 July 2026, camera-ready papers were due on 5 September, and the workshop sits within the EMNLP programme in late October. Four years in, what keeps me involved is that the format has become a shared testbed the community can build on, rather than a one-off contest.