Less is More: Pre-Training Cross-Lingual Small-Scale Language Models with Cognitively-Plausible Curriculum Learning Strategies

Curriculum learning has been popular in BabyLM work but rarely beats a plain baseline. We asked whether the problem is the idea or the level of detail, and built curricula that follow specific acquisition theories closely across four language families.

Ordering training data from simple to complex, in imitation of how children encounter language, is an appealing idea for small-scale language models. It has shown up repeatedly in the BabyLM Challenge. In practice, though, curriculum learning has not delivered much of an improvement over models trained without any ordering at all.1Salhan, Diehl Martinez, Goriely, Buttery. Less is More: Pre-Training Cross-Lingual Small-Scale Language Models with Cognitively-Plausible Curriculum Learning Strategies, CoNLL BabyLM Challenge, 2024. Our starting suspicion was that earlier curricula were too coarse to capture what acquisition theories actually predict.

Building the corpora

To test theories about how children learn language, the training data has to resemble what children hear. We assembled age-ordered corpora of child-directed speech for four typologically distant language families,2Child-directed speech corpora of this kind are drawn from resources like CHILDES, the standard archive of child language transcripts. so that the curricula could be studied beyond English and across genuinely different grammatical systems rather than a single language's quirks.

Three structured curricula

On standard model architectures we tested three curricula, named Growing, Inwards, and MMM. The design principle was to replicate the predictions of acquisition theories precisely, rather than approximate them with a rough ordering. Each curriculum encodes a specific account of the order in which linguistic structure is acquired, so that the training schedule mirrors a theory closely enough to be a fair test of it.

What came out

The fine-grained, acquisition-inspired curricula outperformed the non-curriculum baselines. The reading we take from this is that the earlier disappointment with curriculum learning was about granularity, not about the underlying idea. When the schedule replicates a specific acquisition theory precisely, and is specified per language rather than in one generic form, the benefit shows up.

The conclusion we draw is that the gains from curriculum strategies in small-scale language models come from specifying fine-grained, language-specific curricula that faithfully replicate language acquisition theories. The title is the short version: less data, ordered well, does more than the same data left in an arbitrary order, provided the ordering is detailed and tied to a real theory of how the language is learned.