Beetle: a framework for controlled bilingual & multilingual pretraining

Small models. Big questions. Today I am open-sourcing Beetle, the framework I built to study how language models acquire a second language under tightly controlled, fully reproducible conditions.1Beetle software release — beetlelm.github.io

Beetle is a framework I designed and built for controlled bilingual and multilingual pretraining. It gives you a single, declarative interface in which the tokeniser, the target language, the training budget, and the exposure structure of the data can each be manipulated independently. Everything is configured through declarative YAML, which turns what used to be a tangle of bespoke scripts into a systematic, large-scale ablation of how learning dynamics and data structure shape acquisition. As of today it is open source, and the release — the engine, the curricula, the model suite, and the analysis tooling — is available at beetlelm.github.io.

Although Beetle grew out of computational psycholinguistics — modelling how humans process a second language — it is much more than a psycholinguistics tool. It is a general interpretability and controlled-pretraining infrastructure. Because every model is checkpointed densely across training, and because exposure is a first-class, configurable variable, Beetle lets you ask mechanistic questions about when and how abilities emerge, not just whether a final model has them.

Why controlled pretraining

Most claims about what a language model has learned rest on inspecting the finished model. That misses almost everything interesting about acquisition. I built Beetle so that exposure structure becomes an experimental variable, letting us watch a second language being acquired checkpoint by checkpoint rather than only reading off the end state. The same models, the same budget — only the way the second language is introduced changes. That is what makes the results below controlled rather than anecdotal.

What I trained

Beetle is not a single script but a large, reproducible model suite, and I trained the whole thing. The release includes 285 bilingual and 45 monolingual models (330 in total), with rich checkpoints across a range of typologically-diverse L1s, exposure curricula (B1–B5) and three data scales (100M / 2B / 24B tokens) — released to study multilingual pretraining and the computational modelling of bilingualism and second-language learning.

The 13 L1s span German, Dutch, Chinese, Italian, Hindi, Russian, Turkish, Basque and more, deliberately chosen to vary typological distance from English. Checkpoints follow a log₂ schedule and preserve gradients, activations and circuits, so the suite supports interpretability out of the box; for several of these languages these are the first densely checkpointed models released. The default backbone is a 125M LLaMA-style PicoDecoder,2PicoDecoder / PicoLM — picolm.io. Pico is inherited, collaborative infrastructure; Beetle builds on it. with encoder-decoder, MoE and SSM variants also supported. Models are evaluated on MECO L2 reading-time, BLiSS, CEFR, JFLEG and Mono/Multi-BLiMP.

The five exposure curricula

The heart of the design is five ways of introducing the second language during pretraining. Each is a declarative configuration in Beetle. B1 (Balanced) interleaves L1 and L2 evenly throughout training. B2 (Simultaneous) mixes both languages into a single stream from the first step. B3 (Sequential) establishes the L1 first, then introduces the L2 in a distinct later stage. B4 (Classroom / episodic) delivers the L2 in spaced bursts interspersed with L1, an analogue of classroom instruction rather than immersion. B5 (Late) introduces the L2 only near the end of training, after extensive L1 exposure.

What we find

Exposure structure is not a cosmetic training detail. It changes what the model learns, and how well small models mirror human second-language processing. This work is the subject of my first first-author main-conference paper, currently under review.3Structured Exposure Pretraining in Bilingual Language Models for Modelling L2 Language Processing — Salhan, Arnett, Michaelov, Buttery. Paper available on request.

Small Beetle models rival 8B LLMs on human L2 reading-time

Models trained with structured, staged exposure align with human L2 reading times (MECO L2) as well as, and often better than, multilingual LLMs of up to 8B parameters, at a tiny fraction of the size. Human-scale, controlled pretraining turns out to be competitive with brute-force scale for this cognitive-modelling target.

A dissociation: staged exposure for processing, balanced for grammaticality

The best curriculum depends on what you measure. Staged exposure yields the best fit to online reading-time behaviour, whereas balanced exposure produces stronger offline grammatical judgement (CEFR, JFLEG). This dissociation between online processing and offline competence mirrors a distinction long drawn in the psycholinguistics of second-language acquisition.

Learning dynamics: phase transitions, curricula, and forgetting

Because Beetle checkpoints densely, we can watch these abilities emerge. Acquisition proceeds in phase transitions rather than smoothly; curricula interact with typological proximity to English; and sequential and late curricula expose catastrophic forgetting of the L1 that balanced exposure avoids. These dynamics are invisible if you only inspect the final model.

Reproducible by design

Beetle is not one script but three pip-installable libraries and several supporting infrastructures: a controlled-pretraining engine, an exposure-curriculum system, and an analysis and interpretability toolkit. Everything needed to reproduce the work is released — the 330 model checkpoints, the tokenisers, the decontaminated, pretokenised training data at all three scales, and the full analysis pipelines. The goal is that any result here can be re-derived from the released artefacts alone. Beetle is my own framework: I designed and built the system, I trained the 330 models, and I authored the curriculum engine, the analysis toolkit and the libraries. It builds on the PicoDecoder backbone from PicoLM, which I did not create; everything else is my individual contribution.

The software is live now. If you want to run controlled bilingual or multilingual pretraining, or reuse the checkpoints for interpretability, start at beetlelm.github.io.