Suchir EMNLP 2026 in Budapest: six papers, two orals
EMNLP 2026 was in Budapest in October. Three of our papers were in the main conference: Beetle, LangMAP and a study of cross-lingual alignment between separately trained models. Three more appeared in workshop proceedings: one at the LM Playschool and two in BlackboxNLP's reproducibility track. Beetle and LangMAP were both selected for oral presentations, among roughly the top 3% of a pool of about 17,000 ARR submissions. We also presented a non-archival poster at the MRL workshop, and I co-organised the BabyLM workshop.
Below is one short note per paper: the question, what we did, one result, and where to find the paper, slides and poster.
Beetle
Most work that uses language models to study human language processing is monolingual, and mostly English, even though most people speak more than one language. Beetle asks whether controlling a model's second-language experience (when English arrives, and in what proportion) changes how well it models second-language readers. It was presented as an oral.
With Catherine Arnett, James Michaelov and Paula Buttery, I trained a suite of 330 open models (285 bilingual, 45 monolingual) across five exposure curricula and three training scales, from 100M to 24B tokens. English is always the second language. The curricula run from a balanced 50/50 mix (B1) to a late onset where English only arrives 80% of the way through training (B5).
To compare models with readers, we took first-pass gaze durations from second-language English speakers in the MECO eye-tracking corpus. We used each model's word surprisal to predict them in a mixed-effects regression that controls for word length, frequency and position, and measured the gain in log-likelihood over a baseline without surprisal. Staged exposure (B2 to B5) generally fit second-language reading times better than balanced bilingual training (B1). The poster adds that the Beetle models generally fit better than five massively multilingual baselines (Apertus, Gemma, Llama, Qwen and XGLM), and that no single curriculum is best for every case.
Paper: ACL Anthology · publication entry · Beetle page · slides · poster · framework: beetlelm.github.io
LangMAP
A multilingual tokenizer's vocabulary often already contains good pieces for a language, but it applies one global segmentation policy to every language, so it can pick the wrong ones. Clara Meister led this paper, which asks whether a pretrained tokenizer can be made language-adaptive by changing how it scores existing tokens, without touching the vocabulary. It was presented as an oral.
LangMAP keeps the vocabulary and learns one unigram distribution per language with EM, estimated from 25M tokens of each language. At inference it segments a string under each language's distribution and returns the most likely segmentation overall, so it needs no language label. We applied it to the Qwen, Llama, Gemma and Mistral tokenizers and compared it with the original tokenizer and a language-agnostic UnigramLM, across seven natural languages (German, Finnish, Hungarian, Turkish, Hindi, Arabic and Tamil) and several programming languages.
LangMAP improved alignment with morphological boundaries (MorphScore) across the natural languages and consistently improved alignment with code structure (abstract syntax tree boundaries). Fine-tuning models on LangMAP segmentations did not show consistent improvements, so the gain is in segmentation quality and not, so far, in downstream performance.
Paper: ACL Anthology · publication entry · slides, prepared with Clara · poster · short write-up
Cross-lingual alignment without joint training
Multilingual models usually get their cross-lingual ability from joint training, which makes it hard to say where the alignment comes from. With EJ Zhou, Catherine Arnett and Anna Korhonen, we asked whether language models trained separately on single languages still converge on shared representations, with no joint training at all.
We tested strictly monolingual models, including the Goldfish families and independently developed models from different labs. We measured how alignable their representational geometry is across layers, fit a single Procrustes rotation on parallel sentences to map hidden states between models, and then patched rotated activations from one model into another.
The models develop alignable geometry, and alignment strengthens with data scale, model scale and linguistic proximity. The rotation also carries functional content: in most cases, patching a rotated English residual into a German model on a factual cloze flips its prediction to the donor's capital. In the paper's framing, this suggests cross-lingual alignment can emerge from the structure of language itself rather than from joint training.
Paper: ACL Anthology · publication entry · short write-up
Protocol competence in interactive language games
Before a small language model can learn from interaction, it has to take part in the interaction correctly. At the LM Playschool workshop, Paula Buttery and I asked whether demonstrations from a larger model can teach a small model to take part in interactive language games, and whether it can then improve from its own successful games.
We evaluated Qwen, Llama and Gemma models from 0.8B to 7B parameters on five CLEMBench Playpen games, scored with ClemScore. Small models often failed at interacting correctly rather than at the task itself. We then compared protocol repair, teacher demonstrations and self-ReST (generate, filter and fine-tune on the model's own winning games) on Qwen3.5-0.8B under matched compute budgets.
For that model, ClemScore went from 6.67 with protocol repair to 14.97 with demonstrations and 20.27 with self-ReST. Self-generated data only helped once the model could interact reliably. A credit-weighted variant of ReST had a mean of 19.17 with lower variance, but the differences were not statistically significant.
Paper: ACL Anthology · publication entry · poster
Claude reinvestigates herself
Kryvosheieva et al. (ACL 2026) reported that different kinds of syntactic agreement recruit the same units inside large language models. In BlackboxNLP's reproducibility track, Elena Polyakova and Shivan Arora led a replication with Paula Buttery and me.
We reproduced the main findings with ablation, cross-overlap and cross-validation analyses on BLiMP across seven LLMs. We then added three methods: attention-head analysis, Differential Binary Masking and Distributed Alignment Search. We also extended the analysis to other languages.
In Russian, the units shared across benchmarks overlapped more than chance (3.94% against 1%), but much less than units within one benchmark (15.86 to 26.94%). The attention-head analysis also found sharing across phenomena, but it was broader and less specific to agreement than in the original residual-unit analysis, and its extent varied across architectures and languages.
Paper: ACL Anthology · publication entry · poster
Linear truth encodings
Truth probes are used for hallucination detection and truthfulness interventions. Ravfogel et al. (2025) proposed that linear truth directions emerge because true statements tend to co-occur, which helps a model predict the next token. Mingchuan Zou led our replication, also in BlackboxNLP's reproducibility track, with Paula Buttery and me.
We tested the mechanism in one-layer attention-only toy transformers, with an ablation where the truth of one fact says nothing about the truth of the other. We also probed Llama-3.1-8B and 70B on six CounterFact relations with logistic-regression probes.
In our one-layer fully trainable transformers, truth was not linearly decodable without co-occurrence. We reproduced the 8B false-context gap (1.43 nats with Llama-3.1, against 1.52 reported with Llama-3), but not the original's final probability of the right answer after a false sequence (about 0.48 for us, about 0.25 originally). The limits matter here: we only show necessity in synthetic models, we tested two models on six relations, and probes show that truth is decodable, not that the model uses it.
Paper: ACL Anthology · publication entry · poster · short write-up
Shared units, divergent trajectories
Earlier work on whether syntactic phenomena share components inside a model looks mostly at finished models. At the MRL workshop, Elena Polyakova and I presented a poster asking when those shared or language-specific units emerge during training, and how the order of language exposure shapes them. The poster was non-archival.
This is a pilot on Dutch as the first language and English as the second. We trained GPT-2-style bilingual models with two Beetle curricula: simultaneous (continued mixed exposure) and sequential (a full switch from Dutch to English halfway through). We then tracked syntax-responsive units across four checkpoints from the point English is introduced to the end of training.
Sequential training ended with substantially lower Dutch BLiMP-NL accuracy and weaker overlap among Dutch units, which fits forgetting once Dutch exposure stops. The curriculum effects were specific to each phenomenon rather than a uniform shift across syntax.
What connects them, and BabyLM
Most of these papers vary what a model sees and when, then look at what it ends up representing: the exposure curricula in Beetle and the MRL pilot, per-language token preferences in LangMAP, and separate monolingual training in the alignment paper. The two BlackboxNLP papers check whether published claims about model internals hold up when someone else reruns them.
I also co-organised the BabyLM workshop, which this year opens its sample-efficient pretraining challenge to multilingual training data (call for papers, more on BabyLM 2026).
Thank you to all my co-authors, and to my fellow BabyLM organisers.