Teaching & Supervision

1 · Dynamic Tokenization — Learning What a Model Should Process Together

Research area: Language model architectures, tokenization, adaptive computation, multilingual NLP.

Tokenization is often treated as a preprocessing step. An alternative view is that tokenization is a resource allocation policy: it determines how much computation and memory a model spends processing different parts of its input.

Conventional subword tokenizers assign fixed segmentation rules before a model begins learning. But not all input sequences are equally difficult. A familiar word, an unfamiliar name, a piece of code, and a morphologically complex word may benefit from very different granularities of processing. Could a model learn to allocate its computational resources dynamically, rather than being constrained by a fixed vocabulary? Recent approaches to byte-level and patch-based language modelling, alongside work on transferring pretrained tokenizers into byte-level models, create promising opportunities to investigate this question.

Possible research directions.

• Transferring pretrained tokenization: can we transfer useful segmentation structure from a pretrained BPE language model into a byte-level or tokenizer-free architecture without inheriting its fixed vocabulary?
• Adaptive tokenization: can a model learn when to process text as individual bytes, characters, subwords, or longer patches, using signals such as uncertainty or information content?
• Multilingual resource allocation: does dynamic tokenization allocate computation more effectively across languages with different writing systems, morphological structures, and levels of representation in the training data?
• Interpreting segmentation decisions: can we understand which features of an input cause a model to change its processing granularity, and whether those decisions improve downstream performance?

A particularly interesting direction is to investigate whether tokenization can be transferred as a learned computational strategy, rather than as a fixed collection of boundaries.

What you could do: implement or adapt a tokenization architecture, design controlled comparisons against standard BPE and byte-level baselines, and evaluate the resulting models on language modelling, multilingual tasks, or computational efficiency. Depending on the project, there may also be scope for analysing learned representations and segmentation behaviour.

A good fit if you enjoy: deep learning systems, model architectures, efficient training, multilingual NLP, or the interface between language and computation.

Selected references

• Cognetta et al. (2026), Tokenization: A Survey for Modern NLP.
• Pagnoni et al. (2025), Byte Latent Transformer: Patches Scale Better Than Tokens.
• Meister et al. (2026), LangMAP: A Language-Adaptive Approach to Tokenization.

2 · Sample-Efficient AI — Building Models That Learn More from Less

Research area: Language model training, efficient architectures, cognitive science, multimodal learning.

Modern AI systems are trained on enormous quantities of data, yet humans acquire language and other complex skills from comparatively limited experience. Can we develop learning systems that make better use of the data they receive?

The BabyLM initiative investigates this question by challenging researchers to train language models under constrained data budgets. I co-organise the BabyLM and BabyVLM challenges, which provide benchmarks and shared tasks for investigating sample-efficient language and multimodal learning. The goal is not simply to produce a smaller model; it is to understand which architectural and learning choices allow a model to acquire useful capabilities from limited experience. Cognitive science offers hypotheses about structured learning, curriculum, memory, and inductive biases; machine learning provides the tools to test and extend them.

Possible research directions.

• Architectures for limited-data learning: test whether recurrence, parameter sharing, adaptive computation, or alternative architectural choices improve learning when training data are scarce.
• Cognitively inspired learning strategies: investigate whether structured curricula, linguistic inductive biases, or models of human language acquisition improve sample efficiency.
• From text to multimodal learning: test whether techniques that help language models learn from limited text also benefit models trained on restricted amounts of visual or speech data.
• Evaluating what small models know: develop or apply evaluations that distinguish genuine acquisition of a capability from memorisation, superficial task performance, or reliance on dataset artefacts.

A strong project might start with a specific hypothesis about learning, implement a controlled comparison, and establish not only whether an approach works but under what conditions and why.

What you could do: train small models under controlled data budgets, develop new architectures or training strategies, and evaluate learning curves across model sizes, datasets, or training regimes. Existing open-source frameworks and shared-task infrastructure can provide a starting point.

A good fit if you enjoy: machine learning, neural network architectures, learning theory, cognitive science, or building AI systems under practical resource constraints.

3 · Developmental Interpretability — Monitoring Capabilities as They Emerge

Research area: Mechanistic interpretability, AI safety, training dynamics, capability evaluation.

Most language model evaluations examine a model after training, and most interpretability studies investigate the internal mechanisms of an already-trained system. Both approaches can miss an important question: what happens internally as a model acquires a new capability, and can we detect that development reliably?

A model's capabilities do not necessarily emerge all at once. Training may create internal representations that later support particular behaviours, or produce bottlenecks that limit what subsequent post-training can achieve. Understanding these dynamics could help us identify which capabilities are developing, when existing evaluations become informative, and whether apparent signs of progress or risk correspond to meaningful changes in the model. Rather than treating an internal component as inherently meaningful, we can test whether it predicts behaviour, whether interventions establish a causal relationship, and whether findings replicate across checkpoints, random seeds, or model families.

Possible research directions.

• Monitoring capability emergence: can internal representations or activation patterns reveal that a model is acquiring a capability before standard behavioural evaluations reliably detect it?
• Replicating interpretability findings: do proposed circuits, neurons, or linear representations persist across random seeds, training checkpoints, and model architectures?
• Pretraining and post-training: which capabilities are constrained by representations learned during pretraining, and which can be developed through subsequent optimisation?
• Reliable evaluation and AI safety: when do interpretability metrics provide useful evidence about a model's behaviour, and when can they give false reassurance?
• Cultural and social capabilities: how do models acquire culturally specific knowledge, and can we identify the training conditions or internal representations associated with reliable performance across different cultural contexts?

These questions are particularly relevant to AI safety because an evaluation that works only at the end of training may provide little warning about how a system is developing. Equally, an internal signal that looks promising but does not generalise or predict behaviour can be misleading.

What you could do: analyse model checkpoints, design behavioural and internal evaluations, perform activation interventions, or replicate existing findings under controlled conditions. Projects can range from focused empirical studies to more ambitious investigations of how capabilities develop during training.

A good fit if you enjoy: mechanistic interpretability, AI safety, experimental machine learning, causal inference, or the scientific study of neural networks.

Related research

• Chen et al. (2026), The Coverage Principle: How Pre-Training Enables Post-Training.
• Zou, Salhan & Buttery (2026), Development of Linear Truth Encodings in Language Models: A Replication Study.
• Polyakova-Reed et al. (2026), Claude Reinvestigates Herself: A Replication of Different Types of Syntactic Agreement Recruit the Same Units in LLMs.

The latter two projects form part of our work on reproducibility and reliability in interpretability research.

4 · Beetle — Computational Models of Bilingual and Multilingual Language Processing

Research area: Computational psycholinguistics, multilingual language models, language acquisition, cognitive modelling.

Most people in the world use more than one language, but computational models of human language processing have historically focused heavily on monolingual speakers. This creates a scientific gap: how can we understand the effects of language experience when our computational models do not adequately represent the diversity of that experience?

I developed Beetle, a framework and suite of controlled bilingual language models designed to investigate second-language processing. By controlling which languages a model encounters, when it encounters them, and how much exposure it receives, we can test hypotheses about human bilingualism in a systematic way. This makes it possible to investigate not only whether a model can process multiple languages, but also how the history of its language experience shapes its behaviour.

Possible research directions.

• Code-switching: how does the balance and ordering of language exposure affect a model's ability to switch between languages within and across sentences?
• Language attrition: when a model receives substantially more exposure to a second language, under what conditions does its performance in its first language deteriorate?
• Heritage bilingualism: can controlled training regimes capture aspects of learning a minority language at home while becoming increasingly dominant in a majority language?
• Human-model alignment: do model predictions and internal representations align with human reading times, behavioural experiments, or neural measurements across linguistic tasks?
• Developmental trajectories: do two models with similar final performance develop different internal representations because they encountered their languages in different orders or proportions?
• Language production: how does prior language experience affect syntactic choices, lexical selection, and cross-linguistic influence in model-generated language?

A key advantage of this framework is experimental control. Rather than comparing unrelated models trained on different datasets, we can construct matched populations that differ in a specific aspect of their learning history and test the consequences.

What you could do: train or analyse controlled bilingual models, design computational psycholinguistic experiments, compare model predictions with human behavioural data, or investigate how multilingual training influences representations and language production.

A good fit if you enjoy: language, linguistics, cognitive science, multilingual NLP, experimental psychology, or using computational models to test theories of human learning.

Selected publications

• Salhan, Arnett, Michaelov & Buttery (2026), Beetle: A Bilingual Model Suite for Modelling Second-Language Processing.
• Kubiak, Salhan, Buttery & Moscoso del Prado Martín (2026), Distinct Patterns of Brain Alignment of Language Models across Linguistic Domains.
• Barbenel et al. (2026), L1 Influence in L2 Language Models: A Human-Centric Approach.