Looking to Learn: Token-wise Dynamic Gating for Low-Resource Vision-Language Modelling

A child learns words while looking at the world. This paper asks how a small vision-language model should combine what it sees with what it reads when both the text and the compute are scarce.

This work was led by Bianca-Mihaela Ganescu, an MPhil student I co-supervised, and it grew out of a simple question: under cognitively plausible data constraints, how should a model decide when to trust its eyes and when to trust its words?1Ganescu, Salhan, Caines & Buttery. Looking to Learn: Token-wise Dynamic Gating for Low-Resource Vision-Language Modelling, BabyLM Workshop, EMNLP 2025. Most vision-language systems are trained on far more image-text data than a child ever encounters. The multimodal track of the BabyLM Shared Task turns that assumption around and asks what can be learned from a developmentally realistic budget.

Token-wise dynamic gating

The core proposal is a lightweight decoder-based architecture built around token-wise dynamic gating. Instead of fusing linguistic and visual signals in a fixed way, the model learns a gate that decides, token by token, how much visual information to let in. Two further ingredients support this: feature modulation, which shapes how the visual features are injected, and a contrastive objective that keeps the learned representations aligned. The design stays small on purpose, since the whole point is to work well when resources are limited.

What I find most appealing is not the raw scores but what the gate learns on its own. Without any explicit supervision telling it to do so, the model comes to favour visual information for content words, the nouns and other tokens that name things in a scene, while leaning on linguistic context for function words, the glue of grammar that the image cannot help with. That the split falls out of training is a small piece of evidence that the architecture is capturing something sensible about how vision and language ought to combine.

Evaluation and honest limits

The model was evaluated across five benchmarks: BLiMP, the BLiMP Supplement, EWoK, Winoground, and VQA.2These span grammatical knowledge, world knowledge, compositional image-text matching, and visual question answering. See the ACL Anthology entry. Across this spread the approach reached competitive or superior performance relative to multimodal baselines, which is a meaningful result for an architecture deliberately kept lightweight.

The paper is also candid about where it strains. Working from global image embeddings, a single vector for a whole image, limits how finely the model can ground individual words, and some of the dataset splits introduced training instability. Even so, the dynamic gating mechanism kept working under those constraints, which suggests the idea is robust rather than a lucky artefact of one clean setting.3First BabyLM Workshop, EMNLP 2025, Suzhou, China. The paper received an Outstanding Paper Award.

For me the takeaway is that adaptivity is worth more than scale when the data is small. Letting a model decide, at each token, how much to look versus how much to read gives it a cheap and interpretable way to make the most of limited grounding, and it is a direction I would like to keep exploring with finer-grained visual features.