Pico: A Modular Framework for Hypothesis-Driven Small Language Model Research

A lightweight sandbox and a matched suite of baseline models, built so that a single change to a small model's architecture or training can be isolated and its effect observed.

Most of what we know about scaling comes from large models, where sheer size covers for a lot of design uncertainty. Small and medium-sized models do not have that luxury. Under tight parameter budgets, a choice about tokenisation, an optimiser setting, or a change to the attention block can decide whether a model works at all, and there has been no easy way to test those choices one at a time. This is the gap Pico sets out to close.1Diehl Martinez, Africa, Weiss, Salhan, Daniels, Buttery. Pico: A Modular Framework for Hypothesis-Driven Small Language Model Research, EMNLP 2025 System Demonstrations.

Why a sandbox

The framing in the paper is that developing small models is still closer to an experimental craft than a science. Researchers have intuitions about what should help, but few controlled settings in which to check them. Pico is described as a lightweight, modular framework that makes hypothesis-driven research on small and medium-scale language models practical, by letting you make a targeted change to a model's architecture or training procedure and then watch how its behaviour shifts.

The design splits into two libraries. One provides the sandbox itself, where a modification can be dropped in without rebuilding the whole training stack around it. The other supplies pico-decoder, a suite of baseline decoder models trained under standardised conditions, so that any new result has a fixed point of comparison rather than an ad-hoc one.

Baselines you can trust

The value of the baseline suite is that it removes a common source of noise. When two research groups report different outcomes for what sounds like the same idea, the difference often lives in the training setup rather than the idea. By fixing those conditions and releasing the models trained under them, Pico gives a shared reference so that a reported effect can be attributed to the change being studied.

The paper works through case studies showing how this supports iterative development: make a controlled architectural or procedural change, measure against the standardised baselines, and use the outcome to decide the next step. The emphasis throughout is on reproducibility and on keeping the unit of experimentation small enough that its effect is legible.

Where it sits

For me the appeal is that it treats small-model research as something you can run experiments on properly, rather than a sequence of one-off training runs whose results are hard to compare. The project is maintained with documentation and released models on its own site.2Project site and released models: picolm.io. The claim the paper makes is modest and useful: a controlled setting plus matched baselines turns vague intuitions about small models into questions you can actually test.