Suchir Who Gets to Govern the AI Systems That Assess Us?
We are increasingly expected to trust AI systems that students, teachers and parents cannot inspect. Education needs a different approach.
Artificial intelligence is becoming part of how we learn, teach and assess knowledge. Yet the systems increasingly responsible for these decisions are often developed by organisations whose training data, internal representations and development processes remain inaccessible.
We are being asked to trust systems that we cannot independently inspect, reproduce or meaningfully adapt.
For education, this presents a particularly important problem. Assessment depends on fairness, validity and public confidence. These properties cannot simply be assumed because an AI model performs well on a general-purpose benchmark.
If AI is going to shape how we assess knowledge, who gets to decide whether it is fair? And who gets to check?
I think we need to take a different approach to building AI for education: one based on technological sovereignty, open scientific infrastructure and the ability to independently evaluate how models learn.
From buying AI to governing AI
Educational institutions are increasingly dependent on a relatively small number of companies developing frontier AI systems. They can purchase access to models, integrate APIs and evaluate their outputs. But they often have limited control over how those systems were trained, what data shaped their representations or how their behaviour changes between versions.
This is more than a question of transparency. It is a question of institutional agency.
Technological sovereignty means having the technical capacity to understand, control, adapt and independently evaluate the AI systems an organisation relies upon.
That doesn't mean every university or assessment provider needs to train a foundation model from scratch. It means having meaningful alternatives to depending entirely on external providers.
Open-weight models, training configurations, evaluation datasets and intermediate checkpoints provide part of this foundation. They allow researchers and institutions to investigate systems rather than simply consume their outputs.
Fairness needs to become experimentally measurable
This is particularly important when we consider linguistic diversity.
A model might achieve similar aggregate performance across two populations while exhibiting very different weaknesses in how it processes their languages. Conventional benchmarking can miss these differences.
My research explores how controlled bilingual language models can help us investigate this problem.
In our work on open bilingual model suites, we study how different language-learning experiences shape the capabilities and representations that models develop. By controlling language exposure, training conditions and developmental trajectories, we can investigate how linguistic backgrounds influence subsequent language processing.
This creates an opportunity to move beyond asking whether a model can generate convincing learner-like text. We can instead investigate whether it develops capabilities that meaningfully reflect different language-learning experiences.
For educational assessment, this matters. We need to establish whether AI systems work across different linguistic backgrounds, not simply assume that they do.
Open models make these questions experimentally tractable. Controlled training makes them reproducible.
We need to understand how AI systems develop
There is another important distinction between accessing a finished model and studying how it came to behave as it does.
Most organisations interact with AI as a finished product. They observe its inputs and outputs, perhaps consult external evaluations, and decide whether its performance is sufficient.
But a model's capabilities emerge through a developmental process. Its training data, architecture and optimisation shape the representations it acquires and the limitations it exhibits.
Access to intermediate checkpoints makes it possible to investigate these processes directly.
We can examine when particular capabilities emerge, how different training experiences influence representations, and whether fine-tuning or adaptation introduces new weaknesses.
I think this offers an important opportunity to connect developmental AI research with practical model evaluation. Rather than asking only whether a system works, we can begin asking why it works, how robust its capabilities are and how they change.
This is especially relevant when models are used to evaluate human knowledge and ability.
Education should help shape the future of AI
I see three research ambitions coming together here:
- Sovereign AI: giving educational institutions meaningful control over the systems they depend upon.
- Developmental AI: understanding how training experiences shape the capabilities and representations of models.
- Equitable AI: investigating whether systems work reliably across different languages, populations and educational backgrounds.
None of these ambitions is achieved simply by releasing model weights. Openness does not automatically guarantee fairness, reproducibility or safety.
We also need controlled experiments, independent evaluation and meaningful participation from the institutions and communities affected by these systems.
This is why I believe educational assessment is such an important setting for fundamental AI research. It forces us to confront questions that extend beyond model performance: whose knowledge is represented, what capabilities are being evaluated, and who has the authority and technical expertise to challenge the results?
The organisations responsible for assessing knowledge should not have to surrender control over the intelligence used to evaluate it.
We should be building AI systems that educational institutions can understand, adapt and independently scrutinise, not simply systems they are expected to trust.
Slides from the seminar
This essay grew out of an invited seminar I gave at The Triangle, Cambridge University Press & Assessment (CUP&A), in November 2026: Open Bilingual Model Suite for Modelling Second-Language Processing: Building Open “Sovereign” AI Solutions for Equitable Assessment and Simulation.