The Human in the Loop: Human–AI Collaboration in Language Assessment

A half-day Cambridge symposium on what it means to keep people meaningfully involved as large language models enter language testing.

As large language models are adopted in language testing, they arrive at a point where the stakes are personal: a score can decide a visa, a university place, or a job. "The Human in the Loop" was a half-day symposium held on 21 May at the University of Cambridge, hosted at the Computer Laboratory and run in hybrid format,1Event site. asking how AI is changing assessment practice and what it means to keep humans meaningfully in the loop rather than nominally on top of it.

Aims

The organisers framed the day around reliability, fairness, transparency, and impact on diverse language users, with the connected questions of accountability and equity when automated systems make or shape decisions.2Cambridge Language Sciences report. These are not abstract worries in assessment. A model that is slightly less reliable for one group of test-takers, or whose reasoning cannot be inspected, translates directly into unfair outcomes, so the symposium treated them as questions to be answered rather than principles to be listed.

Format and speakers

The programme brought together four invited speakers from complementary fields, spanning language assessment, applied linguistics, psychometrics, and the assessment industry, alongside a moderated panel, open cross-disciplinary discussion, and a poster session of sixteen accepted posters. The invited speakers were Dr Mark Brenchley of Cambridge University Press & Assessment, Dr Sha Liu of the British Council, Dr Chanjin Zheng of East China Normal University, and Dr Carla Pastorino of Cambridge University Press & Assessment. The line-up deliberately put academic measurement expertise next to people who build and run tests at scale, which is where a lot of the sharper disagreements sit. The day also included an interactive 3D demonstration developed by the Cambridge University EdTech Society, and it drew 219 participants from more than thirty countries in person and online.

Organisers and my involvement

The symposium was organised by Hongyi Yang of the Faculty of Education and Gabrielle Gaudeau of the Department of Computer Science and Technology, supported by Dr Andrew Caines. It was funded by the Cambridge Language Sciences Workshop Fund and by ai@cam's EQUILL project, on improving language equity and inclusion through AI. I took part in the symposium and helped around the event as part of the wider Cambridge language and assessment community; the credit for pulling it together belongs to Hongyi, Gabrielle, and Andrew. What made it worth attending was the refusal to treat "human in the loop" as a slogan. The recurring question across the talks was concrete: which human, doing what, with what information, and with what power to overrule the model? That is the version of the problem that assessment providers actually have to answer, and it is a good one to keep in front of the wider field.