City officials responded to the audit by proposing a reassuring phrase: every algorithmic recommendation would be reviewed by a human.
Elias initially welcomed the safeguard.
Priya asked what the human reviewer would actually do.
If staff merely clicked 'approve' on nearly every recommendation, human oversight would be ceremonial rather than substantive.
Research from the pilot schools already showed signs of automation bias.
Counsellors were more likely to agree with the system when a numerical score appeared on screen, even when narrative notes suggested a different interpretation.
Some reviewers treated the model as a second opinion.
Others treated it as an authority whose disagreement required justification.
The distinction mattered because nominal human control can coexist with practical machine dominance.
The team redesigned the interface so reviewers saw key student information before seeing the model's score.
They were asked to form an initial judgment and record their reasoning independently.
Only then did the interface reveal the algorithmic recommendation.
If the two judgments differed, the reviewer had to examine the discrepancy rather than automatically choose one.
Priya also insisted that reviewers be allowed to override the model without bureaucratic punishment.
An override mechanism is meaningless if staff fear that disagreement will lower performance evaluations.
At the same time, unrestricted overrides could reintroduce human bias the model was partly intended to reduce.
The team therefore monitored override patterns for systematic disparities.
Human oversight was not a magical solution to algorithmic error.
It created a second decision layer with its own biases, incentives, fatigue, and institutional pressures.
Elias began describing the system as a socio-technical process rather than a model.
The final outcome emerged from data collection, prediction, interface design, human interpretation, program capacity, and appeal procedures.
Responsibility was distributed across that chain.
The phrase 'human in the loop' became useful only after the team defined what authority, information, and accountability the human actually possessed.
Human oversight therefore had to be designed against deference as carefully as the algorithm had been designed against statistical error.
The reviewer was neither an ornamental safeguard nor an omniscient corrective but another fallible component whose incentives and cognitive biases required explicit governance.
Without those details, the phrase was merely a comforting abstraction.