The audit began with a question that sounded elementary and proved difficult: what does it mean to say that a machine-learning model has learned something?
Elias explained that CivicLearn had been trained on historical records containing thousands of examples.
For each student, the system received a set of features and an outcome indicating whether the student later required intensive support.
During training, the model adjusted internal parameters to reduce prediction error across the dataset.
Priya interrupted to distinguish statistical association from understanding.
The model did not know what a student, school, family, or hardship was in any human sense.
It detected regularities in encoded data and used those regularities to estimate future outcomes.
This distinction mattered because persuasive language can smuggle anthropomorphic assumptions into technical discussion.
Saying that a model 'understands risk' may encourage decision-makers to trust it as if it possessed contextual judgment.
In reality, the model optimized a mathematical objective defined by the development team.
Elias showed the loss function used during training and explained how optimization rewarded predictions that matched historical labels.
Priya asked a more uncomfortable question: who decided that the labels represented the right thing to predict?
The original team had defined 'need for intensive support' using administrative records of students who had later entered intervention programs.
That sounded objective until they examined how students entered those programs.
Some schools had more counsellors and therefore referred more students.
Others lacked staff and recorded fewer interventions even when students faced comparable difficulties.
The label was therefore partly a measure of student need and partly a measure of institutional capacity to notice and respond.
Training on that label taught the model to reproduce historical referral patterns as though they were neutral truth.
Elias recognized a form of circularity.
The city had asked the algorithm to discover hidden need, yet the algorithm learned from records generated by the city's unequal ability to detect need in the past.
Priya called this a measurement problem rather than a software bug.
Changing code alone would not solve it.
The discussion exposed an ontological ambiguity between the construct of 'need' and the administrative traces used to represent it, a distinction often blurred when databases acquire the authority of reality.
Historical labels thus became a form of institutional testimony rather than neutral observation, carrying the biases, omissions, and capacities of the systems that produced them.
The team would need to reconsider the construct they intended to measure, the evidence available for approximating it, and the uncertainty that remained.