Six months into the audit, a transport strike disrupted bus routes across the city.
Attendance patterns changed within days.
Students who had previously travelled reliably began missing classes, while schools near rail lines experienced smaller effects.
CivicLearn's predictions deteriorated.
The model had been trained on a world in which transport behaviour followed older patterns.
That world had changed.
Priya described the problem as distribution shift: the statistical relationship between past data and present cases was no longer stable.
Machine-learning systems often appear objective because their parameters are fixed, but fixed parameters can become inappropriate when environments evolve.
Elias examined calibration curves and found that a score previously associated with a thirty percent intervention rate now corresponded to a much lower rate in some schools and a higher rate in others.
The model's probabilities had lost consistency.
A second shift appeared after the city introduced free home internet for low-income families.
Internet instability, once a powerful socioeconomic proxy, became less informative almost overnight.
This was a policy success and a model failure at the same time.
The better society became at changing harmful conditions, the less reliable historical correlations could become.
Elias found the paradox encouraging.
A model should not expect inequality to remain stable merely because stability makes prediction easier.
The team established drift monitoring to detect changes in feature distributions, error rates, calibration, and subgroup performance.
They also defined thresholds that would trigger review or temporary suspension.
Priya insisted that retraining should not be automatic.
New data might reflect a temporary disruption, a beneficial policy change, or a new bias.
Blindly retraining could absorb the latest problem faster rather than correct it.
Model maintenance therefore became an interpretive practice, not merely a scheduled computational task.
The lesson was uncomfortable for officials who wanted a system that could be purchased, deployed, and forgotten.
Distribution shift demonstrated the temporality of model validity: performance is always indexed to an environment, population, and institutional regime that may cease to exist.
A successful policy can therefore invalidate yesterday's model, making obsolescence evidence of social improvement rather than technical incompetence.
Any model operating in a changing society would require governance as dynamic as the environment it attempted to describe.