Before redeployment, Priya created a red team whose task was to make the redesigned system fail.
Several developers disliked the adversarial framing.
They had spent months improving the system and found it psychologically difficult to reward colleagues for discovering weaknesses.
Priya argued that defensive confidence without structured challenge becomes complacency.
The red team included security specialists, school staff, students, statisticians, and people who had not participated in the original development.
They tested ordinary software vulnerabilities, but they also searched for conceptual and procedural failures.
Could a malicious employee infer sensitive information from repeated queries?
Could a school manipulate inputs to increase access to scarce tutoring resources?
Could missing data systematically disadvantage students from particular backgrounds?
Could a reviewer bypass the appeal process by recording an unofficial judgment elsewhere?
Could an explanation be technically correct and still predictably misunderstood?
One student tester discovered that entering an implausibly short commute caused the system to treat an incomplete address as reliable data.
A statistician found that one subgroup's calibration deteriorated sharply when records were sparse.
A counsellor showed that staff could unintentionally create a feedback loop by documenting only cases that the model had already highlighted.
None of these failures appeared in the headline accuracy metric.
The red team also performed counterfactual tests, changing selected attributes while holding others constant to see whether predictions behaved unexpectedly.
These tests did not prove discrimination, but they identified cases requiring deeper investigation.
Security specialists simulated attempts to extract information and overwhelm logging systems.
The team documented not only successful attacks but near misses and ambiguous behaviours.
Elias began to appreciate the cultural importance of institutionalized criticism.
A system is safer when finding a flaw is treated as contribution rather than betrayal.
Priya required every high-severity finding to have an owner, mitigation plan, deadline, and verification test.
Several issues delayed deployment.
City officials complained about the schedule.
Priya replied that a deadline is not evidence that a system is ready.
Red teaming institutionalized adversarial inquiry, converting criticism from a reputational threat into a formal mechanism of epistemic resilience.
The practice recognized that failure modes are often sociotechnical: they emerge from interactions among software, incentives, users, procedures, and institutional blind spots.
The red team transformed skepticism from an external threat into part of the development process itself.