When a scoring system's verdict sets what happens to a person, it creates the very outcome it predicted and then counts that outcome as proof. Someone labelled high risk gets a longer prison sentence, comes out with a record and fewer job prospects, and reoffends, and the model logs a success. The model does not measure the cycle. It drives the cycle.
When the real quality can't be measured, models substitute something easier to count, such as a zip code for whether you'll repay a loan. The proxy can outweigh direct evidence. Drivers with clean records but poor credit paid more for car insurance than drivers with excellent credit and a drunk-driving conviction.
A model that never checks its judgments against what really happened cannot find out it was wrong. When teachers flagged as failures are fired, nobody tracks whether they were good teachers, so the label simply becomes the truth. This makes the system's errors permanent.
A scoring model becomes truly harmful when three things combine. The people it judges can't see how it works or challenge it, it is applied to huge numbers of people, and its verdicts hurt them. Scale turns a hidden bias in one formula into the same unfair judgment repeated across millions of lives.
Judging an individual from a handful of data points produces numbers that mostly reflect chance. Rating a teacher on the test scores of 25 or 30 students is statistically unsound, because too many unrelated factors can move the results. Yet those noisy scores still decide careers.