10 ideas
The same person judging the same case can reach different conclusions depending on transient factors like mood, hunger, weather, or the order of prior cases. This within-person variability means a single judge is not a stable instrument, and judgments partly reflect the moment rather than the matter.
To measure hidden disagreement, have multiple professionals independently judge the same set of identical cases and compute the variability of their verdicts. Organizations consistently discover far more scatter than insiders expect, exposing costly inconsistency in decisions presumed to be uniform.
Reduce noise by breaking complex judgments into independent component assessments scored separately, delaying holistic intuition until after the parts are evaluated, and having judges form views independently before discussion. Like handwashing, these practices prevent unseen contamination without anyone knowing which specific error they stopped.
Judge Marvin Frankel documented that defendants with identical offenses received wildly different sentences depending on which judge they drew, with variations of years for the same crime. This disparity motivated U.S. sentencing guidelines and illustrates noise as an injustice, not merely a statistical curiosity.
Executives at an insurance company expected their underwriters' premium quotes for identical cases to differ by about 10%. When the same realistic cases went to many underwriters, the median difference between two quotes was about 55%. Nobody had noticed this hidden variability because no one normally compares independent judgments of the same case.
System noise is the variability among professionals judging the same cases. Part of it is level noise: some judges are consistently harsher or more lenient than others. Part of it is pattern noise: each judge responds idiosyncratically to particular features of particular cases, and this includes occasion noise, where the same person judges differently depending on mood, fatigue, weather or order effects.
Mean squared error breaks down into bias squared plus noise squared, so reducing the scatter of judgments cuts total error exactly as much as removing an equal amount of systematic bias. Because the formula penalizes both components the same way, you can improve accuracy by reducing noise without knowing the true answer or the direction of any bias.
Mechanical models, and even equal-weight rules or a model of the judge's own past decisions, usually predict better than the professionals themselves. They win mainly because they are noise-free: they apply the same weights every time, while humans add random inconsistency that looks like subtle insight. Human judgment beats models less often than experts believe, partly because the world is objectively hard to predict.
Break a complex decision into a small set of predefined dimensions. Have each dimension assessed independently, using fact-based evidence and ideally relative scales. Form the overall intuitive judgment only at the end, after all assessments are in, so that an early impression cannot shape how later evidence is read.
Averaging several independent judgments reduces noise in a predictable way, because each person's random errors partly cancel. Methods like estimate-talk-estimate, and averaging your own two separated guesses ('the crowd within'), keep this cancellation effect.