Promises a Machine Has to Keep

Abstract

Human error is stochastic and local. A hundred agents making judgment calls produce a hundred slightly different outcomes, most acceptable, a few bad, and the bad ones arrive one at a time — which is not just a smaller problem but a differently shaped one. A single bad outcome is a complaint. It gets handled, sometimes generously, and it produces at most a story. The distribution of human error is also, importantly, illegible to customers: nobody can compare their refund to the refund the person before them got, so no one perceives a pattern, because there usually isn't one. Machine error is correlated. It is the same error, in the same direction, applied to everyone who hit the same input pattern, at the same instant, and it does not stop until someone notices and ships. This changes three things at once. The magnitude arrives before the detection does — you can be forty thousand wrong decisions deep before the first ticket escalates.