Knowing Whether It Worked

Abstract

There is a problem at the centre of measuring anything in a complex adaptive system, and it is worth stating before any advice is given. The people inside the system can see the measure. They adapt to it. And once they adapt to it, the measure no longer tells you what it told you before. Two social scientists stated this independently in the 1970s, and their formulations are still the clearest. The psychologist Donald Campbell, in a 1979 paper in Evaluation and Program Planning titled "Assessing the Impact of Planned Social Change", put it as a law of social indicators: the more any quantitative indicator is used for social decision-making, the more subject it will be to corruption pressures, and the more apt it will be to distort and corrupt the social processes it was meant to monitor. The economist Charles Goodhart, writing about monetary policy at the Bank of England a few years earlier, made a narrower observation that has since been generalised under his name: that a statistical regularity tends to collapse once pressure is placed on it for control purposes.