When ROI Hits the Roof


The Coalition for Evidence-Based Policy has a nice compilation of low-cost program evaluations. Example #2 tells us about a $75 million education program that improved nothing. The cost of finding this out was $50K. The simple math says that returns on money invested in evaluation reached 150,000%. It kinda outperforms S&P500.

What’s the trick? First, as mentioned before, maybe the same program completes something else. The program had aimed at improving student results and attendance, and it didn’t improve them. But the teachers got $3K more each and bought themselves useful things. Nothing wrong with that, but we need other ideas to improve education.

Second, so-called unconditional money transfers rarely motivate better performance, though it may seem counterintuitive. Not only in education. Public services just happened to be in full view of everybody. Then ROI in evaluation depends on how much the government or business puts into unchecked programs. This time it was $75 million, next time it’s $750 million. Big policies promise big returns, either due to better selection or faster rejection.

Third, such opportunities exist because big organizations evaluate execution, not impact. Execution is easier to monitor, so public corporations have to have independent auditors who ensure that employees don’t steal. In contrast, efficiency audit requires management’s genuine interest in rigorous evaluation, but there’s no incentives for that. After all, stealing is everywhere a crime, while incompetence is not (despite incompetence being more wasteful).

With that said, ROI of 150,000% is a fact. If you spend on a policy doing X and the policy does nothing to X, you can just leave $75M on the table. Without that $50K evaluation, you’d lose them.