Lant Pritchett asks development economists to fill in a table for impact evaluations (see questions in the header):
Lant has a fair point: Do these popular evaluations measure the drivers of development seen before in now-developed countries. After all, shouldn’t we try the same old drivers in developing countries? Hmmm, many items in the list pass the smell test. Should we doubt the rest? Should, but maybe for other reasons.
Randomized evaluations usually seek to overcome references to experience that Lant’s questions imply. Developed countries have an overwhelming fraction of policies that don’t work when evaluated (see Why It Matters section here). They can afford ineffective policies, but less fortunate countries can’t afford copying them in that.
Nothing is wrong with taking these ideas and testing them in Africa. That happens, as with school meals or conditional transfers (food stamps and government-supported student loans are conditional transfers in the US, for instance). And sure, programs get studied, adapted, adjusted on the way.
The general problem is: If developing countries were like Europe 1870 or Japan 1950, they would grow without much respected economics troops from Boston. Europe had strong states controlling their territories and population in the 19th century. Governments raised taxes when necessary (usually, for wars), workers had strong organizations, inventors created cutting-edge technologies for the world markets. An African may have the same wage that an English worker got in 1820, but they live in different world economies. Plus the development gap is many times wider for the modern English worker.
If you take plain vanilla 19th-century European experience and plant it in Africa, you do what any tribe chief in Congo can do with Adam Smith’s book under his pillow. Naturally, plain vanilla doesn’t work. But if it doesn’t, then what’s wrong with current impact evaluations? Not the past experience—evaluators do check history.
Evaluations have to meet many constraints. Bigger programs, a-la government policies of Asian modernization, have many stakeholders, each with an opinion. Evaluators negotiate the intervention with each one and compromise on the scale. It’s not like you can start an Industrial Revolution in Congo. In this case, RCT designers prefer reliable knowledge about a feasible intervention—well, because that’s what makes a RCT designer. If you can’t negotiate proper design, the result is worth about the same as abundant historical data. Which is low, since that data doesn’t distinguish between impact and noise.