Getting the Counterfactual Right

Good impact measurement is absolutely critical for organizations driven by social missions. Measurement enables organizations to understand their effectiveness, improve on their impact, and actually achieve their mission.

If impact studies are not designed well, program staff will have a false sense of their success and a poor understanding of where impact needs to improve. One area where measurement is often weak is selection bias – the problem of comparing individuals who have self-selected into a program with those who have not, and who might therefore be fundamentally different.

While One Acre Fund had historically assessed our impact quite rigorously – weighing harvest and using a comparison group – we were still concerned that selection bias could be affecting our impact assessments. This memo outlines our experience addressing selection bias through experimenting with four different approaches