How and why we use rigorous evaluations—such as RCTs and diff-in-diffs.
At One Acre Fund, we utilize two major measurement strategies.
First (see Methodology), we routinely conduct roughly 10,000 physical harvest measurements per year, for a group of farmers enrolled in One Acre Fund vs. a comparable comparison group of non-One Acre Fund farmers from the same villages who are subject to the same agro-ecological conditions.
We statistically control for any observable differences between program and comparison farmers. This broad sample enables us to cover every country of operation, nearly every crop, and to understand the differential performance of our operating units within countries. For example, we might find that our impact on bean harvests in a particular district is weak, and therefore switch more energy into another crop for that particular district. This highly-actionable information is relatively simple to execute in the field. We utilize practical analytical tools like propensity score matching to even out any differences between our test and control groups. This broad measurement approach enables us to continually improve our impact.
Second, we periodically check for bias in these estimates, through highly-rigorous measurement methods: Randomized Control Trials (RCT), and difference-in-difference (diff-in-diff) estimates.
We utilize these higher-quality studies to check to make sure that our more routine measurement methods do not have significant bias in their estimates. These studies are substantially harder to execute, and by necessity we do them in one region of a country, and for one crop. These deep evaluations enable us to periodically prove our impact and ongoing measurement.
- In RCTs, we take a group of farmers who have signed up for our program, and randomly assign them to receive the One Acre Fund program (test), vs. a group that instead receives a small compensation package instead of an agriculture program (control). We then see what is the resulting impact on harvests in the test group. This is extremely difficult to execute, because it involves disappointing a group of farmers who have signed up for our program, and withholding what we feel is a beneficial program from them. RCTs are also more vulnerable to region-specific effects (like diseases and drought), which make them less generalizable to program-wide impact. As a result, we use RCTs sparingly.
- In diff-in-diffs, we measure a farmer’s harvest in a baseline Year 1, before joining One Acre Fund, and then we track them into the Year 2, after joining One Acre Fund. We always see a large gain in harvest. Of course, the major criticism of this is that Year 2 might have had better weather which caused a better harvest—not One Acre Fund’s program. Therefore, we also take a comparison group that does not receive One Acre Fund’s program, and measure their harvests in Year 1 vs. Year 2. Any difference here should reflect changes in climatic conditions, and we correct for that. The “difference-in-difference” shows the impact of One Acre Fund’s program. These diff-in-diffs are again hard to execute because they involve doing two consecutive harvest measurements for a panel of farmers that we have to track over multiple years, but are not as difficult to execute as RCTs.
Below is a table summarizing the key rigorous studies One Acre Fund has conducted. Overall, we believe that the body of evidence points to reasonable consistency between our routine measurements and highly-rigorous experimental designs we employ
We do want to caution against putting too much weight into these highly-limited studies. Especially in agriculture, One Acre Fund believes that any single-year, single-crop, single-location study may suffer from a wide variety of unanticipated environmental factors that reduce the general representativeness of results. For example, our 2017 RCT in Teso district of Kenya had an unexpected outbreak of Fall Armyworm —which would tend to reduce harvests and thus the impact of One Acre Fund’s intervention. Also we were surprised to find that more than half of the control group had previously participated in One Acre Fund’s program by “commuting” into neighboring areas, and therefore may have some lingering impact from prior years. For these reasons, we believe the study will likely produce lower results than typical.
Snapshot of One Acre Fund’s rigorous studies
Note: some studies focused on measuring yield improvement, and others income improvement.
|Study Name||Results: Study||Results: Internal M&E in same period||Results: Significance *|
|2017 RCT in Teso district, Kenya||
188 kg/acre yield improvement (weighted average of program and non-program maize land)
24% ($81) profit improvement (maize)**
180 kg/acre yield improvement (weighted average of program and non-program maize land)
27% ($117) profit improvement (maize)
|2014-15 diff-in-diff, Kenya-wide||445 kg/acre yield improvement (maize)||476 kg/acre yield improvement (maize)||p<.01|
|2014-15 diff-in-diff, Tanzania-wide||490 kg/acre yield improvement (maize)||478 kg/acre yield improvement (maize)||p<.01|
|2014-15 diff-in-diff, Burundi-wide||56 kg/acre yield improvement (beans)||49 kg/acre yield improvement (beans)||p<.01|
|2014 RCT in Busia district, Kenya||31% ($91) profit improvement (maize) ***||21% ($87) profit improvement (maize)||p<.01|
|2009 RCT in Chwele district, Kenya||40% ($30) profit improvement (maize) ****||100% ($120) profit improvement (maize)||P<.01|
Detail on rigorous evaluations
2017 RCT in Teso district, Kenya
- Studied maize and beans in 2017 long rains planting season in one district of Kenya. We worked with measurement firm 3ie to validate the experimental design, Intermedia Development Consultants randomly audited data collection, and academic Emilia Tjernstroem of University of Wisconsin-Madison will independently analyze the results.
- Due to lack of expansion territory, the study was launched in an area adjacent to where One Acre Fund services have been offered for several years; we discovered during the study that nearly two-thirds of ‘control’ farmers had previously ‘commuted’ into our program, an unexpectedly high amount. We therefore anticipated our primary sample would show lower impact than our internal M&E, which uses comparison farmers with no prior program exposure. The study also occurred during an outbreak of maize pest Fall Armyworm; in such an environment, we anticipated the absolute difference between One Acre Fund and control farmer profits to be lower.
- Highly statistically significant impacts on both harvests and yields, with One Acre Fund farmers harvesting an additional 180 kg of maize per acre and an extra 344 kg of maize per farmer.
- Highly statistically and economically significant profit impacts on maize ($81 and a 24% improvement) as well as intercropped maize/beans ($72 and a 20% improvement).
- The dollar impact was lower than what One Acre Fund found in the internal country-wide impact that same year, but the study was conducted in a lower yielding district. The authors were not able to conclude how much of this difference was due to inter-district variability vs. methodological differences.
- Former clients in the control group did retain some One Acre Fund practices and used more inputs that those who had never participated; however, this did not translate into a statistically detectable improvement in harvests or profits. This is likely because former clients still under-used inputs, possibly due to cash constraints. This finding supports repeated enrollment of program farmers as well as One Acre Fund efforts to influence policy on access to inputs more broadly.
- Studied maize in Kenya and Tanzania, and beans in Burundi, during primary 2014 and 2015 planting seasons in each country. (Rwanda excluded due to timing reasons).
- We compared baseline harvest of One Acre Fund farmers who enrolled in 2014 (before receiving any program impact), against the same farmers’ harvest one year later (after receiving program intervention). In order to cancel out “year trend,” we took a comparison group and compared their 2014-2015 harvests. Impact was measured as the difference between the change in each group’s yields from 2014 to 2015.
- Per the summary table above, study found both: (1) highly statistically significant impacts (p<.01) of the One Acre Fund program on yields in all three countries; and (2) high consistency with our routine internal M&E over the same period.
See here for more detail on our 2014-2015 diff-in-diff, as well as findings from our 2015-2016 diff-in-diff conducted in Kenya
2014 RCT in Busia district, Kenya
- Studied maize yields in 2014 long rains planting season in one district of Kenya. Evaluation firm ID Insight independently analyzed the data, and independently reported on the findings.
- Using RCT design, the study enrolled 6 “sites” of farmers in Busia district, then randomly dis-enrolled 2 “sites” (control group). The study found results (32% yield improvement, which One Acre Fund converted to a 31% or $91 profit improvement) that were consistent (actually slightly higher) than our routine internal M&E that year Kenya-wide, per the summary table above. Significance level of the yield improvement was p = 0.01, but after adjusting for the low number of randomization units using wild cluster bootstrap, significance level declined to p = 0.09. A key learning was that even with 1,200 individual farmers studied, choosing to randomize at the site level severely under-powered the study, a learning which influenced our 2017 RCT.
- One Acre Fund continued the study into 2015, finding virtually identical average yields for the farmers in the 4 program sites from the prior year (1,031 kg/acre) and those in the 2 control sites from the prior year who were now allowed to enroll in 2015 (1,032 kg/acre). While this finding cannot completely overcome the limits of an underpowered RCT, it lends support to the idea that the observed harvest effects in 2014 were due to our program and not the results of location-specific effects in the 6 areas under study.
2009 RCT in Chwele district, Kenya
- A group of independent researchers studied maize yields in 2009 long rains planting season in one district of Kenya.
- Given this was our third year of operation, the major goals were to determine the effectiveness of our internal M&E and to generate learnings to improve our program.
- Utilizing an RCT design, the study found a 40% (or $30) maize profit improvement from the One Acre Fund program. Independent from the researchers, One Acre Fund believes that some of the low dollar impact is accounted for by unique 2009 conditions, including drought, low maize prices, and farmers changing practice on their ‘control plots’ as a result of the specific approach to randomization that was utilized. Nevertheless, according to our independent analysis, we found the harvest results have a p=0.001 significance level. These results were substantially different from internal M&E results in 2009, which showed a 100% (or $120) maize profit improvement Kenya-wide, leading to several important changes to our internal M&E methodology, including control selection and farming costs captured.
- The RCT also showed some farmers had a negative return on investment from farm inputs and some evidence that One Acre Fund was selecting better-off (though still poor) farmers—both of which were cause for concern. As a result of the study, in ensuing years we diversified our programs into other crops and energy products and created an agriculture innovation team to improve impacts, increase customer screening practices and non-customer studies, and launch our crop insurance program.
Other evaluations (not shown in summary table)
- Solar lamp studies (2011-2013): we are now one of the top retailers of solar lamps in Africa. Before launching these products, we conducted RCTs in 2011 in Kenya and 2013 in Rwanda to estimate the level of cost savings from purchasing a solar lamp. We estimate that by purchasing a solar lamp, the average household saves roughly $0.50 per week on saved kerosene, battery, and mobile phone charging expenditures. Over three years, the NPV (net present value) of those savings is roughly $66, compared to a purchase price of $15-$40, depending on the lamp model. We have found that our internal studies on solar lamp impact are more rigorously designed than most published research on this topic; the summary findings of our study are available in our full paper on RCTs.
At One Acre Fund, we firmly believe measurement is at its best when it drives continuous improvement in the programs we offer and the impact we generate for farm families. Our routine, internal M&E enables an incredible breadth of measurement—across all countries, crop families, and over time, helping us to improve our impact. At the same time, this breadth comes with some sacrifice to rigor because we cannot randomly assign our program to farmers at that scope and completely eliminate selection bias from our estimates. Our periodic evaluations bring more rigorous measurement (RCTs, diff-in-diff studies), helping us to prove our impact, but at some sacrifice to representativeness. As a result, we have used them primarily to test and demonstrate the accuracy of our routine, internal M&E. We maintain a continued commitment—to donors, policymakers, practitioners, staff, and most importantly, our clients—to continually getting better in how we measure our results.