Targeted use of Randomized Controlled Trials (RCTs)
What is an RCT?
The gold standard for determining the effectiveness of a program is a randomized controlled trial. People sign up for a program, and then are randomly assigned either to receive a program (treated or test), or to not receive the treatment (control). The results are then compared. This is the best known way to address “selection bias,” where those who sign up for the program might be systematically different from those they are measured against. One Acre Fund believes in a world where programs are measured for effectiveness using high-quality methodologies like RCTs when possible.
Why it is difficult?
RCTs can be expensive, time-consuming, and operationally complex. Once we have marketed our program to farmers and they have signed up to participate, it can be difficult to randomly exclude some that we would have been able to serve. Because RCTs are expensive and difficult to execute, the results are limited to a small area for a small number of crops.
One Acre Fund's internal M&E strategy
One Acre Fund does not regularly employ RCTs, but rather uses a high-quality quasi-experimental design to test our impact in a broad variety of contexts, for roughly 5,000 test (aka treated) and control farmers per year. This routine measurement activity allows us to measure across all our countries, across regions within countries, and to compare ten different crops. This provides rich, easy-to-use operational data to refine our program. Crucially, we pay a lot of attention to control group selection, and constantly experiment with methods such as propensity score matching, or finding farmers who are “enrolled but not yet treated” or “likely to enroll.” In 2015, we conducted some difference-in-difference studies looking at the change in harvests over time for farmers who entered our program compared to farmers who remained out of the program. This more rigorous design has generally confirmed the impact we typically find in our annual M&E assessment. Please click here to see the results of the study.
One Acre Fund’s increasing use of RCTs
We will increasingly use RCTs to confirm that our regular measurement methods produce correct indications of impact. Below, we summarize one RCT completed in 2009 with independent researchers and another 2014 RCT we conducted internally (with the analysis portion provided by an outside firm). Both studies were conducted in Western Kenya and examine impacts on maize production. Maize is relatively simple to measure (as opposed to many other crops), and in 2014, Kenya maize harvest accounted for almost half of our Kenya impact.
These two RCTs estimate a percent harvest impact that is reasonably consistent with our 2014 internal M&E findings for Western Kenya. However, there are important study limitations and also low statistical significance in the most recent study. Below is One Acre Fund’s interpretation of some of the core study results of interest.
|Estimated % Improvement in Maize Profit From One Acre Fund Program||Translated $ (USD) Impact on Farm Profit, From Maize Alone|
|2014 RCT||31% *||$91 ***|
|Internal 2014 M&E||21%||$87|
|2009 RCT||40% **||$30 ****|
|Internal 2009 M&E||100%||$120|
(*) Significant at the 0.09 level using wild-cluster bootstrap to adjust for low number of randomization units (we believe this to be the correct regression specification). Naive regression specification without adjustment is significant at the 0.01 level.
(**) Cost assumptions use One Acre Fund-obtained farm input prices. One Acre Fund calculated results to be significant at the 0.001 level.
(***) Baseline profit $ was slightly lower in the RCT study area, compared to our internal 2014 M&E, which is why dollar impact is lower than internal 2014 measurement despite higher percent profit improvement.
(****) One Acre Fund (distinct from the independent researchers) believes that 2009 exhibited simultaneous drought and lower maize prices compared to 2014, which caused baseline profits to be low, resulting in lower dollar impact. There was also a study limitation that may have affected $ impact. See below for more details.
We do not believe that either RCT provides definitive evidence of positive harvest impact – they both have important limitations. There is high variability in farmer results, particularly in the 2009 study, resulting in 11 percent of farmers having negative return to farm inputs (although this is lower amongst One Acre Fund-trained farmers). This information was concerning and led to important program changes. As indicated in the above table, the 2009 study also conflicted with our internal 2009 M&E, which led to a significant overhaul in our M&E methodology. Our 2014 internal M&E is now more consistent with both RCTs.
Despite the limitations of any individual study, however, One Acre Fund believes that these two studies largely validate the current internal M&E that we conduct with thousands of farmers every year. Together, we believe that these RCTs plus internal measurement currently add up to a reasonable body of evidence that the One Acre Fund program has positive impact on harvest profits. Here are more detailed write-ups of both studies, including lessons learned by One Acre Fund. Both studies have important drawbacks, but they are also rich in detail that is helpful for program learning.
In particular, the 2009 study drove a lot of programmatic learning at One Acre Fund. On the measurement side, we changed a lot of our M&E practice, made assumptions more conservative, and overhauled our M&E agent hiring and development. In 2010, we started an agriculture innovation team that is dedicated to steadily increasing our agriculture yields. We diversified our programming so that today a majority of our estimated impact comes from non-maize crops and energy products. We increased customer screening to deliberately avoid customers that might have a high propensity to have a negative return to farm inputs. We began a crop insurance program that is now Africa’s largest smallholder crop insurance program. We also conducted several studies on non-customers and our own customers, seeking to better understand their comparative wealth levels.
One Acre Fund has also noticed a lack of high-quality evidence available for many “add on products” that we sell to customers. For example, we are now one of the top retailers of solar lamps in Africa. Before launching these products, we conducted RCTs in 2011 in Kenya and 2013 in Rwanda to estimate the level of cost savings from purchasing a solar lamp. We estimate that by purchasing a solar lamp, the average household saves roughly $0.50 per week on saved kerosene, battery, and mobile phone charging expenditures. Over three years, the NPV (net present value) of those savings is roughly $66 USD, compared to a purchase price of $15-$40 USD, depending on the lamp model.
We have found that our internal studies on solar lamp impact are more rigorously designed than most published research on this topic. Accordingly, the summary findings of two RCTs we conducted on solar lamps are available in the full paper on RCTs. We will publish more of these studies to support our internal M&E, and to provide “placeholder data” until more research emerges on these life-improving products.