Bringing research down to earth

| December 5, 2018

Today is World Soil Day, so it’s an opportune time to discuss some of the work 3ie has been supporting through our Agricultural Innovation Evidence Programme, which is jointly funded by the Bill & Melinda Gates Foundation and the UK Department for International Development. In particular, an evaluation team studying an integrated soil fertility management intervention in Malawi is helping farmers get to know their soil. The programme under study, implemented by the Clinton Development Initiative, seeks to increase the use of improved planting and soil management techniques to boost agricultural productivity, while a parallel set of activities seeks to increase farmers’ market access. As part of the evaluation, the team has amassed a wealth of detailed plot-level data about soil nutrient contents in the study areas. As the evaluation comes to a close, members of the team are visiting villages in these areas to present these data to the farmers, along with recommendations for how they can improve soil fertility.

On a recent field visit, 3ie staff observed one of these soil presentations in Chatambalala village, located in the Dowa district of Central Malawi. An agronomist member of the research team from Bunda College assembled a group of farmers in the centre of the village, described the soil’s characteristics and deficiencies, and provided advice on which fertilisers to buy and how and when to apply them. First, the group discussed issues relevant for the village as a whole and provided general recommendations related to fertiliser application, but also management practices to increase fertility and water retention. Then, each farmer received information about his or her own soils. Importantly, the meetings concluded with discussions among participants on how useful they expected the soil information to be. The research team is planning to use insights from these conversations to improve the design of future projects associated with small holder farmers and soil testing.

Learning now and learning later

An interesting and unique feature of the soil data component of this evaluation is that the research team is providing these data to villages in both the treatment and control groups of the study. The team decided to do this for the simple reason that all the farmers in the study were very keen to learn about their soil. They specifically asked for the results of the soil tests during the study, so they could use these data both to tailor their production practices and to improve their soil quality. (At 3ie, we salute the farmers’ dedication to evidence-informed decision-making!)

This dialogue between participants and researchers about the participants’ needs illustrates an important dilemma when it comes to the role of development research and evaluation. For researchers, it’s easy to get caught up in methodological concerns, like maintaining the separation between treatment and control groups. Researchers also often like to think about long-term research prospects. In this case, if the team had provided soil data only to the treatment group, there may have been greater prospects for research in these areas of Malawi in the future. The difference between treatment and control in their access to soil data would have meant the team could come back some years later and study the effects of soil information in combination with the other intervention components. By providing the same information to the treatment and control groups, the research team has closed off at least some avenues for further research. Yet researchers decided that it was important to provide the information to all farmers given their interest in and demand for the data.

This isn’t to say there aren’t cases where it’s appropriate to maintain a research setup for a long time. Doing so may be one of our best ways of identifying the long-term effects of development programmes, which is evidence we desperately need but don’t have a lot of. It is all the more important the less sure we are about whether an intervention is actually going to have positive effects or may even be harmful – this is known as the principle of equipoise in clinical research. But the risk is that we get so impressed with a nifty research design that we don’t give our full attention to ways we can use a research setup to have an immediate impact on participants.

Evidence for whom?

The experience of the team in Malawi also raises questions about who the data belong to and whether it would have been ethically irresponsible to refrain from sharing the data over any period of time with the soil owners for the larger public good of research. A core principle of impact evaluation is that the evidence we generate should be used to improve the lives of participants. The standard model for how this is supposed to happen can be circuitous: the evidence will make its way to someone with decision-making power over the fate of the programme being evaluated, and based on that evidence, the decision-maker allocates resources in a way that makes constituents better off. And by all means we should continue striving to make that model work as often and as effectively as possible. But we should also be on the lookout for more direct routes for research to benefit those who participate.

With inputs from the research team: Hope Michelson, Eric Kaima, Christopher Phiri, Wezi Mhango, and Annemie Maertens

Tags: , , , , , , ,

6 Comments on “Bringing research down to earth

  1. Jules Marquart

    I applaud this work, in particular, the sharing of data with both the treatment and control groups. However, I am concerned that the intervention mentions only “fertilizer.” Is the intervention missing teaching long-term and cost-effective soil improvement through the use of composting. It seems introducing and including composting is important as a long-term, sustainable fertilization method.

    1. Ohia Dorathy

      I appreciate the study especially as it brings to fore the value of participatory research ana active inclusion of stakeholders especially the farmers who need the information to enhance productivity. I agree absolutely that other ecto friendly interventions like composting should have been included in the study for agricultural sustainability.

  2. HippieKimmy

    Thank you for pointing out that experimental designs have limitations when applied to development work. I wish there were more posts like this and that more researchers were able to embrace this new paradigm in research where knowledge and interventions are co-created with producers, communities, and other counterparts.
    To build on your points:

    First, some of the most egregious negative effects of development often occur ‘outside’ variables of interest that impact evaluations are examining. Unless there are measures for graft, sexual abuse, inequality, dependence, systemic injustice, or other commonly occurring negative results from development the shady-side of programs or interventions can be completely missed.

    Second, stakeholder inclusion in problem definition and iteration of implementation is necessary for effective development. Iterative adjustments to the ‘treatment’ reduce the strength of conclusions drawn from experimental design data, making these designs less cost-effective and less insightful. The World Bank, USAID, DFID, and many other large donor organizations expect programs to adapt interventions during the program making most experimental designs a poor fit.

    Third, There are other options. There are plenty of quasi-experimental designs that enable researchers to gather longitudinal empirical data that quantify intervention effects with high confidence without purposefully excluding people from programs. Many of these techniques fit naturally with staged implementation where new program or project sites are added in subsequent program years.

    To illustrate, here is an example of an impact evaluation that used propensity score matching across a large array of variables.

    Finally, so little development money actually makes it to the communities or individuals who could use it the most. Intentional exclusion further compounds this problem.

  3. Oumoul

    Thank you for this blog post, I came across thru the PELICAN network and got so interested in the process of making development information readily available to those who need it most. This is an interesting way to bridge the gap between Research and Practice, I wonder how much of this should we really encourage and even foster and how to make the best use of research information for development purpose.

  4. Tina wallace

    Good to see a thoughtful reflection on who research is for, how it is of use to the farmers themselves as the key players in this story, why certain methods are chosen that are v narrow in focus and look at just one or two variables not the range of issues that actually shape attitudes and behaviour.
    I also agree there is little long term research and evaluation in development aid.
    Who benefited from this careful research and
    how can it be better used?
    The RCT approach doesn’t serve the purpise of development we’ll and takes time and money. I feel we’re in a quagmire about what rvidrncevis usabke, by whom snd got what and far too much research is for people far from the ground. There’s v little co-creation of knowledge and I fear we’ve learned little about what’s useful for farmers over the last decades.
    Thanks for opening up some of the issues

  5. Peter Caleb Otieno

    well done, it closes the gape between farmers and researchers. Days are gone when blanket use of fertilizer were done. This time we need to know the soil fertility deficit to know the right amount to put, economical. I support this approach


Leave a Comment:

Your email address will not be published. Required fields are marked *