Monthly Archives: December 2018

What’s the deal with Push Button Replications?

Reproduction and replication of research findings can improve the quality and reliability of research. The recent credibility crisis in the field of psychology, has sparked a huge discussion on the reliability of research findings in all fields and critics have expressed strong doubts over replicability of published research. It should be safe to assume that the original data and programming code from a published article would replicate the results presented. However, we regularly find papers mentioning failure to replicate results using original data and code. For instance, Gertler, Galiani and Romero (2018) tried replicating 203 papers with original code and data and found that one out of seven studies replicated perfectly. To mitigate this issue, researchers from all over the world have taken a stand in favor of practicing and promoting transparent research practices.

At 3ie, we use Push Button Replication (PBR) to verify the reproducibility of research findings using the code, data and methodology provided by the original authors. Brown and Wood (2018), point out that the concept of PBR is very simple; replicate research findings using original data and code provided by the researchers. One necessary condition for reproducibility of findings is that data and code should be available. To facilitate verification through PBRs, we mandate submission of all data and programming code from all 3ie-funded studies. We believe that reproduction of results is an important measure to quality assure products such as briefs, reports and other products that are developed based on research findings. To demonstrate our commitment to openness and transparency, we publish all reports and research materials, including data and code in the public domain.

Reluctance to share replication materials

Several institutions do not have a policy on sharing research and most claim intellectual property over the research they fund, and do not make them public. In our interactions with researchers, we found that at the institutional level there are only a handful of people who are aware and willing to work towards openness in research. Although they agree that sharing research materials is important, they are affiliated with institutions that do not support openness, which makes them reluctant to share materials.

A study looking at 141 economics journals found that only 20 per cent of the journals have a functional data sharing policy. The Gertler study actually looked at over 400 studies out of which only 203 studies had both data and code available for replication purposes. The culture of promoting sharing of replication materials isn’t prevalent among journals, in fact, only a handful of them promote sharing which further nurtures the culture of reluctance among researchers. Journals or funding institutions often allow embargoes on the sharing of data and code until a study is published which in turn fuels the reluctance of researchers to share replication materials. By the time the replication materials are shared the data is outdated and of no use to other researchers.

PBR from different perspectives

Why is PBR necessary from the original researcher perspective?

Knowing that a PBR will be conducted when results are submitted makes the researchers write code and transform data in a way that is easily verifiable. PBRs also help correct minor errors such as typos, misreporting of results and so on. It creates awareness among researchers about good data and code sharing practices. It decreases verification times and increases the reusability quotient of the data and code. Researchers refrain from writing illiterate programs which is key in thinking about the long term usage of the code.

Why is PBR necessary from a fellow researcher perspective?

Fellow researchers and students benefit from the PBRs as it helps build an in-depth understanding of the study methodology. They can also reuse the same materials either for replication purposes or as an addition to their own research.

While conducting PBRs, we have interacted with numerous researchers and a common reason for issues with their code or data is recalling what sort of transformation of data was done to get the desired results. Having proper documentation helps trace the changes made to the data and code. Using research logs and version control tools facilitates this process. Documentation is a part of the PBR process which helps original researchers with recall. It helps replication researchers to track any issues in the research lifecycle and validate results.

Why is PBR necessary from a donor perspective?

Donors often seek tangible proof of the authenticity of the study results that they have funded. They often want a quick turn around on the verification of reports after the completion of the studies, and if the data and code is PBR certified the process of verification by the donor is quicker. This third party verification can also help the donor disseminate the findings with greater confidence.

Why is PBR necessary from a policymaker’s perspective?

Policymakers use evidence generated from studies. Replication of the research findings assures them of the quality and builds trust to use evidence to inform policies that affect people’s lives. Quality assuring the research findings also gives policy-makers the confidence to be accountable to the public, whose money is often used to fund these studies.

Good practices that makes studies replication-ready

The four main pillars that ensure reproducibility, replicability, reusability and long-term preservation of data and code are as follows;

  • Submission of raw data
  • Submission of codebook and a Readme file
  • Programming code for cleaning and preparing the data for analysis
  • Programming code for analysis to generate final results

Guidelines such as providing comments, providing codes for user-written commands, proper folder structure, etc., while analyzing the data and writing code, helps organise the data, code and findings. Furthermore, clear documentation of all the processes involved in the data cleaning and analysis by adding comments in the programming code files helps in understanding the rationale behind every line of code.  For all 3ie-funded studies, researchers are given guidance to make their studies replication-ready.

With the new age tools for transparent workflow, like Open Science Framework, Git, R and Stata dynamic documents, it has become easier and more efficient to keep track of all research processes throughout the life cycle of a study. A number of data repositories such as figshare, dryad, etc. to name a few, have their own data curation and submission guidelines which help archive and version control data and code. All these instruments help replication of studies in turn promoting transparency of research practices.

The credibility and ongoing replication crisis is a good wake-up call for researchers to take note and implement practices that would quality assure their research and help in long-term usage of their data and replication materials.

With inputs from Neeta Goel, Marie Gaarder and Radhika Menon

Bringing research down to earth

Today is World Soil Day, so it’s an opportune time to discuss some of the work 3ie has been supporting through our Agricultural Innovation Evidence Programme, which is jointly funded by the Bill & Melinda Gates Foundation and the UK Department for International Development. In particular, an evaluation team studying an integrated soil fertility management intervention in Malawi is helping farmers get to know their soil. The programme under study, implemented by the Clinton Development Initiative, seeks to increase the use of improved planting and soil management techniques to boost agricultural productivity, while a parallel set of activities seeks to increase farmers’ market access. As part of the evaluation, the team has amassed a wealth of detailed plot-level data about soil nutrient contents in the study areas. As the evaluation comes to a close, members of the team are visiting villages in these areas to present these data to the farmers, along with recommendations for how they can improve soil fertility.

On a recent field visit, 3ie staff observed one of these soil presentations in Chatambalala village, located in the Dowa district of Central Malawi. An agronomist member of the research team from Bunda College assembled a group of farmers in the centre of the village, described the soil’s characteristics and deficiencies, and provided advice on which fertilisers to buy and how and when to apply them. First, the group discussed issues relevant for the village as a whole and provided general recommendations related to fertiliser application, but also management practices to increase fertility and water retention. Then, each farmer received information about his or her own soils. Importantly, the meetings concluded with discussions among participants on how useful they expected the soil information to be. The research team is planning to use insights from these conversations to improve the design of future projects associated with small holder farmers and soil testing.

Learning now and learning later

An interesting and unique feature of the soil data component of this evaluation is that the research team is providing these data to villages in both the treatment and control groups of the study. The team decided to do this for the simple reason that all the farmers in the study were very keen to learn about their soil. They specifically asked for the results of the soil tests during the study, so they could use these data both to tailor their production practices and to improve their soil quality. (At 3ie, we salute the farmers’ dedication to evidence-informed decision-making!)

This dialogue between participants and researchers about the participants’ needs illustrates an important dilemma when it comes to the role of development research and evaluation. For researchers, it’s easy to get caught up in methodological concerns, like maintaining the separation between treatment and control groups. Researchers also often like to think about long-term research prospects. In this case, if the team had provided soil data only to the treatment group, there may have been greater prospects for research in these areas of Malawi in the future. The difference between treatment and control in their access to soil data would have meant the team could come back some years later and study the effects of soil information in combination with the other intervention components. By providing the same information to the treatment and control groups, the research team has closed off at least some avenues for further research. Yet researchers decided that it was important to provide the information to all farmers given their interest in and demand for the data.

This isn’t to say there aren’t cases where it’s appropriate to maintain a research setup for a long time. Doing so may be one of our best ways of identifying the long-term effects of development programmes, which is evidence we desperately need but don’t have a lot of. It is all the more important the less sure we are about whether an intervention is actually going to have positive effects or may even be harmful – this is known as the principle of equipoise in clinical research. But the risk is that we get so impressed with a nifty research design that we don’t give our full attention to ways we can use a research setup to have an immediate impact on participants.

Evidence for whom?

The experience of the team in Malawi also raises questions about who the data belong to and whether it would have been ethically irresponsible to refrain from sharing the data over any period of time with the soil owners for the larger public good of research. A core principle of impact evaluation is that the evidence we generate should be used to improve the lives of participants. The standard model for how this is supposed to happen can be circuitous: the evidence will make its way to someone with decision-making power over the fate of the programme being evaluated, and based on that evidence, the decision-maker allocates resources in a way that makes constituents better off. And by all means we should continue striving to make that model work as often and as effectively as possible. But we should also be on the lookout for more direct routes for research to benefit those who participate.

With inputs from the research team: Hope Michelson, Eric Kaima, Christopher Phiri, Wezi Mhango, and Annemie Maertens