Monthly Archives: July 2015

Collaborating with communities to improve vaccine coverage: a strategy worth pursuing?

Gates FoundationIt is unfortunate but distrust of vaccines remains widespread in the world today. In August 2003, a polio vaccination boycott was declared in the five northern states of Nigeria. Political and religious leaders argued that the vaccines could be contaminated with anti-fertility agents, HIV and cancer-causing agents. It took a full year to resolve the boycott but the one year period wreaked havoc on the status of polio across the world. There were polio outbreaks in three continents during that one year period. It ended up costing public health officials more than US$500. This distrust also continues in Afghanistan and Pakistan today.

It is clear that confidence in vaccinations is not just an individual phenomenon but is very much a social phenomenon. Indeed the negative consequences of low vaccine uptake in terms of public health are very large.

Social factors play a very significant role in creating this distrust and affecting vaccine uptake (See here and here for more on factors affecting uptake). Confidence in vaccine efficacy (‘The vaccine will be harmful to us!’) is only one reason affecting uptake. Other reasons include complacency (‘Of course it won’t happen to me!’; ‘It has never happened in our family!’) and convenience (‘The clinic is just too far away’; ‘It is too expensive’). The public good nature of getting vaccinated is also important to note – unvaccinated people are likely to get infected and infect other unvaccinated people. Vaccinated people on the other hand don’t get infected and the cycle breaks.

To change these attitudes towards vaccines, communities rather than individuals, can and should play a big role in in collectively ensuring that a significant proportion of the population gets vaccinated.

In a new 3ie scoping paper, we along with our co-authors discuss the important role that communities can play in not just encouraging demand within communities to take up vaccines and dispel mistrust but also in co-managing and co-delivering immunisation programmes in general. Communities can help implementing organisations plan deliveries of vaccines as well as set up schedules and monitor uptake. Our systematically collected evidence (we include a gap map of evidence on this subject) shows that co-management and co-delivery that actively involve communities in project design, implementation, monitoring and evaluation can be a significant factor in achieving success in increasing vaccine coverage.


Our main findings from the scoping study are:

  • Community engagement is under-used in creating and sustaining demand for immunisation as well as in planning and ensuring the delivery of vaccines.
  • Even if communities are engaged and demand is created, the supply side is almost as important. Organisations need to be cognizant that implementation and service delivery are likely to be a constraint. Communities may be sensitised and involved in ensuring good quality delivery and implementation in various ways. Importantly, if supply-side constraints are not dealt with, the lack of confidence in vaccines is likely to get aggravated.
  • More evidence is required to show how co-delivery and co-management might be organised and ensured. We also need to do more work to understand for which specific types of populations this may be most effective.
  • Not surprisingly, the cultural context within which vaccines are being delivered and the kinds of communities being engaged with is extremely critical.
  • We conclude that one way in which some of these constraints may also be dealt with, is through the use of technology, especially technologies that help to plan and monitor better. But evidence is needed about what works and what does not in using technology for engaging communities to help increase coverage, uptake and ensuring supply.

It is to fill this evidence gap that 3ie, with the support of the Bill & Melinda Gates Foundation, has launched an evidence programme that will generate new evidence about what works to engage communities in increasing immunisation coverage, test the feasibility and effectiveness of these approaches and inform their scale up. Watch this space for what we learn as we go along.

Watch: Shagun Sabarwal, Evaluation Specialist at 3ie, talk about the main findings of this scoping paper in this short video.

Replication research promotes open discourse

Kimberly Burns/USAIDThe just-released International Journal of Epidemiology (IJE) suite of publications reexamining the effectiveness of deworming in Kenya demonstrates the potential impact of replication research. The headline publication is a 3ie-funded replication study. The paper has been published alongside three additional commentaries: a synopsis of a systematic review of deworming evidence, a response from the original authors, and a response from the replication researchers. The publication of these papers in a respected journal puts the role of replication squarely where we think it needs and deserves to be to promote valuable public discourse around highly relevant evaluation evidence. We’re excited about these publications for a number of reasons.

First, replication research is being published! From 3ie’s perspective, this is an accomplishment onto itself. We designed the Replication Programme to help change incentives. In order to encourage more replication papers that will improve the evidence base for policymaking, we need to convince researchers that publication outlets exist for these time-intensive replication studies. The IJE arguably has the highest impact within the field of epidemiology (see information on IJE’s impact factor here). The editors’ decision to publish these papers is a testament to the value they see in replication research. This in turn helps change publication incentives.

Second, the replication studies have sparked a larger conversation around the existing deworming evidence. We’re discovering that a significant grey area exists regarding the ability of replication researchers to recreate the originally published results. However, these replication studies cannot be simply lumped into difficult to define successful or failure categories. We find that replication studies provide researchers with a valuable space to discuss analytical decisions and the robustness of publication results. We believe that these discussions improve the science around these evaluations, which in turn enhances the quality of the evidence on which policymakers rely to spend limited development funds.

Third, these conversations are public, which allows for scrutiny of the findings and a general discussion of the research. Miguel and Kremer helpfully provide their data for replication efforts here and a replication guide for their original paper here, both of which ease the process for replication researchers to reproduce their paper. The original authors have been very open with assisting researchers interested in reanalysing their results (as an example, here’s a report on GiveWell’s replication study). But most of the subsequent replication studies of their original deworming paper don’t appear to be widely circulated or publicly posted. That has now changed with the publishing of the replication results and these commentaries on the deworming evidence in the IJE. The discourse is now open for interpretation by everyone, from researchers and policymakers to funders and implementers. Regardless of which side of the deworming debate one falls, some facts remain irrevocable. The revised tables in the original author response to the replication study correct for agreed upon errors in the original publication. This clearly demonstrates the power of replication research.

Ultimately, 3ie, through our Replication Programme, seeks to change the incentive structure around replication research. If the reanalysis process becomes standardised and journals agree to publish this type of research, there will be a genuine opportunity for researchers to provide more robust evidence for policymaking. This open discourse around replication results will help normalise the replication process. We’re hoping to see more of these open discussions in the future.


Video: Benjamin DK Wood talk about how the publishing of this 3ie-funded study opens up the discourse on replications of impact evaluations.


Replication studies:

Re-analysis of health and educational impacts of a school-based deworming programme in western Kenya: a pure replication

Re-analysis of health and educational impacts of a school-based deworming programme in western Kenya: a statistical replication of a cluster quasi-randomized stepped-wedge trial

Original author’s reply:

Commentary: Deworming externalities and schooling impacts in Kenya: a comment on Aiken et al. (2015) and Davey et al. (2015)

Replication researchers’ response:

Authors’ Response to: Deworming externalities and school impacts in Kenya

Systematic review commentary:

Replication of influential trial helps international policy

Tackling radicalisation through sports

2379611587_b797996183_oTen years ago, on 7 July, four suicide bombers killed 52 people in London. This included the bombing of the number 30 bus at Tavistock Square, just yards away from the building that now houses 3ie’s London Office.

Today, ten years on, in the town of Dewsbury, home to the mastermind of the 7/7 bombings, cricket is being used to combat radicalisation. The NGO Chance to Shine has put together three cricket teams for getting the youth to mix with each other.

The NGO’s approach contrasts the UK government’s stringent new policy for tackling radicalisation. The policy calls on universities to report, monitor and close groups suspected of radicalisation. But does existing evidence really support this policy?

The head coach of the cricket team in Dewsbury says that as a result of the cricket teams, the youth “are rubbing shoulders with people they would never have thought of meeting.” The head coach’s words echo a well-established finding in contact theory in social psychology: increasing contact between groups reduces prejudice.   A systematic review published nearly ten years ago, reporting evidence from 515 studies, showed clearly that contact theory works. Bringing together people from different backgrounds in a variety of settings, such as the classroom, housing projects and the sports field reduces their prejudice about members of the other group.

This finding lends support to the recent growth of sports for development programmes, which are being used , amongst other things, to build trust between youth who have been on opposing sides in a conflict.


Murder rates fell in the area with the sports stadium, but continued to rise elsewhere
Source: Vijayendra Rao and Ana María Ibáñez (2005)

3ie is funding an impact evaluation of life-skills training and psychosocial support through sports groups in in war-torn Liberia, but the results are not yet in. However, in our work on social funds for the World Bank evaluation department, we took a look at the impact of the Jamaican Social Fund on social cohesion. Social funds are meant to build social cohesion, and we found one case where they had clearly done so. It was a project for the renovation of a new sports stadium in the Arnett Gardens area of Kingston, Jamaica. Whilst social funds are meant to build social cohesion through the process of project planning at the community level, the positive effects of this project were from the product not the process. Quite simply, as local residents put it: “The kids play football instead of killing each other.” ”If there are more sporting activities, the youths will have less time to think about guns.” Following the renovation of the stadium, the murder rate in Kingston fell, whereas it continued to grow in the rest of the country.

This evidence thus supports the recent critique of the UK government’s policy for tackling radicalisation. Pulling in universities to keep a close watch on groups will only drive these groups underground and allow small numbers of individuals to feed each other’s extreme ideas. It is better to get people into the open, to mix with others, to reduce prejudice and distrust. If this policy is to be informed by evidence, it needs to be rethought.

Do communities need funds or facilitation?

MUSHENGEZI- BIRINDWA PascalA group of villages in Haryana, India came together to build a much-needed bridge to reduce travel time to the nearest town. This sounds like the sort of community initiative that donors have been supporting for more than a decade with community driven development (CDD) programmes. Except it isn’t.

It isn’t partly because the villagers in Haryana acted on their own initiative without any outside help. Fed up with being ignored by politicians and the local government, the villagers crowdsourced financing for the bridge. But it also isn’t because few CDD programmes actually operate in the way the villagers in Haryana did when they decided to get together and build a bridge.

Staff at 3ie are currently undertaking a review of CDD programmes. Several recent studies, such as the 3ie-funded impact evaluation of the Tungaane programme in the Democratic Republic of Congo and GoBifo in Sierra Leone, have confirmed what earlier (sometimes less rigorous studies) have shown. That is, CDD programmes are successful in building social infrastructure but not in building social capital.

This finding has not surprised me. We found the same in the World Bank review of social funds I worked on fifteen years ago and in a 3ie systematic review done five years ago. CDD programmes use rather than produce social capital. What has surprised me is the extent to which many CDD programmes really aren’t community-driven.

During the World Bank social funds study, I carried out fieldwork in Malawi and Zambia. Especially in Zambia, the social fund operated a pure CDD model. The social fund agency promoted the fund, but it was left to communities to get together, decide on a project and apply for funding. It turns out that this approach is more of an exception than the norm. In many CDD projects, the decision-making and application process is facilitated by outsiders. A chunk of project resources are used not for funding things communities want, but paying NGOs to train communities in how to hold meetings and help communities decide on what they want.

Now, facilitation may be useful. It can help ensure that the voices of the marginalised are heard, that poorer communities without the skills and connections get to apply and develop skills in project management. But I do wonder if communities that already have community-level decision-making bodies need outsiders to help them hold meetings and to decide their priority needs.

We are still analysing the data to test the hypothesis that there has been an innate tendency to bureaucratisation in the design and implementation of CDD projects. More recent projects have greater degrees of external facilitation at every stage of the project cycle. Communities apparently need help to form decision-making structures and determine their priorities. Who says so?

Earlier this year, 3ie hosted a consultative workshop for its Innovations for Increasing Immunisation Thematic Window . At that workshop, I facilitated a session on transferable lessons from development programmes that involved community engagement. The participants were asked to group lessons as being ‘essential’, ‘sometimes useful’ and ‘silly ideas’. The greatest consensus was that external facilitation was essential for community development. That was a consensus amongst researchers and development practitioners. There were no community members from low- and middle-income countries taking part in the workshop to give their views. The villagers in Haryana may not agree.