Monthly Archives: January 2013

Of rumours related to blood, poison and researchers

15A recent analysis of 67 3ie supported evaluations shows that distrust and suspicion among community members can sometimes slam the brakes on the implementation of development programmes. The distrust in almost all cases was sparked off by a ‘rumour’.

The attempt to collect blood samples of children for a malaria treatment intervention in Kenya met with stiff opposition from the study community. There were rumours of blood stealing, covert HIV testing and suspicion about the safety of the study drugs.

It may be quite easy to attribute this rumour to ignorance and superstition. But these rumours do not come out of the blue. Historical, anthropological and sociological accounts can trace the roots of such distrust and suspicion.

In the case of blood samples, the Kachinja or blood stealer is a well-known figure in Kenya and East Africa in general. Potential explanations for the emergence of the Kachinja can be found in stories of the First World War where British occupiers may have forcibly extracted blood from the locals for transfusions to injured white soldiers. Local traditions of djinns and spirits have also cross-fertilized with real fears of blood being sold in the international market. It’s not surprising then that the ‘blood stealing researcher’ became a Kachinja for the local community in Kenya. “Kachinjas are people who drive along the tarmac road in big white Land Rovers and if they meet you there….. they catch you and drain your blood and leave you in the bush,” says a primary teacher in rural Kenya (Geissler, 2005).

The fear of the outsider is not just related to blood stealing. Two more examples from 3ie-supported studies demonstrate that rumours about poisoning can also trigger distrust of implementing agencies and evaluators. On a recent field monitoring visit to Uganda, we found that a rumour helped explain the reduced impact of a food assistance programme delivered to HIV patients. The food provided to the programme participants was an ‘American corn-soya blend’ which was ‘weirdly yellowish’ and quite different in taste from the local diet of cassava, millet bread, sweet potato and beans. The rejection of the food was also attributed to the feeling that the ‘whites were trying to poison the blacks with their food’.

Given the high prevalence of HIV/AIDS in Africa, it is not unusual that many of the local myths are related to the dreaded disease. The belief that HIV/AIDS is the result of a conspiracy that involves poisoning is prevalent in many countries in Africa. Respondents in an anthropological study in Zimbabwe believed that “whites are capable of contaminating the food with a poison that results in AIDS” (Rödlach, 2011). Like in Kenya, historical events intersected with indigenous beliefs to create local idioms. People in Zimbabwe speak of the isidlo phenomenon, in which substances used in sorcerous practices are added to food (Rödlach, 2011). The early days of social marketing of condoms in Africa were plagued by stories that the CIA infected condoms with HIV/AIDS to reduce the black population.

16Other health interventions have also not been free of conspiracy theories. Polio vaccination is now suffering in Pakistan and Nigeria because of rumours that it is a US plot to reduce fertility amongst muslims. These conspiracy theories have triggered violence and led to the murders of health workers in Pakistan. Such rumours also gain credence by media reports that fake vaccination campaigns were concocted in Pakistan to gather intelligence on Osama Bin Laden.

As the last example shows, the suspicion of poisoning by outsiders is not just exclusive to Africa. Tucked away in a grantee progress report submitted to 3ie is yet another stark example from rural Bangladesh that shows the local community’s lack of trust in outsiders. Offering melamine plates as an incentive to young girls for participating in a survey led to several widespread rumours that the plates were being used to either poison them or convert them to Christianity. The community members found it strange that female enumerators were spending considerable amount of time talking to adolescent girls who did not normally receive attention or gifts. The resistance from the community eventually meant that the team had to stop surveying in 13 villages after some enumerators were surrounded by an angry mob.

Development researchers and practitioners have several lessons to learn from these examples. Suspicion of outsiders and the distrust of food for instance have serious implications for the theory of change used to design a food assistance programme. What we found particularly disconcerting was that the impact evaluation of the food assistance programme in Uganda did not pick up on the rumours of ‘whites trying to poison blacks’. While the rumours of blood stealing in Kenya were discussed in the final report, it did not prompt the researchers to reflect on the theory of change behind the malaria treatment intervention.

The little attention to historical, sociological or cultural context in these cases suggests that the theories of change concerning development programmes may be flawed. By not giving due attention to these factors, we will miss obvious explanations for failures of development programmes and end up repeating the same mistakes again and again. In the instances cited here, the inputs of an anthropologist as well as local researchers familiar with the context would have certainly helped pave the way for better understanding of the community.

A poorly designed intervention will also significantly affect the relevance of impact evaluations. The impact of development interventions is bound to decrease if there is no trust in implementing agencies. As development professionals, we should consider securing the trust of the community as an important responsibility.

The field of impact evaluation continues to be dominated by economists. As a result impact analysis does not often focus on cultural and historical processes. There is clearly a strong case here for interdisciplinary research. But do impact evaluators have sufficient incentives to pursue interdisciplinary research? Among behavioral economists, psychology is slowly gaining prominence. However, cultural processes are possibly more difficult to take into account in theoretical economic models. There are however exceptions and these may prove to be useful examples for impact evaluators. Needless to say, inputs from anthropologists are also critical for impact evaluation designs.

Interdisciplinary research could be an important first step towards understanding cultural beliefs. These could be related to poisoning, blood stealing or any other phenomena which we may think of as being just a ‘rumour’.

If you have a story to share with us about how a ‘rumour’ stalled the implementation of a programme or interrupted your research, email us at 3ie@3ieimpact.org. We will post the most interesting stories on our website.

Delivering Global Public Goods

H11Pre-schools in Mozambique boosted children’s social and cognitive development, and increased the likelihood that both they and their siblings will attend primary school.

Independent audits of industrial plant emissions in Gujarat, India produced more accurate audits and also curtailed pollution.

Burdensome registration requirements and high cost of obtaining agricultural electricity connections in West Bengal, India, had unnecessarily restricted access to water for small farmers.

A newly designed cookstove in northern Ghana failed to achieve the expected benefits in reduced exposure to smoke and more efficient use of fuel wood.

Providing social recognition to hairdressers in Lusaka in Zambia proved more effective than cash incentives in encouraging them to sell female condoms, though no mechanism helped in achieving substantial sales.

These are not disparate statements but are lessons learned from 3ie-supported studies released in 2012. 3ie is now making an important contribution to filling the evidence gap of what works in development and why. And it is not that evidence is being produced for evidence’s sake. Pre-schools are being rolled out across Mozambique, independent audits are being adopted in Gujaratand discussed with other states, registration requirements for small farmers in West Bengal have been abolished, and the NGO in northern Ghana has gone back to the drawing board. These are just some of the examples of how evidence from 3ie-funded studies is being used to inform better policies. And better policies can improve lives.

During the course of 2012 3ie clearly started to deliver on its primary purpose of filling the evidence gap in development effectiveness. We have become recognised as an agency with robust, independent and credible review processes. Both governments and development agencies can turn to us for assistance in managing single studies, external peer review of those studies, or a whole programme of studies under our Thematic Windows.

In 2012 we put together new Thematic Windows on the promotion of medical male circumcision and the use of self-testing kits for HIV/AIDS. We have also been working on an ambitious joint programme of agricultural impact evaluations in sub-Saharan Africa and South Asia with AGRA and the International Fund for Agricultural Development, funded by the Bill and Melinda Gates Foundation and the UK Department For International Development. We are also exploring Thematic Windows on climate change and humanitarian assistance. The work done in 2012 will come to fruition in 2013, when all these Thematic Windows will be launched

Another major highlight for us last year was the increasing demand for impact evaluations through3ie’s Policy Window. This is an innovative window through which implementing agencies can come to us to identify researchers for conducting impact evaluations of flagship programmes. We are currently processing grants for the Philippines Department of Education, NEPAD’s e-education programme in Africa, and the Magic Bus sports for girls programme in India, to name a few.

3ie not only funds studies, it also sets international standards for impact evaluation. For the studies we fund, we do this through our review process. For others, we offer quality assurance services and issue conceptual papers and guidelines. We have also launched a replication programme to test the robustness of study findings, and are preparing a registry of planned impact evaluations in low and middle income countries.

We have continued to set standards and push the frontiers of methods and knowledge with our systematic review programme. We made two further rounds of awards for systematic reviews and helped organize the first ever international event on Systematic Reviews in International Development in Dhaka, Bangladesh. We continued to focus on increasing the policy relevance of our studies and completed our first 3ie-style policy friendly systematic review reports which will be published in early 2013.

All in all, 2012 was a very good year for 3ie. Our committed staff in New Delhi, London and Washington DC plan to make 2013 even better.

Matching policymakers and researchers

8

The chasm between policymakers and researchers is frequently observed but seldom addressed. A little over two weeks ago, 3ie organized a matchmaking market place to bridge this gap between research and policy. The setting for this was the Dhaka Colloquium for Systematic Reviews in International Development. The people who came together for this innovative matchmaking exercise were both ‘users’ and ‘doers’ of systematic reviews.

Systematic reviews are exhaustive, unbiased reviews of evidence in a specific area, that summarize, assess and present the state of the evidence from published/unpublished literature. Systematic reviews can help to explain what works, under what circumstances and why.

There are many attributes of systematic reviews that can be useful to policymakers or other users of systematic reviews such as governments, donor agencies, international development agencies and large NGOs. Since systematic reviews are exhaustive reviews of all evidence on a specific topic and are unbiased, they inform all aspects of an intervention including unintended consequences. In many cases where data is available, they use meta-analyses statistical techniques to pool data across all studies and provide a sense of the ‘net’ effect size. Good systematic reviews can thus be very good tools to analyze the effectiveness and cost-effectiveness of development interventions. They can help inform the policy emphasis of a government or a programmatic strategy within an organization.

But putting together systematic reviews is a skill intensive and time consuming task. Hence specialized teams are required to do these. A good systematic review team requires people who’ve done systematic reviews earlier, information specialists, sector experts and very importantly, research assistants who can spend hours and hours poring over papers from all sources (published, grey literature, project documents, design documents and anything else).

At the matchmaking clinic that 3ie organized, around 10 researchers (doers) and 30 possible policymakers (users) participated. They split up in groups, each with some users and some doers. Users presented possible questions that would be useful and relevant to them. Doers discussed steps to produce a policy relevant systematic review. Presentations by mixed teams at the workshop spanned a variety of areas ranging from interventions targeting better irrigation, environmental health, adaptation by farmers in the face of increased weather variability, micro-credit and schemes to increase farmer productivity in Malawi.

There are many steps involved in producing a systematic review. The first step is to identify the question and the population of interest/relevance. Knowing which interventions are going to be examined and who the main stakeholders of interest are, is key. Next, articulating the theory of change and eligible interventions is important. Since systematic reviews cover evidence of impact, understanding comparators, outcomes and the settings is the next step. Once these areas have been defined, a gap map can be put together. A gap map consolidates what we know about ‘what works’ in the sector of interest by drawing out evidence from systematic reviews and impact evaluations. (3ie’s London office has state of the art expertise in systematic reviews and has been working on systematic reviews and refining techniques for gap maps.)

Once the question has been identified, a fairly intensive and long review of the literature is undertaken. A 3ie supported systematic review that assesses the effectiveness of community driven interventions to reduce maternal and neo-natal mortality can provide a good example. In the first step, Haider, Lassi and Buta collected more than 30,000 studies using keyword searches in a variety of libraries. Inclusion criteria and exclusion criteria were used to select within these studies. Inclusion criteria included geographic area, target population, types of intervention and study methods. Studies included in this systematic review included interventions in Africa or Asia, included studies of interventions that were specifically community driven, were undertaken for women in situ (and not in a location outside of the community), and had to include members of the community. Included programmes needed to increase skills of community members in providing support to maternal and neo-natal care. The programme also needed to be a package of interventions that ranged from ante natal, natal and post-natal interventions. Therefore single interventions such as those that solely targeted resuscitation or Vitamin A provision were excluded. Studies that used robust identification strategies such as community based randomized, quasi-randomized and prospective time series were included.

From a population of more than 30,000 studies, 109 were selected for a detailed evaluation. An independent assessment of data quality and particularly bias, attrition, heterogeneity and sub-group analysis resulted in 27 studies being included finally in the systematic review.

The systematic review concluded that overall community based interventions (that included additional training for lady health workers, community mid-wives, village health workers, facilitators and traditional birth attendants) included in the study showed no significant impact on reducing maternal mortality. However it showed that community based intervention packages are associated with significant reductions in neonatal mortality. The systematic review showed that a combination of training in antenatal, natal and post natal care, preventive essential newborn care, breastfeeding counseling, management and referral of sick newborns, skills developing in behavior change communication and community mobilization to promote birth and newborn care preparedness, reduced neo-natal mortality by 27 per cent. This impact was 6 per cent higher in studies that included both therapeutic and preventive care, compared to those that just included preventive care.

Clearly these conclusions can be very useful for overly stretched health ministries that are frequently under-resourced and continually lack good health workers. In developing countries, nearly two-thirds of births occur at home and only half of all births are attended by trained birth attendants. Of the 136 million births every year, more than 7 million are still births or neo-natal births. 98 per cent of these are in developing countries. Community based interventions in developing countries that can reduce this percentage by more than one-fourth are clearly a cost-effective strategy and a potential win-win.

The Dhaka Colloquium featured many ideas for systematic reviews that have the potential for significant policy impact. The workshop concluded two main things: Many more matches between doers and users are needed, and, in many areas, many more impact evaluations (that go onto populate systematic reviews) are required. 3ie will continue to support both of these over the years.

(For those who are interested, 3ie’s newest thematic window on climate change and disaster risk reduction will soon put out a call for systematic reviews in the area of mitigation, adaptation and risk reduction. So watch this space.)