Monthly Archives: August 2015

Using evidence to rebuild lives

Humanitarian-8009363194International humanitarian assistance reached a record high in 2014, with global contributions totalling US$24.5 billion,almost a 19 per cent increase since 2013.However, this isn’t good news for the sector as the demand for humanitarian assistance continues to outstrip supply. Humanitarian financing is now fraught with new challenges as the nature and number of emergencies that come under the realm of humanitarian action is constantly changing. This has resulted in a shift in planning and resources for response and resilience efforts across different contexts.

This World Humanitarian Day, the question that we need to ask is: Is there rigorous evidence (from impact evaluations and systematic reviews) available on the impact of a multitude of interventions aimed at addressing the multi-dimensional needs of over 58 million people displaced by conflicts and over 107 million affected by disasters caused by natural hazards?

The reality looks a bit grim. A recent 3ie scoping study on the evidence landscape in the humanitarian sector showed that there is a lack of reliable and robust evidence from impact evaluations of humanitarian assistance. In other words, we lack sound evidence about what programmes and policies are working or not, how and why they are working or not and at what cost. However, it becomes particularly difficult to fill these evidence gaps because there are several challenges that need to be tackled in humanitarian contexts. We identify a few of these challenges here and explore ways to address them.

Balancing the need to deliver aid speedily versus generating evidence

The need to act quickly and deliver aid to those affected by a crisis often overshadows the need for evidence-informed decisions. But rigorous evidence can help answer crucial questions such as: Is the assistance reaching the target populations in right doses at the right time? Is it being delivered through the right channels in cost-effective ways and with the resources available to an implementing agency? And has the intervention achieved the outcomes described in a programme’s theory of change?

At the recent Global Forum for Improving Humanitarian Action, ALNAP flagged the significant shortfall of high-quality evidence on what works,and argued that the absence of evidence inhibits the delivery of the most effective and efficient interventions in response to particular needs and particular forms of crisis.

Tackling the notion that impact evaluations are unethical

Impact evaluations are often Humanitarian-15154683630equated with randomised controlled trials, where it is assumed that a certain section of the population is denied access to programmes and assistance. This is a mistaken notion because evaluation researchers can harness the potential of how a rigorous study can be designed and applied that is ethical and does not deny anyone any relief they would be getting from a programme. For example, most humanitarian agencies deliver assistance packages that comprise multiple interventions for various sectors. Rolling out programmes with small changes while keeping the basic assistance package the same (a factorial design) or phasing out the roll out of interventions can help assess differences in outcomes and welfare associated with a particular intervention without denying individuals the basic assistance.

In contrast, delivering interventions that are unproven or ineffective can pose a great risk to the lives of people living in contexts where humanitarian assistance is required. In extreme cases, they can even be unethical if those interventions make communities or individuals worse off instead of rebuilding their lives.

Impact evaluations address questions on effectiveness and efficiency unlike others forms of evaluations, which may not be able to assess attributable impact. Impact evaluations can also answer questions on the differential impact of humanitarian assistance on vulnerable populations, such as women, children or chronically poor people, which can provide valuable insights into whether aid is reaching those most in need of this assistance.

Dealing with the dearth of baseline data in humanitarian contexts

In most fragile contexts,census data at the local levels is often not available or reliable. Even if humanitarian agencies collect data, it is often not available to researchers for analysis. Collecting information in humanitarian contexts is challenging. Issues around accessibility, ethical constraints, security concerns and the changing nature of baseline populations constantly present themselves. It thus becomes necessary to make the best possible use of any administrative and monitoring data that are routinely collected by humanitarian agencies. In the absence of baseline information, research teams may use data collected by interviewing respondents, but it, too, is prone to recall errors. Information bias can be reduced by using mixed methods that can help triangulate and validate results.

Impact evaluations that use a range of data sources, such as satellite imagery, geographic information systems and data from other surveys can help address the ethical constraints of collecting information during emergencies.

Lack of incentives and support for generating evidence

Donors can play a key role in incentivising the generation and use of evidence. The International Rescue Committee recently called on donors to incentivise the generation of evidence by ‘fund[ing] only those programmes that are based on the best available evidence or, in cases where impact evaluations (or their equivalent) have yet to be conducted, that are supporting the generation of evidence.’

3ie launched the Humanitarian Assistance Thematic Window in 2014 with the aim to bridge the evidence gap by funding feasibility studies, impact evaluations and systematic reviews across a range of areas in humanitarian contexts.

In the run up to the 2016 World Humanitarian Summit, 3ie along with seven major actors in the humanitarian sector is calling for financial and resource investment in processes for producing and using high-quality, rigorous evidence.

While discussions are now focused on increasing and diversifying funding for humanitarian aid, generating evidence on what works in complex humanitarian situations will mean that scarce resources are invested in programmes and policies that can save millions of lives.

Video lecture: Dr Jyotsna Puri, Deputy Executive Director and Head of Evaluation at 3ie, uses case studies to show how data and methods can be innovatively used for getting around the challenges associated with conducting impact evaluations of humanitarian relief programmes.

Toward evidence-informed policies for achieving the Sustainable Development Goals

So, 2015 has arrived and the Millennium Development Goals (MDGs) are to be replaced by the Sustainable Development Goals (SDGs). But shouldn’t we stop and ask how we have done on the MDGs first? “How we have done” can be seen an outcome monitoring question: have the targets been reached? But since we have fallen far short on some targets, such as access to improved sanitation, we need to dig deeper and ask which policies have been successful in helping achieve the targets. To answer that question we should turn to evidence from systematic reviews because high-quality reviews offer us the best and soundest basis for informing our decisions.

I outline here some things we know from 3ie-funded and other reviews in recent years. This blog summarises systematic review findings from a few sectors. It is the tip of the iceberg, rather than a comprehensive summary.


Microfinance has not been the panacea for poverty reduction as was hoped. It can sometimes even be impoverishing. Attention has turned to asset transfers, particularly for the ultra-poor. But there is not yet a body of evidence on how effective these programmes are. Similarly, very large public works programmes, such as the Mahatma Gandhi National Rural Employment Guarantee Scheme in India and the Expanded Public Works Programme in South Africa are still to be sufficiently rigorously evaluated

Social protection

Cash transfers, both conditional and unconditional, have proved to be effective means of tackling poverty. Conditional cash transfers (CCTs) have also been shown to be effective in increasing school enrolments. Reviews have shown that CCTs are more effective when (1) the transfer is larger; (2) the transfer is less frequent; (3) it is for secondary rather than primary school; and (4) there is good monitoring and enforcement of the conditions for getting the transfer.



The review evidence suggests that programmes that are effective at getting children into school, such as CCTs and some health interventions, are not necessarily the same as the programmes needed to help them learn when they get there, such as learning materials, including computer assisted learning.


Water and sanitation

Access to improved water and sanitation reduces diarrhoea, which in turn lowers infant and child mortality. However, provision of water supply at the community level has no health benefits as the water gets re-contaminated before use. Point-of-use water treatment can reduce child diarrhoea by 40 to 60 per cent. But ensuring sustained proper use has remained a challenge. Improved sanitation has a similarly significant effect on diarrhoea. But both initial and sustained adoption of sanitation is sometimes difficult.

Environment and agriculture

Turning to the environment, payment for environmental services programmes are increasingly common in developing countries. But they have proved to be very cost ineffective. The vast majority of the land for which payments are made would have remained forested in the absence of the payments.

Farmer field schools reduce pesticide use, while maintaining or increasing yields and hence the income of participating farmers (though not their neighbours). But they have not been able to replicate these positive effects when taken to scale.

This brief overview of selected findings from systematic reviews clearly shows that there is a growing useful, high-quality synthesised evidence base. 3ie has been playing a key role in the introduction and development of methods for high-quality systematic reviews in development because reviews, not single studies, provide a sounder base for decisions on policies and programming. Governments and development agencies would do well to start using this evidence to inform their strategies and programmes to realise the improvements we know, from our MDG experience, are needed to achieve the SDGs.