Monthly Archives: April 2015

Trends in impact evaluation: Did we ever learn?

CGIAR Climate_15421304467In 2006, the Evaluation Gap Working Group asked, “When will we ever learn?” This week, 3ie’s Drew Cameron, Anjini Mishra, and Annette Brown (hereafter CMB) have published a paper in the Journal of Development Effectiveness that uses data on more than thirty years of published impact evaluations from 3ie’s Impact Evaluation Repository (IER) to answer the question. We offer a few spoilers below, but if you’d like to read the full paper first, it is available open access here.

There is no denying the huge growth of interest in impact evaluation in the years since the Evaluation Gap Working Group called for more and better evidence of what works in international development. Indeed, the subsequent impact evaluation craze has led to substantial shifts in research priorities among large donors, governments and small NGOs (not to mention the formation of a number of important new organisations). Meanwhile, some like Lant Pritchett have mused that impact evaluation might just be a passing fad; though we sincerely hope that it has more staying power than the selfie stick or instagramming your food. Either way, there is no denying that the body of impact evaluation evidence is growing quickly. In the first four years of this decade, more impact evaluation evidence was published than in the preceding thirty!

A number of people have begun to examine emerging trends within this new evidence base. Eva Vivalt writes about the generalisability of impact evaluations in a number of sectors (here), Ruth Levine and Bill Savedoff look at the importance of impact evaluation for building collective knowledge for policymaking (here), the World Bank provides an overview of the relevance and effectiveness of impact evaluations (here), and David Evan’s recent trip to the annual conference at the Center for the Study of African Economies highlights that even more work is in the pipeline (here).

In early 2013, CMB began an effort to collect all impact evaluations of interventions in low-and middle-income countries (L&MICs) using a systematic search and screening process akin to those typically found in systematic reviews (see their search and screening protocol). The result of that effort is the 3ie IER, launched in 2014. Thanks to these efforts, we now have the first relatively complete picture of how far we have come and where we might be headed. To date, the IER boasts over 2,600 impact evaluations from interventions in L&MICs.

In 2013, using a trend line from indexed articles in 3ie’s old impact evaluation database, Bill Savedoff looked back on a few seminal events in the history of impact evaluation (here). His graphic showed a precipitous rise in the publication of impact evaluations over the previous five years. In the graphic below, we compare his old trend line (in red) to the IER’s more complete body of evidence (in blue). We now know that published evaluation evidence stretches back as far as 1981, and actually began to take off in the mid- to late-1990s, mostly in health. After 2000, social science publications gained a larger share of the total.  Studies in all sectors increased dramatically after 2008. Evidence production still seems to be on the rise (though we’ll know more after this summer’s new round of search and screening).

graph1

We’ve learned a few other things as well. CMB find that most impact evaluations are focused on health, education, social protection, and/or agriculture. Randomised controlled trials dominate the literature in only a few sectors (health, education, information and communications technology, and water and sanitation services). Meanwhile, quasi-experimental methods are employed more frequently in agriculture and rural development, transportation, economic policy, and environment and disaster management.

The following heat map shows that impact evaluations are concentrated mostly in South Asia, East Africa, South and Central America, and Southeast Asia. Meanwhile, notable gaps exist in other regions of the world like North, Central, and West Africa, and the Middle East. A very large share of studies (22.5 per cent) is also concentrated in just three countries: India, China and Mexico.

graph2

Impact evaluation heat map, studies published 1981-2012 (Cameron, Mishra, and Brown 2015)

CMB find a few more interesting (possibly disturbing) trends. Impact evaluations are published at a much slower rate (from end line data collection to publication) among journals in the social sciences (6.18 years) compared to those in the health sciences (3.75 years), from banks and international lending agencies (3.55 years), universities and research institutes (3.54 years), and government agencies (1.00 years). Further, CMB find that impact evaluation authorship is dominated by researchers with institutional affiliations in Western Europe and North America (not including Mexico) (49.7 per cent), and that this trend has only increased over the last 5 years.

In the coming months, we plan to revise our search and screening protocol to increase and improve search results, search for content published in additional languages, such as Spanish and Portuguese, and increase the number of databases and websites we search. We have also developed systems to update our records at a regular time each year and will be expanding the number of sectors and subsectors in which we index studies. We also hope to direct future efforts towards enhancing web capabilities and introducing data visualisation tools to enhance the usability of the IER. Open data and research transparency are also values we embrace at 3ie, which is why we are working to make the repository more accessible to the public by ultimately adopting best practices espoused by the Open Data Institute, Open Science Framework, and the Berkeley Initiative for Transparency in the Social Sciences.

The impact evaluation repository is an important public good. But in all this rush to make more and better evidence increasingly accessible to the international development community, we are eager for suggestions. Is there an area you think we should explore? A service you’d like to see available through the IER? Sectors you’d like to be added to our list? Please suggest them in the comments below or email us at database@3ieimpact.org.

On target? Why participant selection matters for development programmes

icrisat.images_6119486174Many development programmes reach only a fraction of the people they aim to include. One reason for this is that attrition erodes target group participation at various stages between programme conception and completion. Programme targeting using selection criteria, eligibility assessment and participant registration is one of the ways this problem can be addressed. But how far does targeting address this issue?

3ie’s new systematic review on targeting and farmer field schools (FFS) throws up several insights about how targeting decisions can have a real impact on the ground. Evidence shows that many FFS programmes do not necessarily reach the participants they target. The key reason for this was that criteria for eligibility, the selection procedures used and the realities of potential participants were not always compatible.

The targeting debate

Discussions about targeting have typically focussed on the relative merits of universal versus targeted programmes and the problem of how large-scale programmes can reach the poorest in society. However, most development programmes focus limited resources on a specified population, with many targeting women, youth or other groups. The challenge of targeting is one that is of equal importance for these types of programmes.

Amartya Sen pointed out that the word ‘targeting’ turns beneficiaries into passive recipients instead of the agents of change. Targeting also focuses our attention on devising the most accurate way of identifying beneficiaries. However, people act and react to the programmes offered to them. Factors such as implementation, participants’ characteristics and programme context all have an important role to play in determining who participates – and by extension, who it is that stands to benefit.

Targeting analysis: the case of farmer field schools

FFSs have become one of the most common approaches to rural adult education and agricultural extension in the world. They have been implemented in over 90 countries, reaching an estimated 10 to 20 million people.

We found in our systematic review that some FFSs target the poorest or most disadvantaged groups in society on the grounds that they are most in need of the benefits that these programmes provide. However, many FFS programmes target farmers with more resources, more education and greater social agency, with the aim of maximising programme effectiveness.

Programmes targeting more experienced and educated farmers typically succeeded in reaching their target group, irrespective of the procedure they used to select them. The main driver behind this success was that the target groups’ socio-economic characteristics favoured their inclusion.

However, targeting was less successful when programmes were designed to be generally inclusive or to include a specified group such as female farmers. One example from the systematic review is illustrative of a wider trend of an FFS programme in Uganda. The programme intended to be inclusive. But community leaders’ prominent role in selection meant that many final participants had social connections to recruiters. In other cases, it was not selection criteria or procedures that precluded some peoples’ participation, but a lack of time and the resources needed to participate. In Liberia, an FFS programme targeting women failed to reach many female farmers because not enough consideration was given to their limited access to land and tools or their existing household and childcare commitments.

Programmes that were more successful in reaching their target groups were carefully designed, so that selection procedures were not prone to elite capture. They also incorporated additional components to ensure that the poorest or most marginalised had access to the resources needed to participate. Overall, our analysis highlighted the importance of a coherent and consistent logic underlying the targeting process. This encompasses not only the choice of who should participate and the identification procedures used to select them, but also consideration of the characteristics of potential participants. The goal of identifying target groups as accurately as possible should be balanced against the need to ensure that participants are able to participate.

Why targeting matters

In attempting to ensure that a programme reaches its intended audience, targeting aims to minimise resource wastage and increase cost-effectiveness. Of course, targeting is never without cost, and identifying the poor with real precision is likely to be costly. Targeting is also a zero-sum game. Money spent on it reduces the budget for addressing poverty and welfare goals. However, a decision not to spend resources on targeting is still a choice that can have important consequences for programme effectiveness and social equity.

The review of FFS programmes was based on a range of different evaluations that typically did not report on financial considerations or programme budgets. As a result, we were not able to examine how much time and resources were put into planning targeting and selection. However, there was some indication that implementers were sometimes caught between competing demands to deliver a programme that would have an effect on outcomes of interest, to be as inclusive as possible and to meet the practical necessity of reducing costs. Consequently, targeting and selection decisions were not always aligned with programme objectives, target group characteristics and the wider intervention context. Devoting some time and resources to targeting and selection considerations could go a long way to improving programme performance. By ensuring that the target group, selection procedures and target-group characteristics are as aligned as possible, programmes may stand a better chance of reaching those most in need.