Yearly Archives: 2015

Miles before we sleep: building evidence on forest conservation

“The woods are lovely, dark and deep”, wrote Robert Frost.

radhika-3ieThen why have we lost 129 million hectares of forest cover (almost the size of South Africa) in the past decade?

The conference of parties (COP21) meeting in Paris last week arrived at a historic and new global consensus to stop climate change by committing to 1.5 Celsius increase in average temperature target.

Promoting greenhouse gas (GHG) sinks or areas that absorb GHGs are an important way to mitigate climate change (see more in the IPCC report). Forests, along with oceans are the most important sinks in the world. On the face of it, there have been some initiatives that have invested in forestry programmes, e.g. the Global Environment Fund estimates that to date, over US$10 billion has been invested in forest-related programmes. Similarly, the Norwegian aid agency, NORAD have spent $2.5 billion on forest conservation initiatives. (In Paris, they extended a promise to continue to support more such activities till 2030.)

Despite these initiatives, very little is still known about strategies that can help preserve forests even while they continue to provide livelihood benefits. So, why are there so few impact evaluations in this area? Equally important is the question, are impact evaluations even useful for this work?

Mind the gap

3ie’s soon to be published Evidence Gap Maps on forest conservation, and forest and land use change show that there are very few impact evaluations that examine forest conservation and overall land use change. The few impact evaluations that exist provide important pointers on what helps to reduce deforestation and GHG emissions.

An article that one of the blog authors recently published shows that most studies in this area use quasi-experimental methods rather than experimental methods to measure change, which can be causally attributed to forestry and land use programmes. In our review, we found only two impact evaluations that used random assignment (one of them is an evaluation to assess the impact of decentralised forest management in Brazil and the other one assesses payment for ecosystem services in Rwanda). This is surprising because the burden of proof on researchers is much higher with quasi-experimental methods than randomised controlled trials. It is simply easier to explain a randomised evaluation than a quasi-experimental evaluation. However, the main reason for the predominant use of quasi-experimental methods appears to be that evaluators and implementers of forestry programmes don’t usually plan to measure causal change at the design stage of their study. Early planning is critical if a randomised controlled trial is going to be used.

Impact evaluations that use experimental or quasi-experimental methods are also not usually planned at an early-enough stage because they are not popularly acknowledged as tools for measuring changes in final outcomes (such as impact on GHGs and livelihoods). In forestry programmes, output-focused evaluations that just measure the number of trees that stay protected at the end of the programme are far more popular than evaluations that go further along the causal chain.

How can impact evaluations make a difference?

Impact evaluations not only measure attributable change but they can also be very useful in several other ways.

Firstly, they help in dealing with problems related to bias. Biases may arise because forestry programmes are placed in areas where they are likely to be most successful. ‘As they should be’, many would argue. The problem here is that this also means that effects caused by programmes are either overstated or understated. For example, a study in Thailand showed that protected areas (such as national parks or wildlife sanctuaries) are likely to be located in areas that have low agricultural productivity and profitability. This meant that even if there was no designated ‘protected area’, the likelihood that the forest would be cleared, other things held constant, was low. In this case, arguing that reduction in deforestation occurred entirely because of a national park is an overestimation of the effectiveness of national parks.

radhika-m-3ieSecondly, impact evaluations can help to understand if forestry programmes have targeted the right target areas or groups. For instance, with a budget of more than US$5 million, a payment of ecosystem services programme in Mexico, was successful in targeting households that were eligible for the programme and were more likely to clear forests. (See video for highlights from this study). A similar payment for ecosystems programme in Costa Rica, the PSA (Pago por Servicios Ambientales) programme, did not however achieve the same result. This occurred due to wrong targeting, i.e. people who were targeted would not have cleared forests even if they had not received the payment. So, the change caused by the payment system was minimal because the programme did not target people who were most likely to clear forests.

Impact evaluations can also compare the effectiveness of different sorts of programmes in addressing deforestation. So, for instance, an impact evaluation can help examine if programmes that employ government bodies to manage forests are more successful than those that depend on communities to manage forests. Impact evaluations in India and Nepal have found that programmes, in which governments co-manage forests with communities, have a greater impact on reducing deforestation compared to programmes that are solely managed by either the community or the state. This is because communities have incentives to save forests that benefit them, but in many cases do not have the jurisdictional authority to enforce rules. Impact evaluations have also shown that wildlife sanctuaries are more effective in protecting forests than protected areas as more resources are devoted to them.

For researchers, it has now become easier and cheaper to plan impact evaluations for forestry programmes (see here a recently published article that explores this). With advancement in technology and data collecting mechanisms, researchers can now combine the use of satellite imagery, aerial photography, and geographical information systems with social and household data. This data can be used for estimating forest cover, deforestation and land use while also understanding livelihood and welfare effects.

To sum up, ecosystems have non-linear dynamics at various spatial and temporal scales. It is often not clear when and if programmes impact forest cover as well as livelihoods. However, impact evaluations that combine rapid and easy-to-collect, aggregated and geospatial data with household data can build a compelling evidence base that answers crucial questions. As the reality of climate change looms large, it is about time we know whether forestry programmes work, and if so, for whom and why.

Seven impact evaluations on demand creation for VMMC: how a focused thematic window can meet multiple evidence needs

hdptcar-2379802404On World AIDS Day 2015, we are marking the culmination of 3ie’s third thematic window, which funded seven pilot programmes and rapid impact evaluations looking for ways to increase the demand for voluntary medical male circumcision (VMMC). In late 2013, we awarded grants to project teams of implementers and researchers to pilot innovative programs for increasing VMMC demand and to conduct rapid impact evaluations of those programmes. All seven studies were completed in early 2015 and will be published soon in the Journal of Acquired Immune Deficiency Syndromes, which unfortunately means that the results of the individual studies are currently embargoed. But we can share what we learned from the overall grants programme – a highly focused thematic window to test pilot HIV prevention programmes in a rapid way can help to answer both decision-focused and knowledge-focused programme questions.

There were three important features of this grants window. The first is that we designed it from the beginning to support rapid impact evaluations. Our funder, the Bill & Melinda Gates Foundation, wanted the window to produce evidence that can be used to inform programming as soon as possible. The research question – what innovations can increase the demand for VMMC – lends itself to rapid impact evaluation, because the primary outcome, circumcision, is a discrete behavioral response that only needs to happen once and is expected to occur relatively soon after the interventions. In addition, we anticipated that many project teams would be able to use randomised controlled trial designs and be able to randomise treatment assignment at the individual level, both of which allow for smaller sample sizes and thus quicker studies. Based on these factors, we set the funding ceiling and the timelines to require rapid impact evaluations.

The second feature was the specificity of the theme. Although 3ie has many thematic windows, the breadth of the themes for the windows varies widely. In the case of the VMMC window, our funder wanted to know the answer to a very specific question: what innovations can help increase the demand for VMMC? In our scoping report, which informed the request for proposals, we laid out some barriers and facilitators affecting the demand for VMMC that should be considered when proposing innovations. The funded studies included a range of interventions that addressed barriers and facilitators both from a rational choice standpoint (i.e. what are my costs and benefits) and from an emotional choice standpoint (i.e. what might influence me to do this thing I do not feel like doing), as discussed in our blog post last year.

There were similarities across some VMMC interventions. For example, two interventions used a mobile phone lottery (or raffle) as an incentive. Two interventions relied on intimate partners either directly (sending messages about VMMC through women attending ante-natal clinics) or indirectly (using messaging for men about the preferences of intimate partners for circumcised men). Two interventions relied on some kind of peer effects, one through a sports-based program and one through peer referrals. Two interventions used a financial incentive intended to compensate men for some of the costs incurred. You can learn more about one of the interventions and its impact evaluation in this video.

The differences across the seven interventions allow us to examine many possible VMMC programmes, each of which could be a candidate to be scaled up in the same or very similar context. In this way, each study is what Shah, et al. (2015) call a decision-focused evaluation, an evaluation “driven by implementer demand, tailored to implementer needs and constraints, and embedded within implementer structures”. However, the narrow focus of the theme, the similarities between studies, and the grounding on a common framework of barriers and facilitators allow us to examine some of the underlying questions – the answers to which could inform the design of other similar programmes. Taken together, the seven studies then also play the role of what Shah, et al. (2015) call knowledge-focused evaluations, “those primarily designed to build global knowledge about development interventions and theory”. The restriction of the window to a set of countries in Sub-Saharan Africa also narrows the focus of the window in a way that increases the comparability of results across these studies for this region.

world-aids-dayThe third important feature was the coordination. For this funding window, we held 3ie’s first post-award workshop. This event brought together the study teams for two days to refine their intervention and impact evaluation designs based on the feedback from other teams, but also with a view to harmonising the aspects of the projects that would allow for better cross-project comparisons upon completion. Awardees discussed the theories behind their interventions, their sampling and randomisation methods and the measurement of their primary and secondary outcomes. In some cases, the teams continued their communication with each other throughout the implementation of their projects. As the timing of the studies was pre-determined, the studies were all completed at about the same time. This makes it possible for us to synthesise and disseminate the results so that both the decision-focused findings and the knowledge-focused findings can be made available at the same time to complement evidence uptake activities.

The primary challenge for us in implementing a funding window with these features was the long delays many grantees faced in getting their Institutional Review Board (IRB) approvals. These delays slowed the timing of the rapid impact evaluations and interfered with the coordination of completion. We have encountered the same challenge with our other rapid impact evaluation windows; our grantees miss their early project milestones due to delays in IRB approvals.

The results of the VMMC funding window are enlightening. While we cannot share the findings from the individual studies, we can mention some conclusions from our forthcoming synthesis paper “Identifying interventions to increase the uptake of male circumcision in Southern and Eastern Africa: evidence from seven impact evaluation studies”, which is being presented as a poster on World AIDS Day 2015 at the International Conference on AIDS and STIs in Africa:

  • The impact that peers and intimate partners can have in encouraging circumcision appears to be limited. One exception may be sports-based interventions. (See here for results from a precursor study to that funded under our window.)
  • There is evidence that how information about male circumcision is framed and presented can make a difference to the impact it can have.
  • Material incentives based on chance do not seem to have an impact, but financial incentives framed as compensation for costs incurred due to circumcision can have a positive impact on uptake.

3ie is not alone in running highly focused grants windows for impact evaluations with a view to increased comparability moving towards external validity. Evidence in Governance and Politics (EGAP), a 3ie partner organisation, has developed an innovative approach called a metaketa. This approach focuses even more on a single or small set of knowledge-focused evaluation questions by having the research teams work together in the design of the pilot interventions as well as on the design of the experimental evaluations. The principal investigators for the grant programme also plan in advance how they will synthesise the evidence across the individual studies to look for more generalisable answers. 3ie’s VMMC funding (as well as our other rapid impact evaluation windows) and EGAP’s metaketa are two similar approaches to funding a group of impact evaluations with a view to maximising external validity.

What does the evidence say about women’s empowerment and domestic violence?


Approximately 30 per cent of women around the world, aged 15 and above, have experienced physical and/or intimate partner violence. Domestic violence is associated with health and psychological problems for women and adverse consequences for children later in life.

Despite the seriousness of the issue, the empirical evidence on the effectiveness of prevention interventions is still scarce in developing countries. There is also not a lot of evidence on the unintended impacts of social programmes, such as conditional cash transfers, on domestic violence. So, what does the existing evidence say?

Cash transfer programmes can have unintended impacts on domestic violence

Evidence shows that women, who were beneficiaries of the Oportunidades programme in Mexico, were more likely to receive verbal violent threats with no associated physical violence,. Also, the size of the cash transfer seems to matter. Women receiving large transfers seem to be more exposed to domestic violence.

The education status of female beneficiaries of an unconditional cash transfer in rural Ecuador was correlated with domestic violence. There is a significant decrease in psychological violence for women who had received more than primary education, due to the programme. However, there is an increase in emotional violence for women with primary education or less.

Are more empowered women less or more exposed to domestic violence?

The evidence just cited may lead us to think that women who are empowered are more able to stand up for their rights. As a consequence, they experience less domestic violence. But this conclusion may not be necessarily true. More empowered women may deviate from the traditional gender roles in the household. Men, who feel threatened by this, may use domestic violence as a way to restore the male dominance in the household.

The empirical evidence on the association between the status of the women within the household and domestic violence is mixed. Women with partners who have received secondary education or higher, are less likely to experience domestic violence in countries such as Egypt and India, but more likely in countries such as Peru. Women who receive a cash transfer (while not working) are less likely to experience domestic violence in Egypt, but more likely in India and Peru. The evidence suggests that context and social norms matter with respect to domestic violence.

Assessing women’s empowerment is not easy but it is important

Women’s empowerment has multiple dimensions and it can be approached in different ways. Concepts such as access, ownership, entitlement and control are used interchangeably.

In a recent piece of research that I undertook in rural Colombia, I explored different ways of measuring women’s empowerment and domestic violence. I analysed the sensitivity of the relationship between empowerment and domestic violence based on the definition I used. I explored various dimensions of women’s empowerment such as self-esteem, women’s disapproval of domestic violence, participation in household decisions, social capital, income and education. I also differentiated aggressive manifestations of domestic violence, such as hitting and the threat to hit; from controlling behaviour of their partner, which includes restricting them from visiting family and friends.

The results suggest that the different measures of women’s empowerment helped in explaining the more aggressive manifestations of domestic violence in comparison to controlling behaviour. Amongst the different measures of women’s empowerment, the one referring to women’s social capital seemed to be most highly correlated to domestic violence. Women’s self-esteem and disapproval of domestic violence appeared also to be positively and significantly correlated with domestic violence, though it is not as robust as women’s social capital. Surprisingly, I did not find women’s participation in household decisions to play a significant role in determining domestic violence.

A role for impact evaluations

To better understand domestic violence and design better social policies, much more analysis is needed. Impact evaluations can play an important role by providing robust evidence on crucial questions:

  • What are the channels through which social programmes can have an impact on domestic violence?
  • What kind of features should social programmes have to reduce domestic violence effectively?

The usual channel through which social programmes, such as cash transfers, can have an impact on domestic violence is through a change in women’s bargaining power. Therefore women’s empowerment needs to be accurately assessed.

Data collection instruments can also be improved in the following ways:

  • Include questions referring to women’s self-esteem and social capital.
  • Ask both men and women about the acceptance of domestic violence.
  • Explore women’s participation in different kinds of households’ decisions: the kind of food to be purchased, children’s schooling or income.
  • Include dimensions such as how to invest household savings and whether or not to sell an asset.
  • Don’t just assess physical violence. Assess psychological and emotional violence and a partner’s controlling behaviour. Determinants of each type of violence may be different and therefore, the theory of change would also differ for each one.

There are a whole range of social norms and structural determinants that affect behaviour. These need to be considered, not only in designing programmes to reduce domestic violence, but also while assessing the intended and unintended impacts of social programmes on domestic violence.

Reflections on replication research: a conversation with Paul Winters

D15_433_017Replication research is often the space of junior researchers. As a well-cited research economist, with multiple international organisation affiliations, Paul Winters stands out in this space. I recently caught up with Paul to discuss his co-authored replication study of Galiani’s and Schargrodsky’s influential paper Property rights for the poor: Effects of land titling.

Galiani’s and Schargrodsky’s Property rights for the poor: Effects of land titling is considered an important paper in the development literature. Land titling programmes exist throughout the developing world. Whether it is through asset collateralisation or household investment, land titling theoretically combats poverty through numerous channels. Galiani’s and Schargrodsky’s heavily cited study shows that an Argentine land titling programme induced a range of positive effects, from expanding household investment to improving children’s education. Paul’s replication study set out to re-examine intermediate outcomes of this programme to better understand the robustness of the theory of change behind land titling interventions in general.

I was also interested in his perspective on 3ie’s replication programme. We recently held a replication consultation with researchers and multilateral organisation representatives, where some participants suggested we only focus our programme on attempting to exactly reproduce the published results (akin to 3ie’s pure replication section), and not include any robustness checks or explorations of the intervention’s theory of change. Paul’s replication study, like all 3ie-funded replication research, included both a pure replication section and robustness checks of the original research (although data limitations prevent it from fully exploring the intervention’s theory of change).

In our conversation, Paul reflected on his experience in replicating an influential paper and gave me his take on what he thinks about the replication process in general.

Ben: Congratulations on finishing up your replication paper. How would you summarise your replication study of Galiani and Schargrodsky’s Property rights for the poor: Effects of land titling impact evaluation?

Paul: Galiani and Schargrodsky (GS) is a widely cited paper, since they find that providing property rights has an impact on social outcomes. They are able to identify this impact using a natural experiment. In our paper, not only were we able to replicate the results found by GS, but robustness checks using alternative estimates of impact largely confirmed their results, suggesting the outcomes and conclusions presented by GS are reasonable.

Ben: Some of the participants at our replication consultation suggested that 3ie focus its replication programme only on studies that attempt to exactly reproduce the published analysis (similar to Chang and Li’s replication study in economics paper). You have argued against that and said that replication studies should continue to examine robustness checks beyond pure (or exact) replication. Would you please elaborate a bit on your thinking?

Paul: Journals like the American Economic Review (AER) only require researchers to provide the data and analysis files to exactly reproduce original studies and that is the minimum standard. But if you are delving into a data set and considering how variables are created and the analysis being conducted, it is critical to know the literature and to consider the theory of change. Additionally, Brian Nosek’s work that was presented at the replication consultation clearly highlights how decisions on variable creation and model specification can influence results.

Further, if you are interested in designing successful policy interventions linked to an evaluation, you need to understand the mechanisms by which impact occurs and ideally the type of beneficiary most likely to benefit from a programme (by exploring the heterogeneity of impact). A replication is an opportunity for carefully considering an intervention and moving beyond what is done in a paper.

Of course, if the objective is simply to ensure the integrity of the result the AER standard is fine. But 3ie seeks to use evidence to inform policy and programming. The replications you fund should be more critical and look beyond what is done in a paper by reconsidering the theory of change and checking the robustness of results.

Ben: Finally, a few researchers have expressed concerns about the usefulness of internal replication research. As a researcher who approached replication research without many expectations, what have you learned from your experience?

Paul: I believe it is a worthwhile undertaking. At a practical level, it would have been helpful to know what type of data were available for the replication right at the outset. We had hoped to use the data to explore the causal mechanisms of impact and heterogeneity of impact and this was in our replication plan. But those data were not available. It should be clear that GS did provide the data and what they provided is consistent with the AER standard. Further, if someone asked me for data on a previous project, especially a number of years later, as was the case for GS, I probably could not have provided much more. Of course, this points to a potential benefit of such replications. It could make us more careful about documenting all of the steps we take in our impact evaluations.

To download Paul’s co-authored replication study in 3ie’s Replication Paper Series, click here.

For highlights of 3ie’s replication consultation event, click here.

For more information on 3ie’s replication programme, click here and for the summary results of a recent systematic review on land titling click here.

Getting to the goals: what we know and don’t know about sustainable solutions for poverty eradication

Chipiliro-Khonje-15260197844On 25 September 2015, 193 UN member states signed off on a new direction for achieving sustainable development by 2030.  First among the 17 new sustainable development goals is the aim to end poverty in all its forms,and everywhere by 2030. This might seem like a tall order.  As a first and important step, countries need to take stock of effective solutions that have worked and identify areas where more work is required.

3ie’s recent evidence gap map of productive safety-net progammes can help decision-makers do just that because it throws up key pointers for identifying targeted and effective solutions for achieving this ambitious goal.  The map shows existing evidence on the effectiveness of safety net programmes.  These programmes include livelihood or income generating components to expand market opportunities for the extremely poor.

Based on a comprehensive search of the published and unpublished literature, it provides a visual overview of evidence from 248 impact evaluations and 24 systematic reviews, mapped into a framework of relevant interventions and outcomes for this area. The interventions cover six broad categories, such as financial services, land reform and microenterprise support services, and important poverty alleviation outcomes.

What do we know?

The evidence gap map presents what we already know about the impacts of different productive safety net programmes on poverty and human well-being. Here are some of the key findings from a few of the high-quality systematic reviews included in the evidence gap map:

  • A review of the impact of access to formal financial services found that state expansion of formal banking services in rural areas has the potential to increase rural wages, agricultural investment and reduce rural poverty. Innovative savings products showed the potential to increase poor people’s incomes.
  • A review of interventions to strengthen land property rights found that land freehold titling and other forms of rights formalisation improved welfare (consumption or income) by 16 per cent on average.
  • A review of the effects of microcredit on women’s economic empowerment found that there is no consistent evidence of microcredit’s positive effect on women’s control over household spending. There are,however,several mediating factors that might improve effectiveness. For example, the review found that microcredit may be more effective with higher loan sizes, and when given to women who were younger and to women with fewer children.

Gap-mapWhat do we need to know?

The evidence gap map also reveals several blank spaces in existing evidence. Though most impact evaluations report on some measures of income, consumption or savings, a very small proportion of these studies then go on to measure the effects on income poverty, risk management, adaptive capacity and income inequality outcomes. This is particularly surprising because these interventions are frequently implemented with the objective of helping people find a permanent escape out of poverty. Without this information, it is difficult to assess to what extent these intervention meet their ultimate aim of helping people escape poverty and what the effect of these interventions is on income inequality.

Another startling finding from this evidence gap map is that almost half of the included impact evaluations do not report any information relating to the poverty status of the studied populations. Those that do provide this information, use varying definitions of poverty. These gaps in the evidence base make it difficult to reliably assess whether safety-net interventions help people escape poverty.  I have written about the issue of not having these data in an earlier blog.

Where do we go from here?

The evidence gap map shows that there is a wealth of evidence on the effectiveness of productive safety net programmes. However, for designing effective and evidence-informed solutions for ending poverty, there is a lot more that needs to done.

  • We need a consistent definition of poverty that captures its multi-dimensional nature.
  • We need to measure the impact of development interventions on key outcomes such as adaptive capacity, risk management and income inequality.
  • For a comprehensive assessment of impact, it is also important to calculate the cost-effectiveness of interventions.
  • Carrying out sub-group analysis of the studied populations and focussing on all dimensions of disadvantage and marginalisation are integral to producing evidence needed for achieving equitable outcomes. For instance, how do factors such as gender, age and disability influence impacts? Several useful tools, such as the Equity Check list for Systematic Reviews championed by the Campbell and Cochrane Equity methods group can help guide authors of both impact evaluations and systematic reviews on what they should consider and report on in their research.
  • And finally, evidence gap maps can help identify what we know and don’t know, about what works to help people escape extreme poverty, end hunger, improve people’s health, ensure everyone acquires quality education, and gain access to water and sanitation.They can be the roadmap for steering policies, programming and research towards achieving the 17 sustainable development goals by 2030.

Visit the 3ie evidence gap map webpage to learn more about evidence gap maps and explore our gap maps on other topics.