Monthly Archives: November 2015

Seven impact evaluations on demand creation for VMMC: how a focused thematic window can meet multiple evidence needs

hdptcar-2379802404On World AIDS Day 2015, we are marking the culmination of 3ie’s third thematic window, which funded seven pilot programmes and rapid impact evaluations looking for ways to increase the demand for voluntary medical male circumcision (VMMC). In late 2013, we awarded grants to project teams of implementers and researchers to pilot innovative programs for increasing VMMC demand and to conduct rapid impact evaluations of those programmes. All seven studies were completed in early 2015 and will be published soon in the Journal of Acquired Immune Deficiency Syndromes, which unfortunately means that the results of the individual studies are currently embargoed. But we can share what we learned from the overall grants programme – a highly focused thematic window to test pilot HIV prevention programmes in a rapid way can help to answer both decision-focused and knowledge-focused programme questions.

There were three important features of this grants window. The first is that we designed it from the beginning to support rapid impact evaluations. Our funder, the Bill & Melinda Gates Foundation, wanted the window to produce evidence that can be used to inform programming as soon as possible. The research question – what innovations can increase the demand for VMMC – lends itself to rapid impact evaluation, because the primary outcome, circumcision, is a discrete behavioral response that only needs to happen once and is expected to occur relatively soon after the interventions. In addition, we anticipated that many project teams would be able to use randomised controlled trial designs and be able to randomise treatment assignment at the individual level, both of which allow for smaller sample sizes and thus quicker studies. Based on these factors, we set the funding ceiling and the timelines to require rapid impact evaluations.

The second feature was the specificity of the theme. Although 3ie has many thematic windows, the breadth of the themes for the windows varies widely. In the case of the VMMC window, our funder wanted to know the answer to a very specific question: what innovations can help increase the demand for VMMC? In our scoping report, which informed the request for proposals, we laid out some barriers and facilitators affecting the demand for VMMC that should be considered when proposing innovations. The funded studies included a range of interventions that addressed barriers and facilitators both from a rational choice standpoint (i.e. what are my costs and benefits) and from an emotional choice standpoint (i.e. what might influence me to do this thing I do not feel like doing), as discussed in our blog post last year.

There were similarities across some VMMC interventions. For example, two interventions used a mobile phone lottery (or raffle) as an incentive. Two interventions relied on intimate partners either directly (sending messages about VMMC through women attending ante-natal clinics) or indirectly (using messaging for men about the preferences of intimate partners for circumcised men). Two interventions relied on some kind of peer effects, one through a sports-based program and one through peer referrals. Two interventions used a financial incentive intended to compensate men for some of the costs incurred. You can learn more about one of the interventions and its impact evaluation in this video.

The differences across the seven interventions allow us to examine many possible VMMC programmes, each of which could be a candidate to be scaled up in the same or very similar context. In this way, each study is what Shah, et al. (2015) call a decision-focused evaluation, an evaluation “driven by implementer demand, tailored to implementer needs and constraints, and embedded within implementer structures”. However, the narrow focus of the theme, the similarities between studies, and the grounding on a common framework of barriers and facilitators allow us to examine some of the underlying questions – the answers to which could inform the design of other similar programmes. Taken together, the seven studies then also play the role of what Shah, et al. (2015) call knowledge-focused evaluations, “those primarily designed to build global knowledge about development interventions and theory”. The restriction of the window to a set of countries in Sub-Saharan Africa also narrows the focus of the window in a way that increases the comparability of results across these studies for this region.

world-aids-dayThe third important feature was the coordination. For this funding window, we held 3ie’s first post-award workshop. This event brought together the study teams for two days to refine their intervention and impact evaluation designs based on the feedback from other teams, but also with a view to harmonising the aspects of the projects that would allow for better cross-project comparisons upon completion. Awardees discussed the theories behind their interventions, their sampling and randomisation methods and the measurement of their primary and secondary outcomes. In some cases, the teams continued their communication with each other throughout the implementation of their projects. As the timing of the studies was pre-determined, the studies were all completed at about the same time. This makes it possible for us to synthesise and disseminate the results so that both the decision-focused findings and the knowledge-focused findings can be made available at the same time to complement evidence uptake activities.

The primary challenge for us in implementing a funding window with these features was the long delays many grantees faced in getting their Institutional Review Board (IRB) approvals. These delays slowed the timing of the rapid impact evaluations and interfered with the coordination of completion. We have encountered the same challenge with our other rapid impact evaluation windows; our grantees miss their early project milestones due to delays in IRB approvals.

The results of the VMMC funding window are enlightening. While we cannot share the findings from the individual studies, we can mention some conclusions from our forthcoming synthesis paper “Identifying interventions to increase the uptake of male circumcision in Southern and Eastern Africa: evidence from seven impact evaluation studies”, which is being presented as a poster on World AIDS Day 2015 at the International Conference on AIDS and STIs in Africa:

  • The impact that peers and intimate partners can have in encouraging circumcision appears to be limited. One exception may be sports-based interventions. (See here for results from a precursor study to that funded under our window.)
  • There is evidence that how information about male circumcision is framed and presented can make a difference to the impact it can have.
  • Material incentives based on chance do not seem to have an impact, but financial incentives framed as compensation for costs incurred due to circumcision can have a positive impact on uptake.

3ie is not alone in running highly focused grants windows for impact evaluations with a view to increased comparability moving towards external validity. Evidence in Governance and Politics (EGAP), a 3ie partner organisation, has developed an innovative approach called a metaketa. This approach focuses even more on a single or small set of knowledge-focused evaluation questions by having the research teams work together in the design of the pilot interventions as well as on the design of the experimental evaluations. The principal investigators for the grant programme also plan in advance how they will synthesise the evidence across the individual studies to look for more generalisable answers. 3ie’s VMMC funding (as well as our other rapid impact evaluation windows) and EGAP’s metaketa are two similar approaches to funding a group of impact evaluations with a view to maximising external validity.

What does the evidence say about women’s empowerment and domestic violence?


Approximately 30 per cent of women around the world, aged 15 and above, have experienced physical and/or intimate partner violence. Domestic violence is associated with health and psychological problems for women and adverse consequences for children later in life.

Despite the seriousness of the issue, the empirical evidence on the effectiveness of prevention interventions is still scarce in developing countries. There is also not a lot of evidence on the unintended impacts of social programmes, such as conditional cash transfers, on domestic violence. So, what does the existing evidence say?

Cash transfer programmes can have unintended impacts on domestic violence

Evidence shows that women, who were beneficiaries of the Oportunidades programme in Mexico, were more likely to receive verbal violent threats with no associated physical violence,. Also, the size of the cash transfer seems to matter. Women receiving large transfers seem to be more exposed to domestic violence.

The education status of female beneficiaries of an unconditional cash transfer in rural Ecuador was correlated with domestic violence. There is a significant decrease in psychological violence for women who had received more than primary education, due to the programme. However, there is an increase in emotional violence for women with primary education or less.

Are more empowered women less or more exposed to domestic violence?

The evidence just cited may lead us to think that women who are empowered are more able to stand up for their rights. As a consequence, they experience less domestic violence. But this conclusion may not be necessarily true. More empowered women may deviate from the traditional gender roles in the household. Men, who feel threatened by this, may use domestic violence as a way to restore the male dominance in the household.

The empirical evidence on the association between the status of the women within the household and domestic violence is mixed. Women with partners who have received secondary education or higher, are less likely to experience domestic violence in countries such as Egypt and India, but more likely in countries such as Peru. Women who receive a cash transfer (while not working) are less likely to experience domestic violence in Egypt, but more likely in India and Peru. The evidence suggests that context and social norms matter with respect to domestic violence.

Assessing women’s empowerment is not easy but it is important

Women’s empowerment has multiple dimensions and it can be approached in different ways. Concepts such as access, ownership, entitlement and control are used interchangeably.

In a recent piece of research that I undertook in rural Colombia, I explored different ways of measuring women’s empowerment and domestic violence. I analysed the sensitivity of the relationship between empowerment and domestic violence based on the definition I used. I explored various dimensions of women’s empowerment such as self-esteem, women’s disapproval of domestic violence, participation in household decisions, social capital, income and education. I also differentiated aggressive manifestations of domestic violence, such as hitting and the threat to hit; from controlling behaviour of their partner, which includes restricting them from visiting family and friends.

The results suggest that the different measures of women’s empowerment helped in explaining the more aggressive manifestations of domestic violence in comparison to controlling behaviour. Amongst the different measures of women’s empowerment, the one referring to women’s social capital seemed to be most highly correlated to domestic violence. Women’s self-esteem and disapproval of domestic violence appeared also to be positively and significantly correlated with domestic violence, though it is not as robust as women’s social capital. Surprisingly, I did not find women’s participation in household decisions to play a significant role in determining domestic violence.

A role for impact evaluations

To better understand domestic violence and design better social policies, much more analysis is needed. Impact evaluations can play an important role by providing robust evidence on crucial questions:

  • What are the channels through which social programmes can have an impact on domestic violence?
  • What kind of features should social programmes have to reduce domestic violence effectively?

The usual channel through which social programmes, such as cash transfers, can have an impact on domestic violence is through a change in women’s bargaining power. Therefore women’s empowerment needs to be accurately assessed.

Data collection instruments can also be improved in the following ways:

  • Include questions referring to women’s self-esteem and social capital.
  • Ask both men and women about the acceptance of domestic violence.
  • Explore women’s participation in different kinds of households’ decisions: the kind of food to be purchased, children’s schooling or income.
  • Include dimensions such as how to invest household savings and whether or not to sell an asset.
  • Don’t just assess physical violence. Assess psychological and emotional violence and a partner’s controlling behaviour. Determinants of each type of violence may be different and therefore, the theory of change would also differ for each one.

There are a whole range of social norms and structural determinants that affect behaviour. These need to be considered, not only in designing programmes to reduce domestic violence, but also while assessing the intended and unintended impacts of social programmes on domestic violence.

Reflections on replication research: a conversation with Paul Winters

D15_433_017Replication research is often the space of junior researchers. As a well-cited research economist, with multiple international organisation affiliations, Paul Winters stands out in this space. I recently caught up with Paul to discuss his co-authored replication study of Galiani’s and Schargrodsky’s influential paper Property rights for the poor: Effects of land titling.

Galiani’s and Schargrodsky’s Property rights for the poor: Effects of land titling is considered an important paper in the development literature. Land titling programmes exist throughout the developing world. Whether it is through asset collateralisation or household investment, land titling theoretically combats poverty through numerous channels. Galiani’s and Schargrodsky’s heavily cited study shows that an Argentine land titling programme induced a range of positive effects, from expanding household investment to improving children’s education. Paul’s replication study set out to re-examine intermediate outcomes of this programme to better understand the robustness of the theory of change behind land titling interventions in general.

I was also interested in his perspective on 3ie’s replication programme. We recently held a replication consultation with researchers and multilateral organisation representatives, where some participants suggested we only focus our programme on attempting to exactly reproduce the published results (akin to 3ie’s pure replication section), and not include any robustness checks or explorations of the intervention’s theory of change. Paul’s replication study, like all 3ie-funded replication research, included both a pure replication section and robustness checks of the original research (although data limitations prevent it from fully exploring the intervention’s theory of change).

In our conversation, Paul reflected on his experience in replicating an influential paper and gave me his take on what he thinks about the replication process in general.

Ben: Congratulations on finishing up your replication paper. How would you summarise your replication study of Galiani and Schargrodsky’s Property rights for the poor: Effects of land titling impact evaluation?

Paul: Galiani and Schargrodsky (GS) is a widely cited paper, since they find that providing property rights has an impact on social outcomes. They are able to identify this impact using a natural experiment. In our paper, not only were we able to replicate the results found by GS, but robustness checks using alternative estimates of impact largely confirmed their results, suggesting the outcomes and conclusions presented by GS are reasonable.

Ben: Some of the participants at our replication consultation suggested that 3ie focus its replication programme only on studies that attempt to exactly reproduce the published analysis (similar to Chang and Li’s replication study in economics paper). You have argued against that and said that replication studies should continue to examine robustness checks beyond pure (or exact) replication. Would you please elaborate a bit on your thinking?

Paul: Journals like the American Economic Review (AER) only require researchers to provide the data and analysis files to exactly reproduce original studies and that is the minimum standard. But if you are delving into a data set and considering how variables are created and the analysis being conducted, it is critical to know the literature and to consider the theory of change. Additionally, Brian Nosek’s work that was presented at the replication consultation clearly highlights how decisions on variable creation and model specification can influence results.

Further, if you are interested in designing successful policy interventions linked to an evaluation, you need to understand the mechanisms by which impact occurs and ideally the type of beneficiary most likely to benefit from a programme (by exploring the heterogeneity of impact). A replication is an opportunity for carefully considering an intervention and moving beyond what is done in a paper.

Of course, if the objective is simply to ensure the integrity of the result the AER standard is fine. But 3ie seeks to use evidence to inform policy and programming. The replications you fund should be more critical and look beyond what is done in a paper by reconsidering the theory of change and checking the robustness of results.

Ben: Finally, a few researchers have expressed concerns about the usefulness of internal replication research. As a researcher who approached replication research without many expectations, what have you learned from your experience?

Paul: I believe it is a worthwhile undertaking. At a practical level, it would have been helpful to know what type of data were available for the replication right at the outset. We had hoped to use the data to explore the causal mechanisms of impact and heterogeneity of impact and this was in our replication plan. But those data were not available. It should be clear that GS did provide the data and what they provided is consistent with the AER standard. Further, if someone asked me for data on a previous project, especially a number of years later, as was the case for GS, I probably could not have provided much more. Of course, this points to a potential benefit of such replications. It could make us more careful about documenting all of the steps we take in our impact evaluations.

To download Paul’s co-authored replication study in 3ie’s Replication Paper Series, click here.

For highlights of 3ie’s replication consultation event, click here.

For more information on 3ie’s replication programme, click here and for the summary results of a recent systematic review on land titling click here.