Monthly Archives: November 2012

Special feature for World AIDS Day 2012

There has been only a small decline in the prevalence of HIV in the last decade, dropping from 5.9 percent to 5 percent between 2001 and 2009 for those aged 15-49 (UNAIDS, 2010). This decrease, whilst important, does not seem impressive compared to over US$5 billion spent fighting AIDS in low and middle income countries each year (the latest available figure is US$5.1 billion in 2008).

There is a wide variety of HIV prevention interventions including behaviour change communication, biomedical interventions like male circumcision and treatment of sexually transmitted infections, expansion of access to antiretroviral therapy and enhanced prevention of mother-to-child transmission services. Do we know if these interventions work or not?

Behaviour change communication in particular has been considered extremely important given AIDS is a disease that is characterised by ignorance and stigma. Communication therefore seems to have a crucial role in informing, equipping and motivating people to make informed choices about prevention and care.

But the evidence on the effectiveness of behaviour change communication is not good. Systematic reviews, which summarize the available evidence from rigorous impact evaluations, show only a minority of programmes have worked. In one case, only two of nine studies on behavioural interventions showed significant protective effects on HIV incidence among women (McCoy et al., 2009).

What can explain the lack of effectiveness of behaviour change communication?  And what are the implications of this result on our thinking about effective HIV prevention interventions?

Cultural norms and poverty act as barriers to the adoption of safe sex behaviour. “Sugar daddies” in sub-Saharan Africa are an illustration of both these barriers. This is a tradition of sexual reciprocity, where young girls have sex with older men in exchange of money and gifts. Young girls engaging in these relationships do not have much of a choice in negotiating safe sex.

Can structural interventions addressing poverty be a viable approach for HIV prevention? Two recent randomised controlled trials of conditional cash transfers show a significant decline in four sexually transmitted infections (de Walque et al. 2012), and a reduction in HIV infections among adolescent school girls (Baird et al. 2012).

But gender is an important factor in determining impact. Financial rewards can have a negative impact on men. In rural Malawi, conditional cash transfers offered to men led them to engage in more risky sex behaviour (Kohler and Thornton 2011). But when conditional cash transfers were combined with individual and group counselling in Tanzania, the incidence of curable sexually transmitted infections reduced among both young men and women (de Walque et al. 2010).

This initial evidence suggests that structural interventions like conditional cash transfer programmes should be tried and rigorously evaluated to assess if they are viable complements to biomedical interventions. Given that behaviour that is rooted in culture may be harder to change, it is imperative to find interventions that work better.

Field notes on implementing impact evaluations

11

3ie is currently funding 100 impact evaluations in low and middle-income countries spread across Africa, Asia and Latin America. We are now in a unique position to learn a lot about what’s working well in designing and conducting impact evaluations and what can be done better to ensure that research produces reliable and actionable findings.

But as grant makers we usually ‘see and experience’ our projects only on paper. We miss out on listening to the voices and perspectives of field workers, project staff and junior research staff. To get a sense of what has been happening on the ground, we recently carried out a field monitoring visit to four 3ie supported projects in diverse sectors in one African country. Most of the visits were to the field site and the meetings mainly with the implementing agency staff.

And we did learn a lot through this field trip, particularly about the relationship between implementing agencies and researchers, challenges involved in implementing a project and an impact evaluation, and the work being carried out to engage stakeholders and disseminate research findings. Some of the lessons we learned raise further questions.

Local researchers listed as ‘Principal Investigators’ on the 3ie grant application had little engagement in the impact evaluation.
This finding was true for all the projects visited. At 3ie, research teams that include developing country researchers receive higher scores in their grant applications. Not surprisingly then, many of the grant applications we receive usually have researchers from the country of the evaluation listed as Principal Investigators. But on the ground it was a different story.

Local ‘Principal Investigators’ may have been involved in determining the main evaluation questions and giving inputs on the context. However, their involvement was certainly not substantial. In one case, the local ‘Principal Investigator’ was significantly involved in the implementation of the intervention but not the impact evaluation.

So why were the local researchers not involved? Were the local researcher names hurriedly added just to meet 3ie’s requirements? Did the lead Principal Investigators think the researchers lacked the capacity to contribute to the impact evaluation? Funders of impact evaluations like 3ie need to address these questions so that they can tweak their own requirements from grantees. We need to reflect more on the ways in which lead Principal Investigators can involve local researchers and build their capacity to conduct impact evaluations.

Researchers need to work with implementing agencies/governments to address challenges related to the implementation of a Randomised Controlled Trial.
Implementing a Randomised Controlled Trial was quite challenging for one implementing agency. During the course of the implementation, some programme participants in the control group felt that they were ‘discriminated’ against. In one particular instance, the discontent among the participants led to clashes with the project staff. Overall, the implementing agency staff felt that they had to compromise on their integrity to safeguard the integrity of research.

So what are the ways researchers can assuage the fears and concerns of both beneficiaries and implementing agency staff?

Getting the buy-in and involvement of implementing agencies is important for generating actionable evidence from an impact evaluation.

One of the projects visited was a striking illustration of how the disconnect between the researchers and the implementing NGO could well have been the main reason for no take-up of the impact evaluation findings. The impact evaluation did not have a clear theory of change. The intervention that was evaluated was poorly designed. But what makes it worse is that the evaluation does not pick up on the fact that intervention was unsuccessful. The NGO has now changed track and moved on to a new programme. The end result: an impact evaluation with no actionable or credible evidence.

If there are many implementing agencies and stakeholders involved in an impact evaluation, getting them to agree on all aspects of the project could lead to delays in the project.
Getting a project off the ground can be a serious challenge if it requires considerable amount of time and diplomatic effort in getting stakeholders to agree on the design of the intervention. In one case, the delay in implementation reduced the duration of the programme. The delay has implications on the findings of the impact evaluation.

Delays in getting an impact evaluation article published can be an obstacle to using evidence

The long wait (as long as a year or more) to get published in academic journals can be an impediment for implementing agencies since there is an embargo on releasing the findings. Implementing agencies want to cut to the chase. They want to go all out, discuss, disseminate and use the evidence from an impact evaluation.

And finally, some implementing agencies think that by conducting an impact evaluation, they will appear as accountable and credible organisations.
An impact evaluation is seen as a gateway to more project funding. While this is not necessarily bad news, the benefit of conducting an impact evaluation should ideally also extend to the production of evidence that is used for designing more effective policies and programmes.

Impact of daycare interventions in Latin America

mexico_daycare_1.jpg__300x200_q80_crop_upscaleUrbanisation and increased female labour market participation have led to increased demand for daycare services, which in developing countries is partly met by government daycare programmes. Some of these programmes offer subsidised community daycare services, in which women from the community provide full time childcare in their home, food and some recreational or educational activities for the children. Other programmes offer public preschool education to children between 3 and 5 years of age. But do these daycare interventions benefit the child’s development?

Impact evaluations of these programmes were undertaken to assess their effectiveness by comparing the wellbeing of children cared for at daycare (or preschool) to those cared for at home. To synthesise the evidence, researchers of the National Institute of Public Health in Mexico (myself and Jef Leroy, currently at the International Food Policy Research Institute) and the Center for Research and Teaching in Economics (Maite Guijaro), undertook a systematic review. The study, The impact of daycare programmes on child health, nutrition and development in developing countries: a systematic review examined the effects of daycare interventions (formal out-of-home care) on the health, nutrition and development of children under five years of age, in low- and middle-income countries.

The systematic review identified 13,190 studies, but only six, based in Latin America, met the inclusion criteria in terms of scope, type and quality. Four studies evaluated community-based interventions and two looked at preschool interventions.

The findings showed that attending daycare had positive effects on language skills, social and emotional development of children in the short run. In the medium term, school attendance, student behaviour and test scores witnessed a positive trend. In fact, the effects on the children were more pronounced depending on the exposure to the programme. For example, the Bolivia daycare programme had a positive effect (2-11% increase) on bulk (gross) and fine motor, language and psycho-social skills for children with more than seven months of exposure to the programme. On medium-term outcomes, the Argentina study found that one year of preschool increased mathematics and Spanish test scores at third grade of primary education by eight percent. In Uruguay, it was found that children who attended at least one year of preschool, increased their schooling by nearly one additional year by the age of 15.

On child health outcomes, only one study from Colombia evaluated the impact on prevalence of diarrhoea and acute respiratory infections. Although this study found reductions in the prevalence of both diseases with longer exposure to the programme, it is not clear if the results are a true health effect of the programme or if the comparison group of children with less than one month of exposure to the programme, might have suffered from a steep increase in infections right after joining a daycare centre.

However, no conclusions could be drawn with respect to the nutrition outcomes. One study from Guatemala analysed child dietary intake and found positive impacts, a study from Bolivia found no impact on child growth, and two additional studies from Colombia found inconsistent results on child anthropometrics, such as height and weight.

Finally, the reviewed studies did not provide a good description of the type and quality of care children receive in the absence of the programme. This represents an important limitation of the reviewed studies since the potential impact a daycare programme might have is determined by the “net” treatment, which is the difference in the type and quality of care between daycare interventions and the alternative forms of child care in the absence of the programme. For instance, a positive “net” treatment effect can be expected if daycare interventions provide a high quality childcare alternative to mothers who take care of their children while working. However, a negative “net” treatment effect could be anticipated if children who receive adequate family care are enrolled into a low-quality daycare programme.

Policy implications

The evidence shows that daycare interventions in Latin America, community-based or school-based, have had a positive impact on child development. However, there is not enough evidence to conclude that these programmes have improved child health and nutrition. Based on this information, should policymakers decide not to implement daycare interventions until there is conclusive evidence about its impacts?

Considering the proven impact daycare interventions can have on improving child development in the short and medium term and the increasing demand for out-of-home care, these programmes should be implemented if they provide a high quality alternative to the care children normally receive.

However, it is crucial that new programmes are evaluated and closely monitored, not only to add to the very limited knowledge base of programme effectiveness and pathways of impact, but also to guarantee that unintended negative effects are identified and corrected.

(Paola Gadsden is the Coordinator of Analysis and Evaluation of Public Policies for the State of Morelos, Mexico)