Monthly Archives: September 2015

Tips on using impact evaluation to measure agency performance: applying the triple A principles

UN-Women-8067682747Impact evaluation has grown in popularity as governments and development agencies have come to realise that it is the best way to assess if their programmes work or not. But will these evaluations help politicians, managers and funders know if an agency as a whole is ‘working’?

Some years ago, Howard White proposed a triple A standard for agency-wide performance measurement systems (AWPMS). A recent paper we wrote, applies these ideas to the role of impact evaluation in agency performance measurement. The triple A principles are:

  1. Alignment: The outcomes being measured have to be the same as the outcomes reflected in the agency’s goals. If the agency’s vision statement is ‘a world free of hunger’ then impact evaluations have to measure impact on food security, nutrition and so on.
  1. Aggregation: It should be possible to add up across project-level ratings. This has usually been done the way the World Bank does it: projects are rated as satisfactory or unsatisfactory; so, aggregation is the per cent of the portfolio that is satisfactory. Impact evaluation enters this system indirectly by applying the ‘no benefit of the doubt’ principle by which a project can only be rated satisfactory if there is rigorous evidence to back up that claim. But the holy grail of AWPMS is to be able to say how many people have been lifted out of poverty, how many girls empowered and children’s lives saved directly as a result of the agency’s efforts.
  1. Attribution: Evaluation findings should be able to make credible causal statements linking the intervention to the outcome. This is the trump card of impact evaluation and the reason most agencies still need more of them.

In discussing the application of the triple A principles in a range of agencies, we identified a number of key actions to be taken by agencies planning and implementing a programme of impact evaluations. The experiences of a few agencies have been summarised in papers in a special edition of the Journal of Development Effectiveness.

  • Decide how many impact evaluations need to be done, and decide on a system for choosing them. Some impact evaluations are better than none. You can start opportunistically. But once you are doing a considerable number you need to pick them strategically. If you want to make agency-wide statements at the outcome level, then that strategy will involve representative sampling of programmes for impact evaluation. This is what Oxfam GB has done.
  • Value lesson learning as well as accountability. The purpose of AWPMS is to improve the accountability of an agency. However, impact evaluations also provide valuable lesson learning opportunities that can be missed if the sole focus is on the accountability function.
  • Balance independence and influence. Integrity in evaluation is more important than independence. Too great a distance between the evaluator and evaluee can undermine relevance and limit use of the study findings.
  • Build use into evaluation processes. Both of the last points are ways to ensure that we are not just producing reports that sit unread on shelves. User engagement starts at the design stage, ensuring the study design is useful to the commissioning agency. Good communication and engagement planning from the start also helps increase the means for understanding and using findings.
  • Build better agency incentive structures. The penalties should not be for failure but for failing to learn from failure.

3ie has been at the forefront of fostering a steady increase in government agency commitment to doing impact evaluations. The growing acceptance of their value is an important step forward. Just as importantly, agencies need to also think about how they will use their evaluation findings to measure and improve agency performance. And, for most, that journey is just beginning.

Making WASH behaviour stick

Photo by Jim Holmes for AusAIDSmall scale trials have repeatedly demonstrated the health benefits of water, sanitation and hygiene (WASH) programmes. But the benefits disappear when many of these programmes are taken to scale. Large scale handwashing, sanitation and water treatment campaigns in India, Bangladesh, Peru and Vietnam have not led to a change in the behaviour of most of the people they target. So, the health impacts of these programmes are insignificant.

As we gear up for the Sustainable Development Goals, it is time to take stock of the evidence we have on sustained adoption of WASH programmes. Recent systematic reviews and impact evaluations provide some insights on what does or does not make WASH behaviour stick when programmes are taken to scale.

Why just building toilets doesn’t work

A 3ie-supported impact evaluation showed that providing subsidies and mobilising communities for toilet construction led to a substantial increase in latrine coverage in Odisha, India. But the increase in toilets did not lead to a reduction in the exposure to faecal matter. There was hence no reduction in child diarrhoea.

So, why didn’t building toilets have an impact? Many households did not use the toilets even though they had access to them. Men and children in particular continued to defecate in the open. Changing people’s attitudes about open defecation in India requires overcoming significant cultural barriers. Open defecation amongst particular groups in India is seen as a practice that preserves ‘purity’ since it prevents the accumulation of faeces inside or near the house. Just providing subsidies to households for building toilets does not therefore change cultural norms.

In contrast, a community-led total sanitation (CLTS) campaign in Mali did not provide any financial assistance for building toilets or purchasing any other hardware. An impact evaluation showed that the programme was successful in increasing access to private latrines and in reducing self-reported open defecation among men and women. This resulted in a reduction in child stunting and diarrhoea-related deaths of children under age five.

What made the difference? The approach to community engagement was stronger in Mali than in India. The achievement of open-defecation free status in Mali was also acknowledged in a celebratory public ceremony. But it may well have been that there were no cultural barriers to overcome in getting people to use toilets in Mali.

The lesson here: WASH programmes clearly need to understand the barriers to and facilitators of behaviour change. The ‘just build it and they will come’ philosophy misses the fact that human behaviour and the norms that drive it also need to be addressed and changed.

Many factors affect WASH behaviour

©UNICEF Ethiopia/2014/AyeneA recent 3ie-supported review mines through 44 studies to come up with a host of contextual, psychosocial and technological factors that can affect sustained use of WASH practices and technologies.

As is the case with toilets, just providing water treatment technologies does not automatically lead to a change in behaviour. Making people pay even a very small fee puts them off of adopting the technology. The smell and taste of chlorine can also be a deterrent in adopting water treatment.

Age and gender are other crucial factors in determining WASH practice, particularly in latrine use and handwashing. Safety is clearly a key factor that influences the use of toilets for women. In Mali, women in the CLTS villages found it safer and more private to use toilets. But the evidence on these factors is scant. 3ie’s WASH evidence gap map shows that very few impact evaluations examine gendered impacts.

Children are unable to use toilets if they are very small. Even households that have access to toilets dispose child faeces in an unsafe way. Effective disposal of child faeces is therefore a priority area for improved sanitation and hygiene.

Barriers to behaviour change over time

Barriers to behaviour change depend on the stage of the project. Many studies assess the health benefits of initial uptake of safe water, hygiene and sanitation technologies and practices. But few studies consider sustained use. The early project period may be characterised by enthusiasm over the new technology or promotional activities. Enthusiasm may diminish in the late project period but project staff are still around to resolve issues with respect to cost and availability of the hardware supplies. Although external support ends during the early post-project period, the promotional messages may still be fresh in people’s minds. However, influential household members who were sceptical may reassert their domination during this phase. And finally, in the late post-project period stockouts, technology failure or poor maintenance systems can pose a serious threat to sustained adoption.

Incentives for sustained behaviour behaviour change thus differ from what influences initial uptake. A 3ie-supported study in Kenya shows that local government officials valued the opportunity of taking decisions on how and where chlorine dispensers should be installed within their constituencies. But they did not want to take on the responsibility for maintaining chlorine dispensers.

The nature and frequency of behaviour change communication affects uptake

The effects of one-to-one interactions differ from those of group-based approaches and mass media campaigns. A community programme in Peru combined with a mass media campaign to promote handwashing, was more effective at improving handwashing behaviour, compared to the mass media campaign alone.

The 3ie-supported systematic review finds that frequent, personal contact with a health promoter over a period of time is associated with long-term behaviour change. The review suggests that personal follow-up in conjunction with other measures like mass media advertisements or group meetings may further increase sustained adoption.

What next?

Sustained adoption of WASH technologies and practices can lead to lasting health benefits. What we know now is that just focusing on technological and hardware fixes does not produce behaviour change. Multiple factors affect behaviour but not enough attention is being given to these factors and how they change over time.

The recent synthesised evidence throws up a few important pointers about where we need to prioritise getting more evidence:

  • We need more formative research on the cultural, social and psychological factors that affect WASH behaviour. This can in turn inform WASH programme design.
  • Impact evaluations should analyse the impact of WASH programmes on addressing the multiple barriers to and facilitators of sustained behaviour change.
  • WASH programmes need to address the particular needs of women, children, older and differently-abled people. We need to fill the evidence gap on how age and gender affect uptake and outcomes specific to different population groups.