Monthly Archives: March 2014

Institutionalising evaluation in India

UN Women-Flickr-NREGA-7513997028I recently attended the launch of the Independent Evaluation Office (IEO) in India – an event that marks the beginning of an era of evidence-based policymaking in India.

Are we being overly optimistic and maybe somewhat naïve in thinking this? Maybe.

The launch event, which happened in Delhi, included an eclectic mix of presenters and panelists consisting of key policymakers (including the chairperson of India’s planning commission), bureaucrats, India-based researchers and representatives from the Indian media. The discussions at the event brought to the fore several challenges that the IEO will face as it moves forward:

 IEO’s identity: The IEO’s first challenge is to clarify its main role and objectives. The discussion at the launch highlighted divergent expectations about the IEO’s role. While the media expects the IEO to become a channel of accountability for the government, the bureaucrats in the audience emphasised the IEO’s role in increasing the coordination between different ministries and departments for better delivery of social programmes.

However, researchers and monitoring and evaluation experts debated the relative importance of monitoring and process evaluations compared to impact and outcome evaluations .

There are clearly many expectations of the IEO right now. So, what is the IEO’s role and what is its mandate?

A good way to answer this identity question is to have a theory of change. A theory of change for the IEO should explicitly mention the key outcomes it seeks to achieve, its thinking behind those outcomes and how it expects to get to them. A clear theory of change will help define the IEO’s identity and clarify its scope and objectives.

The Implementation challenges:  What is the use of an evaluation if the the programme is not implemented properly? The director of the IEO, Dr. Ajay Chibber, refers to the problem of poor implementation as India’s Achilles heel. Should the IEO make implementation an   important focus of its work? Yes, the IEO will explicitly have to tackle this problem and one way of doing this is by having strong monitoring and evaluation frameworks for different programmes and policies.

Yamini Aiyer, director of the Accountability Initiative at the Centre for Policy Research says that improving implementation would involve increasing the efficiency of the ‘murky middle’. The ‘murky middle’ consists of middle to lower level bureaucrats who are actually tasked with managing the money flow from the planners to the implementers. The issue here is not the amount of money but how the money reaches its intended beneficiaries.

Diverse contexts and programmes: “In India, programmes always work in certain regions and they always fail in some others. Is there something in the DNA of these regions that make implementation a disaster in these regions?,” asked  a speaker at the event. The question resonated quite strongly with many in the audience.  But the question itself missed the point because it was looking through the wrong end of the pipe.  If we have learned only one lesson in poverty reduction programmes and international development, it is that context matters: programming must take into account local social and structural factors, such as culture, gender, poverty, education, as well as how local bureaucracies work (or don’t).  This is true of several national policies, such as the National Rural Employment Guarantee Act, National Rural Health Mission, and the National Rural Livelihoods Mission. Evaluations of these programmes need to be considered at the state, or at least the regional level. It was clear at the event that Dr. Chibber understands this basic requirement for sound programming (and evaluation).   The fact that he envisages regional offices for the IEO is a major step in this direction.

Using evaluation findings:  “India has a small appetite for evidence,” said a researcher from India. The generation of evidence is an important challenge. But its use is an even bigger challenge. Political expediencies are a crucial factor in determining the longevity of programmes. In India, we tend to immortalise programmes by naming them after leaders and politicians. So, you have the Mahatma Gandhi National Rural Guarantee Scheme, Jawaharlal Nehru National Urban Renewal Mission, or the Rajiv Gandhi Scheme for empowerment of adolescent girls.

In a country where policies are determined by political compulsions, Dr. Chibber’s commitment to releasing the IEOs reports simultaneously to the government and to the media is a great idea. This will hopefully lead to better and more informed decision-making.   The IEO can also use its convening power to help generate debates and policy dialogues around its findings.  They will need to find champions at all levels of government and cultivate strong relationships with and find allies among researchers, the media and civil society.

The independence of the IEO: Even though the IEO is a government entity, the word independent in its name is very important. Being truly independent is going to be challenging.

CONEVAL the Mexican government’s independent evaluation office is an illustration of how this can be made possible. Set up about a decade ago, it has now grown to be a successful evaluation institution that is a valued source of evidence that does contribute to evidence-informed policymaking. Dr. Gonzalo Hernández Licona, executive secretary of CONEVAL was at the IEO launch and also spoke about the importance of maintaining independence and how they worked to maintain it in his agency.  It is important for key government players to know that the IEO has equivalents elsewhere in the world that are working well and producing valuable information for governments.

As the world’s largest democracy takes its first steps in the direction of institutionalising evaluation, we at 3ie are hopeful and cheering for the IEO’s success. We think that the IEO can become a powerful and influential institution in its own right.  We realise that policymaking is a complex process and evidence is only one part of the decision-making equation.  But the IEO can ensure that the evidence is there and that it is sound and worth serious consideration in decision-making. We wish the IEO of India lots of success and hope that it becomes the next CONEVAL – a role model that other countries can emulate.

How much evidence is enough for action?

flickr_Krypto_4837923050One of the most useful ways in which evidence from rigorous evaluations can be used is to help policymakers take decisions on going to scale. Notable recent examples of scaled-up interventions based on high-quality synthesised evidence are conditional cash transfers programmes and early child development (pre-school) programmes. Both types of programmes have been piloted and tested in many places effectively, as shown in systematic reviews on cash transfers and early child development.

The global scaling up of development programmes needs to be based on systematic reviews and not evidence from single studies.  This is because single studies – even rigorous studies like randomised controlled trials (RCTs) – can be biased. According to Dr Philip Davies, 3ie’s head of systematic reviews, “Single studies can misrepresent the balance of research evidence. They illuminate only one part of a policy issue, they are sample-specific, time-specific, context-specific and are often of poor methodological quality”. While high quality single studies can still be useful to inform scaling up the specific programmes on which they are based, they are biased when attempting to make more generalisable statements, such as on whether or not to go to scale elsewhere.

The value of systematic reviews has been fully embraced for a number of years now, including major development funders such as the Department for International Development, the Bill and Melinda Gates Foundation, the US Agency for International Development and others. A number of global organisations are dedicated to producing rigorous reviews, including the Cochrane Collaboration, the Campbell Collaboration’s International Development Coordinating Group  and 3ie. Importantly, the systematic review community, thanks to Archie Cochrane, embraces the philosophy that reviews need to be updated regularly, and that including new studies and doing new reviews are crucial to ensuring that the evidence is as robust, validated and useful as possible.

Economists at the Abdul Latif Jameel Poverty Action Lab at the Massachusetts Institute of Technology and its sibling organisation, Innovations for Poverty Action (IPA), have conducted hundreds of RCTs in areas such as agriculture, microfinance, governance and public health. Their focus is on using RCTs to provide rigorous causal evidence on what difference development programmes make. It is not an exaggeration to say that this group has changed the ways in which development micro-economists undertake field research and also how a good number of governments and NGOs conduct pilot programmes. They have made a positive difference to many single programmes based on these rigorous studies.

IPA has just announced a spin off organisation called Evidence Action, launched to scale up programmes to reach millions of beneficiaries. According to the website, the principles of Evidence Action include ‘Only scale interventions whose efficacy is backed by substantial rigorous evidence’. So they are currently focusing on two initiatives they believe to have already been proven effective: Chlorine Dispensers for Safe Water and Deworm the World.

The website states that ‘regular deworming treatment reduces school absenteeism by 25 per cent’. This is an impressive figure, until one realises it is from a single study conducted in Kenya. They also state that ‘chlorination has been shown to reduce diarrhoea by 40 per cent’. The figure is based on results from a systematic review of household water treatment from 2007. Very soon after this review was first published, evidence came to light suggesting child diarrhoea impacts from such studies were severely biased, as also recently discussed by Givewell.

But IPA researchers are scaling up chlorine dispensers in neighbouring Uganda, and have received funding from the Gates Foundation to scale up in Haiti and elsewhere. School-based deworming programmes are being scaled up in Ethiopia and the Gambia, and across India. The World Food Programme is partnering Deworm the World in 12 countries where it operates school feeding programmes.

IPA states that it bases its scaling up campaigns on substantial evidence, which we take to mean more than single studies, which is good. It’s just that in these cases, it doesn’t look as though they are looking at the full body of substantial and rigorous synthesised evidence for either intervention.

In contrast, the balance of evidence does not support scaling up of either one.

The impacts of deworming are context specific, and sustaining impacts means treating children every year. As the Cochrane Collaboration stated “it is probably misleading to justify contemporary deworming programmes based on evidence of consistent benefit on nutrition, haemoglobin, school attendance or school performance as there is simply insufficient reliable information to know whether this is so.” IPA and the World Bank have commented on that review here and here.

Health impacts of water treatment are likely to be smaller, and may not exist. And sustaining adoption (which is necessary for sustaining impacts) is problematic. People frequently don’t like the taste of chlorinated water. It is also very difficult to get children’s carers to change behaviour when the main benefits of a new technology, such as reducing a child’s disease rate, are hard for them to observe. Moreover, while a single study in Kenya may have suggested impacts, the balance of systematic evidence suggests adoption and health impacts from water treatment typically are not sustained.

One of the problems we face in international development is that the results of effective interventions are not shared in useful ways with the right decision-makers, in the right places at the right times. So, the idea of an organisation getting the word out about proven interventions and building support for scaling up is a welcome one.

Our concern, based on Evidence Action’s initial campaigns, is that the evidence demands firstly that they consider more than single studies, and secondly they use the most up-to-date synthesised evidence. We hope IPA and Evidence Action take context into account when campaigning for particular interventions, do sufficient further research on single studies (not just from their own work) and use the latest systematic reviews before deciding that they have enough evidence to go to scale.