Monthly Archives: October 2017

Strengthening impact evaluation ecosystems by supporting local research teams

IFLA-HQBuilding a culture of evidence is a tall order, one that demands the engagement of different stakeholders committed to evidence-informed policy. While we embed capacity-building activities in our grant programmes, we continue to explore alternative approaches beyond our grants to increase local researchers’ familiarity with impact evaluation, so the pool of research centers able to provide impact evaluation services in a given country expands. Such an approach has the potential of building a community of professionals around the generation and use of rigorous evidence; and even building or expanding the local market for impact evaluation services.

We initially attempted to increase capacity with workshops, sharing knowledge and awarding bursaries. But we deemed our response insufficient, given the potential demand and the need for advisory services customized to local needs.

Our answer? 3ie’s policy window advanced (PWA) program.

Local researchers have a deep understanding of context, political economy and the development challenges in their countries. It makes total sense to commission local research teams to design and implement impact evaluations of local interventions. However, one of the key challenges is the limited number of local researchers familiar with the wide range of methodological tools at their disposal for impact evaluation. Even where strong and experienced research centres exist in L&MICs, such as GRADE in Peru, they may be unable to respond the local and regional demand for impact evaluation services. This limitation often forces government agencies and others to look abroad for such expertise.

In the last couple of years, 3ie collaborated with the National System for Evaluation of Management and Results (SINERGIA-DNP) in Colombia, to tackle this challenge. SINERGIA-DNP understands the importance of impact evaluation and funds several studies each year. However, they have struggled to find enough local researchers familiar with innovative approaches to assess government programs being evaluated.

PWA is one answer to the problem. The program we piloted in Colombia paired local research teams with an international expert, who serves as a team mentor or advisor throughout the design and/or implementation of the impact evaluation. The expert became part of the research team and collaborated on design ideas, suggested alternative methodological approaches, and helped solve implementation challenges. Under PWA 3ie contracts the international experts directly, and they report to 3ie. This set-up reaffirms the independence of the evaluation and the perception of the expert as a peer.

Staff from 3ie and SINERGIA-DNP first reviewed upcoming evaluation tenders and identified three projects for the PWA pilot: a family housing subsidy, the Mandatory Healthcare Plan, and an internet access expansion initiative (summaries of each experience can be found here). We posted descriptions of the programmes to be evaluated and a request for expressions of interest from international experts on the 3ie’s website.

3ie and SINERGIA-DNP then selected the mentors based on both their impact evaluation and sectoral expertise, and matched them to respective projects. In the case of Colombia, it was crucial for the mentors to be fluent in Spanish. Our first mentors were Jose Galdo (Carleton University), Antonio Trujillo (John Hopkins University) and Alberto Chong (Georgia State University).

The experience has been very encouraging. The experts gave helpful guidance and inputs to revise the evaluation research strategies, drawing from their own experiences with similar programmes elsewhere. They guided teams through several approaches to robustness checks, set up protocols on how to implement matching techniques, and worked with their teams to address statistical issues. Throughout implementation, the research teams acknowledged the value experts added.

Aligning incentives

PWA has concrete benefits for all parties involved. The model allows government agencies to commission local teams to conduct impact evaluations and still benefit from the contextual knowledge such teams bring to the studies. The commissioning agency also feels more confident about the quality of the final evaluation. Local research teams gain experience in and further exposure to impact evaluation approaches, analysis and reporting that the mentor offers as advice and guidance, which they can use in future contracts. They also have a greater probability of publishing their work. Finally, the experts have a chance to build new or strengthen existing relationships in the country, as well as have an additional publication opportunity.

The PWA model supports 3ie’s objective of strengthening the capacity of impact evaluation researchers in L&MICs, and responds to local needs for expertise cost-effectively and collaboratively. We look at PWA as an instrument for increasing the drive for impact evaluation in L&MICs, by strengthening the supply side of the local impact evaluation market. Additionally, PWA allows 3ie to reach a large number of countries at a fraction of the cost of a full impact evaluation.

The pilot PWA has generated interest amongst other 3ie L&MIC members. Spurred by these initial inquires, 3ie has recently launched a new request for qualifications for international experts. We look forward to collaboratively find different ways to enable the local environment for rigorous evidence and its use in policy. Stay tuned.

Reflections on the impact of agricultural certification on well-being

Neil-Palmer-(CIAT)Carlos Oya and colleagues recently published a systematic review of agricultural certification schemes that stands out for me as useful research for informing policy and programming. Why do I say that? Agricultural certification schemes set and monitor compliance to voluntary standards with the objective of making production socially sustainable and terms of trade fairer for smallholder farmers and workers. The rise of voluntary agricultural production standards and their respective certification schemes has accelerated in recent years. This makes this systematic review both necessary and timely. There have also been several important recent developments in certification standards that increase the review’s relevance. Supermarket chain Sainsbury has launched its own certification scheme, while the Rainforest Alliance and Utz certification standards have merged.

This review is useful for other reasons as well. A single impact evaluation can give us answers that may hold true for a specific context; but the findings may not be more widely generalisable. A quality, mixed-method systematic review, however, draws on a wide variety of evidence to explore whether and how a programme works at each stage of the causal chain. It can also tell us for whom the programme works and why. Carlos’s review took a considered approach to a complex area. It drew on a wide evidence base – 43 quantitative impact evaluations that explored the effectiveness of certification schemes; and 136 qualitative studies that shine the light on the barriers to and enablers of effective certification schemes.

I recently had interesting email exchanges with Carlos, Antonio Capillo (Fairtrade Foundation) and Harveen Kour (Fairtrade International).  I asked them to comment on the review’s findings and give their views on wider implications for agricultural certification schemes. All of them reviewed this blog and approved of its content. Here are highlights.


Daniel: Congratulations, Carlos, on completing your review on the impact of certification systems on well-being. Can you tell us a little about the main findings from the review for policy, certification schemes and for research?

Carlos: Evidence suggests that although certification schemes improve prices and income from agriculture, they do not automatically lead to an increase in household income and wages, nor do they always improve education and health outcomes. This may reflect the fact that certification schemes are more likely to be effective in improving some immediate outcomes, while more indirect outcomes and longer term impacts are affected by a wide array of contextual factors that may make certification less effective than expected.

Certification schemes can be quite diverse. For instance, some schemes implement capacity-building initiatives that focus on empowering producer organisations and strengthening their position in the value chain, while others aim to contribute to yield improvements. Some include direct price interventions while others focus on interventions to improve quality and help producers access more remunerative markets.

The review also highlights the need for certification to extend relevant labour standards to all forms of agricultural production and establish enforceable labour standards. This would of course need to be complemented by strong national and international systems regulating trade and labour conditions in global supply chains.

The review also shows the paucity of rigorous evidence on impact. The number of certification schemes for which evidence is available is limited, reflecting a bias towards a few schemes, especially Fairtrade. More importantly, much can be done in terms of improving the quality of existing impact evaluations through theory-based approaches that ensure outcomes are measured consistently along the causal chain. We also need better reporting of methods and statistical procedures to improve the credibility of the evidence base.

Daniel: What do Fairtrade regard as the most important findings from the review, and how might you use them to inform your policies and programmes, and your monitoring, evaluation and learning framework?

Fairtrade: While it is positive that certification schemes can lead to better prices and increased incomes, the study is a crucial step to inform the way forward. It helps the Fairtrade movement and the sector focus greater attention on challenging the market dynamics that still limit farmers’ and workers’ access to decent livelihoods in global value chains.

It is of great relevance to us that the review did not find rural certification schemes to positively impact primary wages for workers engaged in certified production – winning progress towards living wages is a cornerstone of Fairtrade’s global strategy. We have been strengthening our work in this area through our regular Standards reviews and engagement with partner organisations, companies, producer representatives and trade unions. For example, the recent revision of our standards requirements for flower farms aims to tackle extremely low wages in the sector. But it is important we don’t do this in isolation, and that’s why we are actively involved in the Global Living Wage Coalition.

We wholeheartedly agree with the finding that context is particularly important in achieving sustainable impact. It supports our advocacy efforts and our drive for more tailor-made programmes to complement certification. Fairtrade recognises that the systemic and pervasive nature of certain core challenges in supply chains – such as gender inequalities and poor livelihoods for informal labour force – require context-driven interventions to address these deep rooted issues. We are therefore designing programmatic interventions that aim to achieve sustainable impact, are scalable and address endemic issues within key supply chains.

Last but not the least, the overall conclusion of the review is that there is limited and inconclusive evidence on the effects of certification schemes.  This finding is crucial for the sector. These recommendations are particularly helpful to Fairtrade as we renew our commitment to promote reliable and open-access research and evaluation and strengthen our research protocols and policies. We are a learning organisation. That’s why we will continue to invest in robust monitoring, evaluation and learning systems to assess quality and effectiveness of our interventions and adapt according to what works.

Daniel: The sustainability landscape is changing, with some certification schemes merging. At the same time, major companies are launching their own in-house schemes. What lessons can we draw from the review regarding the rapid developments in this field and how could future research address the new challenges and opportunities they bring?

Carlos: Our review documents the growth and proliferation of certification schemes and standards. This growth has led to instances of producers being certified by different standards systems simultaneously, despite sharing many objectives and requirements in common.

As the demand for ethically-produced products expands, more actors, including retailers, are compelled to certify compliance with ethical standards across a wider range of products. Since certification is costly for both producers and standards systems, pressures to merge or rationalise the adoption of standards are understandable and may potentially lead to less complex options for consumers. This can be a positive development as long as new (merged or joint) standards add to what is already being achieved by existing standards systems.

From the point of view of researchers, overlapping processes and schemes make impact evaluation more challenging as the number of confounding factors multiply, making it harder to attribute outcomes to particular schemes or interventions. A narrower set of standards systems with more clearly defined interventions could facilitate the evaluation of each intervention within a given certification scheme. Clearer distinctions between schemes and types of interventions would make it easier to attribute impacts to certification. Of course, all certification systems must be held to the highest standards of performance and be subject to rigorous and independent assessment.

Fairtrade: Our message to brands, traders and retailers is that to be truly successful in dealing with today’s most urgent issues, such as living incomes and wages for producers, none of us can act alone and everybody has to play their part. Our offer is that of collaboration and partnership.

Whilst we welcome the ambition of companies to do more to take responsibility for sustainability in their supply chains, establishing in-house initiatives requires significant investment and expertise.  We would urge any business to work with the sector and to ensure they have processes in place that mean they can be genuinely accountable for their ethical claims to farmers, producers and consumers. Hence, we would also encourage companies to open their doors to independent researchers to assess and measure the effectiveness of their schemes, too.

We also have common interests with many certification schemes. ISEAL provides an important platform for all of us to exchange ideas, learn from best practices and explore innovative ways to disaggregate impact, which is particularly relevant, as many producers opt for more than one scheme. Given the complexity of the changing landscape we find ourselves in, we need more collaboration between researchers and certifiers. We would also be open to exploring investing in methodologically sound research jointly with others to increase our effectiveness and to deliver learning that is applicable at scale and can be put into practice as soon as possible.

To find out more, readers can explore the full systematic review, a summary or a brief.