Evaluation departments of development agencies have traditionally jealously guarded their independence. They are separate from the operational side of agencies, sometimes entirely distinct, as in the case of UK’s Independent Commission for Aid Impact or the recently disbanded Swedish Agency for Development Evaluation. Staff from the evaluation department, or at least the head, are often not permitted to stay on in any other department of the agency once their term ends. The reasoning is that their work should not be compromised by wanting to please future colleagues or bosses in other departments. The main argument for independence is that it is necessary to ensure high-quality evaluations that tell it like it is.
But the argument for independence is overstated for many reasons.
Institutional independence does not necessarily safeguard against biases toward positive evaluation. These can stem from poor quality evaluation designs, contract renewal bias, and ‘friends’ bias (an evaluator forms friendships with those responsible for the programme). These biases are best overcome by having a good quality assurance system that includes external peer reviewers.
And independence comes at a cost. If the evaluation agency is totally outside of the agency it is evaluating it can lack access and influence. That is, it will not have the same understanding of how the agency works. It will also not have complete access to internal project documentation for the purposes of the evaluation. I have evaluated World Bank programmes from both inside and outside the World Bank and can testify that there is a huge difference in the documentation to which you have access under these different circumstances. Influence is greater as an internal agency can keep up sustained communication during and after the study. And all agencies have a tendency to take reports from their own agency more seriously than they do those by outsiders, who can readily be written off as not really understanding the programme or even the institution.
And finally, what agency evaluation departments do is only a small part of the evaluation story. The vast majority of evaluations in all governments and agencies are those commissioned and managed directly by line ministries and operational departments. If independence means having nothing to do with these ‘operational’ or ‘self’-evaluations, then that is possibly a great loss for the quality of evaluations.
Luckily, some agencies see this differently. At the Inter-American Development Bank, the Office of Evaluation and Oversight reviews the proposed monitoring and evaluation (M&E) systems of projects at the design stage, rather than waiting around to criticise them once the project is over. At the UK’s Department for International Development the role of the evaluation department has evolved considerably in the last few years through the creation of an accredited evaluation cadre across the agency and running a quality assurance panel for operational evaluations.
Going further still, Mexico’s CONEVAL (National Council for the Evaluation of Social Development Policy) is responsible for reviewing the M&E plans of all government agencies implementing social programmes. CONEVAL sits institutionally in the Ministry of Social Development, thus seeming to compromise the principle of independence. When I asked its director, Gonzalo Hernández Licona, where he stood on the independence versus influence issue, he replied it is far better to be inside to get access and possible influence. I agree with him.
Tags: evaluation, government