Failure is the new black in development fashion: Why learning from mistakes should be more than a fad.

| April 22, 2014

Blog PixDuring a meeting at the Inter-American Development Bank (IADB) last week, I mentioned the UK Department for International Development’s moves toward recognising failure, and the part recognizing failure has in learning (see Duncan Green’s recent blog on this).   Arturo Galindo, from IADB’s Office of Strategic Planning and Development Effectiveness, responded by picking up a copy of their latest Development Effectiveness Overview, and opening it to show the word FAILURE emblazoned across the page in large yellow letters.  The word failure appears 27 times in the report, compared to just 22 times for the word success.  This doesn’t mean that IADB is in any way a failing institution. Quite the opposite. They are putting into action the same principle emphasised by Ian Goldman from South Africa’s Department of Performance Monitoring and Evaluation: incentives should not penalise failure but failing to learn from failure.

Impact evaluations have an important part to play in learning from failure.  Eighty per cent of new businesses fail in the first five years: a fact that holds true generally across the world. Do we really think that public sector and NGO programmes do any better?  Failing development programmes survive because they don’t face the bottom line in the way unsuccessful businesses do. So, how does one estimate the bottom line for development programmes? Traditional process evaluations are subject to various biases which make them less likely to point to the harsh reality of failure (as discussed in my 3ie working paper with Daniel Phillips). Impact evaluations are not subject to these biases. Impact evaluations are the bottom line for development programmes.

In the past, development agencies have shied away from acknowledging failure. They have cherry picked the projects and programmes to learn from best practice. In doing this, they ignore the fact that we should also learn from our mistakes.  But now, some agencies are doing impact evaluations on a serious scale. A few years ago Oxfam GB, instituted a new results system which includes 30 new impact evaluations a year on a random sample of their projects. Last year, close to 60 new IADB projects – that is nearly half of their total projects – included impact evaluations.  The systematic collection of results from these studies as they are completed will start to give more accurate pictures of the agency’s performance.

The intention to learn from failure signals an important step in the use of impact evaluation. Agencies have started producing impact studies, but not really thought about how they fit into their overall learning and accountability frameworks. In an earlier blog I wrote about the unhappy marriage between results frameworks and impact evaluation. In fact, they have not even been dating.  So, systematic attempts to draw lessons from impact evaluations – as IADB does in the 2013 Development Effectiveness Overview – should be lauded.  These attempts take us further down the road of improving lives through impact evaluation.



Tags: ,

5 Comments on “Failure is the new black in development fashion: Why learning from mistakes should be more than a fad.

  1. Heather

    Hi Howard: Thanks for bringing up the importance of discussing successes as well as failures in development. It’s not clear to me, at least yet, that organizations aren’t able to cherry pick their reported failures, too, when they are under no mandate to report on everything.

    Reported “fails” often seem to tend towards identifiable problems with clear lessons-learnt and relatively easy tweaks. They focus on implementation #fails over design fails, and design fails over any consideration of the underlying theory of change, philosophy, or ethics.

    Perhaps I am being cynical in thinking that organizations tend to only address failure when it makes for an amusing story at a conference or TED talk, when there is a key teachable moment, and when the organization has largely already figured out the way forward.

    Also, in writing this, I found myself tending towards “organizations” — and that is my sense, that the (selective) embrace of failure tends to be a donor phenomenon, maybe followed by new/innovative/agile NGOs and NGOs that are relatively secure in their funding and can afford a teachable fail or two, especially if it yield a new insight that should definitely be funded and tried (!).

    One test of the productivity of the trend of failing will be, I think, how donors react to more open discussion of their recipients’ different types of failures.

    But I suspect the bigger test of whether the present framework of admitting and addressing failure is productive, if we are trying to move towards policy relevance and the government/endogenous scaling of programs, is whether government bureaucracies and ministries can cultivate a culture of innovation, failing, and learning (and whether donors and other agencies can support this).

    Otherwise, I wonder if most of the failure stories will be told in DC- or London-conference halls, about that one time, in that one context, that John thought blue cookstoves would be a good idea, and it was so funny because everyone wanted red cookstoves and that would have totally solved whatever problem it is that cookstoves were positioned as the solution to at that moment…

    Reply
  2. Pingback: Failure is the new black in development fashion...

  3. Luc Lapointe

    Dear Howard!

    Having had the opportunity to look closely at some of the work of these institutions it’s very easy to see why projects fail and why most often impact is associated with the accounting process than meaningful long lasting change. Most of the traditional donors don’t have the ability to adapt to a fast changing situation and hardly recognize soft skills of local contractors ….which quite often would work produce better results. They are also engrained in the idea and the need that every projects needs to scale up instead of adjusting to the different realities of a complex world and diversity of communities.

    I am not a fan of celebrating failure with public money when we still have so much to do. Maybe we should have a “ticker box” on tax filing that allows us to pay more taxes for those who think that failure is a good thing. I am glad the airline industry is not using the same principle.

    Luc Lapointe

    Reply
  4. Taitos Matafeni

    Hi Howard
    Great blog on an interesting subject. I definitely see the role of impact evaluations in helping organisations evidence success and failures in development interventions. However the cost and technical design for such evaluations is often times prohibitive as a source of evidence for a lot of development agencies coz donors are not willing to invest in such. In such a context its important to explore other areas to document, share and learn from failure.

    @Sightsavers I have led a process where we have documented our failures through our inhouse learning publication Insight Plus, which we have also publicly made available to anyone who wants to know where, why and how we failed across a number of initiatives. Just a word of encouragement to other development managers that you do not need huge financial outlay to learn from failure in your development interventions.

    Reply
  5. Ehtisham ul Hassan

    Dear Howard,
    Hi, Many thanks for raising the flag high once again. It’s really interesting to read your thoughts on a very important issue. I think most of the time when we design M&E system we focus on Key Performance Indicators of success chosen for a project and always try to keep our discussion/analysis within the scope of that pre-defined performance markers only because that is a measure of performance due to which Monitoring reports did not report on failures.
    I think when we report success its only one side of the picture where as, there is a flip side of the coin as well where we can find more about the failures. i some times feel that It is just like Probablity of success and filure will be equal to 1 (p+q=1). so reporting on success is only 50% of report.

    Another key challenge is that most of the time a report on failures might be considered as an indicator of performance of an organisation that’s why most of the organisations are not keen to publish about their failures. Therefore, donor organisations should also should be keen to learn from failures and challenge their theory of change in practice.

    All best wishes

    Reply

Leave a Comment:

Your email address will not be published. Required fields are marked *