Let’s bring back theory to theory of change

| April 19, 2016

Anyone who has ever applied for a grant from 3ie knows that we care about theory of change. Many others in development care about theory of change as well. Craig Valters of the Overseas Development Institute explains that development professionals are using the term theory of change in three ways: for discourse, as a tool, and as an approach. And indeed a google search for theory of change in international development returns images of elaborate diagrams of logical frameworks, long documents describing policies and approaches, manuals presenting tools, and blog post after blog post. Theory of change has come to mean many things to many people, and in the process it has lost its soul: theory.

We need to bring theory back. Theory is what allows us to learn cumulatively from one programme to another and from one evaluation to another. Theory is our best hope for the external validity of any impact evaluation and for using small pilots to indicate what might happen at scale. Theory is the best way to design a programme with the highest probability of working and the lowest probability of causing unintended bad things to happen.

What is theory?

Academic circles take the concept so much for granted that I had to go looking for a good definition. I found this one: a theory is ‘a well-substantiated explanation of some aspect of the world, based on a body of facts that have been repeatedly confirmed through observation and experiment.’ (Thank you, Wikipedia.) Wikipedia goes on to explain, ‘The strength of a theory is related to the diversity of phenomena it can explain, which is measured by its ability to make falsifiable predictions with respect to those phenomena. Theories are improved (or replaced by better theories) as more evidence is gathered, so that accuracy in prediction improves over time…’ This definition reminds us about what is needed for an explanation to be considered a theory and also why theories are so useful. A well-founded theory is one based on evidence gathered over time, and it can be used to make predictions. This predictive power of a theory, which we get from repeated tests of it, is a key element to establishing external validity, or the ability to apply what we learn about what works from one situation to another.

Assumptions ≠ theory

Quote-1We often see ‘theory of change’ presented simply as a results framework or logical framework, sometimes augmented by assumptions, which are intended to provide the theory. Vogel (2012) explains, ‘The central idea in theory of change thinking is making assumptions explicit. Assumptions act as “rules of thumb” that influence our choices, as individuals and organisations.’ But assumptions are not theory. Assumptions do not influence our choices. The predictive powers of most theories rely on making assumptions, but simply assuming that an activity will lead to a result is no better than employing the good-things-must-cause-good-things theory or the if-you-build-it-they-will-come theory.

From arrows and assumptions to theory

The figures below depict the distinctions I am making. Figure 1 shows a logical framework for a programme to provide literacy training to adults. Figure 2 shows some of the assumptions that we make in following the logic of the arrows in the framework. Figure 3 shows the underlying theories that explain (or predict) whether the programme will work.

Figure 1. Logical framework for an adult literacy programme

Figure1-logical-framework

Figure 2. Assumptions implicit in the logical framework

figure2-assumptions-logical-framework

Figure 3. Theories underlying the design of the adult literacy programme

figure3-theories-adult-literacy-programme

When we think about the programme in terms of the theories underlying the causal chain, we have a richer picture of what is required for the logical framework to work. We should not just assume that adults will seek training to become literate. We should understand how they are able to optimise their welfare in their current environment. This allows us to predict whether they will make the investment in literacy training as the result of a rational choice. We should not just assume that any training about reading will lead to literacy. We should understand how adults learn to read so that we can design a training programme most likely to achieve an improvement in literacy outcomes. We should also not just assume that individuals who learn how to read will be able to find and get better jobs for higher incomes. We should understand the features of the local labour market so that we know there is an excess demand for literate workers at jobs with higher wages. We should also understand the local labour market well enough to predict whether or not an influx of new literate workers will drive down wages.

You may be thinking that’s a lot of theories for one simple programme. You are right. Development is complicated.

How we get it wrong when we ignore theory

Here are two examples of how we can get things wrong when we ignore theory. The first example is an edutainment intervention. I am on the research team for the impact evaluation of a television soap opera designed to address violent extremism (not funded by 3ie). When we began to design the impact evaluation, we looked carefully at the programme – the television programme, that is. The outline was the same as any soap opera. There are bad people; there are good people; as viewers, we are supposed to dislike the bad people and like the good people. I wondered, why should we believe that bad people viewing this soap opera will dislike the bad people and change their own behaviour as a result? I never became a better person from watching all those episodes of All My Children in graduate school.

I looked to my own discipline, economics, for an explanation and couldn’t find one. I turned to my political scientist colleague on the study team. He didn’t have an explanation either. So we turned to psychology. It turns out the theories in psychology do address how observing others’ behaviour might influence our own, but the theory supported by empirical evidence, i.e. social learning theory, does not support the standard soap opera approach. It predicts that we model behaviours we observe, but that we model both the good and the bad. There is no reason to predict that when we observe both the good and the bad in a soap opera, we only model the good, even when the bad might be punished. Instead social learning theory supports what my colleague and I call the Remember-the-Titans approach, where the drama depicts primarily the positive behaviour that we want observers to model (albeit with some tension to make the story interesting). After identifying the relevant theory, we suggested some changes in the soap opera story to the programme team.

The second example comes from the agriculture sector. An intervention that has been replicated in many countries over the years is the farmer field school. The farmer field school is a programme designed to increase the productivity of small-scale farmers by teaching them better agricultural practices, such as fertiliser use. The design of the training itself is based on adult learning theory. The premise behind the sustainability of the farmer field school model, and the claim for its cost effectiveness, is that there will be positive spillovers. Farmers who attend the school and learn the better practices directly will teach these practices to their neighbours who do not attend the school so that all farmers in the community will ultimately benefit from the programme.

The question is, why would any self-respecting economist predict that a rational agent who gains a productive advantage would share that advantage with her competitors? In fact many training programmes rely on the assumption that people who receive training will pass on that knowledge to others creating positive spillovers, or indirect benefits. For some kinds of training (e.g., political mobilisation, sexually-transmitted disease prevention) that assumption makes sense. But when markets are involved, we should not ignore competition unless we have a good reason to rule it out.

In the case of farmer field schools, a 3ie systematic review shows that the intervention produces positive outcomes for those who attend the school, providing evidence in support of the adult learning theory. But there is no evidence that neighbouring or non-participant farmers have improved outcomes, supporting, in a sense, economic theory.

Whose job is it anyway?

As I mentioned above, we at 3ie often press impact evaluators to provide a detailed theory of change for the programmes they are evaluating. Unless they designed those programmes, however, the responsibility for the theory is not really theirs. They have to take the programme as it is, and all too often, we see development programmes with no more than (and sometimes less than) results frameworks supporting them. Does that mean we should not require the impact evaluators to design their evaluations around theories?

No, it doesn’t. Remember that a theory is an explanation that is repeatedly confirmed using experiment and observation. Whenever possible, we want to increase the return on investment from individual impact evaluations by having them contribute evidence that helps to improve the predictive power of these explanations, even where the improvement means showing that the theory is wrong or does not apply. The more evidence we can lend to theories, the better the theories we have for programme designers to use the next time around. If an impact evaluator cannot suggest a clear theory or set of theories on which the programme is based, then we need to recognise that this limits (but does not eliminate) what can be learned from that impact evaluation.

So what can we do?

How to fully solve the problem is a longer discussion for perhaps another blog post. One part of the solution, though, is that we need to make sure that our programme design teams and our impact evaluation teams are able to draw on the social scientific expertise needed to explore all the relevant theories. They need to draw on this expertise to understand the body of evidence that supports, or not, the application of those theories to the challenge at hand. This is not easy to do. Development is complicated, and often theories from several disciplines apply. But we are not going to be successful if we rely on the ‘logic’ that good things cause good things to happen or if you build it they will come. Let’s bring theory back to theory of change.



Tags: , , ,

3 Comments on “Let’s bring back theory to theory of change

  1. Steve Powell

    Annette, it is great what you are saying … it isn’t enough to want A, be able to do B, and then just hope that B will lead to A. In order to claim such a link it has to stand up to the cold light of evidence. But I think you might be guilty of a bit of a related kind of wishful thinking. Namely, how much uncontested evidence do we really have in social, political and developmental sciences which might be used to reinforce theories of change (or not?) It would be great if programme planners and managers could just click through Google Scholar or 3ie and peel off the relevant evidence for any given issue, but it isn’t always that simple is it? I happen to be a psychologist and there really isn’t a single uncontested theory or evidence base for social learning / model learning … the trouble is that in some areas, the more you know about it, the less convinced you are that everything is that simple. In my turn, I have surely been guilty of oversimplification when looking for theories and evidence from your field, economics. Particularly in the social sciences as I am sure you know there is currently such a crisis of replicability that some wonder whether we will ever be able to say we have assembled a body of reliable knowledge on, say, learning theory.
    3ie does a great job of generating and collating evidence about highly reproducible causal links, i.e. links which where there are plenty of cases and contexts are similar to one another, e.g. education. But even with these kinds of contexts, applicability to an individual case or program isn’t always that straight forward is it?

  2. heather

    thanks for a nice post, annette. i just want to re-highlight a few points that i think you make well but can perhaps be even more pointed.
    1. theory and theory of change are not equivalent. theories of change order and deploy theories to help explain how change is expected to be implemented, engaged with, and successful in a specific context. theory strengthens theories or models of change substantially but this doesn’t mean that the two are interchangeable. it does not suffice to say ‘well we have a theory section’ in our proposal or the resultant report or publication.
    2. theories of change benefit from bringing together theory and theoretical constructs from multiple disciplines.
    3. logic and reasoned intuition still plays an important role in fleshing out a theory of change that well-describes why and how a program is anticipated to bring about change — again, theory alone is incredibly important but learning transferable lessons (or at least lessons that those in other contexts can decide whether they may transfer successfully) also depends on identifying and reporting contextual features, etc, that also determine whether a program worked ‘there’ and may also work ‘here.’ continuously doing development better requires learning about when theoretical and operational assumptions don’t hold and why.

  3. Wayne C White

    In addition to laziness and wishful thinking, I think the industry often dumbs down theory of change to avoid conflict. Our modern western industry avoids possible contention on the role of culture, host country power realities, etc. Would be great if we were mature enough to air and maturely debate real drivers of dysfunction or institutionalized poverty.

    It would be great to have a rational theory of change on the subset of how people can be positively influenced. This actually could become generally accepted following a healthy debate.

    Where a large bilateral donor “signals” their ToC in the RFP, it becomes bid-suicidal for the bidders to actually suggest a fresh, localized ToC rather than echo what the donor wants.

    You are onto something, but there are these obstacles, and probably more, requiring innovation and courage to pass.

Comments are closed.