Monthly Archives: October 2018

Measuring open defecation behaviour in India

Open defecation poses significant health risks for individuals and communities across the globe. The practice affects vulnerable populations through diseases such as diarrhoea, schistosomiasis and trachoma, which often lead to stunting and malnutrition in children. Open defecation is particularly prevalent in India, which is home to 59 per cent of the 1.1 billion people in the world who practice open defecation. It is a major cause of diarrheal deaths among children under age five in India, and constituted 22 per cent of the global disease burden in 2015 (UNICEF 2018).

To overcome the challenges associated with open defecation, we need, first, accurate estimates of its prevalence, and second, rigorous evidence on what works to promote latrine use. This information would be imperative to inform sanitation policy within and beyond the purview of the ongoing national sanitation campaign, the Swachh Bharat Mission (SBM, Clean India Mission), which aims to make the country free of open defecation by 2 October 2019.

Responding to the need for rigorous evidence on what works to promote latrine use in rural India, 3ie launched an evidence programme that supports four impact evaluations of contextually informed interventions, rooted in behavioural science, to promote latrine use. These evaluations are underway in Bihar, Gujarat, Karnataka and Odisha and will be completed in 2019.

The challenges of measurement

Measuring sanitation coverage and uptake is imperative to shaping policy in a country that hopes to be open defecation free (ODF) in a short span of time. Recent estimates provided by the Ministry of Drinking Water and Sanitation, and the National Annual Rural Sanitation Survey (NARSS 2017), report that the SBM (rural) has been successful in improving latrine coverage and use by a significant margin since 2014. However, from ongoing projects, we learn that there are several challenges to the accurate measurement of defecation behaviour in India. Some of the key barriers to this include:

1. Proxies for latrine use and social desirability bias

Latrine use has typically been measured through the construction of latrines. This could be because the presence of infrastructure is relatively easy to verify. However, latrine presence does not necessarily amount to use. (3ie 2017)

Measuring the sustained use of water, sanitation and hygiene (WASH) infrastructure is uniquely challenging in the midst of a nationwide campaign advocating for the importance of ‘correct’ defecation behaviour. More recently, surveys have relied on respondent reported usage of toilet infrastructure, which can be prone to social desirability bias. (Coffey and Spears 2014). Asking questions about inherently private behaviours such as defecation practices could elicit socially desirable responses, especially in ODF declared and verified areas.

For example, in the surveys that feed into the WHO/UNICEF Joint Monitoring Program for Water Supply and Sanitation (JMP), such as the Census 2011, the National Family Health Survey (NFHS-4) and National Sample Surveys (NSS), open defecation is deduced once all other forms of access to sanitation infrastructure have been negated. In addition, the survey questions may lead to biased measurements of latrine use, as they refer to ‘usual’ usage or ‘always’ usage (see box 1).

Box 1: Latrine use questions in NFHS-4 and NARSS 2017

What kind of toilet facility do members of your household usually use?’ (NHFS-4)
‘Does (name) use the latrine always?’ (NARSS 2017)

In order to tackle inflated responses, surveys have begun to rely on the use of structured observations and spot checks; for example, checking for physical signs of use, such as slippers, water containers near the latrine, etc. However, using one or the other may lead to inaccurate estimations. Combining structured observations and spot checks with self-reported latrine use may be one possible solution. But employing the use of both on a large scale may have significant cost implications.

2. Capturing variations within households

Another challenge to the measurement of latrine use lies in the difficulty of adequately capturing variations within households. Surveys frequently fail to unpack usage to investigate individual latrine use due to resource constraints and logistical timelines. Better understanding individual use is important, and a recent investigation of sanitation coverage, usage and health in India concludes that surveys that aggregated to the household level revealed lower levels of partial usage, while surveys investigating individual use found a higher incidence of partial use (Viswanathan 2017). National surveys such as the Census 2011, the NFHS 4 and the NSS also tend to aggregate measured behaviour either to the household, sub-household or demographic group level, overlooking individual behaviour.

While the latest Census (2011) collected usage data indirectly through latrine ownership, the 69th round of the NSS finally introduced questions on use, in particular asking if “all household members of categories specified are using a latrine, followed by the list of categories: men/women above 15 years of age, and men/women below 15 years of age”.

Learning from experiences in measurement, the NARSS (2017) measured both individual and subgroup latrine use. However, it subsumed latrine use into the categories of always, rarely, sometimes and never, conditional on shared and unshared latrines owned by households. This may lead to inaccurate responses, as the terms are not precisely quantifiable.

3. Seasonality implications

Whether or not a latrine is being used depends on water supply, which differs by season. Furthermore, the rainy season could deter many from defecating in the open. Dry season bias in household survey data collection (Wright et al., 2012), coupled with seasonal shifts in sanitation practices (Routray et al., 2015; Sahoo et al., 2015), are likely to cause some distortion in reported use. In order to tackle this, questions with response categories such as “all the time/some of the time/ none of the time” could be better answered if they are framed around the time of day. In case of seasonality, respondents should be asked about their toilet usage in the recent past or within a specific time frame that can be accurately recalled. (Coffey and Spears 2014).

Toward more accurate measurements of latrine use

There is no ‘right question’ to capture latrine use in rural India. However, survey questions can be designed in a manner that allows for triangulation to determine latrine use, where self-reported use may be combined with structured observations in sample areas, or in sub-samples. These questions could also be framed while considering the nuances of seasonality and recall periods, and be administered at the individual level. These designs would have to be informed by evidence, the need for which has spurred a healthy debate among researchers and academics of late. In an attempt to address these issues, the four evaluation teams are using a standard list of questions. These questions (hereafter referred to as the 3ie-r.i.c.e. standard questions) were developed in collaboration with the four teams.

Additionally, 3ie is supporting an independent measurement project, which estimates the prevalence of reported latrine use in a sub-sample of existing project areas in rural India. It compares the prevalence of reported latrine use in the same sample areas by asking two distinct survey questions. One question is taken as is, from the National Family Health Survey (NFHS-4), which draws on the recommendations of the JMP on water and sanitation. The other is a standard 3ie-r.i.c.e. question developed through formative research in nine states in India. (See box 2).

Box 2: 3ie r.i.c.e. standard question on latrine use

“For every household member five or older, as part of a household roster (where household is defined as living under this roof): “The last time [NAME] defecated, did [NAME] defecate in the open or use the latrine?” (3ie-r.i.c.e. standard latrine use question)

In the interim, 3ie has worked on a compilation of latrine use questions across major studies and national surveys. The compilation lists existing latrine use questions across key national surveys and studies, and hopes to aid researchers designing studies in the future. In the long term, it hopes to set the path for standardised questions for measurement. This is a living document, open to updates by researchers, implementers, policymakers and other stakeholders.

India represents an amalgamation of cultures, identities and social values that influence private behaviour currently under the spotlight. Much depends on researchers, implementers, policymakers and citizens to explore what works, why and in which contexts to reduce open defecation. This relies fundamentally on objective and balanced measures of the successes and failures of sanitation policy in the country.

With input from Neeta Goel and Radhika Menon

Innovating to learn

Innovating to learnWe are in the midst of a global learning crisis. This is the clear message from recent major reports: According to the World Bank’s 2018 World Development Report on learning, “hundreds of millions of children reach young adulthood without even the most basic life skills.” And the Education Commission’s 2016 Learning Generation report estimates that “over three-quarters of a billion young people in low- and middle-income countries will not be on track to acquire basic secondary-level skills.” Alarms are being raised that while the substantial resources being spent on education by governments and families have succeeded in increasing enrollments to unprecedented levels, schools are failing to produce more learning, including the skills needed to lift the productivity and competitiveness of economies.

The challenge of the classroom

As enrollments in schools and universities rise, so does the diversity of those who attend them. It has become patently unrealistic to expect today’s classrooms to cater for the needs of every student. The tendency to teach to the ‘median student’ has been a powerful impetus to reach targets like ‘education for all.’ Even those admirable teachers who succeeded in one-room schoolhouses a century ago would be hard-pressed to impart the knowledge required in today’s curriculum. Both the scale and scope of instruction have changed dramatically. Enrollment numbers have expanded in all countries, resulting in larger class sizes, with students from a wider range of economic and cultural backgrounds. In addition, teachers are expected to cover increasingly heavier curricula that include instruction of the 3 R’s as well as a variety of subjects including computer literacy, civics, fitness and health, sex education, and environment. These changes reflect the important roles that we want schools to play in our societies and economies, but many schools are ill-prepared and ill-equipped to do so.

As we look into the future, it’s hard to predict what knowledge and skills students will need. A 2017 McKinsey report on jobs in 2030 says that, because of automation, the “transition will be very challenging – matching or even exceeding the scale of shifts out of agriculture and manufacturing we have seen in the past.” The 2019 World Development ReportThe Changing Nature of Work warns of “the vast uncertainty involved in making predictions about the future” and how technological change is making it “harder to anticipate which job-specific skills will thrive and which will become obsolete in the near future.” So, how can schools change to deal with such disparity and uncertainty?

Technology can help. Recent years have seen promising education technologies develop remarkably fast. Adaptive learning through artificial intelligence and big data, social learning through digital platforms, immersive learning through virtual reality (VR), augmented reality (AR), and gamification are among the smart tools that can provide personalized learning for every student. These technologies deliver an unlimited world of knowledge to connected students, as well as tools for honing specific problem-solving skills. Support outside the classroom, like the online videos on numerous subjects produced by the Khan Academy, can be accessed for free by students and teachers anywhere on the planet as long as web connectivity is available. In classrooms at Arizona State University (ASU), students are using a computer-based adaptive learning system that provides personalized feedback and suggests learning pathways based on their performance. This “high-tech learning” system has enabled ASU professors to focus their teaching time on “high-touch learning” such as project-based learning and hands-on learning through laboratory experiments in small groups.

For now, research has shown that the use of new technologies in schools is not a panacea. A recent 3ie review on what works to improve education shows that the impact of computers in classrooms on learning from rigorous studies is not statistically indistinguishable from zero. The studies that show a positive effect are those in which the technological intervention is linked to teacher training and curriculum redesign. The lesson from the review is that investments in technological innovations must go hand-in-hand with changes in pedagogy that involve more than just putting digital devices in classrooms or the hands of students. Indeed, it takes a system – and more specifically, an innovation system.

An innovation system for education

A 2011 report by Deborah Jackson of the National Science Foundation argues that fostering innovation requires a system which “models the … complex relationships that are formed between actors or entities whose functional goal is to enable technology development and innovation.” Several frameworks have been developed for what such an innovation systems might look like. These frameworks share four common components that need to be in place to nurture innovation: people, infrastructure, economic resources, and an enabling environment. How do these four components in education systems measure up? In our opinion, not very well…yet.

  1. People. Unlike entrepreneurs who try hard to create new products, services, or procedures to maximize profits, teachers don’t have any incentive to find new ways of providing a learning experience for their students. Most teachers consider it their job to teach curriculum designed by others. So there is a critical need to redefine the education workforce, and figure out the roles teachers and principals can play to transform schools so they deliver the learning goals of every student. The Education Commission’s Education Workforce Initiative (EWI) aims to bring fresh thinking and new approaches to this key area.
  2. Infrastructure. We’ve seen the rapid emergence of new smart technologies, often called the “fourth industrial revolution,” which refers to the greater and wider availability of the internet of things, artificial intelligence, big data, 3-D printing, mobile devices (e.g., e-books), and 5G wireless communications that can be harnessed in schools for instruction and learning. However, schools are often the last places where teachers and students can find these technologies. For example, while Korea has some of the best ICT infrastructure in the world, including free Wi-Fi in every subway line, only 19 percent of classrooms had Wi-Fi connection in 2016. This is due to both the resistance of principals and teachers to embrace new technologies, and the concerns of parents about student addiction to internet use and gaming. But in lower-income countries, poor general infrastructure – for instance, lack of dependable power supply and telecommunications towers, especially in rural areas – are the binding constraint, coupled with school buildings that cannot keep technological devices safe from extreme weather or theft. In addition, the absence of computer know-how and technical services outside cities can render existing school infrastructure such as computer labs almost useless after an initial period of use.
  3. Economic resources.There is often very little or no discretionary funding that innovative teachers and principals can work with. The Education Commission estimates that getting more children in school and learning will require governments to double their spending from just 2 percent of GDP to 4-5 percent and the international aid community to supplement this increased level of domestic funding. However, the budget for innovations in teaching need not overload the education budget. With connectivity, content libraries are available for free to teachers and students. These innovations can also make it possible to reach excluded learners at lower cost while enhancing their learning through more targeted instruction and content material.
  4. Enabling environment.Despite the uncertainty about the impact of education technologies on learning, there’s an unstoppable positive buzz about them and a growing number of projects that apply these new technologies in education systems. Mike Trucano of the World Bank who has been following this field closely notes that “the overall volume of such projects, and the sophistication of many of them, are quite notable,” and that “[t]here is more happening, in more places, than ever before.” He identifies 20 emerging education technologies from around the world including Mindspark, an adaptive-learning product from Educational Innovations that aims to help children in India improve their skills in mathematics, and Eneza, which offers low-cost quizzes and related products to help learners in Kenya prepare for exams. Having this abundance of information easily available to students is a boon to learning, but teachers and school leaders must be able to help students use the information. Inertia is powerful, and vested interests within the education system may be reluctant adopters of innovations. System changes rather than piecemeal reforms could more effectively overcome this resistance.

Innovating to leapfrog

Unless education systems embrace new technologies and turn them into effective tools of teaching and learning, schools are likely to remain mired in the past – with a teaching model that was good for the Industrial Revolution but not for the Digital Revolution. Transforming systems is a daunting task—but the returns would be enormous. Education systems that develop an innovation ecosystem are equipped with the tools to leapfrog, allowing underperforming education systems to bound forward faster than the progress that rich countries have made over 150 years. They may even be able to outpace them, according to Rebecca Winthrop at the Brookings Institution. In industry, by taking advantage of new digital technologies within a robust innovation ecosystem, Korea surpassed Japanese and Western companies in the manufacture of mobile phones, TVs, and computer displays. Why couldn’t this work for education? There are many challenges that education systems around the world must address, but they can hardly go wrong by investing in a learner-centered innovation system.

Learn more about the Education Commission’s Education Workforce Initiative, and read the initial literature review. 

This article was originally published on the Education Commission’s blog

3ie’s Agricultural Risk Insurance Evidence Programme: a structured approach to impact evaluations

Agricultural Risk Insurance Evidence Programme: a structured approach to impact evaluations With climate change becoming a reality, agricultural productivity has suffered considerably. This has put at risk the livelihood of the majority of the world’s poor, who are dependent on agriculture and related activities. Various risk mitigation solutions such as improved seeds and drought irrigation have shown promising results, but the role of transferring risk via agricultural insurance demands deeper exploration. To address this need, 3ie’s Agriculture Risk Evidence Programme funded evaluations of various interventions related to risk mitigation in agriculture and related activities in low- and middle-income countries. This programme examines innovative risk mitigation products, insurance delivery mechanisms, and the role of technology and novel information dissemination processes that can generate effective demand for agricultural insurance products among smallholder farmers. This programme helps identify interventions that have the potential to contribute to the effectiveness of agricultural insurance.

In 2017, 3ie published an evidence gap map, which carefully captured existing studies, as well as evidence gaps in this sector. The gap map showed that interventions around innovative or improved and bundled products are limited. This field is also notorious for low take-up, which makes impact evaluations less useful and/or more costly. To address both these issues, we awarded grants in two phases. In phase one, we funded formative and process evaluations, which required research teams to assess the uptake of interventions. These evaluations were helpful in testing implementation feasibility and assessing opportunities and challenges that are likely to arise during impact evaluations. Additionally, this phase also informed programme, implementation and research designs for the full impact evaluations that have been funded in the second phase. The impact evaluations will generate high-quality and policy-relevant evidence on what works, why and for whom, to increase the uptake of agricultural insurance among smallholder farmers and its impact on their welfare and vulnerability.

Our formative and process evaluations have demonstrated some encouraging and useful findings. For instance, an India-based study demonstrates how to capitalize on the availability of low-cost internet and the rising use of smartphones. The novel picture-based insurance product marketed in this intervention welds technology with weather index-based insurance. Farmers were instructed to take pictures from the same site two to three times a week throughout the cropping season. Based on the assessment of these images, payments for losses were directly issued to their bank accounts. Initial findings have revealed that farmers in the evaluation sample demonstrated high uptake of picture-based insurance.

Innovative studies funded under this programme

One Kenya-based study combined risk mitigation and m-learning gamification in its intervention tool. Insurance agents play a crucial role in generating awareness about insurance products among the pastoralists in this region. Recognising their significance, this study tested m-learning solutions in the formative phase. M-learning tools are being touted as a cost-effective tool to train agents who may not be able to travel long distances for face-to-face trainings. The formative phase revealed that m-learning alone, though cheaper, was not effective. This was likely to be successful only if combined with an initial face-to-face training and other tools such as interactive voice response and SMS contents for sustained support. Thus, the formative phase helped the teams modify their intervention for the impact evaluation phase.

An evaluation based in Burkina Faso experimented with an interesting marketing strategy. By keeping community ties at the very core of its intervention, the study merged traditional relations with modern-day risk mitigation solutions. The intervention sought to give urban migrants the opportunity to purchase insurance products for agricultural plots farmed by their rural relatives. The assumption for the success of such an intervention is the existence and quality of such urban-rural linkages. In the formative phase, the research team examined the existence and strength of these social ties using secondary and qualitative primary data. They found that a significant percentage of urban migrants send remittances to their rural relatives regularly, substantiating their most important assumption.

Risk Contingent Credit (RCC) is an interesting agricultural insurance product being tested in Kenya. It offers a credit solution embedded within the insurance product. In RCC, even though farmers have to pay a risk premium during normal circumstances, they are insured against adverse circumstances. Since the insurance component of RCC is substituted for collateral, it is financially more inclusive than conventional credit products. Thus, RCC could bring risk-rationed farmers (who did not borrow or borrowed less than optimal for fear of losing collateral and falling into a credit-driven poverty trap) into the credit market. The research team used the formative phase to test the uptake of RCC and were able to establish that this was much higher than for traditional credit.

A similar bundled product was tested in Senegal. In countries that are drought-prone or experiencing rainfall deficit, such as Senegal, bundling weather index-based insurance with credit and other services as a package is considered as a possible pathway to address low insurance take-up. The intervention compares mandatory or voluntary insurance bundled with credit to farmers affiliated to two different aggregators. The research team used a randomised design to assess uptake and found that mandatory bundling led to participants forgoing their loan applications. They also discovered that implementing a randomised evaluation was not feasible given sample size requirements and proposed an alternative evaluation design.

With such a structured approach to impact evaluation, this programme attempts to produce useful evidence on what works, why, how and for whom in agricultural risk mitigation. By extensive scoping of the evidence gaps, we were able to define the scope of this programme to address policy-relevant questions. We focused on funding studies that address the key gaps highlighted in the evidence gap map. Formative evaluations using mixed-methods helped us determine which programmes were impact evaluation-ready and under what conditions these were feasible. With this programme, we will be generating high-quality evidence in sectors where evidence is limited.

(With inputs from Bidisha Barooah)