Of metrics and moustaches: Measuring social change

by S Giridhar

It was a searing hot afternoon in Hospet. We were recruiting around 70 volunteers for a three month assignment and more than 300 candidates were being screened and interviewed in the district education office. It was dusk when we were done and as we walked out of the gate, we saw Lingappa, head teacher of the Government Primary School Vaddarahatti. We knew him well because Azim Premji Foundation had worked with his school for over three years in a program.

“What brings you here Lingappa?” we asked in chorus and he replied, “My younger brother was one of the aspirants and I am waiting to take him home on my moped.”

Our next query was why did you not tell us earlier? Lingappa replied that if he had told us in advance, he would in some way have been guilty of trying to influence us. When people ask us to describe excellent government schools in rural India with whom we have worked, Lingappa’s school figures because it has ensured enrolment of every child in the village, demonstrated that every child is attending regularly and the learning achievement of the children is outstanding. But our narration is never complete without recounting this attribute of Lingappa, the head teacher.

The nature of work in the domain of social development is such that it is often impossible to observe visible change or impact for some years. And yet measurements are important for we need to know whether the journey is in the right direction. The challenge is to walk the fine line between loosely articulated ‘good intentions’ and a rigid set of quantitative parameters (the classical example of a rigid and unrealistic metric when implementing school improvement programs is “learning outcomes will improve by 15% within one year of the intervention”). To insist on merely a set of woolly indicators is as rigid and self-defeating an approach as the bull headed insistence on purely quantitative indicators as though one were doing experiments in a laboratory.

For us at the Foundation, to understand and try and achieve this balance between the two extremes, has been a continuous learning curve. To ensure that we can have good meaningful indicators to tell us whether we are on the right track and metrics that provide substantive facts and figures.

With more people with corporate experience taking the mid-career plunge into the non-profit and social change domain, these are times of great churn but also opportunities for mutual learning. These people bring into the social sector, into the NGOs and civil society organizations their convictions and insistence for milestones and metrics, so that organizations at all times can know whether they are “on plan”. On the other hand, they must imbibe the complexities of social change, to recognize that while activities can be planned and executed, while indicators of “improvement” over some time can also be developed, ‘impact’ needs time and will reveal itself in myriad ways only to those who look for them.

Let me narrate something, from our work. We run a program that aims at a holistic improvement in all 350 government schools in Surpur, the second most disadvantaged block in all of Karnataka. We began work in 2004 and articulated expected outcomes in unambiguous quantitative terms such as x% improvement in learning outcomes, y% improvement in attendance and Z% improvement in school completion rates of students and so on. Umashanker Periodi, my colleague in charge of this program, always had a mischievous smile whenever we discussed these metrics to show us how we were faring. For about five years at Surpur, we had no visible improvement in learning outcomes or in attendance.

Yet Periodi and our team knew we were doing well. How did we know? We knew because we saw and observed changes that do not reveal themselves in the metrics. What did we see? We would go to the schools and see how the children were so engaged in the class room, we saw how they conversed with each other and with their teachers; we saw the efforts teachers were putting in. We saw the way the children had their mid-day meals. We knew how different all this was in Surpur from the time we began or when we visited neighbouring blocks.

In the same program, the “school mela” is an event where the children and teachers share their work, their creations and their activities with the parents and members of the community to create a healthy connect between school and community. How do we measure this? The number of such melas or the crowds they attract are measurable, but the more difficult and valuable metric is whether the teachers and community work together to hold such melas. Understanding the impact of interventions in a school in all its complexity, including attitudes, beliefs, processes, and effects is possible. The approach is one that resolves the informality and indefensibility that anecdotes often have, while not getting blinkered by simplistic metrics.

Another example, another story. In Uttarakhand as in all the states that we work in, we have been advocating that exams and class room processes should move away from rote learning to a space where teachers and students engage with each other in dialogue. We introduced the concept of tests that are child friendly and kindle the spirit of enquiry in the child. I have talked about this many times.

When people ask me about its impact, I explain how this has taken root in the state, how exams and tests have moved this way, how oral tests are done in such a friendly atmosphere and so on. But what opened my eyes to measurement and reporting of impact is the way my colleague Anant Gangola who has been leading our work there, once reported impact. He said, with complete seriousness and conviction that the program has had great impact because Ranbir Singh, the Block Resource Coordinator of Gadarpur Block in Udhamsingh Nagar, has shaved off his huge fierce moustache. He said the man realized that his moustache frightened the young children and prevented him from establishing rapport with them. For him to remove a moustache that he flaunted with such pride for twenty years was, for Anant, the single biggest indicator that our program was having impact. Seeing Ranbir Singh thus, was a signal for all the teachers in his block about how much the program meant. No rational metric could ever capture that. It remains for me one of the most insightful lessons.

Lest you think that I am trivializing the importance of measurements and metrics let me now add that measuring children’s learning outcomes, attendance, school completion rates are all relevant and critical. But we need the wisdom to realize that these metrics will yield and reveal in their own ways and in their own time. It has taken the program at Surpur six years to show 40% improvement in learning outcomes, but we had the wise Periodi who insisted that ‘the signs we saw on the ground with our own eyes’ was telling us that the change will come.

If we had merely relied on improved learning achievements, we might well have lost our way or thrown in the towel after a few years. Surpur represents for all of us the tortuous journey of change. We need to measure but we must have the sagacity to realize that metrics do not tell the whole story and that they need to be read together with the other kinds of measurement that see, observe and record the imperceptible, but continuous movement of the needle.

(The author is Registrar and Chief Operating Officer of Azim Premji University. He can be contacted on giri@azimpremjifoundation.org)