How do we measure impact?
There is an increasing trend, particularly within the for purpose or not for profit sector, to measure the “impact” of what we do and the difference our contribution makes. But measuring impact is a lot harder than it sounds and there aren’t too many people who can put their hand on their heart and declare they have the formula all sorted.
We can readily measure the inputs, the outputs but measuring the impact is a bit trickier that you might think.
For example, there is a major telco here in Australia that supporters a program called Student2Student. How it works is that the telco supplies hand sets and phone credit to kids in rural disadvantaged communities who have a below average literacy standard. Then kids who are often the children of the employees of the telco, volunteer to be part of a program. The kids will then connect on a scheduled basis and those with the below average literacy standard will read to the kids who are doing ok at school. There is a set program of content and duration that is followed and at the end of the twelve week program the literacy standards of the kids who had the challenges are measured and compared to when they started the program.
The program has proven to be hugely successful with a very high completion rate and an even higher improvement rate in reading of the kids who stayed till the end of the program.
So when we consider a program such as the Student2Student program it is relatively easy to measure the inputs. They consist of:
• number of students participating in the program
• number of students acting as mentors in the program
• number of phones distributed
• number of hours the program ran for, etc.
The outputs are similarly equally easy to measure:
• how many students completed the program
• the improvement in the literacy standards of the participants
• the number of students who remained on the telco’s network after the program (new customers)
But the impact of the program becomes harder to measure. Let me explain. If one of the kids who was struggling at school due to their low literacy standards, failing at exams, getting into trouble and low self esteem drops out of school and enters into the criminal justice system via minor leading to major crime there is a huge cost to society. But what if, the improvement in their literacy meant they started passing exams, decided to stay at school and avoided a life of crime, how do we measure the impact of something that didn’t occur?
This is an example of great work that is so often done in the not for profit sector by the money making end of town but the true measure is difficult to identify.
I run a charity called Hands Across the Water in Thailand for kids. It was established after the tsunami in 2004 when I was deployed there to identify bodies of those who died. The charity has grown somewhat and we now have seven different homes in areas all across Thailand. Each night around 300 kids will go to bed in one of our homes in a safe, loving and secure environment. Due to the unique operating model we established we have raised over $15million Australian dollars without spending a cent of donors money on administration or fundraising. I can even tell you that last year we supplied around 320,000 meals to the kids in our care and the communities that we support.
All of these measures are either inputs or outputs. How much money do we put into the community, how many meals we provide, how many local Thai staff are employed, all measures of inputs and outputs. But what is the true impact of what we are doing?
There is a Chinese proverb that goes something like this:
“If you want to plan for a year; sow rice,
If you want to plan for a decade; plant trees,
If you want to plan for a lifetime; then educate children”
This Chinese proverb brings me back to what we are doing in Thailand, the value of education and a long term commitment and a further example of how hard it is to measure impact with any real accuracy.
If the purpose of our existence at Hands is to change lives in a meaningful way, what impact have we had if the kids leave our homes at the age of 18 or thereabouts, and then go into a very low skilled job, or worse still the sex industry without choice?
As our older students who first came to live with us ten years ago now graduate from University they have choices around not just jobs, but careers. They now get to choose what type of “career’ they will pursue and the real impact is not just on them but it will inevitably change their social circles and it’s not hard to conceive how it then impacts the next generation.
If as a charity our measure is on changing lives, how difficult is it for us to report to our supporters if we have to wait until our kids graduate from University, have families of their own who are then educated and the previous cycle of poverty has been broken? How do you measure and monetise the investment we have made in the community?
As Hands has recently passed its tenth year of operation I feel we are now able to start reporting, in loose terms, on the impact we are having. Sure we can report on dollars raised, homes built, meals provided, but as I said they are inputs and outputs. The true measure of our success is the investment we have made in education. Planning for a lifetime, not just of the present generation is about investing in education.
Eleven years ago I was working identifying the thousands of bodies recovered after the tsunami in Thailand. Now I have the absolute honour and incredibly humbling experience of attending graduation ceremonies of the kids who we started supporting and now are leaving university.
The true impact might be difficult to measure but I don’t need a matrix to tell me the lives of the kids graduating with qualifications that will set them up for life, have been immeasurably improved for the better. It’s in their smiles, you can tell.