Impact – More Than a Hot Topic

In social sector work, it is no longer good enough to try hard and hope for the best. Organization executives, boards, funders, individual investors, and communities are demanding proof of impact. People want to know whether their time, money, and efforts are making a difference – and why shouldn’t they? Understanding the type and amount of impact we are making helps us figure out what works and what we need to change to help our communities thrive.

The academics agree. Alnoor Ebrahim, associate professor of business administration at Tufts University, in an article for the Harvard Business Review, says there is a discernible trend in the social sector: “Claims about making a difference are no longer sufficient; evidence of how much difference you’re making is now required.” According to a research paper on impact developed by the Overseas Development Institute, private foundations talk of ‘impact investing’, social change actors talk about ‘collective impact’ and ‘social impact’, and academics are being asked about their ‘research impact’. The international development community is also increasingly preoccupied with impact. Since the early 2000s, the terms ‘impact’ and ‘impact evaluation’ have skyrocketed in use and have become common parlance.”

What is Impact?

So, impact may be trending, but there are still many questions surrounding the term. For starters, how do we define impact? Is it just a positive effect? Or is it something more concrete? Can it be proven? Can it be measured?….Should it be measured? So let’s start with the first. There are two general dictionary definitions for “impact,” but in the social sector, we’re mostly talking about the second: to have a strong effect on someone or something. Obviously, we all want to have a positive impact on our communities, but impact goes both ways. Our work may have a negative impact or no impact at all. This is why impact measurement is so important. It helps us clearly see the effects of our strategies so that we can reinforce what is working and change (or eliminate) what is not.

Defining Impact Using Measurement

So what constitutes proof of impact? There are several types of “proof” or “evidence” we can gather to determine our individual and collective impact. Some organizations tell stories about specific clients who have benefited from a program or service. Some use general community feedback to gauge impact. And some use “hard data” (performance measures, indicators, etc.) to measure whether things are getting better or worse. All of these types of evidence are important, but all organizations should be using hard data to measure their performance and impact on key indicators of community wellbeing.

You probably agree with the above, but you may also be aware of the criticisms surrounding impact measurement. One of the most common criticisms is that data doesn’t give us a full picture of our community’s wellbeing. Many feel that data is “cold” or “impersonal.” Some claim that there are of some aspects of society that are impossible to measure.

It’s true. Numbers aren’t enough for us to understand our community’s mood and gauge the quality of the lives of individuals we serve. It can be difficult to measure things like “gender equality” or “happiness.” But there’s no way we will know if anything is getting any better without using measures. It is also important that we always provide the “story behind our data.” Maybe we’re not doing so hot on one of our performance measures. Providing the story behind this measure (contributing and limiting factors) can help prevent our programs from getting the boot simply because they aren’t going as planning. Maybe we’re doing the best we can considering external circumstances. Looking at our story helps us figure out what can and needs to be changed in order to do better.

Measuring something is always better than measuring nothing. The important thing here is measuring whether our customers (or clients) are better off as a result of our programs or services. How are we going to gauge the effectiveness of our job-training program if we don’t measure the “% of job training program participants who obtain and keep good jobs?” How will we know if our reading programs are working unless we measure things like “% of reading scores at or above grade level”?

There are 2 Types of Impact

When we talk about impact, we usually think about how our individual programs and services are impacting the larger community. But, there are actually 2 different types of impact we need to think about. Understanding the distinction between the two is critical; not understanding the difference can be disastrous. Program-level impact is the impact our individual services have on the people who directly participate in them. In contrast, population-level (or community-level) impact is the impact that many different organizations have on the community through coordinated strategies. Individual organizations and agencies cannot and should not be held solely responsible for creating population-level impact. They can only be held responsible for the impact on their customers and clients. Think about it. How many different people and organizations are involved in making sure our children graduate on time? Parents, schools, educational nonprofits, education agencies, the students themselves….All play a role in achieving education outcomes. This brings us to our next discussion: establishing causality between our individual efforts and population-level impact.

Defining Impact by Establishing Causality

When gauging our impact, focusing on establishing causality is flawed at best. This is simply due to the fact that we cannot hold our individual organizations or programs responsible for community-level well being. What we can do is hold our communities collectively responsible for creating this impact, through coordinated strategies. We should look at our programs as having a contributory relationship to our community-level impact.

Consider the following excerpt from a report by the Center for High Impact Philanthropy:

“A third common misconception is that there are some impacts that simply can’t be measured; this stems in part from a fourth misconception, which is that impact must be attributed to a particular actor or action. In reality, some impacts—such as a change in attitudes towards women—are very difficult or perhaps impossible to attribute to specific causes, but the impact itself can still be measured, even without a clear causal attribution.”

Mark Friedman, Director of the Fiscal Policy Studies Institute and author of Trying Hard is Not Good Enough, unabashedly argues that “The relationship between programs and populations has been poorly understood in the past. This has led to repeated demands that programs prove their worth by showing their “impact” at the population level. This is a bogus requirement. Programs can and should show their effect on their customers. They should be able to articulate how their work fits with the work of other partners in a strategy to improve community quality of life. But it is extremely rare that any one program can change population conditions. We must stop asking programs to validate their worth by demonstrating such effects.”

So, what have we learned?

  • Impact can be positive, negative, or neutral.
  • To truly understand our impact, we need to be using hard data.
  • Data really isn’t enough. We need to talk about the story behind our data.
  • There are 2 levels of impact – population-level and program-level.
  • Individual organizations cannot be held responsible for population-level impact.
  • Because of the above, we should not attempt to establish causality between our programmatic – level efforts and our impact on the larger community.
  • Instead, we can establish a contributory relationship to help us understand how our actions fit into a larger strategy implemented by many partners.

Learn More About Achieving Impact

Download our free Results-Based Accountability Guide to learn a simple and practical way to measure and improve your performance and impact.

For those in the Collective Impact arena, check out The Components of Effective Collective Impact.