Last updated: 1/30/22

In public sector work, we can’t just try hard and hope for the best. An organization’s executives, boards, funders, investors, and communities demand and deserve proof of impact. People want to know whether their time, money, and efforts are making a difference – and why shouldn’t they? Understanding the type, quality, and quantity of impact achieved helps us figure out what’s working to change communities (and equally important – what isn’t).

The academics agree. For example, in an article for the Harvard Business Review, associate professor of business administration at Tufts University, Alnoor Ebrahim, says there is a discernible trend in the social sector: “Claims about making a difference are no longer sufficient; evidence of how much difference you’re making is now required.”

According to a research paper on impact, developed by the Overseas Development Institute, private foundations talk of “impact investing,” social change actors talk about ‘collective impact’ and ‘social impact’, and academics are asked about their ‘research impact’. The international development community is also increasingly preoccupied with impact. Since the early 2000s, the terms ‘impact’ and ‘impact evaluation’ have skyrocketed in use and have become common parlance.”

What is Social Impact?

How do we define social impact in general? Is it just a positive effect on our community? Or is it something more concrete? Can it be proven? Can it be measured? Should it be measured?

There are two general dictionary definitions for “impact,” but in the social sector, we’re mostly talking about the second: to have a strong effect on someone or something.

We define positive impact as “turning the curve” (or beating the data baseline) on a community indicator of wellbeing or a program performance metric that answers “are our customers/clients actually better off as a result of our efforts?”

We all want to have a positive impact, but impact can go both ways. It’s unfortunate (but extremely important) to understand that our work may have a negative impact or no impact at all. This is why impact measurement is so important. It helps us clearly see the effects of our strategies so that we can reinforce what is working and change (or eliminate) what is not.

The 2 Types of Social Impact

When organizations talk about impact, they usually talk about how an individual program or service is creating community-level change. But this can be a misleading way to think about impact.

There are actually 2 different types of impact we need to think about. Understanding the distinction between the two is critical; not understanding the difference can be disastrous.

1. Program-level impact is the impact that individual services/organizations have on the community members who directly participate in them.
2. Population-level (or community-level) impact is the impact that many different partners, working in collaboration, have on a specific population (community, town, state. etc.)

The reason we separate these types of impact is that:

Individual organizations and agencies cannot and should not be held solely responsible for creating population-level impact. Community wellbeing is complex and relies on factors involving a wide range of groups, individuals, agencies, organizations, and industries. Individuals and organizations can only be held responsible for the performance of the programs and services that they manage.

Think about it. How many different groups are involved in making sure children graduate on time? Parents, schools, educational nonprofits, educational agencies, teachers, textbook manufacturers, the students themselves… all play a role in the status of high school graduation. Responsibility is shared, and true impact is generated when diverse groups come together with a common goal. An individual organization should never be held solely responsible for community-level impact like this.

Should We Measure Impact?

There are many criticisms regarding impact measurement. One of the most common criticisms is that “data doesn’t give us a full picture of a community’s wellbeing.” It may also be extremely difficult (or even impossible) to quantitatively assess certain aspects of wellbeing. For example, measuring human emotion and perception is notoriously difficult. Do people think their lives have improved as a result of your program? Think about your own experiences and how your perception of a situation may change by the day or even the hour.

But there’s no time for excuses when it comes to the wellbeing of our children, adults, families, and communities.  Measuring something is always better than measuring nothing. Luckily, there are some more objective measures we can look at in addition to perception and emotion.

So what constitutes proof of impact? There are two kinds of “proof” or “evidence” we can gather to determine our individual and collective impact:

• Anecdotal evidence: Some organizations tell stories about specific clients who have benefited from a program or service. Some use general community or staff feedback to gauge impact.
• Quantitative measurement: using numerical data (performance measures, indicators, etc.) to measure whether things are getting better or worse. (Tip: What is the difference between indicators and performance measures?)

All of these kinds of evidence are important, but:

All organizations should be using quantitative (numerical) data to measure their programmatic impact. Why? Tracking data can help ensure our programs are appropriately designed, reflect true community conditions, and are actually making a difference. (Tip: The Basics of Identifying Performance Measures for Your Organization.)

This means measuring whether customers (or clients) are better off as a result of our programs or services (that’s our individual impact). How are we going to gauge the effectiveness of our job-training program if we don’t measure the “% of job training program participants who obtain and keep good jobs?” How will we know if our reading programs are working unless we measure things like “% of reading scores at or above grade level”?

Partners should also continuously monitor key indicators of community wellbeing to inform collaborative improvement strategies and provide a sense of whether individual programmatic efforts correlate with a positive impact on the population. Correlation…NOT causation.

Regarding the criticisms, it’s true that numbers aren’t enough to fully understand community conditions of wellbeing. It can be difficult to measure things like “gender equality” or “happiness.”

This is why we should always provide the “story behind our data.” Maybe we’re not doing so hot on one of our performance measures. Providing the story behind this measure (contributing and limiting factors) can help prevent our programs from getting the boot simply because they aren’t meeting arbitrary targets. Maybe we’re doing the best we can considering external circumstances.

Looking at the story behind our data brings precision and leverage to our work by helping us figure out exactly what needs to change in order to do better.