This blog is part two of a mini-series on program evaluation. Read part one.
Good intentions and reasonable approaches can’t guarantee an effective program, but systematic evaluation can help us get closer to achieving our objectives.
Program evaluation is the process of investigating the effectiveness of programs in ways that are adapted to their specific contexts and are designed to inform decision-making. Program evaluations can help organizations increase impact and decrease cost, which is essential — especially in the under-resourced realm of nonprofits and local governments.
The good news? Program evaluation doesn’t have to be expensive or fancy; in fact, you’re probably already doing it, or have the means to do it in-house. Below is a list of considerations to weigh (plus resources!) before you start the evaluation process.
Step 1: Identify the why.
You likely have more than one reason for wanting to conduct a program evaluation, but recognizing your primary purpose will help you navigate the next steps. Check which purpose below best aligns with your organization’s reason for program evaluation.
- Program Improvement | Maybe you’re looking to increase clients served, maybe you’re looking to decrease inefficiencies internally, but either way, you need your program to perform better. You’re looking for a formative evaluation that will provide timely, concrete, immediately useful results.
- Accountability | Program managers are expected to use their resources effectively and efficiently. Stakeholders need to know how your program is performing so they can make critical decisions on things like the continuation of the program, allocation of resources, or restructuring. You’re looking for a summative evaluation that produces results that can credibly answer questions and withstand critique.
- Knowledge generation | Program innovation can’t happen without new information. Maybe your program is tweaking service delivery or a funder wants you to try something completely new — regardless, you’ll want to know if it works. You’re looking for an evaluation that will show others what went wrong, what went right, and what should be considered in the future.
- Requirement | If we’re being honest, a lot of program evaluation is done simply to satisfy an obligation. In this case, you’re wanting an evaluation that understands the requirements you have to fulfill and deliver with precision.
Step 2: Determine the what.
The five types of evaluations below aren’t static categories; they overlap and rely on each other. They aren’t the only ones out there, but they do provide a good survey of options available.
- Needs Assessment | Programs are typically designed to fill in a gap or ameliorate a problem — but what if that need had transformed since the program was developed? Has the nature, magnitude, or distribution of the problem changed? How and why have those conditions changed, and what are the implications of these circumstances for the design of the program? What are the characteristics of the population served? All of these questions (and more!) can be answered with a needs assessment.
- Theory & Design | Maybe you have a good handle on the need because an external party keeps tabs on it for you (e.g., the ALICE Report), but you’re wondering if your approach is maximizing the likelihood of success. In other words, does your design hold up in theory? Does it rely on valid assumptions? What outcomes does the program intend to effect, and is the program sufficiently staffed, resourced, organized, and delivered to effect that change? Before you consider how and if your program is working in practice, you have to make sure you have a good idea to start with.
- Process | The best program design in theory won’t work unless it’s implemented well. Often, this is where the bugs in a program are found! This probably isn’t a surprise given that we all know program management is hard work. Sometimes there isn’t enough funding, personnel, or other resources. Other times, program staff lack the training or motivation. Maybe your program isn’t reaching your intended target population at the rate you had anticipated.
Regardless of the reason, process evaluations help you understand how your program is working in reality. Though they can be the easiest to overlook because we all want to jump to impact, it’s the type of evaluation you’re probably already doing without even knowing it just by monitoring your employees or collecting administrative data.
- Impact | Assessing program efficacy is the most popular part of systematic evaluation, but it can only be done well if all of the types above have happened or are happening. Impact evaluations show if your program is doing what it’s supposed to do — and if it has any unintended effects! A good impact evaluation will also look for differential impact, or if it affects different types of people differently.
- Cost-Benefit Analysis | Last but not least, CBAs assess a program’s efficiency by weighing all of the costs and benefits. CBAs identify inefficiencies, who pays the costs, who benefits, and alternative options that could prove more efficient while yielding equivalent benefits.
Step 3: Consider the who.
Regardless of type, all evaluations must be done in a way that recognizes the political and organizational context present. To that end, you must identify and understand the stakeholders involved. The stakeholders that influence your decisions might include the following:
- Evaluation sponsors
- Program sponsors
- Target participants
- Actual participants
- Program managers
- Program staff
There’s a lot to consider when it comes to program evaluation, but being able to make informed decisions is worth it. Our next blog will cover Step 4: Selecting the how. We’ll talk about 1.) how you decide whether an internal or external evaluation is right for you, 2.) what you’re already doing internally that is evaluation, and 3.) how you can use it to make better decisions.