Measuring Impact

Formal Evaluations and Types of Study Designs

 

Introduction

Social enterprises, just like social programs of all types, want to know whether or not their program is driving real change towards the mission of the organization. In our first learning guide on impact measurement, we introduced the idea that impact measurement exists on a spectrum, with monitoring and measurement initiatives on one end and formal evaluations on the other.

This learning guide will explore the topic of formal evaluations in greater detail, including learning about:

• Observational studies
• Quasi-experimental studies
• Experimental studies
• What to consider before pursuing

 

Evaluations

Formal evaluations are conducted by third-party evaluators and are geared towards attributing impact to your social enterprise’s intervention and isolating the root causes that lead to that impact. Just as all impact measurement initiatives exist on a spectrum, formal evaluations also fall on a spectrum of the level of evidence they provide.

 

On the one hand, we have observational study designs which provide moderate level evidence of a program’s efficacy – on the other, we have truly experimental designs that produce a high level of evidence. While social enterprises will likely want to maximize the level and rigor of the evidence, it is worth keeping in mind the tradeoffs that come with conducting more rigorous evaluations.

 

Observational Study Designs

Observational study designs attempt to demonstrate positive impacts from an intervention, without necessarily attributing impact solely to it. This type of study design looks solely at the intervention and the participants to draw conclusions about impact. Some examples include:

Cost benefit analysis (CBA)

Compares the costs and benefits of social enterprise employment for workers, the social enterprise, and taxpayers (outside of those directly involved in the social enterprise), and society as a whole (total benefit).

Implementation study

An implementation study of a social enterprise that documents the implementation of the social enterprise model, work experience, and surrounding employee supports.

Outcomes study

Compares information collected before the social enterprise job began and one year later to determine whether changes in outcomes over time are associated with social enterprise employment.

 

Quasi-Experimental Designs

Quasi-experimental designs attempt to increase the level of evidence of the effectiveness of a given intervention by incorporating aspects of experimental studies, without necessarily modifying the program delivery in order to do so.

These types of studies often involve comparing treatment and comparison groups, where the treatment group receives a given intervention, while the comparison group does not. Monitoring these groups over time allows researchers to compare outcomes between the two and then draw reasonable conclusions about the efficacy of the intervention. If the treatment group has better outcomes (however they may be defined) than the comparison group, researchers can draw conclusions about the impact the intervention had towards causing those outcomes – up to a certain level.

For example, an employment social enterprise may be interested in understanding the effect that employment has on a person, above and beyond other programmatic supports they may receive. Researchers could observe a cohort of people who were employed by the social enterprise (the treatment group) with a cohort of people who only received other program services (the comparison group).

However, the reason that these are still quasi experimental is largely because the treatment and comparison groups are not perfectly identical. In the above example, people in the comparison group are chosen to be as close a comparison to the treatment group as possible. However, there may be important differences between people seeking social enterprise employment and people who are only seeking other support services – differences such as job readiness, motivation, or life stability could contribute to a difference in overall outcomes.

 

Experimental Designs

Experimental designs attempt to achieve the highest level of evidence by replicating the rigor of a truly scientific study. The most common type of experimental design is a randomized control trial (RCT), which is widely considered the gold standard of evidence. Similar to a quasi-experimental design, a RCT examines the outcomes of two groups – one that has received the intervention (the treatment group) and one that has not (the control group).

The important difference here is that the control group is designed to be as close to possible as the treatment group, but has simply been randomized away from receiving the intervention. In other words, these are people who would otherwise be receiving an intervention but aren’t because of their randomization into the control group of the ongoing study. This understandably makes some organizations uncomfortable, however it is the only way to achieve this high level of evidence.

 

Key Considerations

While evaluations are invaluable ways to get to high levels of evidence about the efficacy of your specific programmatic model, they are also considerable investments of time and resources. The following are some key considerations to consider before your social enterprise pursues an evaluation:

Maturity – Evaluations are appropriate for mature, established social enterprises where both the program model and business are on a firm footing so the evaluation is measuring the impact of a fully implemented intervention that won’t change significantly over the course of the study.

Staffing – Dependent on the type of Evaluation, they can be enormously time intensive in both the planning and implementation phases – a social enterprise undertaking one must be able to staff accordingly. There will be a lot of liaison work with the external evaluator, setting up and monitoring the data collection systems, and general project management. The social enterprise will need someone on staff whose role is either specifically oriented around learning and impact, or at least has a space in their job to focus on the evaluation.

Cost – While there is a lot of internal staff time that is required to run an evaluation, the actual research will be done by third-party evaluators. This is a highly specified and, therefore, expensive type of work.

Time – These initiatives are highly complex and will take years to yield meaningful data. Many evaluations will produce interim reports roughly half way through the project with preliminary findings. But even these will only materialize years after the start of the project, with final reports coming years afterwards.

Goals – With the above considerations in mind, it is important that your social enterprise has a clear understanding of its goals before pursuing an evaluation. Given the investment in time, staffing, and money that an evaluation will require, you will want to make sure that your social enterprise will get what it wants out of an evaluation.

 

Conclusion

Evaluations are an incredibly effective way for a social enterprise, and even the social enterprise field, to build its evidence base. Individual social enterprises, as well as funders like REDF, are interested in demonstrating the efficacy of the social enterprise model, and evaluations are a crucial way in doing so. However, as we have seen, they are not easy to do and require a considerable investment in time and resources.

In upcoming learning guides on this topic, we will conduct a literature review of available studies on the social enterprise model, and provide you with strategies for using this data, or internally collected data, when approaching funders.