How to Isolate Your Program’s Unique Impact
Social change is complex, and there are many moving factors that impact a specific client outcome. Even when rigorous program evaluation methods including a control group are in place, questions may linger regarding if it was the program or some other factor that made the ultimate difference or influenced the desired change. For example, if a school-based program is able to demonstrate that their participants had increases in school attendance and increased graduation, it is unclear if these improvements were due to the programing or activities conducted at the school level, the home level, or the community level.
A student who is at risk of dropping out of school may receive multiple interventions intended to keep the child in school and on track to graduate from high school. His parents may intervene by spending more time with the child working on homework and school projects. He may also be involved in an afterschool program, which gives him a mentor. Or he may have had previous involvement with the court and have an assigned probation officer. Finally, the school may be providing extra help. Given all these interventions in the student’s life, how can the school know for certain that it was their program that impacted the student’s attendance and graduation outcomes?
Unlike medical trials, where doctors can control a single pill or dosage an individual receives to evaluate the effect of a health intervention, social program evaluation makes it nearly impossible and often undesirable to isolate the treatment group and ignore other interventions. After all, social programs are designed to work in the real world not in a laboratory. There is, however, a relatively easy and practical way to attempt to isolate a program or training effect by conducting a secondary evaluation.
Once program staff knows the difference in the outcomes achieved, gathering data sets from both the control group and the treatment group, they can conduct a survey of key stakeholders to achieve a score that can help isolate the treatment effect. For the school program highlighted above, key stakeholders may be teachers, parents, and other individuals involved with the students in the school program. These people would be administered a short questionnaire that contains these three questions:
- What percent of the student’s improvement can be attributed to the application of skills, techniques, and knowledge gained through this school program?
- What confidence do you have in this estimate, expressed as a percentage?
- What other factors contributed to this student’s improvement in performance?
To examine how this impacts the interpretation of data, let us assume that once these surveys are collected from all stakeholders involved with the student, the average answer to the first question related to the percent of student improvement attributed to the application of the skills learned in the program is 50 percent. We will also assume that the average answers to question two related to the rater’s confidence in their estimate is 70 percent, which would mean that the stakeholders are 70 percent confident that the program contributed to about half of the student outcome success.
Other factors such as parental involvement, court intervention, or church activities may have been listed as the other factors that made up the additional 15 percent. The confidence percent (70 percent) is then multiplied by the estimate (50 percent) to produce a usable intervention factor of 35 percent. This will then be multiplied by the actual intervention results to isolate the effects of the program.
We will consider that this school program learned that only 55 percent of at-risk students who did not participate in the program graduated, compared to 85 percent who were involved in the program. Therefore, the comparison results would indicate that the program increased graduation rates by 30 percent. In order to estimate the program effects to allow for other outside factors that influenced the success of these participants, the intervention effect could be used to further reduce this number.