Program evaluation refers to a systematic method of gauging the effectiveness and efficiency of a project, policy, or program. The rationale for program evaluation stems from public and private stakeholders’ interest in selecting, funding, and supporting only the programs that are sustainable and yield the intended results. Kellog (2017) writes that in the United States, program evaluation is a relatively new phenomenon that might not have established traditions yet. One tradition that Kellog (2017) finds as not exactly consistent with the original purpose and the definition of program evaluation is using evaluation to judge grantees’ performance and worth. However, at its core, evaluation is not a punitive measure but rather an opportunity for an open dialogue with funders. A good program evaluation helps to create a vision for the future and move on to new ventures.
Program Evaluation vs. Performance Measurement?
Program evaluation and performance measurement are complementary practices that are distinct from each other. Performance measurement works with data that is used to understand the output of a specific program, while program evaluation is the process of data collection, analysis, interpretation, and dissemination itself (Royse, Thyer & Padgett, 2009). To measure performance means to continuously monitor program accomplishments and passing pre-planned milestones. The key performance measurement concepts are input (resources), output (products and services), and outcome (the result of an activity) (Royse, Thyer & Padgett, 2009). The latter may refer to a change in behavior, attitude, skill, and knowledge at different levels, be it an individual, an organization, or a community. In turn, program evaluation concerns itself with process, outcome, and impact (Royse, Thyer & Padgett, 2009). Process evaluations seek to assess the inner workings of a program and compare them to a pre-established standard. Like performance measurement, program evaluation finds outcomes essential, but it also puts an emphasis on the impact or the long-term effects of a program.
Evaluative Thinking and Its Importance
Program evaluation is impossible without evaluative thinking that concerns itself with setting expectations, goal setting, and selecting data collection methods to ensure the possibility of data-driven decisions in the future. Kellogg (2017) explains that evaluative thinking is not an external phenomenon; in fact, it needs to be embedded into the internal and external processes of an organization and become a part of its culture. Evaluative thinking is critical, reflective, and systematic in nature. According to Kellogg (2017), an evaluator should not be afraid of challenging tradition, myths, and assumptions. It is his or her reflective ability that should get to the core of organizational practices and discern between those that are beneficial and those that are obsolete. Kellogg (2017) emphasizes the importance of being systematic in evaluative thinking because it helps to organize data and give it structure. An organization that can move from disorganized subjective opinion toward unbiased insights has a better chance at carrying out successful programs.
Program Evaluation Study
An example of a program evaluation study is a study conducted by Shannonhouse, Lin, Shaw, Wanna, and Porter (2017). Shannonhouse et al. (2017) report that in the United States, suicide remains a public health concern for college communities as it takes the lives of 1,100 students each year. Surveys show that every fifth student contemplated suicide during their lifetime, and two-thirds of college students are familiar with a feeling of complete hopelessness. Shannonhouse et al. (2017) argue that suicide prevention on campuses might have been more effective if staff had access to better training programs. Unfortunately, students at risk of suicide are least likely to contact mental health professionals (Shannonhouse et al., 2017). Therefore, it is the people that see the said students every day that may be able to spot concerning signs and symptoms first. Shannonhouse et al. (2017) chose ASIST (Applied Suicide Intervention Skills Training), a 14-hour training program certified by the Centers for Disease Control and Prevention. The program provides sufficient knowledge to staff to become both competent and confident in approaching a student at risk.
Type, Methodologies, and Approaches
In their study, Shannonhouse et al. (2017) performed a summative or outcome evaluation. Kellogg (2017) states that this type of evaluation helps to understand whether a particular program has reached the desired results and whether the effort is replicable and sustainable at all. In particular, Shannonhouse et al. (2017) measured the improvement in four dimensions: (1) responding to students at risk; (2) attitudes toward suicide; (c) knowledge about suicide; and (d) the perceived competence and confidence when providing support. Though Shannonhouse et al. (2017) do not specify the evaluation approach that they used, it can be inferred that they made an appreciative inquiry. At the beginning of the study, the researchers created a specific vision that they wanted to adhere to in their efforts. The goal was to test the ASIST program, and the expectation was that the training will help participants develop the desired traits and skills. As for methodology, as stated by the authors themselves, the study had a quasi-experimental design as there was no separation into control and intervention groups.
References
Kellogg, W.W. (2017). The step-by-step guide to evaluation. How to become savvy evaluation consumers. W.K. Kellogg Foundation.
Royse, D., Thyer, B.A., & Padgett, D.K. (2009). Program evaluation: An introduction. Cengage Learning.
Shannonhouse, L., Lin, Y. W. D., Shaw, K., Wanna, R., & Porter, M. (2017). Suicide intervention training for college staff: Program evaluation and intervention skill measurement. Journal of American College Health, 65(7), 450-456.