Evaluation questions
Process evaluation
Evaluation questions form the basis on which a health promotion program is driven (Issel, 2009). The following questions have been refined to ensure they provide significant feedback on the program’s quality, utility and impact on the target population
Process evaluation examines the delivery of a program to its target population (Nutbeam, 2009). In this regard the following questions have been formulated:
- Has the program been implemented as planned?
- What factors influenced the implementation (negative and positive)
- What proportion of the target group received the program?
Impact evaluation
This type of evaluation is aimed to assess the immediate effect of the health promotion program on the priority population (Farell, et al., 2002). Such effects may include levels of knowledge acquired, skills, attitudes and change in behavior. Key questions here are:
- Have the program impacts been achieved?
- What difference did the program make on populations facing the greatest risk?
- Have all strategies been appropriate and effective in achieving the impacts.
Outcome evaluation
Outcome evaluation is normally carried out at the maturity stage of a program. It seeks to assess long term outcomes such as mortality, morbidity, quality of life and equity (Silva-Sanigorski, Bolton, & Meloni, 2009). Questions prepared for outcome evaluation are:
- To what extent were intermediate outcomes achieved?
- How were the intermediate outcomes achieved?
- Did the primary target group receive the greatest benefits of the program?
STD/HIV Health Promotion Evaluation Plan
Goal
The primary goal of this program is to reduce prevalence of STD/HIV in men who have sex with other men (MSM).In the case of this program MSM group comprise of gay and bisexual men. Recent Data by CDC, indicate that gay and bisexual men though making up only about 2% of US population, account for most new HIV infections(61% in 2009) (Prejean,et al.,2011; CDC,2011).
Target populations
The primary target population is men who have sex with men (MSM) (gays and bisexuals between 20 and 40 years of age without any racial exclusion) living in urban centers. Others are health service provider of gay and bisexual men, gay and bisexual right groups as well as family and friends of gay and bisexual men).
Consistent with best practices and laid down guidelines of similar programs, the objectives of the program are:
- Discover behaviors likely to contribute to reducing incidences of new infection of STD/HIV in MSM.
- Increase rates of condom use in gay and bisexual men.
- Increase STI and HIV testing in MSM.
The desired outcomes of the Health Promotion Program are:
- Condom culture in gay and bisexual men is developed and maintained.
- Condom use among gay and bisexual men is increased.
- Condom use is supported by those who influence sexual activity-porn stars.
- Regular condom users maintain healthy behavior.
- Testing and testing services and accessibility to testing is increased.
- Adoption of healthy sexual behavior that will promote good health.
Activities
Activities are the operations that are necessary in order to achieve the objectives and overall goal of a health promotion program (Holt, 2009 ; Mckenzie, Neiger, & Thackeray, 2009).
Outcome: Adoption of healthy sexual behavior that will promote good health
Key activities:
- Sponsorship and participation in gay and bisexual community events.
- Celebrity modeling of behaviors i.e. endorsement from prominent gay figures.
- Interaction and engagement with key community leaders e.g. religious leaders.
- Sponsorship of gay events or related social activities to communicate safe sex behaviors messages.
- Mass communication targeting the gay and bisexual men on safe sex.
Outcome: Promote and increase access to HIV/STI testing services for MSM
Key activities:
- Increase access to testing services online and offline.
- Rewards for testing.
- Social movement to encourage testing, text reminders, viral online marketing.
- Promote “bring your partner” testing schemes.
Outcome: Increase condom use between MSM
Key activities:
- Media campaign countering unprotected sex in pornography.
- Promote condom culture through organizations highly influential among MSM.
- Use of mass communication channels to promote condom use.
- Partnership with key health providers for gay and bisexual men.
Primary evaluation questions
The primary evaluation questions for this program are:
- Have all the program impacts been achieved?
- Was the program implemented as planned?
These questions will be used to assess whether all the desired short term and intermediate outcomes have been achieved within the stipulated time and whether the program has remained on course as planned. These questions are important because they ultimately determine the preliminary success and/or failure of the program.
Process , Impact and outcome variables/indicators
An indicator is a measurable variable of a program activity (Keleher, Marshall, Murphy, & Round, 2003 ; The Health Communicaton Unit (THCU), 2007. The measureable variables of the program are outlined below:
Process evaluation measurable variables
- Participation: Number of gay and bisexual men absorbed into the program.
- Resources distributed: Number of condoms distributed within a particular period.
- Training session held: Number of training sessions conducted for priority groups.
- Reach: Number of gay and bisexual men reached per month.
- Client/staff satisfaction: Benefit /challenges of the program reported by the target population and implementation staff.
Impact evaluation (Immediate impact variables)
- Levels of Improved health knowledge on gay sexuality. Assessed through knowledge test at the end of a discussion session/discussion. Program aims scores of over 60%.
- Levels of Improved protective skills and confidence as reported by participants in open-ended questionnaires. Program aims for high improvements in these areas.
- Participants report on their quality of life through survey questionnaires. Project aims for Improved quality of life in the target group.
Outcome evaluation: short term impact variables
- Organizational development: Policy changes, reward systems, number of legislations, made in favor of promoting the health of gay and bisexual men.
- Benefits to participants; participant to report at least five benefits.
- Participants’ satisfaction; program aims for high satisfaction among the respondents.
- Barriers to participants. As few as possible should be reported.
- Increase in number of people reached -Target is an increase of about 10% /month.
Intermediate outcomes (impact measures)
- Changes in service utilization: the rate of condom use and positive change in testing numbers in gay and bisexual men.
- Changes in sexual behavior in gay and bisexual men in pre/post introduction of the program. Change should favor adoption of safer sex practices.
- State of knowledge, skills or attitudes; assessed through observation and demonstration of skills and self-report records.
- The extent of policy implementation. New policies enacted during on implementation.
Outcome evaluation: long term
- New infection rate and mortality rate of gay and bisexual gay men. Programs targets double digit decrease in new infections by the completion of the program.
- Morbidity/mortality: Death related to HIV/STI positive gay and bisexual men before and after the program.
- The quality of life according to participants own views.
Evaluation design
The evaluation will be largely process and impact-based and will be executed using a mix of descriptive/non experimental and experimental designs. Process evaluation will emphasize on quality and suitability of the interventions and approaches of the program. A critical aspect of process evaluation will be to determine and track the ‘reach’ of the program.
Impact assessment will be carried out to determine the immediate effects of program’s interventions on the priority population. The evaluation will not involve comparison of the target with control groups as this approach is simply unethical. The choice of this design model is due to the fact that such designs are easier to implement and least expensive (Mckenzie, Nieger, & Thackeray, 2009; Victoria Department of Health, 2009).
The benefits of the program will be measured before and after implementation of the programs. The program will employ a number descriptive evaluation methodology such as case study, cross-sectional design as well as correlational design and will borrow significantly from established protocols to accelerate implementation and meet reliability and validity requirements.
Methods of data collection
Individual and community level impacts will be measured using a mix of qualitative and quantitative methods. Qualitative methods entail gathering detailed information that is contextual in nature (Issel, 2009). Quantitative methods present quantifiable result with high degree of generalizability (K.Farell, et al., 2002). The quantitative methods to be used in the program include: mail/telephone surveys, process tracking records, analysis of large datasets as well as direct measure of health indicators mainly STI/HIV.
Some of the qualitative methods expected to be used include: focus groups, in-depth interviews, open-ended survey questions, dairies, consensus building as well as forums/discussion groups/audio/video recording.
Actual collection of information will entail questionnaire, surveys, interview conducted in person, by telephone or email, analysis of documents or records to obtain opinion, practices, or belief from the target population. The questionnaires will used to access individual level impacts of sexual behavior, sexual health literacy and quality of life.
Measurement tools
Measurement tools expected to be developed and used in the evaluation program include questionnaires, moderators guide for forum sessions, recording forms, observation forms, dairies, survey software, audio recording systems, and government health statistics. Screening tools for STI/HIV will need to be developed to enable on-site assessments. Focus groups, in-depth interviews and observation will be used gather data with behavioral dimensions such as attitudes, opinions, knowledge and skills.
Sampling
Cluster sampling will be employed as the program aims to cover a large geographical area. Survey areas will be picked from county clusters in target areas. Random sample will be made from these cluster and all individuals included in the survey.Based on funding, the program expects to reach as many gay and bisexual men as possible in major urban cities.
A sample size, sufficiently large enough to produce significant changes will be selected. The sample population will be recruited via online campaign and through gay and bisexual organization. The timeframe for sample recruitment will be defined before embarking on the exercise.
Strength ,Weaknesses & Threats to reliability and validity of measurements
Strengths: The program will be run by highly skilled and knowledgeable coordinators to ensure realization of its intended goals. The program also boasts of a clear goal and objectives augmented with necessary procedure and policy guidelines.
Weakness: Potential weaknesses are limited funding and time constraint. The absence of a control in the program, owing to the nature of the program, means that results will be open to different interpretations.
There are several possible confounding factors that may affect the reliability and validity of the measurements. History may serve to confound the findings when other events occur between the first and the second measurement. Others are changes in instrument calibration, non-response and changes in participants or values between successive measurements.
The program coordinator will ensure that all interview themes are approved by statutory vetting agencies. Interview and focus groups will be analyzed by an independent research fellow who will not be involved in other affairs of the program.
Poor response rate are a common problem in many programs (Round, Marshall, and Horton, 2005). Counter measure for poor response include: providing stamps for mail surveys, reward schemes as well as SMS and emails reminders. The program will also favor the use of widely used quality-proven existing data collection tools rather than developing new ones.
Application of evaluation results
The evaluation results will be used for:
- Making progress decision such as abandoning the less successful interventions while concentrating and sustaining the more receptive and successful ones.
- Device best ways to enhance service utilization.
- Making necessary changes so as to spearhead realization of the desired impacts.
- Providing accountability for program funding.
References
CDC (Centre for Disease Control). (2011). Fact Sheet: Estimates of new HIV infections in the United States.
Holt, L. (2009). Understanding program logic. Web.
Issel, L. (2009). Health Program Planning and Evaluation. A practical systematic approach for. Sudbury, Massachussets: MA. Jones and Bartlett Publishers.
Farell,K., Kratzmann, M., McWilliam, S., Robinson,N., Saunders,&.,Ticknor,J. (2002). Evaluation Very easy Accessible ,and Logical. Atlantic centre of excellence for women’s health, Halifax.
Keleher, H., Marshall, B., Murphy, B., & Round, R. (2003). Evaluation Report: Victorian DHS Short Course in Health Promotion. Melbourne: Deakin University.
Mckenzie, J., Neiger, B., & Thackeray, R. (2009). Planning, Implementing, and Evaluating Health Promotion Programs:A Primer (5th Edition ed.). San Francisco: Pearson Benjamin C.
Nutbeam, D. (2009). Program management guidelines for health promotion. Sidney: Lindwall and Ward Pty Ltd.
Prejean J et al. (2011). Estimated HIV incidence in the United States. 2006-2009. PLos ONE, 6(8), 433-487.
Round, R, Marshall, B & Horton, K. (2005). Planning for effective health promotion evaluation. Melbourne. Australia: Victorian Government Department of Human Services.
Silva-Sanigorski, A. d., Bolton, K., & Meloni, L. (2009). Evaluation plan for Health Promoting Communities. Geelong: D e a k i n U n i v e r s i t y.
The Health Communicaton Unit (THCU). (2007). Web.
Victoria Department of Health. (2009). Web.