Introduction
Weeks 10-12 cover contrasting topics, but they are significantly related to materials we have studied before – ethical considerations, the concept of counterfactual, research design, and other issues. Topics discussed below include qualitative research tools, causal inference and counterfactuals, and randomized assignment. They are significantly related to the design of the evaluation and its initial stages, which we have studied previously within the framework of the theory of change and logic models. Selecting tools for evaluation, considering the fundamental problem of causal inference, and working with the sample in the first stages significantly influence the study’s success, which justifies their importance.
Qualitative Tools
Some evaluation features determine which tools to use for it. As we identified in the first two weeks, two data types are used to answer specific evaluation questions – quantitative, transmitted through measurements and numbers, and qualitative, presented through language. For every kind of data, particular methods allow its obtaining. The need for qualitative data and, therefore, the use of qualitative tools may arise when the focus of the evaluation is the experience of research participants (EDEP 657, 2021a). For instance, such methods are applicable when documenting processes or evaluating implementation. When selecting tools, it is also necessary to consider intended users – if the answers that can be obtained in this way satisfy them – it is another reason to use qualitative tools.
There are three main types of qualitative design strategies and analysis approaches:
- Naturalistic inquiry occurs when the program develops in its way without a preselected focus.
- Emergent design flexibility is the ability to modify the design of the study depending on the information received.
- Purpose sampling is a focus on participants who have an evaluation- or program-relevant experience. At the same time, sampling strategies also differ – the evaluator can set a specific criterion for selecting participants or, conversely, include the possible number. Strategies are extreme or deviant case sampling, intensity, maximum variation, criterion, and snowball sampling.
The main qualitative methods of data collection are surveys, interviews, and focus groups. Surveys usually include questionnaires and, depending on the answers – close-ended or open-ended – can provide quantitative and qualitative data. Interviews, in turn, more deeply assess the experience or perspective of participants concerning a specific program. There are several types of interviews, for example, structured, open-ended interviews or conversational interviews.
The focus group is an extended interview when the answers are given simultaneously by a group of participants with the help of the facilitator. Patton (2018) highlights open-ended interviews, observations, and written documents as kinds of qualitative data collection. Observations include notes from the evaluator’s fieldwork on behavior, interactions, and other aspects, and documents are written materials, such as memoranda, letters, or articles.
Causal Inference and Counterfactuals
The most frequent questions in the evaluation are causal, which are aimed at learning whether a particular intervention leads to the desired results. The 19th-century philosopher John Mill set the conditions for identifying the cause and effect:
- The cause must go before the effect in time.
- When the cause level differs, the effect changes accordingly.
- Ability to discount other plausible explorations (Murnane & Willett, 2010).
The most favorable circumstances in which all three conditions will be satisfied is conducting an experiment in which cause manipulation is possible.
There is a fundamental problem of causal inference that affects evaluation. In particular, to understand the causal impact, it is necessary to see the difference between the outcome in which a certain intervention or treatment took place and the outcome in which they were not. Gertler et al. (2016) represent this calculation as follows: Δ = (Y | P = 1) − (Y | P = 0). The formula includes Δ as causal impact, Y – the outcome, and P as the program. The fundamental problem is missing information because we can never observe two conditions simultaneously – with and without a program.
The fundamental problem, the inability to analyze one object in two conditions, leads to the concept of counterfactual, the evaluation of which can give complete information. Counterfactual is an unobserved outcome – the one in which the program was absent. So both conditions cannot be applied to the same studied object, and one must refer to the average values. In detail, assessing the causal effect means comparing the average result of a group of individuals who participated in the program with a counterfactual – the average result of the group that did not participate (EDEP 657, 2021b). These groups correspond to the treatment group and control group, whose presence is necessary for the purity of the study results.
Researchers need to choose a counterfactual that matches the study, and there are specific requirements for it:
- The characteristics of the groups should be the same in the baseline, for example, age or the studied course.
- Researchers need to make sure that the program has no direct or indirect impact on the control group.
- The outcome of the control group should be the same as treatment groups if both participated or did not participate in the program.
In some cases, the counterfactual is inappropriate to be a convincing argument and represents counterfeit estimates. For example, if the critical discussion of the study is a comparison of conditions before and after the intervention. Using this method for the causal effect is incorrect since everything is developing, and the program would not be the same even without intervention. Another mistake is Enrolled-and-nonenrolled (or self-selected) comparison – when the participants’ desire to participate affects the result (Gertler et al., 2016). Such errors in establishing causal relationships are common, and, therefore, it is necessary to be careful when interpreting findings.
Randomized Assignment
If the program has a positive impact, extending it to the most significant possible number of people is essential. However, resources are usually limited, and, therefore, evaluations are needed, and since the potential number of participants is restricted, it is vital to involve them most honestly. The randomized assignment is applicable when an eligible population is larger than the program can accommodate, or it develops gradually and in the initial stages can accommodate only a part of the eligible candidates. At the time we studied the ethical considerations of evaluations, we encountered randomized assignment. The concept involves randomly selecting participants for groups – control and treatment, with equal chances to be in any of the groups. In other words, it is a lottery that honestly determines who will have access to the program.
The method’s advantages are simplicity for understanding, the absence of subjective criteria, and the impossibility of manipulation. With honesty and transparency, the randomized assignment also enhances the estimation of counterfactuals since groups are as similar as possible to each other. This feature refers to external validity, which the method provides and thereby allows generalizing the result. Internal validity is ensured by eliminating other influences besides the research’s focus due to the use of two groups (Gertler et al., 2016). Such features provide significant benefits of using randomized assignment for research, programs, and evaluations.
The first step in using the randomized assignment tool is to define the eligible population. This step is critical because it affects program performance and effectiveness (Murnane & Willett, 2010). The second step is to determine the sample from the eligible candidates.
Finally, a randomized assignment is used, for example, a coin flipping, a random number generator, or other methods. An eligible population is also interconnected with levels of randomization – for example, a person, group, or institution are different levels, and the higher the level, the less comparability (EDEP 657, 2021b). At lower levels, there are more common characteristics, and therefore the comparison is more accurate. With all the advantages of randomized assignment, it does not entirely warn against possible errors, and it is necessary to choose the study’s design carefully. For example, potential problems include spillover, when a member of one group affects a member of another one or imperfect compliance.
Conclusion
Evaluators need to be careful about all the details of their studies. The materials of the last three weeks draw attention to essential elements of the research design, which are crucial to identify in the initial stages. In particular, the choice of qualitative tools that include methods for collecting data presented through language rather than measurement depends on the study’s objectives and intended users. When the evaluation focuses on establishing causal relationships, it is vital to consider the fundamental problem of causal inference and select counterfactual relevant for the study. Participants in the study determine how balanced the various conditions which are necessary for comparison.
The choice of participants should be transparent and honest, which can be ensured through the randomized assignment. Thus, all the themes of past weeks are closely interrelated and provide a common picture of the evaluation process.
References
EDEP 657. (2021). Week 10 [Video]. YouTube. Web.
EDEP 657. (2021). Week 11 [Video]. YouTube. Web.
EDEP 657. (2021). Week 12 [Video]. YouTube. Web.
Gertler, P. J., Martinez, S., Premand, P., Rawlings, L. B., & Vermeersch, C. M. J. (2016). Impact evaluation in practice (2nd Ed.). International Bank for Reconstruction and Development / The World Bank.
Murnane, R. J., & Willett, J. B. (2010). Methods matter: Improving causal inference in educational and social science research. Oxford University Press.
Patton, M. Q. (2018). Qualitative evaluation checklist. Western Michigan University. Web.