Introduction
In this interview, a well-known researcher kindly agreed to share some knowledge about the merits and weaknesses of quasi-experimental designs. Additionally, we will talk about the statistical analysis methods used for each of the designs under discussion. Finally, we will explore research questions for these designs.
Merits of Designs
So the first question is – what are the merits of each of the specific designs?
Good question! Let us start with the single group posttest only designs. They are particularly suitable for times when the results of a very unconventional intervention need to be measured. The major merit occurs in the situation when there is a substantial body of background knowledge that can signify the previous dynamics in the participating group and issue under research (Follmer, 2003). This is often the case in educational research projects.
The single group pretest/posttest design is beneficial due to its focus on the changes that occur within one group of participants who undergo an intervention. The merit of this design is in its concentration on a single case and the observation of effects. Further, in the single-group time-series design, the main advantage is in the series of measures that allow researchers to assess the stability of a certain behavior before the implementation of treatment and in various periods after it was put into practice (Clark & Shadish, 2007). In this manner, the exact dynamic of changes can be detected and assessed.
The nonequivalent control group posttest-only design is advantageous because it includes a control group; however, the two participating groups are not equal, and their members are not assigned to the groups randomly (Follmer, 2003). Additionally, results are measured only once to assess the intervention effect. In the nonequivalent control group pretest/posttest design, the results of two non-equal groups are measured twice which helps notice the change they experienced throughout the research compared to their state before the intervention. Finally, the multiple-group time-series design also requires the participation of nonequivalent groups whose results are measured several times before and after the intervention (Clark & Shadish, 2007). This approach is beneficial due to multiple measures that help to track changes.
Application of Designs
When should you use each of the specific designs?
The single group posttest only designs are used when the effectiveness of a certain intervention needs to be measured quickly and simply (Clark & Shadish, 2007). A good example is a research in education where a new teaching strategy needs to be tested. A single group pretest/posttest design is used when a single case is explored. In such projects, only the results of one group measured before and after intervention are compared (Clark & Shadish, 2007). This design should be used when the participating group is unique so its response to the intervention needs to be measured only within this group.
The single-group time-series design is applied when a change needs to be explored more broadly. As a result, in this design, measures are taken several times before and after the intervention to detect the pattern. Also, the duration of the effect of an intervention can be measured using this design (Follmer, 2003). The nonequivalent control group posttest-only design is based on the comparison of the posttest results of two groups. This design can be used when the equivalency of two groups or their random assignment is not an option. This is often the case in research projects where due to the nature of the study, its participants are provided with the right to choose whether or not they want to undergo an intervention.
The nonequivalent control group pretest/posttest design is usually applied in research projects where not only the effect of a certain treatment needs to be measured but also the performance of the compared groups during the intervention (Follmer, 2003). This approach is particularly important in nonequivalent groups because it allows researchers to see whether the nonequivalence was a factor that may have influenced research results. Finally, multiple-group time-series design is implemented when the intervention or treatment under research is anticipated to produce an immediate and noticeable effect. Otherwise, if the effect is not as strong, the measures might not show any statistically significant changes.
Statistical Analysis
What is the statistical analysis that is used for each of the specific designs?
In the single group posttest only designs, researchers are commonly forced to rely on descriptive statistics. In such cases, pattern-matching logic is employed as a means of analysis and interpretation of the acquired statistical data. In the single group pretest/posttest design, paired-samples t-test is a suitable solution. However, if the differences scores are not distributed evenly, this measure may be replaced with a Wilcoxon signed-rank test.
In a single-group time-series design, the single-sample t-test can be used when the collected data is compared to a certain fixed value. The absence of such value will call for the use of ANOVA with repeated measures. A nonequivalent control group posttest-only design a simple t-test with independent samples can be used for the analysis of data. This test will allow researchers to compare the results in the two participating groups (Johnson, 2000). A nonequivalent control group pretest/posttest design will have to rely on the analysis of covariance that is also referred to as ANCOVA. This is a highly advantageous test in the projects where the randomization was unavailable (Johnson, 2000). In a multiple-group time-series design, just like in a single-group time-series design, ANOVA needs to be selected as the method of statistical data analysis.
Limitations
What are the limitations of each of the specific designs?
This is interesting because most of these designs have numerous limitations. Specifically, a single group posttest only design is vulnerable to almost all kinds of threats to internal and external validity. It is widely criticized for the absence of comparison (Follmer, 2003). Moreover, the lack of a comparison group presents a limitation in the single group pretest/posttest design. Under such circumstances, the data interpretation may be biased because it is impossible to predict whether or not an outside influence affected the final results. The limitation of a single-group time-series design is known as regression to the mean. This is a threat to internal validity that occurs due to the inclusion of extreme scores that tend to regress to the mean throughout the research.
In a nonequivalent control group posttest-only design, the nonequivalence of the groups and the lack of a pretest measure present the main limitations as it is impossible to tell whether or not the performance of the groups was different before the intervention was implemented (Clark & Shadish, 2007). In a nonequivalent control group pretest/posttest design, the nonequivalence can be tested during a pretest, and if it turns out to impact the dependent variable, it has to be taken into account when interpreting the results. In a multiple-group time-series design, the limitation in the data interpretation as the inclusion of extreme scores may distort the overall picture and thus, serve as a source of bias or error in analysis.
Research Questions
What is one research question for each of the specific designs (using a topic area that you are interested in)?
The research question for a single group posttest only design may be the following: “Does a course of counseling help improve the academic performance and social confidence of children affected by bullying?” Next, the same research question may be used for a research project organized according to the single group pretest/posttest design. However, the pretest measures can encourage the students to show better academic results throughout the project. This will result in the distortion of data. A single-group time-series design may be focused on the exploration of the following research question: “Does playing computer games affect schoolchildren’s academic performance?”.
One of the potential research questions that could be explored with the help of the nonequivalent control group posttest-only design is “In patients with PTSD, does a combination of cognitive processing and prolonged exposure techniques produce a more significant effect than cognitive processing alone?” To address the issue of nonequivalence and its potential impact on the performance of the compared groups and the research results, the aforementioned research question can be used for the nonequivalent control group pretest/posttest study design as well. The results of participants and their PTSD levels will need to be measured before the intervention implementation to see whether the nonequivalence of the groups affects the member’s performance. Finally, a research question for the multiple-group time-series design could be: “Does playing computer games affect academic performance differently in boys and girls of school age?”
References
Clark, M., & Shadish, W. (2007). Quasi-experimental method. In N. J. Salkind & K. Rasmussen (Eds.), Encyclopedia of measurement and statistics (pp. 805-807). Thousand Oaks, CA: SAGE Publications, Inc.
Follmer, A. (2003). Design and analysis of experimental and quasi-experimental investigations. In M. C. Roberts & S. S. Illardi (Eds.), Blackwell handbook of research methods in clinical psychology (pp. 92-115). Hoboken, NJ: John Wiley & Sons.
Johnson, B. (2000). Toward a new classification of nonexperimental quantitative research. Educational Researcher, 30(2), 3–13.