Introduction
Twenty-first century education has a wealth of technology available to draw on to enhance student engagement in the learning of new knowledge. Understanding of the best ways in which to deliver learning materials to students is essential in narrowing the gap between those of different learning styles who find traditional lecture methods inadequate in engaging them with learning of materials (Bayek & Layne, 1988; Greenhalagh, 2001). Given that modern workplaces utilize computers to a greater degree than traditionally it is likely that use of computer assisted programs to complement lectures will broaden undergraduate generic skills needed in the workplace, as well as enhance the learning experience of those who are ill-suited to traditional lecture methods.
To evaluate the efficacy of a computer assisted instructional method used as a complement to traditional lectures at Bryston College an empirical study has been designed. The research question for the study is: Does the use of computer assisted instructions enhance student learning of developmental basic algebra, as compared to traditional lecture methods? It is anticipated that the mean test scores on the Descriptive Test of Mathematics Skills (DTMS) (Bryston College Board, 1995) of students receiving computer assisted instruction in conjunction with the traditional lecture will be significantly higher (alpha =.05) as compared to the mean scores of students receiving only traditional lectures. As such the null hypothesis was that there would be no difference in mean scores across the groups.
Method
Participants
140 undergraduate students taking the developmental basic algebra course at the Bryston College will participate in the study. Seventy students have been registered for the course on Monday/Wednesday mornings with Instructor 1, and seventy students have registered for the course on Monday/Wednesday evenings with Instructor 2. To control for pre-existing differences across students, the students will be randomly allocated to their class using a “number from the hat method”. Each student will draw a number, either “1” or “2” from the hat which will indicate their allocation to either the traditional lecture group (1) or the traditional lecture and computer assistance (2). The class days will be randomly selected using the number form the hat method, to be either the control group or the experimental group.
Materials
The Descriptive Test of Mathematics Skills (DTMS) was published by the Bryston College Board of Education (1995).The test measures skills of math ability and is untimed. The passing score is 506.This test has been found to have high comparative validity across a range of studies. Bridgeman(1982) compared the DTMS with scores from the College Board Scholastic Aptitude Test-Mathematics (SAT-M) in predicting grades in freshman level mathematics courses across two institutions; his research has been supported in more recent literature (Gray & Sheehan, 1992; Suddick & Collins, 1992; Sireci & Talento-Miller, 2006). The DTMS has also been found to have high predictive validity (Meyer, Woodard & Suddick, 1994).
The reliability coefficient for the Basic Arithmetic section of the Descriptive Tests of Mathematical Skills is.84 (Cronbach’s alpha) with a 2.1 standard error (Bryston College Board of Education, 1995).
Design
This study will use a 2 x 2 mixed method experimental design; this being Lecture type (traditional or traditional + computer assisted) x Instructor (Instructor 1 and Instructor 2). This design will control for differences across Instructors of their method of delivery. The IVs in this study are the Lecture methods and the assigned Instructors. The DV will be the student’s total score on the DTMS.
Descriptive statistics such as frequency and percentages will be used to make general comparisons across groups and to describe the data. Inferential statistics will be the use of mixed-method ANOVA (SPANOVA) to compare variance across the groups and to identify the main effects (if any) and interactions (if any) of variables.
Procedure
Students will arrive at their assigned class and asked by their assigned Instructor to complete the DTMS individually, without consulting with each other. The test is untimed, so students will be given the entire class period to complete the questions. The Instructor will then collect the test and thank the students for their participation.
Discussion
Threats to Internal Validity
An experiment is valid if it has internal and external validity. Internal validity is the extent to which any change in the dependent variable is actually due to the independent variable. As there are two Instructors there will obviously be slight differences in the delivery of materials, regardless of attempts to standardize delivery. Other differences including the gender of the Instructor and their age may also influence student engagement with the material. For example, Patten (2002) states that in western cultures men tend to be seen as figures of authority as compared to women and older persons are more likely to be perceived as knowlegable about a topic as compared to younger persons. Another threat to internal validity is demand characteristics on the part of the students. They may try to guess the hypothesis and so seek to “please” the Instructor by answering in a way that y think is wanted. A further threat to internal validity is that of researcher bias in that the Instructors may not present the learning material in a standard format as they want a particular lecture method to be successful. The aptitude-treatment-interaction occurs when the sample has certain characteristics that interact with the independent variables that limit generalization (Shutt, 1999). So that those students who have a better grasp of developmental basic algebra prior to the learning material exposure will likely do better on the final test.
As no other instruments are being used to pre-test students as to their current knowledge of mathematics it may be that the students are unable to grasp the topics covered in the Lectures and so at the experimental test they are unable to adequately answer the DTMS and so the test would not measure what it is meant to, this would be a threat to internal validity. Also there is no pre-test of students and or Instructors computer abilities, as those who are computer illiterate would skew the data in that they would not know what they are doing as compared to students/Instructors who do.
The history of the students is unknown throughout the course, as is that of the Instructors (Patten, 2002). Personal circumstances could negatively impact on a student’s ability to concentrate in class or an Instructor’s ability to remain consistent in the delivery of materials. This does not make the DTMS test itself inaccurate, rather it shows a flaw in the design. Maturation effects of students and Instructors across time could also negatively influence final test scores, as the course runs over the duration of a semester of college (Shutt, 1999). For example, aging and development could cause an older female student to be going through menopause, resulting in flushes (a common source of embarrassment) and/or swings in her emotions due to hormonal changes. These states could affect her ability to concentrate in class, or to interact with other students or the Instructor to discuss her questions about the material learnt. Also, taking the DTMS test itself may negatively impact on a student’s ability to concentrate and so they do not perform to the best of their ability.
Threats to External Validity
Turning now to external threats of validity that could impact on how the results affect the wider population of undergraduates taking developmental basic algebra courses, these include; selection bias; reactive effects of experimental arrangements; and timing of the experiment (Patten, 2002; Schutt, 1999). External validity is the extent that the relationship observed between the independent and dependent variable during the experiment is generalizable to the “real world.”
Firstly, selection bias could occur in that the sample in this study is not in fact representative of the wider population of students taking developmental basic algebra courses in North America. This could be due to the sampling technique used (Meyer, Woodard, & Suddick, 1994). Although students were randomly allocated to groups, the sample itself was a convenience sample in that the students were already enrolled in the course. Also, the demographics of students enrolled at Bryston College may be diametrically different to those of students in other parts of the USA. For example, the college may only take students with an above average GPA score, predisposing the students to an above average ability to grasp class materials irrespective of the Lecture delivery method. Alternatively the students may come from predominantly higher socio-economic brackets predisposing them to access to resources and materials that aid their learning experience. A student who does not have to worry about rent, full-time work or lack of child care facilities will have more time and energy to devote to study materials in their own time (Meyer, Woodard, & Suddick, 1994).
Secondly, the experimental setting itself could influence student scores. If the test does not take place in the classroom in which material was learnt there is evidence that students may not do as well. Physical and ambient cues within the classroom, even something as simple as sitting in the same seat where one learnt most class material, can have a dramatic effect on how a student answers questions during a test (Patten, 2002; Solso, Johnson, & Beal, 1998). In this way the final scores on tests may not be a true reflection of the target population.
Finally, the timing of the experiment could threaten external validity (Solso, Johnson, & Beal, 1998). For example, if the experiment takes place after a major incident, such as a terrorist attack as experienced by the USA in New York in 2001, the students may not be in the frame of mind to concentrate on the test. If they were to go ahead with the test, those within the target population that take the test at another time when such major incidents have not occurred they will not have the socio-environmental influences to impact on their test taking abilities.
References
Baek, Y. & Layne, B. (1988). Color, graphics, and animation in a computer-assisted learning tutorial lesson. Journal of Computer-Based Instruction, 15(4), 131-35.
Bridgeman, B. (1982). Comparative validity of the College Board Scholastic Aptitude Test-Mathematics and the Descriptive Tests of Mathematics Skills for predicting performance in college mathematics courses’. Educational and Psychological Measurement, 42 (1), 361-366.
Gray, M. W. & Sheehan, K. R. (1992). Sex bias in the SAT and the DTMS. The Journal of General Psychology, 12(1).
Greenhalagh, T. (2001). Computer assisted learning in undergraduate medical education. BM, 322 (6), 40–44.
Meyer, J., Woodard, P., & Suddick, D. E. (1994). The Descriptive Tests of Mathematics Skills: Predictive validity for an elementary mathematics concepts and structures course. Educational and Psychological Measurement, 54(1), 115-117.
Patten, M. L. (2002). Understanding research methods: An Overview of the essentials (3rd ed.). Los Angeles: Pyrczak Publishing.
Schutt, R. K. (1999). Investigating the social world: the Process and practice of research (2nd ed.). Thousand Oaks: Pine Forge Press.
Sireci, S. & Talento-Miller, e. (2006). Evaluating the predictive validity of graduate management admission test scores. Educational and Psychological Measurement, 66, 305 – 317.
Solso, R. L., Johnson, H. H., & Beal, M. K. (1998). Experimental psychology: A case approach (6th ed.). New York: Longman.
Suddick, D. & Collins, B. A. (1982). The Descriptive Tests of Mathematics Skills: A follow-up of performance of older upper division students. Educational and Psychological Measurement, 42, 559 – 561.