Introduction
Conducting a critique on research studies is an important skill. Although many research articles are peer-reviewed and have significant contributions to knowledge, it does not mean that they lack some weaknesses and limitations. Critique on research articles helps to identify such weaknesses and ways through which the studies could be improved and strengthened. The main aim of this paper is to carry out a critique on two peer-reviewed research articles.
The first research study was conducted by Rau, Gao and Wu (2006) and aimed at determining how modern technology particularly mobile technology affects the motivation, pressure and learning performance of high school students.
The second research study was conducted was conducted by Rodriguez, Ooms and Montanez (2008) and aimed at determining the perceptions of students towards the quality of online learning as influenced by comfort, motivation, satisfaction and experience. The major areas of critique are the ethical issues surrounding the research studies, the research approach used and research methodology and methods.
Ethics
Evidence of issues of power in the research
There are various attributes of power in any research. These attributes include: age, race/ethnic group, education level, social status, religion and gender among others. In the study conducted by Rau, Gao and Wu (2006) issues of power are evident through gender, race, and educational level.
The study was conducted in a predominantly Taiwanese vocational senior high school. Although the participants were both males and females, the females were the majority (142) while the males the minority (34). All the participants were of the same educational level since they were all selected from the same academic grade.
In the study conducted by Rodriguez, Ooms and Montanez (2008), issues of power are evident through age and educational level. The age of the participants ranged from 16 to older than 46. The majority of the participants were aged between 16 and 25 years, followed by 26-35 years (38%), 36-45 years (13%) and lastly 46 years and above (10%). The participants were enrolled in different academic programs and were on different academic levels.
Situation of the research in relation to broader academic debates
The study conducted by Rau et al. (2006) has the objective of determining how modern technology particularly mobile technology affects the motivation, pressure and learning performance of high school students. This study is highly significant because of the modern technology era.
More so it is based on the fact that motivation and pressure substantially affect the learning experience of senior high school students. Given the rapid advancement of information technology, educational institutions are not left behind in the application of the technology to enhance the experiences of all stakeholders in the system. The study by Rau et al. (2006) is thus appropriate and further contributes to the debate surrounding the role of information technology in educational institutions.
The study by Rodriguez et al. (2008) had the objective of determining the perceptions of students towards the quality of online learning as influenced by comfort, motivation, satisfaction and experience.
The study is significant chiefly because the rapid development of technology has increased the number of online courses as well as the number of colleges and universities willing to offer online courses. But as this trend continues to grow, there is a heated debate about the quality of the online courses particularly because such technologies are impersonal and reduce the level of personal interactions between students and their instructors and between students themselves.
In addition, the experience of online learning varies among the learners depending on their grasp of the new technologies. The study by Rodriguez et al. (2008) is therefore significant because it provides additional knowledge about factors that influence the quality of online learning.
Recruitment of participants in relation to research ethics
The study by Rau et al. (2006) had two different experiments. In the first experiment, the researchers selected “on hundred and seventy six juniors majoring in accounting in Taiyuan vocational senior high school (Rau et al. 2006, p. 5). 142 of the participants were female while 34 were male.
In the second experiment, the researchers used a sample of forty-five students, 29of who were females and 16 males. Based on their usage of SMS, email and internet, the participants were then classified into four groups namely: SMS, email, online forum and control group. The researchers ensured that the classification of the participants was not influenced by prior learning performance of the students. In this study, the ethical issues surrounding the selection of participants have not been made explicit by the researchers.
However, like any other ethical research, it is possible that the researchers selected the participants based on their informed consent and voluntary participation. It is possible that the researchers educated the school about the study they wanted to undertake, its potential benefits and potential harm to the potential participants.
The juniors majoring in accounting were then invited to take part in the study only if they were willing to do so. Out of all those who volunteered to take part in the study, the researchers selected a suitable sample that would help them to achieve their goal.
In the study conducted by Rodriguez et al. (2008), the participants were selected based on their ability to complete the online survey sent to them. The final sample was taken from post-BA, professional and graduate school of Midwest Research-I University which had approximately 3000 students enrolled in the programs.
The survey was posted online and the students were invited to fill them in. however before this took place, the researchers sent the students e-mail notices which informed them of the study, its potential benefits and potential harm. Invitations were then sent to the students with the link of the survey as well as reminders about the study. The survey was left open for 4 weeks.
This process of selecting participants is ethical because it ensured that the students who agreed to participate in the study did so out of informed consent. The informed consent was guaranteed by the e-mail notices sent to the students prior to the survey. The selection process is also ethical because it upheld voluntary participation. This is illustrated by the fact that the survey was left open for four weeks so as to ensure that any student who wished to take part in the study did so.
Use of research methods in relation to research ethics
The research methodology used in the study by Rau et al. (2006) is a one factor experiment design. The researchers divided the participants into two groups: the experimental and the control group.
Participants in the experimental group received all the learning instructions through the experimenting media (SMS, email and online forum) while those in the control group received the same information through face-to-face interaction with the instructor in a classroom setting. The use of the experimental research methodology has significant ethical implications particularly if the experiment involves human subjects.
The research methodology employed by Rodriguez et al. (2008) is theory verification. The researchers begin by giving an introductory paragraph about the problem under investigation after which they present several perspectives that explain the influence of several factors on the quality of online learning.
The researchers then conduct the survey to test the perspectives discussed. The theory verification methodology used by Rodriguez et al. (2008) present a number of ethical issues all of which pertain to the measurement techniques used. Specifically, the study raises doubt about whether or not the right constructs were used and whether the constructs were measured correctly. These doubts can be tackled by the study’s replication and modification.
Significance of the research question and its evaluation in terms of research ethics
The study by Rau et al. (2006) has six main research questions. The first three research questions deal with the impact of using SMS, email or online forum separately on student pressure, students’ learning motivation and students’ exam performance respectively. The other three research questions deal with the impact of combining wireless communication and online communication on student pressure, students’ learning motivation and students’ exam performance respectively.
The major research question addressed by Rodriguez et al. (2008) revolves around the perceived quality of online learning by students. However, this question is addressed through a number of other research questions which pertain to: the general learning experiences and technological support needs of students in online learning, the perception of students registered in online courses towards comfort with technology, satisfaction with earlier online learning experiences and motivation to develop their computer-based skills, the general perception of students towards the quality of different formats and levels of online learning, and the relationship between the perceptions of quality of online learning and comfort, satisfaction and motivation to develop computer-based skills.
Ethical issues in analysis and reporting
Researchers have an ethical obligation to their fellow researchers to report any technical limitations or any unexpected or contrary result that may have been experienced in their studies. This helps other researchers to see another side of a phenomenon and also forms the platform upon which future studies can be conducted.
In the study by Rau et al. (2006), this ethical obligation has been made explicit because the researchers truthfully reported the findings as they were rather than as they expected. In addition, the researchers outlined the limitations of the study which included the short time period of the study, and the lack of differentiation between intrinsic and extrinsic motivation.
The limitations help the readers to have a deeper understanding of the findings. They also help other researchers to replicate the study taking into consideration the limitations. For instance, a researcher may be interested in conducting a similar study but using a longer time period.
The reporting and analysis of data by Rodriguez et al. (2008) has been done ethically. First and foremost, the researchers make explicit the fact that the analysis could raise doubts about the measurement used. In order to erase such doubts, the researchers recommend replication and modification of the study.
Second, the researchers report the findings as they are and not how they thought they would be. This means that both positive and negative correlations between the variables were reported irrespective of the researchers’ prior anticipation. This is ethical because the researchers have not altered the results to make them more acceptable to themselves.
Unfortunately, the researchers failed to make explicit the limitations of the study particularly as far as the methodology, data collection and analysis are concerned. This makes it difficult for other researchers interested in a similar study to replicate the research taking into consideration the limitations.
Research Approach
Research approach used in the study and evidence
The study by Rau et al. (2006) made use of the experimental research approach. This approach classifies participants into two major groups: the experimental and the control group. This is evidenced in the study because the researchers divided the participants into the two groups.
In the experimental group, the researchers applied the experimenting media (SMS, email and online forum) to the participants. On the other hand, the researchers applied the traditional communication medium (face-to-face interaction) to the participants in the control group. The use of experimental research approach enables the researchers to compare differences or similarities in the results between the two groups.
The study by Rodriguez et al. (2008) made use of the theory verification approach. This approach entails the testing of various theories or perspectives to determine if they hold in different situations. In the Rodriguez et al.’s study, several perspectives touching on the problem under investigation were first presented and then later tested through the sample of the graduate students.
Theoretical perspectives/theories used by the researchers and its implication on the approach
In the study by Rau et al. (2006), the researchers used the social presence and information richness theories and their application on media and how they affect the instruction and learning processes. Using the two theories, the researchers reviewed several literatures on: motivation in learning, media richness and social presence, the use of computer mediated communication in education, and the use of mobile communication technologies in education.
All the literatures reviewed in the study showed that information technology media have some influence on student’s learning process as well as the instruction process adopted by the instructors. Based on the rich background information, the researchers then proceeded with their study.
The theoretical perspective influenced the choice of the research approach used by the researchers. Because the literature review showed some significant influence, the researchers had to make use of the experimental research approach to determine if indeed the information technology media have any effect on student pressure, students’ learning motivation and students’ exam performance.
This could only be achieved by the use of the experimental design in which the participants are divided into two groups: the experimental and the control group. The experimental group makes use of the technology media under investigation while the control group makes use of the traditional face-to-face communication medium. Differences in the scores of the participants’ pressure, learning motivation and exam performance are then compared between the two groups (Carter & Little, 2007).
Rodriguez et al. (2008) used several theoretical perspectives in their study. These perspectives include: the multidimensional concept of online courses which encompasses the interaction between students and instructors, support services, technical support and the mode of delivery among others.
Other perspectives include the issue of comfort with technologies and computer, and satisfaction of students with web- and computer-based technologies. These perspectives help to illustrate the different perceptions of students enrolled in online courses towards the quality of the courses. These perspectives helped to determine the research approach used by Rodriguez et al. (2008). Specifically, the researchers used the theory verification approach to test whether or not the perspectives hold.
The theory verification approach entails collecting data from sample to test the correlation between the different variables of the perspectives. In the study by Rodriguez et al. (2008), the variables tested included comfort level with computer-based technologies, satisfaction with online courses, online delivery modes, level of face-to-face interaction between instructors and students, flexibility, self reliance, physical location, technical support, and quality standards of online courses.
In order to achieve their objectives and to answer the research questions, the participants were classified into three groups namely: participants with prior online-learning experience, participants with hybrid learning experience, and participants without any online learning experience. The variables were then tested in each of these groups to determine any differences or similarities.
Research paradigms used in the research studies and their justifications
The research paradigm used in the study conducted by Rau et al. (2006) is the post-positivist research paradigm. This research paradigm is based on the notion that theories developed to explain phenomena are only provisional and can therefore be challenged through research and new theories developed to modify the original ones. The research paradigm therefore calls for the testing of existing theories through observation and measurement.
The study by Rau et al. (2006) uses the post-positivist research paradigm because its aim is to test the social presence and information richness theories and their application on media and how they affect the instruction and learning processes. In short, the researchers want to know if indeed the theories hold and can be used to explain the differences between traditional and mobile/online communication media on students’ pressure, learning motivation and exam performance.
The post-positivist research paradigm used in the study by Rau et al. (2006) can also be justified by the research methodologies and methods used. The researchers use the experimental research methodology which is one of the methodologies used by post-positivists. In addition, quantitative research methods (collection of quantitative data and the use of statistical data analysis techniques) have been used by the researchers which further justify the research paradigm (Bernard, 2005).
Like in the study conducted by Rau et al. (2006), the research paradigm used by Rodriguez et al. (2008) is post-positivist paradigm. This research paradigm is based on the notion that theories developed to explain phenomena are only provisional and can therefore be challenged through research and new theories developed to modify the original ones.
The research paradigm therefore calls for the testing of existing theories through observation and measurement. Rodriguez et al. (2008) conducted their study to test the different perspectives discussed that relate the quality of online courses to comfort with technologies and computer, and satisfaction of students with web- and computer-based technologies.
Methodology and Methods
Relationship between the methodology and research methods
The methodology used in the study by Rau et al. (2006) is experimental which classifies participants into two groups: experimental and control groups.
The experimental group is the one to which the variables under investigation are applied while in the control group the status quo is maintained. In the Rau et al.’s study, the media under investigation (SMS, email and online forum) were applied to the experimental group while the traditional face-to-face communication was applied to the control group.
Because the researchers wanted to compare the differences between these two groups, they had to use quantitative research methods (Mackenzie & Knipe, 2006). Quantitative data were collected pertaining to the number of participants in both groups who perceived that the medium of communication increased pressure and enhanced their learning motivation. Data were also collected on the participants’ scores in the exams administered to them at the end of every week.
The methodology used in the study by Rodriguez et al. (2008) is a theory verification methodology which tests the different perspectives that help to explain the phenomenon under investigation. In order to achieve this, the researchers used survey as the research method and analyzed the data using statistical tests such as Chi-square, percentages, Spearman’s rho, standardized path coefficients, and fit statistics.
Methods used to generate and analyze data
Quantitative research methods were used to collect and analyze data in the study by Rau et al. (2006). A quantitative research is one that is interested in numerical data (Mackenzie & Knipe, 2006). To collect the data, the researchers made use of different measurement scales depending on the variable.
Data on motivation and pressure were collected using a five point scale that had six items while data on the learning performance of the participants were collected using scores of a quiz administered at the end of every week. In this study, the researchers make use of statistical tests such as one way ANOVA, student’s t-test, mean and standard deviations to analyze data.
The one way ANOVA was used to test the differences in leaning motivation and pressure between the two groups prior to the experiment. No significant difference was found between the two groups. Mean and standard deviation were used to test the exam performance of the four groups and also the perception of the participants in the two groups towards learning motivation and pressure.
The student’s t-test as also used to test the differences in the exam scores between the experimental and the control group. The use of the statistical tests lies in the fact that the researchers wanted to compare differences (or lack thereof) between the experimental and the control groups as far as the impact of wireless and online communication media on the students’ pressure, learning motivation and exam performance is concerned (Zikmund, 2003).
In the study conducted by Rodriguez et al. (2008), survey was the instrument used to collect data. The data were then analyzed using various statistical tests including Chi-square, percentages, Spearman’s rho, standardized path coefficients, and fit statistics. The Chi-square was used to test “differences between those students with online-learning experience, those with hybrid-learning experience, and those with no online-related learning experience,” (Rodriguez et al., 2008, p. 108).
The Spearman’s rho was used to test “correlations between motivation to learn and level of comfort with advanced technology skills and correlation between motivation to learn and age” (Rodriguez et al., 2008, p. 109). The fit statistics were used to test the goodness of fit of the models while the standardized path coefficients were used to make comparisons of the results between the groups.
Validity, Reliability and Rigor of the research
Validity of the research
Both research studies have ensured their internal validity. The internal validity of the study by Rau et al. (2006) and Rodriguez et al. (2008) is illustrated by the fact that their findings accurately describe the phenomena under investigation.
This implies that the two studies have systematically presented the results of their analysis in the order of the research questions they are trying to address (O’Toole & Beckett, 2010). Second, both studies are face valid. Face validity implies that a research study appears to measure what it says it will measure on the face value.
In the Rodriguez et al.’s (2008) study, the main objective is to determine the perceptions of students towards the quality of online learning as influenced by comfort, motivation, satisfaction and experience.
On face value, this seems to be the actual case and the reader does not necessarily have to go through the entire document to determine this. The same case applies to the study by Rau et al. (2006) whose main objective is to determine how modern technology particularly mobile technology affects the motivation, pressure and learning performance of high school students.
Third, the study by Rodriguez et al. (2008) is externally valid while that of Rau et al. (2006) lacks external validity. External validity is gauged by the extent to which a study’s findings can be generalized to the entire population. The study by Rau et al. lacks external validity because it used a homogenous sample, that is, a sample of participants with the same ethnic background, the same academic level and the same age bracket.
The findings might therefore have been different if a heterogeneous sample was used. On the other hand, external validity was ensured in the study by Rodriguez et al. (2008) because the researchers used a heterogeneous sample, that is, a sample of participants belonging to a wide range of age bracket and different academic levels. The findings could thus be generalized to the target population (Golafshani, 2003).
Reliability of the research
In the study by Rau et al. (2006) reliability as stability has not been ensured. This is because the researchers used two different samples in the two experiments yet the research questions in the two experiments were related. Reliability as stability would thus have been ensured if the researchers used the same sample in the two experiments (Golafshani, 2003).
In the study by Rodriguez et al. (2008), reliability as stability has been ensured because the researchers classified the participants into three groups based on their experience in online learning and then conducted the tests on the three groups separately.
Reliability as equivalence has been ensured in one of the research studies. In the study by Rau et al. (2006), this type of reliability has been ensured by measuring motivation and pressure of the participants by the same instrument both before and after the experiment. The same however cannot be said of the study conducted by Rodriguez et al. (2008).
Reliability as internal consistency has been ensured in both studies through the Cronbach alpha, or simply the alpha coefficient (O’Toole & Beckett, 2010). This coefficient has been used in both studies to enhance the reliability of the measurement instruments particularly those that measure multiple items.
Rigor of the research
A rigorous research is one that uses tools and techniques that are fit to meet the study’s objectives (Neergaard & Ulhoi, 2007, p.113). The studies conducted by Rau et al. (2006) and Rodriguez et al. (2008) are rigorous. This is because both studies have made explicit the main objectives of the studies and the research questions to be addressed by the studies.
In addition, sufficient background information and literature review have been presented to support the study. Based on the objectives and research questions, the researchers in both studies adopted the most appropriate research paradigm and research methodology and used the most appropriate data collection and analysis tools and instruments (Leedy & Ormrod, 2005).
Conclusion
This paper has critiqued two peer-reviewed research articles. The ethical issues, research approach and research methodology and methods used in the studies have been reviewed. On the ethical issues, the paper has identified ethical practices surrounding the selection of participants, power in the studies, research question, and ethical practices in data analysis and reporting.
On the research approach, the paper has identified the research approach used, the paradigms used, and theoretical perspectives/theories utilized by the researches and how they influenced the approach.
On methodology, the paper has identified the relationship between methodology and methods, methods of collecting and analyzing data as well as issues of reliability, validity and rigor of the research studies. Through the critique, several weaknesses and strengths have been identified and recommendations made on how the studies could have been improved.
Reference List
Bernard, H.R. (2005). Research methods in anthropology: qualitative and quantitative approaches. London: Rowman Altamira.
Carter, S., & Little, M. (2007). Justifying knowledge, justifying method, taking action: Epistemologies, methodologies, and methods in qualitative research. Qualitative Health Research, 17, 1316-1328.
Golafshani, N. (2003). Understanding reliability and validity in qualitative research. The Qualitative Report, 8(4), 597-607.
Leedy, P., & Ormrod, J. (2005). Practical research: Planning and design (8th ed.). Upper Saddle River: Pearson.
Mackenzie, N., & Knipe, S. (2006). Research dilemmas: Paradigms, methods and methodology. Issues in Educational Research, 16(2), 193-205.
Neergaard, H., & Ulhoi, J. (2007). Handbook of qualitative research methods in entrepreneurship. London: Edward Elgar.
O’Toole, J., & Beckett, D. (2010). Educational research: Creative thinking and doing. Melbourne: Oxford University Press.
Rau, P., Gao, Q., & Wu, L. (2006). Using mobile communication technology in high school education: Motivation, pressure, and learning performance. Computers & Education, 1-24.
Rodriguez, M., Ooms, A., & Montanez, M. (2008). Students’ Perceptions of Online-learning Quality given Comfort, Motivation, Satisfaction, and Experience. Journal of Interactive Online Learning, 7(2), 105-125.
Zikmund, W. (2003). Business research methods (7th ed.). Thousand Oaks, CA: Thomson/South-Western.